Add lessons learned

ai-workflow-development/01-vscode-on-cluster/SKILL.md

@@ -196,7 +196,9 @@ non-repetitive; point at an existing tutorial instead.
   over a `list` input fails over across resources (see
   `workflow/tutorials/round-robin-failover/`; reference §3 "Step retries").
 - **Make scripts idempotent and traceable:** `set -o pipefail`, `set -x`; check
-  before installing; safe to re-run.
+  before installing; safe to re-run. Guard every download with an existence check,
+  and prefer resumable fetches (`hf download` re-fetches only missing files) over
+  delete-and-redownload.
 - **Stream progress and emit structured results.** Print incremental progress (it
   streams to `run.<JOBID>.out` / the page) and write a machine-readable result
   (JSON) — don't make the user guess whether it's alive or done.

ai-workflow-development/references/activate-platform.md

@@ -801,6 +801,12 @@ subdomain URL (`https://<name>.activate.pw/<slug>`; `--slug` may be a query stri
   `skip_cleanups_file` and `parallelworks/cancel-jobs` the submitter); the service
   outlives the run. `pw endpoints delete <name>` tears down the whole remote process
   tree (verified: the `pw endpoints run` child dies with it).
+- **`pw endpoints http [--link] <port>` serves an already-running local server**,
+  decoupling registration from process start: a slow-starting service (LLM loading
+  weights) can come up first, pass a health check, and register only when it can
+  answer — an endpoint registered before the app serves shows "Session is not
+  reachable" in the platform chat. `--link` stops the local server when the client
+  exits, so `pw endpoints delete` also kills the app (verified: vllm on awsgpu).
 - **Env-var auth for containers/sidecars (v7.79.0):** the CLI authenticates from
   `PW_API_KEY` + `PW_PLATFORM_HOST` env vars with no config file — this is how to run
   it in a pod. `ghcr.io/parallelworks/pw-cli:<ver>` is distroless (entrypoint

ai-workflow-development/references/singularity-sif-containers.md

@@ -117,6 +117,14 @@ Using n8n as the template (diff `controller-v3.sh` → `controller-v4.sh`):
 
 - `singularity build` from `docker://` needs internet — build on the login node or
   your own machine, never in the start template.
+- Squashfs-less kernels are common, not exotic (e.g. AWS RHEL9 cluster images):
+  the sandbox fallback can be the primary path on a whole cluster. The probe
+  handles it unattended.
+- Apps that create Unix sockets under `$TMPDIR` (vLLM's ZMQ `ipc://`) crash when it
+  points into the deep job dir — socket paths cap at 107 chars. Bind a per-job dir
+  to container `/tmp` and set `TMPDIR=/tmp` inside the container instead.
+- Derive the cached SIF filename from the tag (`vllm:v1.0` → `vllm-v1.0.sif`) so
+  changing the artifact URI re-pulls while old versions stay cached.
 - Set `SINGULARITY_TMPDIR` **and** `SINGULARITY_CACHEDIR` under `${HOME}` for both
   build and fallback: `/tmp` is often too small for image builds and is not shared
   across nodes. (Apptainer also accepts the `APPTAINER_*` names; the `SINGULARITY_*`

ai-workflow-development/references/v4-to-v5-endpoints-upgrade.md

@@ -79,6 +79,16 @@ Contract differences vs v3:
   `SINGULARITY_TMPDIR`/`SINGULARITY_CACHEDIR` under `${HOME}` (both paths verified on
   gcpsmall, n8n conversion). Full build/push/convert recipe:
   [singularity-sif-containers.md](singularity-sif-containers.md).
+- **Slow-starting services (model servers): register the endpoint only when ready.**
+  `pw endpoints run` registers the session at launch, so the platform chat shows
+  "Session is not reachable" until the app actually serves (minutes for an LLM
+  loading weights). Instead: start the server in the background with output to a log
+  file, poll its health URL — if the process dies, print the log tail and
+  `pw workflows runs cancel ${PW_RUN_SLUG}` — then foreground
+  `pw endpoints http --link <port>`. `--link` stops the server when the endpoint
+  client exits, so `pw endpoints delete` also kills the app; still write a
+  `cancel.sh` that kills the background pid to cover cancel-before-ready.
+  (Verified: activate-rag-vllm `yamls/general_v5.yaml` on awsgpu, login + SLURM.)
 - Password/token: optional password → hash it (`jupyter_server.auth.passwd` via a
   python heredoc — avoids shell-quoting the `$`-laden hash); none → `token = ''` is
   fine because the endpoint already requires platform login.
@@ -186,3 +196,6 @@ full ask_cluster round trip).
   read-only without `-W`. No writable flag needed (verified over the ws protocol).
 - A root-slug service (webshell) just omits `--slug` from `pw_endpoints_args` —
   `--slug ""` risks the empty token being eaten by the arg parser.
+- Renaming a job breaks silent references: `wait_for_endpoint`'s
+  `parallelworks/cancel-jobs` lists the submitter job by name and `--dry-run` does
+  not catch a stale entry — the run then never completes.