diff --git a/ai-workflow-development/01-vscode-on-cluster/SKILL.md b/ai-workflow-development/01-vscode-on-cluster/SKILL.md index 7a0194fb..e9afc091 100644 --- a/ai-workflow-development/01-vscode-on-cluster/SKILL.md +++ b/ai-workflow-development/01-vscode-on-cluster/SKILL.md @@ -196,7 +196,9 @@ non-repetitive; point at an existing tutorial instead. over a `list` input fails over across resources (see `workflow/tutorials/round-robin-failover/`; reference §3 "Step retries"). - **Make scripts idempotent and traceable:** `set -o pipefail`, `set -x`; check - before installing; safe to re-run. + before installing; safe to re-run. Guard every download with an existence check, + and prefer resumable fetches (`hf download` re-fetches only missing files) over + delete-and-redownload. - **Stream progress and emit structured results.** Print incremental progress (it streams to `run..out` / the page) and write a machine-readable result (JSON) — don't make the user guess whether it's alive or done. diff --git a/ai-workflow-development/references/activate-platform.md b/ai-workflow-development/references/activate-platform.md index 9e7a858f..9dcaffbe 100644 --- a/ai-workflow-development/references/activate-platform.md +++ b/ai-workflow-development/references/activate-platform.md @@ -801,6 +801,12 @@ subdomain URL (`https://.activate.pw/`; `--slug` may be a query stri `skip_cleanups_file` and `parallelworks/cancel-jobs` the submitter); the service outlives the run. `pw endpoints delete ` tears down the whole remote process tree (verified: the `pw endpoints run` child dies with it). +- **`pw endpoints http [--link] ` serves an already-running local server**, + decoupling registration from process start: a slow-starting service (LLM loading + weights) can come up first, pass a health check, and register only when it can + answer — an endpoint registered before the app serves shows "Session is not + reachable" in the platform chat. `--link` stops the local server when the client + exits, so `pw endpoints delete` also kills the app (verified: vllm on awsgpu). - **Env-var auth for containers/sidecars (v7.79.0):** the CLI authenticates from `PW_API_KEY` + `PW_PLATFORM_HOST` env vars with no config file — this is how to run it in a pod. `ghcr.io/parallelworks/pw-cli:` is distroless (entrypoint diff --git a/ai-workflow-development/references/singularity-sif-containers.md b/ai-workflow-development/references/singularity-sif-containers.md index 0f7d3fdd..ef07c7dd 100644 --- a/ai-workflow-development/references/singularity-sif-containers.md +++ b/ai-workflow-development/references/singularity-sif-containers.md @@ -117,6 +117,14 @@ Using n8n as the template (diff `controller-v3.sh` → `controller-v4.sh`): - `singularity build` from `docker://` needs internet — build on the login node or your own machine, never in the start template. +- Squashfs-less kernels are common, not exotic (e.g. AWS RHEL9 cluster images): + the sandbox fallback can be the primary path on a whole cluster. The probe + handles it unattended. +- Apps that create Unix sockets under `$TMPDIR` (vLLM's ZMQ `ipc://`) crash when it + points into the deep job dir — socket paths cap at 107 chars. Bind a per-job dir + to container `/tmp` and set `TMPDIR=/tmp` inside the container instead. +- Derive the cached SIF filename from the tag (`vllm:v1.0` → `vllm-v1.0.sif`) so + changing the artifact URI re-pulls while old versions stay cached. - Set `SINGULARITY_TMPDIR` **and** `SINGULARITY_CACHEDIR` under `${HOME}` for both build and fallback: `/tmp` is often too small for image builds and is not shared across nodes. (Apptainer also accepts the `APPTAINER_*` names; the `SINGULARITY_*` diff --git a/ai-workflow-development/references/v4-to-v5-endpoints-upgrade.md b/ai-workflow-development/references/v4-to-v5-endpoints-upgrade.md index 6fb910e3..d66f62d0 100644 --- a/ai-workflow-development/references/v4-to-v5-endpoints-upgrade.md +++ b/ai-workflow-development/references/v4-to-v5-endpoints-upgrade.md @@ -79,6 +79,16 @@ Contract differences vs v3: `SINGULARITY_TMPDIR`/`SINGULARITY_CACHEDIR` under `${HOME}` (both paths verified on gcpsmall, n8n conversion). Full build/push/convert recipe: [singularity-sif-containers.md](singularity-sif-containers.md). +- **Slow-starting services (model servers): register the endpoint only when ready.** + `pw endpoints run` registers the session at launch, so the platform chat shows + "Session is not reachable" until the app actually serves (minutes for an LLM + loading weights). Instead: start the server in the background with output to a log + file, poll its health URL — if the process dies, print the log tail and + `pw workflows runs cancel ${PW_RUN_SLUG}` — then foreground + `pw endpoints http --link `. `--link` stops the server when the endpoint + client exits, so `pw endpoints delete` also kills the app; still write a + `cancel.sh` that kills the background pid to cover cancel-before-ready. + (Verified: activate-rag-vllm `yamls/general_v5.yaml` on awsgpu, login + SLURM.) - Password/token: optional password → hash it (`jupyter_server.auth.passwd` via a python heredoc — avoids shell-quoting the `$`-laden hash); none → `token = ''` is fine because the endpoint already requires platform login. @@ -186,3 +196,6 @@ full ask_cluster round trip). read-only without `-W`. No writable flag needed (verified over the ws protocol). - A root-slug service (webshell) just omits `--slug` from `pw_endpoints_args` — `--slug ""` risks the empty token being eaten by the arg parser. +- Renaming a job breaks silent references: `wait_for_endpoint`'s + `parallelworks/cancel-jobs` lists the submitter job by name and `--dry-run` does + not catch a stale entry — the run then never completes.