serving-llms-on-epyc skill - address feedback

skills/serving-llms-on-epyc/SKILL.md

@@ -53,29 +53,45 @@ Returns `cpu_model`, `is_amd_epyc`, `epyc_generation`
 `zen_arch`, `is_supported_epyc`, `avx512`, `logical_cores`, `physical_cores`,
 `sockets`, `numa_nodes`, `memory_gb`.
 
-Three hard gates -- stop if any fails:
+Route from detect.py -- decide the serving path:
 - `is_amd_epyc` is `false` -> stop: this skill targets AMD EPYC. (Other x86 may work
   but is unsupported here.)
-- `is_supported_epyc` is `false` -> stop: this recipe supports only the **AMD EPYC
-  9000 series** for now -- Genoa (9004), Turin (9005), and Venice (9006). Other EPYC
-  (Bergamo, Siena, EPYC 4004/4005, pre-Zen4) may even expose AVX-512, but ISA
-  compatibility alone does not make them supported targets for this skill; stop.
-- `avx512` is `false` -> stop: the zentorch CPU path **requires AVX-512**, i.e. Zen4+
-  on the supported 9000-series parts above. Pre-Zen4 EPYC (Naples / Rome / Milan) is
-  not supported -- say so and stop rather than launching into a load-time failure.
+- `avx512` is `false` -> **zentorch cannot run** on this CPU (its bf16 path needs
+  AVX-512 BF16, `avx512_bf16`, which only lands on Zen4+). This is a pre-Zen4 EPYC
+  (Naples / Rome / Milan, 7000 series). Do **not** dead-end -- it is still an EPYC
+  host: **offer the stock vLLM CPU path** (plain vLLM, no zentorch acceleration --
+  slower, but verified working on EPYC 7763/Milan). Proceed only on the user's explicit
+  OK, launching the official stock vLLM CPU image (Step 6 "Stock vLLM" variant); if the
+  user declines, stop.
+- `is_supported_epyc` is `false` but `avx512` is `true` (e.g. Bergamo / Siena /
+  EPYC 4004/4005) -> the zentorch path is **not validated** for this generation. Get
+  explicit confirmation to try zentorch unvalidated, or take the same stock vLLM offer.
+- else (**9000 series** -- Genoa/Turin/Venice -- with AVX-512 BF16) -> the validated
+  zentorch path. Proceed.
+
+`validate.py` (Step 2) reports `zentorch_capable` and sets `requires_confirmation`
+for the stock/unvalidated paths, so this routing is enforced there too.
 
 Carry `epyc_generation` / `avx512` through the later phases -- e.g. Venice packs up
 to 256 cores/socket, which the thread-binding in Step 5 sizes from.
 
 ## Step 2: Validate the runtime and environment
 
 ```bash
-python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect>
+python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect> --avx512 <avx512 from detect>
 ```
 
-Returns `ready`, `requires_confirmation`, `runtime` (`docker`, `podman`, or null),
-`runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `ram_gb`, and
-`errors/warnings/advisories`. Pick the path:
+This also **hard-enforces the AVX-512 gate**: on a CPU without AVX-512, validate.py
+returns a blocking `error` (`ready: false`) so the flow stops here regardless of the
+Step 1 prose -- no image pull, no launch. (It reads the local CPU itself if `--avx512`
+is omitted, so the gate holds even if the value was not passed through.)
+
+Returns `ready`, `requires_confirmation`, `zentorch_capable` (false -> zentorch
+can't run; use the stock vLLM variant in Step 6), `runtime` (`docker`, `podman`, or
+null), `runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `hf_cache`
+(resolved HF cache mount -- use `hf_cache.mount` at launch), `ram_gb`, and
+`errors/warnings/advisories`. If `requires_confirmation` is set (stock/unvalidated
+path), surface that and get the user's OK before launching. Pick the path:
 - `runtime` is `docker` or `podman` -> container path (Step 6), used verbatim.
 - `runtime` null but `conda_path_available: true` -> conda/host path.
 - `runtime` null and no conda -> `ready` is false. Report the one-time
@@ -187,6 +203,7 @@ not launch unprompted. This is the human gate before anything runs:
 | Field | Value |
 |---|---|
 | Model / kind | `<model>` -- `text` or `multimodal` (from `check_model.py`) |
+| Backend | zentorch-accelerated (Zen4+) **or** stock vLLM CPU / no zentorch (`zentorch_capable:false` -- unaccelerated; verified on Milan) |
 | Path | container (`<runtime>`, image from `data/epyc.json`) or conda/host |
 | Precision | `bfloat16` (or the user's choice) |
 | Fit | required `<required_gb>` GB vs `<ram_gb>` GB RAM |
@@ -204,15 +221,24 @@ Build the launch from `data/epyc.json`. The CLI is `vllm serve <model>`.
 auto-selects the CPU platform and `vllm serve` rejects the flag. Only add it if
 `vllm serve --help` lists it (older vLLM).
 
+**Pick a free port first.** With `--network=host` the port is bound directly on
+the host, so a busy port is a **hard failure** (no remapping). Choose one that is
+free -- e.g. `PORT=8000; while ss -ltn "sport = :$PORT" | grep -q LISTEN; do PORT=$((PORT+1)); done`
+-- and use `$PORT` in the launch, health poll, and handover.
+
+**Mount the HF cache that `validate.py` resolved.** Use `hf_cache.mount` from
+validate.py (it follows symlinks and flags NFS/root-squash) rather than a raw
+`~/.cache/huggingface`, or the bind-mount can fail at container start on NFS homes.
+
 **Container path** (`runtime` from validate.py). The agent runs these itself,
 including the pull. `RT` is the resolved runtime verbatim:
 ```bash
 RT="<runtime from validate.py: docker | podman>"
 $RT rm -f vllm-epyc 2>/dev/null               # clear any leftover container from a prior run (name collision otherwise)
 $RT pull <image from data/epyc.json>          # agent pulls; do not ask the user to
 $RT run -d --name vllm-epyc \
-  <run_flags from data/epyc.json>            # --ipc=host --network=host (NO --shm-size: it conflicts with --ipc=host on podman)
-  <hf_cache_mount> \
+  <run_flags from data/epyc.json>            # --ipc=host --network=host --cap-add=SYS_NICE (SYS_NICE = NUMA membind; NO --shm-size with --ipc=host)
+  <hf_cache.mount from validate.py> \        # resolved real path, e.g. -v /scratch/you/hf:/root/.cache/huggingface
   <container_cpuset from cpu_tune>             # --cpuset-cpus=<cores> --cpuset-mems=<nodes>
   --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \
   --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE \
@@ -229,6 +255,28 @@ from cpu_tune so memory is bound to the chosen socket (empty → unpinned, with
 # e.g. numactl --cpunodebind=0 --membind=0 vllm serve ...
 ```
 
+**Stock vLLM path (no zentorch)** -- only when `validate.py` reports
+`zentorch_capable: false` (pre-Zen4 EPYC like Milan) **and the user confirmed** the
+unaccelerated fallback. Use the **official stock vLLM CPU image**, not the zentorch
+image: `vllm/vllm-openai-cpu:latest-x86_64` (its ENTRYPOINT is `vllm serve`, so pass
+`<model> --dtype ... --port ...` as args). Same sized env + flags as the container
+launch above (`VLLM_CPU_OMP_THREADS_BIND`, `--cpuset-cpus/--cpuset-mems`,
+`--cap-add=SYS_NICE`, `--ipc=host --network=host`, the resolved HF cache mount):
+```bash
+RT="<runtime>"
+$RT rm -f vllm-epyc 2>/dev/null
+$RT run -d --name vllm-epyc \
+  --ipc=host --network=host --cap-add=SYS_NICE \
+  <container_cpuset from cpu_tune> <hf_cache.mount from validate.py> \
+  --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \
+  --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE --env HF_TOKEN=${HF_TOKEN} \
+  vllm/vllm-openai-cpu:latest-x86_64 \
+  <model> --dtype bfloat16 --port <port> --max-model-len <len>
+```
+This path is **unaccelerated** (no zentorch), but **verified working** on EPYC 7763
+(Milan/Zen3) with `--dtype bfloat16` -- bf16 runs on stock vLLM CPU without AVX-512, so
+no fp32 is needed. If it still fails at load, apply the no-retry rule (report + stop).
+
 Optional throughput flags are **opt-in and must move together** (see Gotchas):
 `TORCHINDUCTOR_FREEZING=1` + `VLLM_USE_AOT_COMPILE=0` (+ `ZENTORCH_WEIGHT_PREPACK=1`).
 The base launch sets none of them.

skills/serving-llms-on-epyc/data/epyc.json

@@ -6,14 +6,17 @@
     "comment": "Pinned public vLLM + zentorch CPU image on Docker Hub (amdih/zendnn_zentorch) -- no internal-registry access needed. Update this image and vllm_version together so model-registry checks match the served runtime. Both docker and podman are supported; the skill prefers docker and falls back to podman.",
     "run_flags": [
       "--ipc=host",
-      "--network=host"
+      "--network=host",
+      "--cap-add=SYS_NICE"
     ],
     "hf_cache_mount": "-v ~/.cache/huggingface:/root/.cache/huggingface",
+    "hf_cache_note": "The default assumes the HF cache is a real local-disk dir at ~/.cache/huggingface. It BREAKS on NFS homes / symlinked caches / root-squash (docker's root cannot traverse the dir or resolve the symlink -> permission-denied mkdir at 'docker run'). Use the RESOLVED path validate.py reports in `hf_cache.mount` (it follows symlinks and flags NFS), or fall back to a local-disk cache (e.g. export HF_HOME=/scratch/$USER/hf and mount that).",
     "flag_notes": {
       "--ipc=host": "vLLM workers need a large /dev/shm; --ipc=host shares the host's (large) shared memory, which covers it. Do NOT also pass --shm-size: podman rejects '--shm-size' together with '--ipc=host' (cannot set shmsize in host IPC namespace), and it is redundant on docker too.",
       "shm_alternative": "If you must isolate IPC (drop --ipc=host), then add --shm-size=16g instead (the 64MB container default is too small for vLLM). Use one or the other, never both.",
-      "--network=host": "Expose the served port directly. Alternative: -p <port>:<port>.",
-      "numa": "A single instance is pinned to ONE socket plus its memory. cpu_tune.py picks a free socket by CPU load on dual-socket hosts (warns if both busy; --socket N forces), sizes KV from that socket's local RAM, and emits --cpuset-cpus + --cpuset-mems (container) or numactl --cpunodebind/--membind (conda). True multi-socket scaling = multiple instances (one per socket), out of scope here."
+      "--network=host": "Expose the served port directly (no -p needed). Because the port is bound directly on the host, a busy port is a HARD failure (no remap) -- pick a free one (see the port note in SKILL.md Step 6). Alternative: drop --network=host and use -p <hostport>:<port>.",
+      "--cap-add=SYS_NICE": "Required so the container can call set_mempolicy/numa_set_membind for NUMA memory binding. Without it those calls fail with 'Operation not permitted' and the socket-local memory pin (which cpu_tune.py treats as load-bearing) is silently lost -- the server still looks healthy but loses NUMA locality.",
+      "numa": "A single instance is pinned to ONE socket plus its memory. cpu_tune.py picks a free socket by CPU load on dual-socket hosts (warns if both busy; --socket N forces), sizes KV from that socket's local RAM, and emits --cpuset-cpus + --cpuset-mems (container) or numactl --cpunodebind/--membind (conda). Memory binding needs --cap-add=SYS_NICE (above). True multi-socket scaling = multiple instances (one per socket), out of scope here."
     }
   },
   "launch": {
@@ -25,6 +28,13 @@
     "default": "bfloat16",
     "notes": "bf16 is the throughput default on EPYC (Zen). fp32 is slower and for debugging only. WOQ (per-channel/per-group int) is supported by zentorch but out of scope for the base recipe."
   },
+  "no_zentorch_fallback": {
+    "when": "detect.py avx512 == false (pre-Zen4 EPYC: Naples/Rome/Milan, 7000 series) -- zentorch's bf16 CPU path needs avx512_bf16 (Zen4+), so zentorch cannot run.",
+    "offer": "Do not dead-end -- it is still an EPYC host. Offer STOCK vLLM CPU serving (no zentorch), on explicit user confirmation (validate.py sets requires_confirmation + zentorch_capable:false).",
+    "stock_image": "vllm/vllm-openai-cpu:latest-x86_64 -- the official stock vLLM CPU image on Docker Hub (pin a version with vllm/vllm-openai-cpu:v<ver>-x86_64). Use THIS for the no-zentorch path; do NOT pull the zentorch image just to disable it. Its ENTRYPOINT is `vllm serve`, so pass `<model> --dtype ... --port ...` as args (no leading 'vllm serve').",
+    "verified": "VERIFIED on AMD EPYC 7763 (Milan/Zen3, AVX2, no AVX-512), 2026-09-17: vLLM 0.29.0 came up healthy and returned a real Qwen/Qwen3-0.6B chat completion with --dtype bfloat16. bf16 WORKS on stock vLLM CPU without avx512_bf16 (no fp32 needed). The same sized env applies: VLLM_CPU_OMP_THREADS_BIND, --cpuset-cpus/--cpuset-mems, --cap-add=SYS_NICE.",
+    "caveat": "Unaccelerated (no zentorch) -- slower than the zentorch path. If it fails at load, apply the no-retry rule (report + stop). Note the stock image tracks upstream vLLM (e.g. 0.29.x), which may differ from vllm_version above."
+  },
   "model_support": {
     "check_script": "scripts/check_model.py",
     "policy": "Do NOT blanket-block multimodal. check_model.py reads the model's HF architectures and checks them against vLLM's model registry for the pinned vllm_version. Text and multimodal generation endpoints are allowed; pooling/embedding/reranker and non-LLM architectures are rejected (not chat/completion endpoints).",

skills/serving-llms-on-epyc/reference.md

@@ -89,9 +89,10 @@ From `data/epyc.json`. Unlike the Instinct (GPU) skill there are **no**
 |---|---|
 | `--ipc=host` | vLLM workers need a large `/dev/shm`; sharing the host IPC namespace provides it. **Do not also pass `--shm-size`** -- podman rejects the combination, and it is redundant on docker |
 | `--shm-size=16g` | **only if you drop `--ipc=host`** (isolated IPC). The 64MB container default is too small for vLLM. Use one or the other, never both |
-| `--network=host` | expose the served port directly (or use `-p <port>:<port>`) |
+| `--network=host` | expose the served port directly (or use `-p <port>:<port>`). Port is bound on the host, so a busy port is a **hard failure** -- pick a free one |
+| `--cap-add=SYS_NICE` | lets the container call `set_mempolicy`/`numa_set_membind`; without it NUMA memory binding fails with *"Operation not permitted"* and socket-local memory is silently lost |
 | `--cpuset-cpus` / `--cpuset-mems` | pin the container to the chosen socket's physical cores and its NUMA node(s); from `cpu_tune.py` |
-| `-v ~/.cache/huggingface:/root/.cache/huggingface` | reuse the host model cache |
+| `-v <resolved HF cache>:/root/.cache/huggingface` | reuse the host model cache -- use the real path `validate.py` resolves (`hf_cache.mount`), not a raw `~/.cache/huggingface` symlink (breaks on NFS) |
 
 Image: `amdih/zendnn_zentorch:<tag>` -- the public vLLM + zentorch CPU image on
 Docker Hub (no internal-registry access needed). The exact tag lives in
@@ -211,3 +212,24 @@ its memory**: CPU bind (`VLLM_CPU_OMP_THREADS_BIND` + `--cpuset-cpus`), memory b
 the KV pool never lands on the other socket. The socket is chosen by load (free socket
 preferred; warns if both busy). True multi-socket throughput = **multiple instances**
 (one per socket) -- out of scope for this single-instance recipe.
+
+**NUMA membind needs `--cap-add=SYS_NICE`**
+The socket memory pin (`--cpuset-mems` / the app's `numa_set_membind`) requires the
+`SYS_NICE` capability inside the container. Without it the calls fail with *"Operation
+not permitted"*, and -- because they surface only as warnings -- the server looks
+healthy while silently losing the NUMA locality the sizing depends on. The base
+`run_flags` include `--cap-add=SYS_NICE`.
+
+**HF cache bind-mount breaks on NFS / symlinked homes**
+The default mounts the HF cache into the container. On shared/HPC hosts the cache is
+often on an NFS home, or `~/.cache/huggingface` is a symlink to `/scratch`; docker's
+root-squash over NFS then can't traverse the dir or resolve the symlink, and the run
+dies with a permission-denied mkdir -- something `ready:true` does **not** catch.
+`validate.py` resolves the real path (follows symlinks), flags NFS, and returns the
+mount to use in `hf_cache.mount`. If the resolved path is NFS/root-squashed, fall back
+to a local-disk cache: `export HF_HOME=/scratch/$USER/hf` and mount that.
+
+**`--network=host` port collisions are hard failures**
+With host networking the port binds directly on the host -- a busy port fails outright
+(no remap). Pick a free port before launching (Step 6) and use it consistently in the
+launch, health poll, and handover.

skills/serving-llms-on-epyc/scripts/detect.py

@@ -12,8 +12,9 @@
 Output: JSON with cpu_model, is_amd_epyc, logical_cores, physical_cores,
 sockets, threads_per_core, numa_nodes, memory_gb, epyc_generation
 (Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005),
-zen_arch, is_supported_epyc, and avx512. Exits 0 on success, 1 if no CPU info
-could be read.
+zen_arch, is_supported_epyc, and avx512 (true = has AVX-512 BF16 / avx512_bf16,
+the extension zentorch's bf16 CPU path requires). Exits 0 on success, 1 if no CPU
+info could be read.
 
 Env vars (used when --host is not given):
     ZEN_SSH_HOST, ZEN_SSH_USER, ZEN_SSH_PORT
@@ -147,7 +148,10 @@ def _int(label, default=0):
     is_epyc = vendor == "AuthenticAMD" and "EPYC" in model.upper()
     generation, zen_arch = _epyc_generation(model)
     is_supported_epyc = is_epyc and generation in SUPPORTED_EPYC_GENERATIONS
-    avx512 = "avx512f" in _lscpu_field(lscpu_out, "Flags").split()
+    # zentorch's bf16 CPU path needs AVX-512 BF16 specifically (avx512_bf16), not
+    # just base AVX-512 (avx512f). On EPYC these coincide (Zen4+ has both, pre-Zen4
+    # neither), but gate on the exact flag zentorch requires.
+    avx512 = "avx512_bf16" in _lscpu_field(lscpu_out, "Flags").split()
 
     print(json.dumps({
         "cpu_model": model,