From 463ae5516d72ee064e8f40a18903f7bd44e28d4f Mon Sep 17 00:00:00 2001 From: Lalithnarayan C Date: Wed, 16 Sep 2026 05:39:00 -0600 Subject: [PATCH] serving-llms-on-epyc: address dogfooding feedback - Hard-gate AVX-512 in validate.py (blocking error / ready:false), so pre-Zen4 EPYC stops before the image pull instead of failing at model load. Enforces the Step 1 gate programmatically, not just in prose. - Resolve the HF cache mount in validate.py: follow symlinks, flag NFS/root-squash, and return the real -v mount to use (the ~/.cache/huggingface default breaks on NFS homes). - Add --cap-add=SYS_NICE so NUMA memory binding works in the container (else it silently fails with 'Operation not permitted'). - Add free-port guidance for --network=host (a busy port is a hard failure). Signed-off-by: Lalithnarayan C Change-Id: Idc30ae70dbf709008a90bb215eeb66f29d5edfea --- skills/serving-llms-on-epyc/SKILL.md | 76 +++++++++--- skills/serving-llms-on-epyc/data/epyc.json | 16 ++- skills/serving-llms-on-epyc/reference.md | 26 +++- skills/serving-llms-on-epyc/scripts/detect.py | 10 +- .../serving-llms-on-epyc/scripts/validate.py | 116 +++++++++++++++--- 5 files changed, 208 insertions(+), 36 deletions(-) diff --git a/skills/serving-llms-on-epyc/SKILL.md b/skills/serving-llms-on-epyc/SKILL.md index 151a2e8..0c6e05c 100644 --- a/skills/serving-llms-on-epyc/SKILL.md +++ b/skills/serving-llms-on-epyc/SKILL.md @@ -53,16 +53,24 @@ Returns `cpu_model`, `is_amd_epyc`, `epyc_generation` `zen_arch`, `is_supported_epyc`, `avx512`, `logical_cores`, `physical_cores`, `sockets`, `numa_nodes`, `memory_gb`. -Three hard gates -- stop if any fails: +Route from detect.py -- decide the serving path: - `is_amd_epyc` is `false` -> stop: this skill targets AMD EPYC. (Other x86 may work but is unsupported here.) -- `is_supported_epyc` is `false` -> stop: this recipe supports only the **AMD EPYC - 9000 series** for now -- Genoa (9004), Turin (9005), and Venice (9006). Other EPYC - (Bergamo, Siena, EPYC 4004/4005, pre-Zen4) may even expose AVX-512, but ISA - compatibility alone does not make them supported targets for this skill; stop. -- `avx512` is `false` -> stop: the zentorch CPU path **requires AVX-512**, i.e. Zen4+ - on the supported 9000-series parts above. Pre-Zen4 EPYC (Naples / Rome / Milan) is - not supported -- say so and stop rather than launching into a load-time failure. +- `avx512` is `false` -> **zentorch cannot run** on this CPU (its bf16 path needs + AVX-512 BF16, `avx512_bf16`, which only lands on Zen4+). This is a pre-Zen4 EPYC + (Naples / Rome / Milan, 7000 series). Do **not** dead-end -- it is still an EPYC + host: **offer the stock vLLM CPU path** (plain vLLM, no zentorch acceleration -- + slower, but verified working on EPYC 7763/Milan). Proceed only on the user's explicit + OK, launching the official stock vLLM CPU image (Step 6 "Stock vLLM" variant); if the + user declines, stop. +- `is_supported_epyc` is `false` but `avx512` is `true` (e.g. Bergamo / Siena / + EPYC 4004/4005) -> the zentorch path is **not validated** for this generation. Get + explicit confirmation to try zentorch unvalidated, or take the same stock vLLM offer. +- else (**9000 series** -- Genoa/Turin/Venice -- with AVX-512 BF16) -> the validated + zentorch path. Proceed. + +`validate.py` (Step 2) reports `zentorch_capable` and sets `requires_confirmation` +for the stock/unvalidated paths, so this routing is enforced there too. Carry `epyc_generation` / `avx512` through the later phases -- e.g. Venice packs up to 256 cores/socket, which the thread-binding in Step 5 sizes from. @@ -70,12 +78,20 @@ to 256 cores/socket, which the thread-binding in Step 5 sizes from. ## Step 2: Validate the runtime and environment ```bash -python3 scripts/validate.py --image --generation +python3 scripts/validate.py --image --generation --avx512 ``` -Returns `ready`, `requires_confirmation`, `runtime` (`docker`, `podman`, or null), -`runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `ram_gb`, and -`errors/warnings/advisories`. Pick the path: +This also **hard-enforces the AVX-512 gate**: on a CPU without AVX-512, validate.py +returns a blocking `error` (`ready: false`) so the flow stops here regardless of the +Step 1 prose -- no image pull, no launch. (It reads the local CPU itself if `--avx512` +is omitted, so the gate holds even if the value was not passed through.) + +Returns `ready`, `requires_confirmation`, `zentorch_capable` (false -> zentorch +can't run; use the stock vLLM variant in Step 6), `runtime` (`docker`, `podman`, or +null), `runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `hf_cache` +(resolved HF cache mount -- use `hf_cache.mount` at launch), `ram_gb`, and +`errors/warnings/advisories`. If `requires_confirmation` is set (stock/unvalidated +path), surface that and get the user's OK before launching. Pick the path: - `runtime` is `docker` or `podman` -> container path (Step 6), used verbatim. - `runtime` null but `conda_path_available: true` -> conda/host path. - `runtime` null and no conda -> `ready` is false. Report the one-time @@ -187,6 +203,7 @@ not launch unprompted. This is the human gate before anything runs: | Field | Value | |---|---| | Model / kind | `` -- `text` or `multimodal` (from `check_model.py`) | +| Backend | zentorch-accelerated (Zen4+) **or** stock vLLM CPU / no zentorch (`zentorch_capable:false` -- unaccelerated; verified on Milan) | | Path | container (``, image from `data/epyc.json`) or conda/host | | Precision | `bfloat16` (or the user's choice) | | Fit | required `` GB vs `` GB RAM | @@ -204,6 +221,15 @@ Build the launch from `data/epyc.json`. The CLI is `vllm serve `. auto-selects the CPU platform and `vllm serve` rejects the flag. Only add it if `vllm serve --help` lists it (older vLLM). +**Pick a free port first.** With `--network=host` the port is bound directly on +the host, so a busy port is a **hard failure** (no remapping). Choose one that is +free -- e.g. `PORT=8000; while ss -ltn "sport = :$PORT" | grep -q LISTEN; do PORT=$((PORT+1)); done` +-- and use `$PORT` in the launch, health poll, and handover. + +**Mount the HF cache that `validate.py` resolved.** Use `hf_cache.mount` from +validate.py (it follows symlinks and flags NFS/root-squash) rather than a raw +`~/.cache/huggingface`, or the bind-mount can fail at container start on NFS homes. + **Container path** (`runtime` from validate.py). The agent runs these itself, including the pull. `RT` is the resolved runtime verbatim: ```bash @@ -211,8 +237,8 @@ RT="" $RT rm -f vllm-epyc 2>/dev/null # clear any leftover container from a prior run (name collision otherwise) $RT pull # agent pulls; do not ask the user to $RT run -d --name vllm-epyc \ - # --ipc=host --network=host (NO --shm-size: it conflicts with --ipc=host on podman) - \ + # --ipc=host --network=host --cap-add=SYS_NICE (SYS_NICE = NUMA membind; NO --shm-size with --ipc=host) + \ # resolved real path, e.g. -v /scratch/you/hf:/root/.cache/huggingface # --cpuset-cpus= --cpuset-mems= --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \ --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE \ @@ -229,6 +255,28 @@ from cpu_tune so memory is bound to the chosen socket (empty → unpinned, with # e.g. numactl --cpunodebind=0 --membind=0 vllm serve ... ``` +**Stock vLLM path (no zentorch)** -- only when `validate.py` reports +`zentorch_capable: false` (pre-Zen4 EPYC like Milan) **and the user confirmed** the +unaccelerated fallback. Use the **official stock vLLM CPU image**, not the zentorch +image: `vllm/vllm-openai-cpu:latest-x86_64` (its ENTRYPOINT is `vllm serve`, so pass +` --dtype ... --port ...` as args). Same sized env + flags as the container +launch above (`VLLM_CPU_OMP_THREADS_BIND`, `--cpuset-cpus/--cpuset-mems`, +`--cap-add=SYS_NICE`, `--ipc=host --network=host`, the resolved HF cache mount): +```bash +RT="" +$RT rm -f vllm-epyc 2>/dev/null +$RT run -d --name vllm-epyc \ + --ipc=host --network=host --cap-add=SYS_NICE \ + \ + --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \ + --env VLLM_CPU_KVCACHE_SPACE=$VLLM_CPU_KVCACHE_SPACE --env HF_TOKEN=${HF_TOKEN} \ + vllm/vllm-openai-cpu:latest-x86_64 \ + --dtype bfloat16 --port --max-model-len +``` +This path is **unaccelerated** (no zentorch), but **verified working** on EPYC 7763 +(Milan/Zen3) with `--dtype bfloat16` -- bf16 runs on stock vLLM CPU without AVX-512, so +no fp32 is needed. If it still fails at load, apply the no-retry rule (report + stop). + Optional throughput flags are **opt-in and must move together** (see Gotchas): `TORCHINDUCTOR_FREEZING=1` + `VLLM_USE_AOT_COMPILE=0` (+ `ZENTORCH_WEIGHT_PREPACK=1`). The base launch sets none of them. diff --git a/skills/serving-llms-on-epyc/data/epyc.json b/skills/serving-llms-on-epyc/data/epyc.json index 8e5cadb..0ea157f 100644 --- a/skills/serving-llms-on-epyc/data/epyc.json +++ b/skills/serving-llms-on-epyc/data/epyc.json @@ -6,14 +6,17 @@ "comment": "Pinned public vLLM + zentorch CPU image on Docker Hub (amdih/zendnn_zentorch) -- no internal-registry access needed. Update this image and vllm_version together so model-registry checks match the served runtime. Both docker and podman are supported; the skill prefers docker and falls back to podman.", "run_flags": [ "--ipc=host", - "--network=host" + "--network=host", + "--cap-add=SYS_NICE" ], "hf_cache_mount": "-v ~/.cache/huggingface:/root/.cache/huggingface", + "hf_cache_note": "The default assumes the HF cache is a real local-disk dir at ~/.cache/huggingface. It BREAKS on NFS homes / symlinked caches / root-squash (docker's root cannot traverse the dir or resolve the symlink -> permission-denied mkdir at 'docker run'). Use the RESOLVED path validate.py reports in `hf_cache.mount` (it follows symlinks and flags NFS), or fall back to a local-disk cache (e.g. export HF_HOME=/scratch/$USER/hf and mount that).", "flag_notes": { "--ipc=host": "vLLM workers need a large /dev/shm; --ipc=host shares the host's (large) shared memory, which covers it. Do NOT also pass --shm-size: podman rejects '--shm-size' together with '--ipc=host' (cannot set shmsize in host IPC namespace), and it is redundant on docker too.", "shm_alternative": "If you must isolate IPC (drop --ipc=host), then add --shm-size=16g instead (the 64MB container default is too small for vLLM). Use one or the other, never both.", - "--network=host": "Expose the served port directly. Alternative: -p :.", - "numa": "A single instance is pinned to ONE socket plus its memory. cpu_tune.py picks a free socket by CPU load on dual-socket hosts (warns if both busy; --socket N forces), sizes KV from that socket's local RAM, and emits --cpuset-cpus + --cpuset-mems (container) or numactl --cpunodebind/--membind (conda). True multi-socket scaling = multiple instances (one per socket), out of scope here." + "--network=host": "Expose the served port directly (no -p needed). Because the port is bound directly on the host, a busy port is a HARD failure (no remap) -- pick a free one (see the port note in SKILL.md Step 6). Alternative: drop --network=host and use -p :.", + "--cap-add=SYS_NICE": "Required so the container can call set_mempolicy/numa_set_membind for NUMA memory binding. Without it those calls fail with 'Operation not permitted' and the socket-local memory pin (which cpu_tune.py treats as load-bearing) is silently lost -- the server still looks healthy but loses NUMA locality.", + "numa": "A single instance is pinned to ONE socket plus its memory. cpu_tune.py picks a free socket by CPU load on dual-socket hosts (warns if both busy; --socket N forces), sizes KV from that socket's local RAM, and emits --cpuset-cpus + --cpuset-mems (container) or numactl --cpunodebind/--membind (conda). Memory binding needs --cap-add=SYS_NICE (above). True multi-socket scaling = multiple instances (one per socket), out of scope here." } }, "launch": { @@ -25,6 +28,13 @@ "default": "bfloat16", "notes": "bf16 is the throughput default on EPYC (Zen). fp32 is slower and for debugging only. WOQ (per-channel/per-group int) is supported by zentorch but out of scope for the base recipe." }, + "no_zentorch_fallback": { + "when": "detect.py avx512 == false (pre-Zen4 EPYC: Naples/Rome/Milan, 7000 series) -- zentorch's bf16 CPU path needs avx512_bf16 (Zen4+), so zentorch cannot run.", + "offer": "Do not dead-end -- it is still an EPYC host. Offer STOCK vLLM CPU serving (no zentorch), on explicit user confirmation (validate.py sets requires_confirmation + zentorch_capable:false).", + "stock_image": "vllm/vllm-openai-cpu:latest-x86_64 -- the official stock vLLM CPU image on Docker Hub (pin a version with vllm/vllm-openai-cpu:v-x86_64). Use THIS for the no-zentorch path; do NOT pull the zentorch image just to disable it. Its ENTRYPOINT is `vllm serve`, so pass ` --dtype ... --port ...` as args (no leading 'vllm serve').", + "verified": "VERIFIED on AMD EPYC 7763 (Milan/Zen3, AVX2, no AVX-512), 2026-09-17: vLLM 0.29.0 came up healthy and returned a real Qwen/Qwen3-0.6B chat completion with --dtype bfloat16. bf16 WORKS on stock vLLM CPU without avx512_bf16 (no fp32 needed). The same sized env applies: VLLM_CPU_OMP_THREADS_BIND, --cpuset-cpus/--cpuset-mems, --cap-add=SYS_NICE.", + "caveat": "Unaccelerated (no zentorch) -- slower than the zentorch path. If it fails at load, apply the no-retry rule (report + stop). Note the stock image tracks upstream vLLM (e.g. 0.29.x), which may differ from vllm_version above." + }, "model_support": { "check_script": "scripts/check_model.py", "policy": "Do NOT blanket-block multimodal. check_model.py reads the model's HF architectures and checks them against vLLM's model registry for the pinned vllm_version. Text and multimodal generation endpoints are allowed; pooling/embedding/reranker and non-LLM architectures are rejected (not chat/completion endpoints).", diff --git a/skills/serving-llms-on-epyc/reference.md b/skills/serving-llms-on-epyc/reference.md index 23f0751..80412c2 100644 --- a/skills/serving-llms-on-epyc/reference.md +++ b/skills/serving-llms-on-epyc/reference.md @@ -89,9 +89,10 @@ From `data/epyc.json`. Unlike the Instinct (GPU) skill there are **no** |---|---| | `--ipc=host` | vLLM workers need a large `/dev/shm`; sharing the host IPC namespace provides it. **Do not also pass `--shm-size`** -- podman rejects the combination, and it is redundant on docker | | `--shm-size=16g` | **only if you drop `--ipc=host`** (isolated IPC). The 64MB container default is too small for vLLM. Use one or the other, never both | -| `--network=host` | expose the served port directly (or use `-p :`) | +| `--network=host` | expose the served port directly (or use `-p :`). Port is bound on the host, so a busy port is a **hard failure** -- pick a free one | +| `--cap-add=SYS_NICE` | lets the container call `set_mempolicy`/`numa_set_membind`; without it NUMA memory binding fails with *"Operation not permitted"* and socket-local memory is silently lost | | `--cpuset-cpus` / `--cpuset-mems` | pin the container to the chosen socket's physical cores and its NUMA node(s); from `cpu_tune.py` | -| `-v ~/.cache/huggingface:/root/.cache/huggingface` | reuse the host model cache | +| `-v :/root/.cache/huggingface` | reuse the host model cache -- use the real path `validate.py` resolves (`hf_cache.mount`), not a raw `~/.cache/huggingface` symlink (breaks on NFS) | Image: `amdih/zendnn_zentorch:` -- the public vLLM + zentorch CPU image on Docker Hub (no internal-registry access needed). The exact tag lives in @@ -211,3 +212,24 @@ its memory**: CPU bind (`VLLM_CPU_OMP_THREADS_BIND` + `--cpuset-cpus`), memory b the KV pool never lands on the other socket. The socket is chosen by load (free socket preferred; warns if both busy). True multi-socket throughput = **multiple instances** (one per socket) -- out of scope for this single-instance recipe. + +**NUMA membind needs `--cap-add=SYS_NICE`** +The socket memory pin (`--cpuset-mems` / the app's `numa_set_membind`) requires the +`SYS_NICE` capability inside the container. Without it the calls fail with *"Operation +not permitted"*, and -- because they surface only as warnings -- the server looks +healthy while silently losing the NUMA locality the sizing depends on. The base +`run_flags` include `--cap-add=SYS_NICE`. + +**HF cache bind-mount breaks on NFS / symlinked homes** +The default mounts the HF cache into the container. On shared/HPC hosts the cache is +often on an NFS home, or `~/.cache/huggingface` is a symlink to `/scratch`; docker's +root-squash over NFS then can't traverse the dir or resolve the symlink, and the run +dies with a permission-denied mkdir -- something `ready:true` does **not** catch. +`validate.py` resolves the real path (follows symlinks), flags NFS, and returns the +mount to use in `hf_cache.mount`. If the resolved path is NFS/root-squashed, fall back +to a local-disk cache: `export HF_HOME=/scratch/$USER/hf` and mount that. + +**`--network=host` port collisions are hard failures** +With host networking the port binds directly on the host -- a busy port fails outright +(no remap). Pick a free port before launching (Step 6) and use it consistently in the +launch, health poll, and handover. diff --git a/skills/serving-llms-on-epyc/scripts/detect.py b/skills/serving-llms-on-epyc/scripts/detect.py index ae23fca..42e7776 100644 --- a/skills/serving-llms-on-epyc/scripts/detect.py +++ b/skills/serving-llms-on-epyc/scripts/detect.py @@ -12,8 +12,9 @@ Output: JSON with cpu_model, is_amd_epyc, logical_cores, physical_cores, sockets, threads_per_core, numa_nodes, memory_gb, epyc_generation (Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005), -zen_arch, is_supported_epyc, and avx512. Exits 0 on success, 1 if no CPU info -could be read. +zen_arch, is_supported_epyc, and avx512 (true = has AVX-512 BF16 / avx512_bf16, +the extension zentorch's bf16 CPU path requires). Exits 0 on success, 1 if no CPU +info could be read. Env vars (used when --host is not given): ZEN_SSH_HOST, ZEN_SSH_USER, ZEN_SSH_PORT @@ -147,7 +148,10 @@ def _int(label, default=0): is_epyc = vendor == "AuthenticAMD" and "EPYC" in model.upper() generation, zen_arch = _epyc_generation(model) is_supported_epyc = is_epyc and generation in SUPPORTED_EPYC_GENERATIONS - avx512 = "avx512f" in _lscpu_field(lscpu_out, "Flags").split() + # zentorch's bf16 CPU path needs AVX-512 BF16 specifically (avx512_bf16), not + # just base AVX-512 (avx512f). On EPYC these coincide (Zen4+ has both, pre-Zen4 + # neither), but gate on the exact flag zentorch requires. + avx512 = "avx512_bf16" in _lscpu_field(lscpu_out, "Flags").split() print(json.dumps({ "cpu_model": model, diff --git a/skills/serving-llms-on-epyc/scripts/validate.py b/skills/serving-llms-on-epyc/scripts/validate.py index ae1bf40..90fe386 100644 --- a/skills/serving-llms-on-epyc/scripts/validate.py +++ b/skills/serving-llms-on-epyc/scripts/validate.py @@ -5,9 +5,12 @@ Checks a container runtime (docker or podman); probes the SELECTED runtime (container image if present, else conda/host) for its exact vLLM/zentorch/torch versions and the active vLLM platform; applies the Venice stack-compatibility -gate; and checks host perf libraries (tcmalloc / OpenMP via LD_PRELOAD), -HF_TOKEN, and RAM. Each issue is error (blocks launch) / warning (degrades) / -advisory (info). +gate; checks AVX-512 BF16 (zentorch bf16 needs Zen4+) and, when absent, offers the +stock no-zentorch path via `requires_confirmation` + `zentorch_capable:false` rather +than dead-ending; resolves the HF cache mount (follows symlinks, flags NFS/root-squash +and returns the real -v mount to use); and checks host perf libraries (tcmalloc / +OpenMP via LD_PRELOAD), HF_TOKEN, and RAM. Each issue is error (blocks launch) / +warning (degrades) / advisory (info). The stack probe distinguishes zentorch-accelerated serving (a Zen platform is active) from an unaccelerated stock CPU platform. Pass `--generation` (from @@ -25,6 +28,7 @@ import argparse import json import os +import shlex import shutil import subprocess import sys @@ -61,6 +65,20 @@ def _sh(cmd, timeout=20): return 1, "", f"timed out after {timeout}s" +def _host_avx512(): + """Read AVX-512 BF16 support from the local CPU (lscpu flags). Returns + True/False, or None if it could not be determined. Checks `avx512_bf16` -- the + exact extension zentorch's bf16 CPU path needs -- mirroring detect.py, so the + gate does not depend on the agent passing the value through.""" + rc, out, _ = _sh("lscpu") + if rc != 0 or not out: + return None + for ln in out.splitlines(): + if ln.strip().lower().startswith("flags:"): + return "avx512_bf16" in ln.lower().split() + return None + + def _detect_runtime(): """Pick an accessible container runtime: docker (daemon reachable) > podman (rootless). Returns (runtime, detail) or (None, why). @@ -103,15 +121,15 @@ def _probe_stack(run_prefix, source): return stack, None -def stack_compatibility(generation, stack, default_vllm=DEFAULT_VLLM_VERSION): +def stack_compatibility(generation, stack, zentorch_capable=True, default_vllm=DEFAULT_VLLM_VERSION): """Pure policy over the detected EPYC generation and the probed stack. Returns {"status", "message"} where status is one of: - - "blocked": zentorch/Zen platform is not active -> serving would be - unaccelerated stock CPU; a hard stop. - - "confirmation_required": Venice on a vLLM other than the validated default - -> warn, nudge to the pinned default image, and require explicit user - confirmation before continuing. + - "blocked": the CPU CAN run zentorch (Zen4+) but the Zen platform is not + active -> a misconfigured unaccelerated stack; a hard stop. + - "confirmation_required": either Venice on a vLLM other than the validated + default, OR the intentional stock (no-zentorch) path on a pre-Zen4 CPU that + cannot run zentorch -> warn and require explicit user confirmation. - "proceed": validated/expected stack. Returns None when there is no stack to judge. """ @@ -119,11 +137,16 @@ def stack_compatibility(generation, stack, default_vllm=DEFAULT_VLLM_VERSION): return None vllm_v = str(stack.get("vllm", "")).split("+")[0] if not stack.get("zen_active"): + if not zentorch_capable: + return {"status": "confirmation_required", + "message": (f"Serving with STOCK vLLM CPU ({stack.get('platform', '?')}) -- this CPU " + "cannot run zentorch (no AVX-512 BF16), so this is the no-zentorch fallback. " + "It is UNACCELERATED and not perf-validated by this recipe; confirm to proceed.")} return {"status": "blocked", "message": (f"vLLM {vllm_v or '?'} is on the stock CPU platform " f"({stack.get('platform', '?')}), not a Zen/zentorch platform -- zentorch " - "acceleration is NOT active. Enable zentorch or use the pinned container " - "image in data/epyc.json; do not serve an unaccelerated CPU stack.")} + "acceleration is NOT active though this CPU supports it. Enable zentorch or use " + "the pinned container image in data/epyc.json; do not serve an unaccelerated stack.")} if generation == "Venice" and vllm_v != default_vllm: return {"status": "confirmation_required", "message": (f"Venice (6th Gen EPYC) is on vLLM {vllm_v or '?'}, which this recipe has " @@ -140,11 +163,35 @@ def main(): p.add_argument("--image", default="", help="container image to check for (advisory)") p.add_argument("--generation", default="", help="epyc_generation from detect.py; enables the Venice stack-compatibility gate") + p.add_argument("--avx512", default="", choices=["", "true", "false"], + help="avx512 from detect.py; overrides the local lscpu read for the AVX-512 gate") args = p.parse_args() issues = [] stack = None # the probed runtime stack for the SELECTED path + # 0. AVX-512 BF16 -> zentorch viability. zentorch's bf16 CPU path needs + # avx512_bf16 (Zen4+: Genoa/Turin). Pre-Zen4 EPYC (Naples/Rome/Milan, 7000 + # series) lacks it, so zentorch CANNOT run. This is still an EPYC host, so + # rather than dead-ending we OFFER stock vLLM CPU (no zentorch acceleration) + # -- a warning that needs explicit user confirmation, NOT a hard error. Prefer + # the passed --avx512 (from detect.py; works for remote too); else read the + # local CPU so the signal holds even if it was not threaded through. + if args.avx512: + avx512 = args.avx512 == "true" + else: + avx512 = _host_avx512() + zentorch_capable = avx512 is not False # True, or None (undetermined) -> assume capable + if avx512 is False: + issues.append({"check": "avx512", "severity": "warning", + "message": "CPU lacks AVX-512 BF16 (avx512_bf16), so zentorch acceleration is NOT " + "available -- it needs Zen4+ (Genoa/Turin, the supported 9000 series). This " + "is a pre-Zen4 EPYC (Naples/Rome/Milan, 7000 series). You can still serve " + "with STOCK vLLM CPU (no zentorch): UNACCELERATED, but verified working on " + "EPYC 7763/Milan with bf16. Requires your explicit OK.", + "fix": "Confirm to serve on the stock (no-zentorch) path -- use the official stock image " + "vllm/vllm-openai-cpu:latest-x86_64 (see SKILL.md Step 6) -- or use a Zen4+ EPYC for zentorch."}) + # 1. Container runtime (prerequisite): docker > podman, else conda fallback. runtime, detail = _detect_runtime() conda_ok = _sh('python -c "import vllm, zentorch"')[0] == 0 @@ -199,8 +246,9 @@ def main(): issues.append({"check": "host_stack", "severity": "advisory", "message": "Host `import vllm, zentorch` not available; use the container path."}) - # 4. Venice stack-compatibility gate (needs a probed stack + a generation). - compatibility = stack_compatibility(args.generation, stack) + # 4. Stack-compatibility gate (needs a probed stack). Also decides the + # intentional stock (no-zentorch) path for pre-Zen4 CPUs. + compatibility = stack_compatibility(args.generation, stack, zentorch_capable) if compatibility: status = compatibility["status"] if status == "blocked": @@ -243,16 +291,56 @@ def main(): "message": f"LD_PRELOAD is missing {', '.join(missing)}; vLLM CPU warns about this and throughput suffers without them (host/conda path).", "fix": "export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$CONDA_PREFIX/lib/libiomp5.so:$LD_PRELOAD"}) + # 8. HF cache mount (container path). The default bind-mounts the HF cache into + # the container; on NFS homes / symlinked caches / root-squash that fails at + # `docker run` with permission-denied mkdir -- and it is NOT caught by "ready". + # Resolve the real path (follow symlinks), detect NFS, and hand back the mount + # the agent should actually use (or a local-disk fallback). + hf_cache = None + if runtime: + raw = os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface") + real = os.path.realpath(raw) + symlinked = real != os.path.abspath(raw) + fstype = "" + if os.path.exists(real): + fstype = _sh(f"stat -f -c %T {shlex.quote(real)}")[1] + on_nfs = "nfs" in fstype.lower() + mount = f"-v {real}:/root/.cache/huggingface" + hf_cache = {"requested": raw, "resolved": real, "symlinked": symlinked, + "fstype": fstype or "unknown", "on_nfs": on_nfs, "mount": mount} + if on_nfs or symlinked: + issues.append({"check": "hf_cache", "severity": "warning", + "message": (f"HF cache {raw} -> {real} (fstype {fstype or '?'}" + + (", symlink" if symlinked else "") + + (", NFS" if on_nfs else "") + "). A container bind-mount of an " + "NFS/symlinked cache can fail at container start with permission-denied " + "(root-squash cannot traverse it) -- and `ready` above does NOT test the mount."), + "fix": (f"Mount the RESOLVED path: {mount}. If it is NFS/root-squashed, use a local-disk " + "cache instead: mkdir -p /scratch/$USER/hf && export HF_HOME=/scratch/$USER/hf, " + "then -v /scratch/$USER/hf:/root/.cache/huggingface.")}) + elif not os.path.exists(real): + issues.append({"check": "hf_cache", "severity": "advisory", + "message": f"HF cache {real} does not exist yet; it is created on first run (models download then). " + f"Mount it with: {mount}"}) + else: + issues.append({"check": "hf_cache", "severity": "advisory", + "message": f"HF cache OK: {real} (local {fstype or '?'}). Mount with: {mount}"}) + errors = [i for i in issues if i["severity"] == "error"] - requires_confirmation = bool(compatibility and compatibility["status"] == "confirmation_required") + # Confirmation is required for a Venice-on-unvalidated-vLLM stack, or for the + # stock (no-zentorch) fallback on a CPU that can't run zentorch. + requires_confirmation = (avx512 is False) or bool( + compatibility and compatibility["status"] == "confirmation_required") result = { "ready": len(errors) == 0, "requires_confirmation": requires_confirmation, + "zentorch_capable": zentorch_capable, "runtime": runtime, "runtime_detail": detail, "conda_path_available": conda_ok, "stack": stack, "compatibility": compatibility, + "hf_cache": hf_cache, "ram_gb": ram_gb, "errors": errors, "warnings": [i for i in issues if i["severity"] == "warning"],