Enntity · data-angel · Sep 8, 2026 · Sep 8, 2026
@@ -0,0 +1,115 @@
+# SparkGLM under LLooM
+
+Build and qualify SparkGLM separately. LLooM manages the resulting Docker
+runtime: admission, worker-first TP2 startup, readiness, stop, routing and
+telemetry. This adapter uses the patchers baked into the selected SparkGLM
+image; it does not overlay the older Mia launcher implementation.
+
+Materialize a private, immutable-image recipe from the installed LLooM checkout:
+
+```sh
+mkdir -p "$HOME/.lloom/sparkglm-recipes"
+node backends/sparkglm/materialize.mjs \
+  --image-id "$HEAD_IMAGE_ID" --worker-image-id "$WORKER_IMAGE_ID" \
+  --source-revision "$SPARKGLM_COMMIT" \
+  --output "$HOME/.lloom/sparkglm-recipes/linux-nvidia-dgx-spark-2x-sparkglm-exl3.json"
+```
+
+Image IDs must be complete `sha256:` identities present on their respective
+nodes; source revision must be a complete commit. When the images are built
+independently, verify their serving source and extension hashes match. Omit
+`--worker-image-id` when both nodes have the same image. Keep the generated
+recipe directory and this backend directory identical on both installed nodes.
+Distributed setup invokes the same recipe on the worker.
+
+Review the ordinary setup plan, then apply and explicitly start:
+
+```sh
+lloom setup --recipe linux-nvidia-dgx-spark-2x-sparkglm-exl3 \
+  --recipes-root "$HOME/.lloom/sparkglm-recipes" --additive --no-auto-host --json
+lloom setup --recipe linux-nvidia-dgx-spark-2x-sparkglm-exl3 \
+  --recipes-root "$HOME/.lloom/sparkglm-recipes" --additive --no-auto-host --apply --yes --json
+lloom runtime-start glm53-flash-exl3-cluster --json
+lloom runtime-status glm53-flash-exl3-cluster --json
+lloom runtime-stop glm53-flash-exl3-cluster --json
+```
+
+Installing the profile does not set keep-warm or resume suspended routes.
+Independent Docker restart is disabled: LLooM owns distributed retries so a
+failed head cannot repeatedly reload beside a stale NCCL worker.
+Cluster and head readiness use `/v1/models` with the exact served-model ID.
+The EXL3, NVFP4 and fixture profiles share a port; a generic `/health` response
+from another active profile must not make a stopped runtime appear loaded.
+Use `lloom route glm53f-local --json` to inspect the strict local canary alias.
+Verify a streamed tool-call through that alias and confirm gateway metrics
+attribute it to `glm-5.3-flash-exl3`, in addition to checking both rank image
+identities and backend health. Run SparkGLM's documented shape warmup and
+qualification harness; a successful LLooM health check does not qualify a
+kernel or a release.
+
+`--e3` explicitly enables the unqualified E3 large-prefill experiment and must
+only select an image containing its source-locked adapter. `--tiny` generates
+an isolated `sparkglm-tiny` model/runtime with no production aliases. First
+build the documented tinyGLM fixture on each host and expose its snapshot at
+`${modelRoot}/sparkglm--tinyglm`. The launcher rejects dummy loading unless
+the config identifies the synthetic `tinyglm-v1` fixture and speculation is
+disabled. Stop the full runtime before starting the fixture.
+
+`--e3-policy concurrent` selects the newer experimental CPU-metadata policy
+that keeps solo prefill on the reference path. It requires the corresponding
+image helper; `--e3-trace` records initial selection decisions for qualification.
+
+`--nvfp4` generates the isolated `sparkglm-nvfp4` runtime for the separately
+pinned current compressed-tensors checkpoint. It cannot be combined with
+`--tiny` or `--e3`. This initial lane uses 8 GiB KV per rank and 65536 context;
+The original 262144 context failed vLLM cache admission with this budget.
+Use matched limits for diagnostic comparisons and independently tune finalists. Native kernel
+execution and model quality require separate evidence. None of these options
+changes the production Presence alias or promotes an experimental backend.
+Use `--nvfp4-budget` when materializing that EXL3 comparison arm; it selects
+the same explicit 8 GiB KV per rank, 65536 context, and admission reservation.
+It can be combined with `--e3`, but cannot be used with a tiny fixture.
+
+`--nvfp4-tiny` selects the separate `sparkglm-tiny-nvfp4` fixture. Build its
+reviewed dummy-loader image and metadata from SparkGLM's NVFP4 experiment
+first. The image must contain the guarded deterministic FP4 initializer;
+ordinary integer dummy weights are uninitialized in the underlying loader.
+This option uses no production aliases and cannot enable E3 or real NVFP4
+loading. Finish model downloads before full-model cold starts: downloader
+buffers count against GB10's shared memory budget.
+
+For performance experiments, `--mxfp8-draft` selects the separately pinned
+MXFP8 DFlash2 checkpoint and requires an image with its quantized selector,
+grouped-convolution and fused context K/V support. `--prefill-tokens N` varies prefill chunking. Draft TP must remain 2: the
+inherited DFlash loader preserves the target parallel group even if vLLM accepts
+a TP1 configuration flag. This adapter rejects that misleading request early. NVFP4 also
+accepts an explicit `--moe-backend`; vLLM must still validate its compatibility,
+including SwiGLU clamping and mixed quantization. These are experiment controls,
+not recommended defaults. Final performance selection may use different tuned
+settings for each target; the matched-budget option is a diagnostic control.
+
+`--context-tokens N` and `--kv-cache-gib N` explicitly tune context and per-rank
+KV allocation. Context capability must be reported with results; vLLM retains
+its cache admission check. Increasing KV requires actual host memory headroom.
+Budgets above 8 GiB also increase LLooM's per-rank admission reservation;
+smaller budgets preserve the existing conservative reservation.
+MoE overrides also accept the guarded NVFP4 tiny fixture for backend validation.
+
+`--exl3-temp-rows 32|64|128` tunes the thin/fat expert threshold. The default
+remains 128; a lower threshold requires operator and TP2 integration evidence.
+
+Large prefill chunks also increase the padded sliding-window draft cache
+reservation. The observed 16K-chunk profile needed 15.2 GiB even for one 64K
+request; smaller chunks are essential to explore on memory-limited NVFP4.
+
+Explicit `--context-tokens` also sets `maxPromptTokens: 0` on the gateway model.
+This delegates token-budget validation to vLLM's exact rendered-token check; it
+does not change or disable `max_model_len`. The gateway's character estimate and
+98% rejection margin otherwise reject some valid near-window requests. The
+backend still rejects prompt plus output exceeding its configured context.
+
+For explicit contexts above 524288 tokens, the managed no-output watchdog budget
+is 30 minutes, matching the backend request deadline. The inherited ten-minute
+budget interrupted a valid progressing 1M cold prefill. Watchdog recovery remains
+enabled; this change permits the initial prefill silence and does not claim
+endurance qualification or make a cold 1M request interactive.
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: MIT AND Apache-2.0
+# Adapted from LLooM's Mia launcher; SparkGLM supplies the built runtime.
+set -euo pipefail
+
+log() { printf '[sparkglm rank=%s] %s\n' "${NODE_RANK:-?}" "$*"; }
+
+: "${NODE_RANK:?NODE_RANK is required}"
+: "${CLUSTER_NODE_COUNT:?CLUSTER_NODE_COUNT is required}"
+: "${MASTER_ADDR:?MASTER_ADDR is required}"
+: "${MODEL_DIR:?MODEL_DIR is required}"
+
+# The inherited DFlash loader preserves the target parallel configuration;
+# accepting draft TP1 here would silently run TP2 and mislabel measurements.
+if [[ "${SPEC_METHOD:-dflash}" == "dflash" && "${DFLASH_DRAFT_TP:-2}" != "${CLUSTER_NODE_COUNT}" ]]; then
+  log "independent DFlash draft TP is not implemented by this adapter; use target TP=${CLUSTER_NODE_COUNT}"
+  exit 1
+fi
+
+[[ -f "${MODEL_DIR}/config.json" ]] || {
+  log "missing target config: ${MODEL_DIR}/config.json"
+  exit 1
+}
+
+if [[ "${SPARKGLM_EXL3_E3:-0}" == "1" ]]; then
+  [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_e3.py && -f /usr/local/lib/python3.12/dist-packages/exl3_fat_moe_ext.so ]] || {
+    log "selected image does not contain the E3 adapter and extension"; exit 1;
+  }
+  if [[ "${SPARKGLM_EXL3_E3_POLICY:-large}" == "concurrent" ]]; then
+    [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_e3_policy.py ]] || {
+      log "selected image does not contain the concurrent E3 policy"; exit 1;
+    }
+  fi
+fi
+if [[ "${SPARKGLM_NVFP4_TINY:-0}" == "1" ]]; then
+  [[ -f /usr/local/lib/python3.12/dist-packages/sparkglm_nvfp4_tiny.py && -f /usr/local/lib/python3.12/dist-packages/sparkglm_nvfp4_tiny.pth ]] || {
+    log "selected image does not contain the guarded NVFP4 fixture initializer"; exit 1;
+  }
+fi
+if [[ "${SPARKGLM_MXFP8_DRAFT:-0}" == "1" ]]; then
+  grep -q '_fused_kv_weight_scale' /opt/glm53/patch_dflash2.py || {
+    log "selected image does not contain MXFP8 DFlash2 context projection support"; exit 1;
+  }
+fi
+
+if [[ "${SPEC_METHOD:-dflash}" == "dflash" && ! -f "${DFLASH_MODEL_DIR:-}/config.json" ]]; then
+  log "missing DFlash2 config: ${DFLASH_MODEL_DIR:-unset}/config.json"
+  exit 1
+fi
+
+# Runtime patches come from the selected SparkGLM image, never the legacy
+# Mia files installed alongside LLooM. Image identity is pinned by the recipe.
+for patch in \
+  patch_glm_video_placeholders.py \
+  patch_suppress_stops_in_reasoning.py \
+  patch_scheduler_decode_floor.py \
+  patch_glm5_drafter_group.py \
+  patch_hybrid_prefix_hit.py \
+  patch_xgrammar_termination.py \
+  patch_kpool_tail_slotmap.py \
+  patch_spinwait.py \
+  patch_indexer_workspace.py \
+  patch_ablit.py; do
+  [[ -f "/opt/glm53/${patch}" ]] || {
+    log "missing SparkGLM runtime patch: /opt/glm53/${patch}"
+    exit 1
+  }
+  # These source patchers use only the standard library. Avoid importing the
+  # serving stack through site .pth hooks ten times during each cold start.
+  python3 -S "/opt/glm53/${patch}"
+done
+
+if [[ -z "${LIMIT_MM_PER_PROMPT:-}" ]]; then
+  LIMIT_MM_PER_PROMPT='{"image":4,"video":1}'
+fi
+
+args=(
+  --served-model-name "${SERVED_MODEL_NAME:-glm-5.3-flash-exl3}"
+  --host "${VLLM_HOST:-0.0.0.0}"
+  --port "${VLLM_PORT:-8890}"
+  --tensor-parallel-size "${CLUSTER_NODE_COUNT}"
+  --nnodes "${CLUSTER_NODE_COUNT}"
+  --node-rank "${NODE_RANK}"
+  --master-addr "${MASTER_ADDR}"
+  --master-port "${MASTER_PORT:-29521}"
+  --distributed-executor-backend mp
+  --tool-call-parser glm47
+  --enable-auto-tool-choice
+  --reasoning-parser glm45
+  --enable-prefix-caching
+  --no-enable-flashinfer-autotune
+  --quantization "${QUANTIZATION:-exl3}"
+  --max-model-len "${MAX_MODEL_LEN:-1000000}"
+  --gpu-memory-utilization "${GPU_MEMORY_UTILIZATION:-0.87}"
+  --max-num-seqs "${MAX_NUM_SEQS:-4}"
+  --max-num-batched-tokens "${MAX_NUM_BATCHED_TOKENS:-7168}"
+  --kv-cache-dtype "${KV_CACHE_DTYPE:-fp8}"
+  --chat-template /opt/glm53/chat_template.jinja
+  --limit-mm-per-prompt "${LIMIT_MM_PER_PROMPT}"
+  --skip-mm-profiling
+)
+
+if [[ "${SPARKGLM_TINY_DUMMY:-0}" == "1" ]]; then
+  python3 -S - "${MODEL_DIR}/config.json" <<'PYSAFE'
+import json, sys
+c = json.load(open(sys.argv[1]))
+if c.get("quantization_config", {}).get("version") != "tinyglm-v1":
+    raise SystemExit("dummy loading requires the synthetic tinyGLM fixture")
+PYSAFE
+  [[ "${SPEC_METHOD:-dflash}" == "none" ]] || { log "tinyGLM requires SPEC_METHOD=none"; exit 1; }
+  args+=(--load-format dummy --generation-config vllm)
+fi
+if [[ "${SPARKGLM_NVFP4_TINY:-0}" == "1" ]]; then
+  python3 -S - "${MODEL_DIR}/config.json" <<'PYSAFE'
+import json, sys
+if json.load(open(sys.argv[1])).get("_sparkglm_fixture") != "tinyglm-nvfp4-v1":
+    raise SystemExit("NVFP4 dummy loading requires the synthetic fixture")
+PYSAFE
+  [[ "${SPEC_METHOD:-dflash}" == "none" && "${QUANTIZATION:-exl3}" == "compressed-tensors" ]] || { log "invalid NVFP4 fixture options"; exit 1; }
+  args+=(--load-format dummy --generation-config vllm)
+fi
+if [[ "${LANGUAGE_MODEL_ONLY:-0}" == "1" ]]; then
+  args+=(--language-model-only)
+fi
+
+if [[ -n "${KV_CACHE_MEMORY_BYTES:-}" ]]; then
+  args+=(--kv-cache-memory-bytes "${KV_CACHE_MEMORY_BYTES}")
+fi
+if [[ -n "${MOE_BACKEND:-}" ]]; then
+  args+=(--moe-backend "${MOE_BACKEND}")
+fi
+
+if [[ "${NODE_RANK}" != "0" ]]; then
+  args+=(--headless)
+fi
+
+case "${SPEC_METHOD:-dflash}" in
+  dflash)
+    dflash_tokens="${DFLASH_TOKENS:-7}"
+    dflash_draft_tp="${DFLASH_DRAFT_TP:-2}"
+    [[ "${dflash_tokens}" =~ ^[0-9]+$ ]] || { log "invalid DFLASH_TOKENS=${dflash_tokens}"; exit 1; }
+    [[ "${dflash_draft_tp}" =~ ^[0-9]+$ ]] || { log "invalid DFLASH_DRAFT_TP=${dflash_draft_tp}"; exit 1; }
+    # Do not launch Python here: Mia's installed video .pth emits a status line
+    # on interpreter startup, which would contaminate command-substitution JSON.
+    printf -v spec '{"method":"dflash","model":"%s","num_speculative_tokens":%d,"kv_cache_dtype":"auto","draft_sample_method":"probabilistic","rejection_sample_method":"standard","draft_tensor_parallel_size":%d}' \
+      "${DFLASH_MODEL_DIR}" "${dflash_tokens}" "${dflash_draft_tp}"
+    args+=(--speculative-config "${spec}")
+    ;;
+  mtp)
+    args+=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS:-2}}")
+    ;;
+  none) ;;
+  *)
+    log "unsupported SPEC_METHOD=${SPEC_METHOD}"
+    exit 1
+    ;;
+esac
+
+if [[ "${ENFORCE_EAGER:-0}" == "1" ]]; then
+  args+=(--enforce-eager)
+else
+  args+=(--cudagraph-capture-sizes 1 2 4 8 16 24 32)
+fi
+
+log "starting SparkGLM quant=${QUANTIZATION:-exl3} TP=${CLUSTER_NODE_COUNT}, spec=${SPEC_METHOD:-dflash}, draft-tp=${DFLASH_DRAFT_TP:-2}, mnbt=${MAX_NUM_BATCHED_TOKENS:-7168}, E2=${EXL3_FAT_KERNEL:-1}"
+exec vllm serve "${MODEL_DIR}" "${args[@]}"