From a7cf64a252c7294fdaf28970b14ece3d3a17fcf5 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Tue, 11 Aug 2026 21:30:00 -0700 Subject: [PATCH 1/5] feat: bump mlx-swift-lm for glm_moe_dsa (GLM-5.2) support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Points at bfc2462, which brings two things: - SharpAI/mlx-swift-lm#48 — glm_moe_dsa / deepseek_v3_2 load and run with dense attention (stage 1 of #111). GLM-5.2 is DeepSeek V3.2, whose indexer is inert below index_topk (2048), so output is exact for the first 2048 positions of context and diverges beyond them. That is enough to exercise --stream-experts against the 308GB checkpoint, which is what the issue actually asks for. - SharpAI/mlx-swift-lm#47 — the all-KV-shared assistant regression tests, which had not been picked up by a bump yet. #48 also generalises a latent trap in DeepseekV3.sanitize, which dropped `model.layers.61` by string literal. That number is just numHiddenLayers; on GLM-5.2's 78 layers it would have deleted a real layer while keeping the MTP block. Verified past the registry: pointing the binary at a glm_moe_dsa config constructs the model and fails only on absent weights — Key model.embed_tokens.weight not found in DeepseekV32Model.DeepseekV3ModelInner.Embedding so the architecture is reachable end to end, not merely registered. No real weights have been run: the smallest glm_moe_dsa checkpoint is 308GB. Refs #111 Co-Authored-By: Claude Fable 5 --- mlx-swift-lm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlx-swift-lm b/mlx-swift-lm index 6a2c179..bfc2462 160000 --- a/mlx-swift-lm +++ b/mlx-swift-lm @@ -1 +1 @@ -Subproject commit 6a2c179998723107bb3d271963eeb9061056bee5 +Subproject commit bfc2462b975bc278411be041d5761299e836c846 From 59df52c7104df9a61256673c854dcdcf78a884f5 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Wed, 12 Aug 2026 11:57:40 -0700 Subject: [PATCH 2/5] fix: make the dependency automation work without a PAT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dependency Automation has failed all 12 times it has run since 2026-04-27 — it has never once succeeded. Every failure is the same: ##[error]Input 'token' not supplied. Unable to continue. The Create Pull Request step reads secrets.SWIFTLM_PR_TOKEN, which is not set in this repository. The dispatch side is fine: mlx-swift-lm's auto_release does hold a token that can dispatch cross-repo, so the event arrives and the job runs, does its work, and dies at the last step. Rather than add the secret, stop trying to open the PR. A workflow needs a personal access token to open one usefully because GitHub does not start workflow runs for events raised by GITHUB_TOKEN — a bot-opened PR would arrive with no checks at all, permanently pending rather than green, and release.yml gates releases on CI concluding successfully. A pushed branch plus a compare link in the job summary costs one click and gets real CI, because the PR event is then the human's. Keeping a human in that loop is not a consolation prize. Bumps here have needed a pointer check, an umbrella build and a smoke test before they were trustworthy; this does the mechanical part and leaves the judgement. Three further problems fixed while in here: - The mlx-swift branch ran `swift package update mlx-swift`, which does nothing: both dependencies are `.package(path: "./…")` local paths backed by submodules, and SwiftPM takes whatever is on disk for a path dependency. It could only ever have produced an empty commit. Both are now handled the same way, as the pointer move they are. - client_payload was interpolated straight into run blocks, so a crafted new_tag would have been executed rather than compared. Values are now validated (source_repo against an allowlist, new_tag against a plain-tag pattern) and passed through the environment. Verified rejecting `b554; rm -rf /`, `$(whoami)`, `b554 && curl evil.sh`, `../../../etc/passwd`, `-x` and empty, while accepting b554, b459 and v1.2.3. - A re-dispatch for a tag already checked out produced an empty commit; that case now reports and stops. Exercised against the real submodule: an already-current tag (b500) takes the no-op path, a nonexistent tag (b99999) fails with a clear message, and a real older tag (b497) computes bfc2462 → b320bc4. Co-Authored-By: Claude Opus 5 --- .github/workflows/update_dependencies.yml | 137 +++++++++++++++++----- 1 file changed, 108 insertions(+), 29 deletions(-) diff --git a/.github/workflows/update_dependencies.yml b/.github/workflows/update_dependencies.yml index 66d0271..85a599c 100644 --- a/.github/workflows/update_dependencies.yml +++ b/.github/workflows/update_dependencies.yml @@ -1,12 +1,25 @@ name: Dependency Automation +# Prepares a submodule bump when mlx-swift or mlx-swift-lm cuts a release, and stops +# at a pushed branch rather than opening a pull request. +# +# Opening the PR from a workflow needs a personal access token, because GitHub does +# not start workflow runs for events raised by GITHUB_TOKEN. A bot-opened PR would +# therefore arrive with no checks at all — permanently pending, never green — and this +# repository gates releases on CI concluding successfully (see release.yml). A branch +# is the honest stopping point: opening the PR yourself takes one click, and CI then +# runs normally because the event is yours. +# +# That a human sees the bump before it merges is a feature. Bumps here have needed a +# pointer check, an umbrella build and a smoke test to be trustworthy; the automation +# does the mechanical part and leaves the judgement. + on: repository_dispatch: types: [dependency_bump] permissions: contents: write - pull-requests: write jobs: bump-dependencies: @@ -18,34 +31,100 @@ jobs: submodules: recursive fetch-depth: 0 - - name: Update swift-mlx dependencies - if: ${{ github.event.client_payload.source_repo == 'mlx-swift' }} + # client_payload is attacker-controlled in principle — anything able to dispatch + # to this repository chooses these strings — and they end up in shell and in a + # ref name. Validate them here and pass them onward through the environment + # rather than interpolating ${{ }} into a run block, where a crafted tag would + # be executed rather than compared. + - name: Validate dispatch payload + env: + PAYLOAD_SOURCE_REPO: ${{ github.event.client_payload.source_repo }} + PAYLOAD_NEW_TAG: ${{ github.event.client_payload.new_tag }} run: | - echo "Bumping mlx-swift dependency to ${{ github.event.client_payload.new_tag }}" - # In Package.swift we depend on branch main, but if we wanted to depend on a tag: - # SwiftPM resolves "main" to the latest commit automatically, but updating the SPM resolved file ensures deterministic builds: - swift package update mlx-swift + set -euo pipefail + case "$PAYLOAD_SOURCE_REPO" in + mlx-swift|mlx-swift-lm) ;; + *) + echo "::error::unexpected source_repo '$PAYLOAD_SOURCE_REPO' — expected mlx-swift or mlx-swift-lm" + exit 1 + ;; + esac + if ! printf '%s' "$PAYLOAD_NEW_TAG" | grep -Eq '^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$'; then + echo "::error::refusing tag '$PAYLOAD_NEW_TAG' — not a plain tag name" + exit 1 + fi + { + echo "SOURCE_REPO=$PAYLOAD_SOURCE_REPO" + echo "NEW_TAG=$PAYLOAD_NEW_TAG" + } >> "$GITHUB_ENV" - - name: Update swift-mlx-lm dependencies - if: ${{ github.event.client_payload.source_repo == 'mlx-swift-lm' }} + # Both dependencies are `.package(path: "./…")` in Package.swift, backed by git + # submodules, so bumping either one is a pointer move. `swift package update` + # does nothing for a path dependency — SwiftPM takes whatever is on disk — which + # is why the mlx-swift branch of the previous version of this workflow could only + # ever have produced an empty commit. + - name: Move submodule to the released tag run: | - echo "Bumping local mlx-swift-lm submodule to ${{ github.event.client_payload.new_tag }}" - git submodule update --remote mlx-swift-lm - # Force the submodule onto the specific new release tag - cd mlx-swift-lm - git checkout ${{ github.event.client_payload.new_tag }} - cd .. - git add mlx-swift-lm - - - name: Create Pull Request - uses: peter-evans/create-pull-request@v6 - with: - token: ${{ secrets.SWIFTLM_PR_TOKEN }} - commit-message: "chore(deps): bump ${{ github.event.client_payload.source_repo }} to ${{ github.event.client_payload.new_tag }}" - title: "Update ${{ github.event.client_payload.source_repo }} Dependency to ${{ github.event.client_payload.new_tag }}" - body: | - Automated dependency update triggered by release `${{ github.event.client_payload.new_tag }}` in `SharpAI/${{ github.event.client_payload.source_repo }}`. - - This PR ensures SwiftLM is tracking the latest validated architectural improvements. - branch: "auto-update/${{ github.event.client_payload.source_repo }}-${{ github.event.client_payload.new_tag }}" - base: main + set -euo pipefail + git -C "$SOURCE_REPO" fetch --tags --force origin + if ! git -C "$SOURCE_REPO" rev-parse -q --verify "refs/tags/${NEW_TAG}^{commit}" >/dev/null; then + echo "::error::tag $NEW_TAG does not exist in $SOURCE_REPO" + exit 1 + fi + before=$(git rev-parse "HEAD:$SOURCE_REPO") + git -C "$SOURCE_REPO" checkout --detach "refs/tags/$NEW_TAG" + after=$(git -C "$SOURCE_REPO" rev-parse HEAD) + { + echo "BEFORE_SHA=$before" + echo "AFTER_SHA=$after" + } >> "$GITHUB_ENV" + if [ "$before" = "$after" ]; then + echo "ALREADY_CURRENT=1" >> "$GITHUB_ENV" + fi + + - name: Report an already-current submodule and stop + if: env.ALREADY_CURRENT == '1' + run: | + { + echo "### Nothing to bump" + echo + echo "\`$SOURCE_REPO\` is already at \`$NEW_TAG\` (\`${AFTER_SHA:0:7}\`)." + } >> "$GITHUB_STEP_SUMMARY" + + - name: Push the bump branch + if: env.ALREADY_CURRENT != '1' + run: | + set -euo pipefail + branch="auto-update/${SOURCE_REPO}-${NEW_TAG}" + git checkout -B "$branch" + git add "$SOURCE_REPO" + git \ + -c user.name='github-actions[bot]' \ + -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \ + commit -m "chore(deps): bump $SOURCE_REPO to $NEW_TAG + + Moves the $SOURCE_REPO submodule from ${BEFORE_SHA:0:7} to ${AFTER_SHA:0:7}, + the commit tagged $NEW_TAG. + + Prepared automatically; opened by hand so that CI runs against it." + # The auto-update/* namespace belongs to this workflow, so replacing a branch + # left by an earlier run for the same tag is safe and keeps re-runs idempotent. + git push --force origin "$branch" + echo "BUMP_BRANCH=$branch" >> "$GITHUB_ENV" + + - name: Summarise, with a link that opens the pull request + if: env.ALREADY_CURRENT != '1' + run: | + { + echo "### \`$SOURCE_REPO\` → \`$NEW_TAG\` is ready" + echo + echo "Branch \`$BUMP_BRANCH\` pushed, moving the submodule from" + echo "\`${BEFORE_SHA:0:7}\` to \`${AFTER_SHA:0:7}\`." + echo + echo "**[Open the pull request](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/compare/main...${BUMP_BRANCH}?expand=1)**" + echo + echo "No PR is opened here on purpose: GitHub does not start workflow runs" + echo "for events raised by \`GITHUB_TOKEN\`, so a bot-opened PR would never" + echo "get CI. Opening it yourself gets the checks this repository gates" + echo "releases on." + } >> "$GITHUB_STEP_SUMMARY" From 0afb039a4477d6dc9c45328b4976e5360efca3ce Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Wed, 12 Aug 2026 13:58:30 -0700 Subject: [PATCH 3/5] fix: pin the vision test's LFM2.5 model to an immutable revision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Main went red at 19:28 today with nothing changed on our side — the merge that preceded it touched only a workflow file. The failing job was integration_matrix (vision), and it reproduced on re-run, so it was not a flake. LiquidAI republished LFM2.5-VL-450M-MLX-4bit at 19:23, five minutes earlier. The new revision's chat template is one brace short of valid: old: {{- bos_token -}} new: {- bos_token -}} Every request against it returns HTTP 500, `parser('Unexpected token type: closeExpression')`. Confirmed by reproducing locally against the new revision, then restoring that single brace in a copy — same weights, same request, HTTP 200 with identical token counts. The fault is upstream, not a compatibility gap on our side, and no code change here would be the right response to a malformed template. CI never noticed the substitution because the vision job did not prefetch this model at all: the server fetched it mid-test and resolved the floating id to whatever was newest. So the job's result depended on what a third party published that afternoon. Pins the revision, prefetches it, and teaches ci-download-models.sh a `repo@revision` spec so any model can be pinned the same way. The test resolves the pinned snapshot on disk and falls back to the floating id with a printed note, so a local run without a prefetch still works but cannot quietly test a different revision than CI did. The test-vision.sh edit rotates the job's model cache key, so CI re-downloads rather than restoring a cache that now holds the broken revision. Verified: the vision test passes locally with the pin, both cases; the `repo@revision` split parses correctly for pinned and unpinned specs; the fallback path triggers and warns when the pinned snapshot is absent. Worth reporting upstream — LiquidAI's template is broken for every consumer, not just this repository. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 7 ++++++- scripts/ci-download-models.sh | 18 +++++++++++++++--- tests/test-vision.sh | 31 ++++++++++++++++++++++++++++++- 3 files changed, 51 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aa2b23e..177236a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -93,8 +93,13 @@ jobs: name_suffix: -gemma-e2b models: mlx-community/gemma-4-e2b-it-4bit test_model: mlx-community/gemma-4-e2b-it-4bit + # LFM2.5 used to be absent here and was fetched by the server mid-test, which + # is how it silently picked up a republished revision whose chat template was + # one brace short of valid and took main red. Prefetching it at a pinned + # revision is what makes this job depend on our code rather than on what + # upstream published that afternoon. - modality: vision - models: mlx-community/Qwen2-VL-2B-Instruct-4bit + models: mlx-community/Qwen2-VL-2B-Instruct-4bit LiquidAI/LFM2.5-VL-450M-MLX-4bit@10ce3604e42cd595497c47aaf67b7890e1e2a3b4 - modality: audio models: mlx-community/gemma-4-e4b-it-4bit - modality: graph diff --git a/scripts/ci-download-models.sh b/scripts/ci-download-models.sh index a2fa868..72c48e3 100755 --- a/scripts/ci-download-models.sh +++ b/scripts/ci-download-models.sh @@ -30,12 +30,24 @@ has_partial_files() { } download_one() { - local repo="$1" + local spec="$1" + # `repo@revision` pins to an immutable commit. Upstream repositories are mutable: + # LiquidAI republished LFM2.5-VL-450M-MLX-4bit with a malformed chat template + # (`{- bos_token -}}`, one brace short), which turned every request into an HTTP 500 + # and took main red with no change on our side. A floating tag means CI results + # depend on what a third party did that afternoon. + local repo="${spec%@*}" + local revision="" + if [ "$spec" != "$repo" ]; then + revision="${spec##*@}" + fi local dir="$HUB_DIR/models--${repo//\//--}" + local rev_args=() + [ -n "$revision" ] && rev_args=(--revision "$revision") for attempt in $(seq 1 "$ATTEMPTS"); do - echo "--- $repo (attempt $attempt/$ATTEMPTS, per-request timeout ${HF_HUB_DOWNLOAD_TIMEOUT}s)" - if hf download "$repo"; then + echo "--- $repo${revision:+ @ $revision} (attempt $attempt/$ATTEMPTS, per-request timeout ${HF_HUB_DOWNLOAD_TIMEOUT}s)" + if hf download "$repo" "${rev_args[@]}"; then if has_partial_files "$dir"; then echo "::warning::$repo downloaded but .incomplete files remain; retrying" else diff --git a/tests/test-vision.sh b/tests/test-vision.sh index c33276d..13e0f39 100755 --- a/tests/test-vision.sh +++ b/tests/test-vision.sh @@ -89,8 +89,37 @@ mkdir -p /tmp/vision_test # 28x28 black PNG (requires multiple of 28 for Qwen2-VL patch embedder) BASE64_IMG="iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAIAAAD9b0jDAAAAGUlEQVR4nO3BMQEAAADCoPVPbQdvoAAA6DQJTAABRMAOLAAAAABJRU5ErkJggg==" +# A model id names a moving target. On 2026-08-12 LiquidAI republished +# LFM2.5-VL-450M-MLX-4bit with a chat template one brace short of valid — +# `{- bos_token -}}` where the previous revision had `{{- bos_token -}}` — so every +# request became `parser('Unexpected token type: closeExpression')`, HTTP 500, and main +# went red five minutes later with nothing changed on our side. Restoring that single +# brace locally makes the same revision answer normally, so the fault is upstream, not +# a compatibility gap worth chasing here. +# +# Pinning is what stops a third party's afternoon from deciding whether this repository +# has a green build. Re-point it deliberately, when someone means to test a newer +# revision, and treat the failure that follows as a real result. +LFM_REPO="LiquidAI/LFM2.5-VL-450M-MLX-4bit" +LFM_REVISION="10ce3604e42cd595497c47aaf67b7890e1e2a3b4" + +# Resolve to the pinned snapshot on disk. Falling back to the bare repo id keeps a local +# `./tests/test-vision.sh` working without a prefetch, but says so — a run that quietly +# tested a different revision than CI did is worse than one that took a moment longer. +pinned_snapshot() { + local repo="$1" revision="$2" + local hub="${HF_HUB_CACHE:-${HF_HOME:-$HOME/.cache/huggingface}/hub}" + local dir="$hub/models--${repo//\//--}/snapshots/$revision" + if [ -d "$dir" ]; then + echo "$dir" + else + log "note: pinned revision ${revision:0:7} of $repo is not in the cache; using the floating id" + echo "$repo" + fi +} + run_case "mlx-community/Qwen2-VL-2B-Instruct-4bit" "$BASE_PORT" "yes" -run_case "LiquidAI/LFM2.5-VL-450M-MLX-4bit" "$((BASE_PORT + 1))" "yes" +run_case "$(pinned_snapshot "$LFM_REPO" "$LFM_REVISION")" "$((BASE_PORT + 1))" "yes" rm -rf /tmp/vision_test exit 0 From 1e93e37311c8a30ef023e575fc32ddb02aa56685 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Wed, 12 Aug 2026 15:51:22 -0700 Subject: [PATCH 4/5] fix: don't expand an empty array under set -u in the download script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first version of the revision-pinning change assembled the optional `--revision` flag into an array and expanded it unconditionally. The runners are macOS, which ships bash 3.2, where expanding an *empty* array under `set -u` is an unbound-variable error rather than expanding to nothing. Every unpinned download therefore failed, which took out every job that prefetches a model — speculative-decoding, dflash, ssd-draft-memory-guard — while the pinned path would have worked fine. Spelled the two calls out instead. Verified by running the script under /bin/bash 3.2 with `set -u` for both shapes: unpinned resolves to the current snapshot, `repo@revision` resolves to the pinned one. CI caught this, which is the system working; worth noting the local `bash -n` syntax check could not have, since the failure is a runtime expansion. Co-Authored-By: Claude Opus 5 --- scripts/ci-download-models.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/scripts/ci-download-models.sh b/scripts/ci-download-models.sh index 72c48e3..e063234 100755 --- a/scripts/ci-download-models.sh +++ b/scripts/ci-download-models.sh @@ -42,12 +42,14 @@ download_one() { revision="${spec##*@}" fi local dir="$HUB_DIR/models--${repo//\//--}" - local rev_args=() - [ -n "$revision" ] && rev_args=(--revision "$revision") for attempt in $(seq 1 "$ATTEMPTS"); do echo "--- $repo${revision:+ @ $revision} (attempt $attempt/$ATTEMPTS, per-request timeout ${HF_HUB_DOWNLOAD_TIMEOUT}s)" - if hf download "$repo" "${rev_args[@]}"; then + # Spelled out rather than assembling an args array: the runners are macOS, which + # ships bash 3.2, where expanding an empty array under `set -u` is an unbound + # variable error rather than nothing at all. That fails every unpinned download. + if { [ -n "$revision" ] && hf download "$repo" --revision "$revision"; } \ + || { [ -z "$revision" ] && hf download "$repo"; }; then if has_partial_files "$dir"; then echo "::warning::$repo downloaded but .incomplete files remain; retrying" else From a54a256d60bb1357756ed6efac2740642dc2d3b3 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Thu, 13 Aug 2026 18:15:05 -0700 Subject: [PATCH 5/5] fix: name a malformed chat template instead of failing every request MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When a checkpoint ships a chat template the Jinja parser rejects, SwiftLM used to load cleanly, report ready, open the port, and then return HTTP 500 on every request with parser('Unexpected token type: closeExpression') That message names neither the chat template, nor the model, nor the fact that the offending file came out of someone else's checkpoint. It reads as a broken server. Diagnosing the real instance of this — LiquidAI republishing LFM2.5-VL-450M-MLX-4bit with `{- bos_token -}}`, one brace short — took CI logs and a bisect across two model revisions, and that was with far more to work with than a user reporting it would have. Two changes: - Template failures now surface as MalformedChatTemplate, which names the model, points at chat_template.jinja / tokenizer_config.json, says the defect belongs to whoever publishes the checkpoint, and mentions pinning as the workaround. - The template is rendered once during load, before the port opens. A checkpoint that cannot produce a prompt now refuses to start rather than serving 500s indefinitely across restarts. A model with no chat template at all stays legitimate — base models ship without one and /v1/completions does not need it — so only a template that exists and fails to parse is treated as fatal. The startup probe is shaped like the simplest real request (one user turn, add_generation_prompt) rather than a bare minimum, so a failure is the template's rather than the probe's. Verified: the broken revision now exits 1 with the diagnostic and never opens the port. Five cached models covering both modalities, thinking and non-thinking, and two model families all still start normally — LFM2.5-VL-450M (good revision), Qwen2-VL-2B, Qwen2.5-0.5B, Qwen3-1.7B, LFM2-VL-1.6B. Contract suite: 10 passed, 0 failed, 2 skipped. Co-Authored-By: Claude Opus 5 --- Sources/SwiftLM/Server.swift | 85 ++++++++++++++++++++++++++++++++---- 1 file changed, 77 insertions(+), 8 deletions(-) diff --git a/Sources/SwiftLM/Server.swift b/Sources/SwiftLM/Server.swift index 626e4b8..f876a62 100644 --- a/Sources/SwiftLM/Server.swift +++ b/Sources/SwiftLM/Server.swift @@ -43,15 +43,50 @@ private struct HubDownloader: Downloader, Sendable { } private struct TransformersTokenizerLoader: TokenizerLoader, Sendable { + /// Carried purely so template diagnostics can name the offending checkpoint; the + /// on-disk directory is a snapshot hash and means nothing to the person reading it. + let modelId: String + init(modelId: String = "this model") { self.modelId = modelId } + func load(from directory: URL) async throws -> any MLXLMCommon.Tokenizer { let t = try await AutoTokenizer.from(modelFolder: directory) - return TransformersTokenizerBridge(t) + return TransformersTokenizerBridge(t, modelId: modelId) + } +} + +/// A chat template that the Jinja parser rejected. +/// +/// Worth its own type because the underlying error says only what the parser choked on +/// — `parser('Unexpected token type: closeExpression')` — with no hint that the subject +/// is a file shipped inside someone else's checkpoint. A model published with a +/// malformed template loads fine, starts fine, and then fails every request, which +/// reads as "the server is broken" rather than "this checkpoint is broken". That +/// happened for real: LiquidAI republished LFM2.5-VL-450M-MLX-4bit with +/// `{- bos_token -}}` (one brace short) and identifying it took a bisect across two +/// model revisions. +struct MalformedChatTemplate: Error, CustomStringConvertible { + let modelId: String + let underlying: String + + var description: String { + """ + The chat template shipped with \(modelId) is not valid Jinja and could not be \ + parsed: \(underlying). This is a defect in the model's chat_template.jinja (or \ + the chat_template field of its tokenizer_config.json), not in the request — \ + report it to whoever publishes the checkpoint. Pinning to an earlier revision \ + of the model is the usual workaround. + """ } } private struct TransformersTokenizerBridge: MLXLMCommon.Tokenizer, Sendable { let upstream: any Tokenizers.Tokenizer - init(_ upstream: any Tokenizers.Tokenizer) { self.upstream = upstream } + /// Only used to name the model in template diagnostics. + let modelId: String + init(_ upstream: any Tokenizers.Tokenizer, modelId: String = "this model") { + self.upstream = upstream + self.modelId = modelId + } func encode(text: String, addSpecialTokens: Bool) -> [Int] { upstream.encode(text: text, addSpecialTokens: addSpecialTokens) } @@ -73,6 +108,12 @@ private struct TransformersTokenizerBridge: MLXLMCommon.Tokenizer, Sendable { messages: messages, tools: tools, additionalContext: additionalContext) } catch Tokenizers.TokenizerError.missingChatTemplate { throw MLXLMCommon.TokenizerError.missingChatTemplate + } catch { + // A missing template is a legitimate state (base models have none) and keeps + // its own error above. Anything else here means the template exists but the + // parser would not take it, which is worth naming precisely. + throw MalformedChatTemplate( + modelId: modelId, underlying: String(describing: error)) } } } @@ -598,7 +639,7 @@ struct MLXServer: AsyncParsableCommand { print("[SwiftLM] Loading Omni-Language Model (Text + Vision + Audio)...") container = try await OmniModelFactory.shared.loadContainer( from: downloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: modelConfig ) { progress in tracker.printProgress(progress) @@ -607,7 +648,7 @@ struct MLXServer: AsyncParsableCommand { print("[SwiftLM] Loading VLM (vision-language model)...") container = try await VLMModelFactory.shared.loadContainer( from: downloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: modelConfig ) { progress in tracker.printProgress(progress) @@ -618,7 +659,7 @@ struct MLXServer: AsyncParsableCommand { // and the native prepareForMultimodal path extracts real mel features. container = try await OmniModelFactory.shared.loadContainer( from: downloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: modelConfig ) { progress in tracker.printProgress(progress) @@ -627,7 +668,7 @@ struct MLXServer: AsyncParsableCommand { print("[SwiftLM] Loading LLM (large language model)...") container = try await LLMModelFactory.shared.loadContainer( from: downloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: modelConfig ) { progress in tracker.printProgress(progress) @@ -674,7 +715,7 @@ struct MLXServer: AsyncParsableCommand { let draftDownloader = HubDownloader(hub: HubApi(downloadBase: cacheRoot)) let draftContainer = try await LLMModelFactory.shared.loadContainer( from: draftDownloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: draftConfig ) { progress in // Silent loading for draft model @@ -709,7 +750,7 @@ struct MLXServer: AsyncParsableCommand { let assistantDownloader = HubDownloader(hub: HubApi(downloadBase: cacheRoot)) let assistantContainer = try await LLMModelFactory.shared.loadContainer( from: assistantDownloader, - using: TransformersTokenizerLoader(), + using: TransformersTokenizerLoader(modelId: resolvedModelId), configuration: assistantConfig ) { _ in } mtpAssistantModelRef = await assistantContainer.perform { assistantContext in @@ -837,6 +878,34 @@ struct MLXServer: AsyncParsableCommand { print("[SwiftLM] 🧠 Auto-calibration (Wisdom) bypassed for SSD Streaming") } + // Render the chat template once before opening the port. A checkpoint whose + // template does not parse otherwise loads cleanly, reports ready, and then fails + // every single request — a shape that reads as a broken server rather than a + // broken model, and that survives restarts without ever explaining itself. + // Failing here instead costs one template render and makes the cause the first + // thing anybody sees. + // + // A model with no chat template at all is a different and legitimate case (base + // models ship without one, and /v1/completions does not need it), so that is + // passed over rather than treated as a defect. + do { + let probeTokenizer = await container.tokenizer + // Shaped like the simplest real request rather than a bare minimum: one user + // turn with a generation prompt is what every chat template is written to + // handle, so a failure here is the template's, not the probe's. + _ = try probeTokenizer.applyChatTemplate( + messages: [["role": "user", "content": "ping"]], + tools: nil, + additionalContext: ["add_generation_prompt": true] + ) + } catch let error as MalformedChatTemplate { + print("[SwiftLM] ❌ \(error.description)") + throw error + } catch { + // Missing template, or a template that needs context this probe does not + // supply. Neither is a reason to refuse to start. + } + print("[SwiftLM] Model loaded. Starting HTTP server on \(host):\(port)") // ── Capture CLI defaults into a shared config ──