feat(serve): the server now supports vision for streaming-experts (larger than RAM) models

CHANGELOG.md

@@ -17,6 +17,10 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   remap puts them back into MoonViT's interleaved layout.
 - You can now use `--stream-experts` with `--mmproj` with the cli run/chat
   commands. Server support for vision + streaming still to come.
+- The server accepts `stream: experts` on a VLM entry, so you can serve an
+  over-RAM VLM. gmlx puts the placement on the text tower, and the vision
+  tower stays resident. The server refuses `stream: cpu` on a VLM entry.
+  Send only one request at a time to a streamed entry (see streaming.md).
 
 ### Changed
 

docs/server-config.md

@@ -644,10 +644,15 @@ cache stay on GPU. With the decode feeder (default, below) it matches
 quantized KV cache extends the advantage to long context. `stream: cpu`
 instead runs the whole model on the CPU device: weights stream from the page
 cache, so a MoE bigger than the wired-memory budget stays serveable.
-Load-affecting (part of the residency identity), text-only models; rejected
-on VLM and speculative/MTP entries. A `stream: cpu` entry switches the whole
-process to the CPU device, so it suits a single-model server rather than
-mixing with GPU-resident models. (The old key `cpu_moe: full | hybrid` is a
+Load-affecting (part of the residency identity). `stream: experts` also
+applies to a VLM entry. gmlx puts the placement on the text tower, and the
+vision tower stays on the GPU. The server refuses `stream: cpu` on a VLM
+entry, and it refuses both values on a speculative/MTP entry. A `stream: cpu`
+entry switches the whole process to the CPU device, so it suits a
+single-model server rather than mixing with GPU-resident models. Send only
+one request at a time to a streamed entry. Concurrent requests turn off the
+streaming tier's decode accelerations, which need a one-token step (see
+[streaming.md](streaming.md)). (The old key `cpu_moe: full | hybrid` is a
 deprecated alias for `stream: cpu | experts` and warns at config load.)
 
 `moe_expert_mass: P` (a share in `(0, 1]`) installs the adaptive lossy

docs/streaming.md

@@ -59,8 +59,10 @@ model's hot set. The decode feeder's exit stats (printed at `-v` on
 `run` and `chat`, always in server logs) show the arena hit rate a
 session settled at.
 
-Streaming applies to text models - the server rejects `stream` on VLM
-and speculative entries. On the CLI, MTP composes with
+You can stream a text model, or the text tower of a vision model. The
+server accepts `stream: experts` on a VLM entry, and puts the placement
+on the text tower. The server refuses `stream: cpu` on a VLM entry. It
+also refuses `stream` on a speculative entry. On the CLI, MTP composes with
 `--stream-experts` (not `--stream-cpu`) but defers by default: auto-MTP
 stays off and an explicit `--speculative` opts in. The lossy `--moe-*`
 levers below are hard-incompatible with MTP and force plain decoding.
@@ -176,6 +178,14 @@ In server configs the placement is the per-model `stream: experts | cpu`
 key and the feeder opt-outs are `prefill_feeder: false` /
 `decode_feeder: false`.
 
+Send only one request at a time to a server that streams a model. The
+engine can batch concurrent requests, but the streaming tier makes decode
+faster only when a step contains one token. The wired-memory refresh, the
+lookahead prestage, and the GPU-side token path each test the step width,
+and each stays off when a step contains more than one token. A second
+concurrent request thus puts both requests on the slow path, and the
+arena loses the hit rate that it built. This does not affect prefill.
+
 When a larger-than-RAM model is released, its page cache is also released, via
 `msync(MS_INVALIDATE)` over the shards - at process exit, or at unload on a
 running server (`GMLX_RELEASE_PAGECACHE=0` disables).

docs/vlm.md

@@ -69,9 +69,7 @@ llama.cpp's `build_rope_2d` reads. The remap puts them back into the
 interleaved layout of MoonViT. A mis-decoded mmproj thus gives confidently
 wrong image descriptions, and not a load error. GLM-5.2-V has the same vision
 encoder on a different text arch, and gmlx refuses it by name. K2.x is an
-over-RAM MoE, so use `--stream-experts` with `--mmproj`. gmlx puts the
-placement on the text tower after it loads the model, and the vision tower
-stays resident.
+over-RAM MoE, so it needs the streaming placement in the caveats below.
 
 Qwen2-VL / Qwen2.5-VL mmprojs (`qwen2vl_merger`) are not supported yet. The
 load fails up front with the family named. LLaVA's image processor isn't
@@ -111,6 +109,13 @@ consumer.
   runs as a plain text model (the vision side is simply absent).
 - Adapters (`--adapter`) don't combine with `--mmproj` yet -- live GGUF LoRA is
   text-path-only and errors loudly.
+- You can use `--stream-experts` (server key `stream: experts`) with
+  `--mmproj`. Use this combination for an over-RAM multimodal MoE. gmlx puts
+  the placement on the text tower after it loads the model, and the vision
+  tower stays on the GPU. You cannot use `--stream-cpu` (server key
+  `stream: cpu`) with `--mmproj`. That mode moves the process to the CPU
+  device, and it moves the vision tower with it. Send only one request at a
+  time to a server that streams a VLM (see [streaming.md](streaming.md)).
 - Speculative decoding (`--speculative`) *does* combine with `--mmproj` when a drafter
   is available -- a native MTP head (e.g. Qwen3.5/3.6) or `--draft-gguf`. Text-only
   turns speculate; media turns fall back to plain decode. With `--mmproj` but no

gmlx/server_bridge_vlm.py

@@ -480,6 +480,61 @@ def _apply_gguf_adapter(raw_model, config, adapter_gguf: str,
                               base_arch=base_arch)
 
 
+def _install_stream_placement(
+    text_model,
+    *,
+    gguf_path: str,
+    stream,
+    feeder_prefill: bool | None,
+    feeder_decode: bool | None,
+    moe_experts: int | None = None,
+    moe_expert_mass: float | None = None,
+    moe_miss_shed: float | None = None,
+    moe_prestage: str | None = None,
+    moe_layer_shed: float | None = None,
+) -> None:
+    """Put the execution placement and the lossy MoE levers on a text tower.
+
+    ``text_model`` must be the text tower. On a VLM, use the
+    ``language_model``. The vision tower is small, and it stays on the GPU.
+
+    The levers come after the placement, because each one hooks the streamed
+    layers that the placement makes.
+    """
+    if stream == "cpu":
+        # The whole model runs on the CPU device, and this device change
+        # applies to the process. Use it for one over-RAM model. Do not mix
+        # it with GPU-resident models in one config-mode server.
+        from .loader import configure_stream_cpu
+        configure_stream_cpu(
+            text_model, gguf_path=gguf_path,
+            feeder_prefill=feeder_prefill, feeder_decode=feeder_decode)
+    elif stream:  # "experts": routed experts stream; rest of model + KV on GPU
+        from .loader import install_expert_streaming
+        install_expert_streaming(
+            text_model, gguf_path=gguf_path,
+            feeder_prefill=feeder_prefill, feeder_decode=feeder_decode)
+    if moe_experts is not None:
+        from .loader import install_moe_experts_override
+        install_moe_experts_override(text_model, moe_experts)
+    if moe_expert_mass is not None:
+        from .moe_experts import install_moe_expert_mass
+        install_moe_expert_mass(text_model, moe_expert_mass)
+    if moe_miss_shed is not None:
+        from .moe_experts import install_moe_miss_shed
+        install_moe_miss_shed(text_model, moe_miss_shed)
+    if moe_prestage == "keepers":
+        if moe_miss_shed is None:
+            print("[stream] moe_prestage: keepers ignored: it needs "
+                  "moe_miss_shed")
+        else:
+            from .moe_experts import install_moe_prestage_keepers
+            install_moe_prestage_keepers(text_model)
+    if moe_layer_shed is not None:
+        from .moe_experts import install_moe_layer_shed
+        install_moe_layer_shed(text_model, moe_layer_shed)
+
+
 def load_serveable_model(
     gguf_path: str,
     *,
@@ -524,11 +579,13 @@ def load_serveable_model(
     text path (no merge, no requant). The VLM and speculative/MTP paths don't yet wire
     adapter apply, so an adapter on those raises rather than silently dropping it.
 
-    ``stream`` selects the text-path execution placement: ``"experts"`` streams
+    ``stream`` selects the execution placement: ``"experts"`` streams
     only the routed-expert stacks from disk while the every-token layers + KV
     cache stay on GPU; ``"cpu"`` runs the whole model on the CPU device (all
-    weights streamed through the page cache). Like the adapter, the VLM and
-    MTP paths raise rather than silently dropping it.
+    weights streamed through the page cache). ``"experts"`` also applies to a
+    VLM base. It goes on the text tower, and the vision tower stays on the
+    GPU. The MTP path, and ``"cpu"`` on a VLM, raise rather than silently
+    dropping it, as the adapter does.
 
     The lossy MoE levers (config ``moe_experts: K`` / ``moe_expert_mass: P`` /
     ``moe_miss_shed: P`` / ``moe_layer_shed: P``) install their filters/hooks
@@ -538,14 +595,18 @@ def load_serveable_model(
     prestage through the miss-shed policy and additionally needs
     ``moe_miss_shed`` (announced as ignored without it).
     """
-    def _reject_unwired(base_kind: str) -> None:
+    def _reject_unwired(base_kind: str, *, streamable: bool = False) -> None:
         # Raising beats silently dropping the option on bases that don't
-        # wire it yet.
+        # wire it yet. A streamable base accepts stream: experts, which goes
+        # on the text tower. It refuses stream: cpu, because that mode moves
+        # the process to the CPU device and moves the vision tower with it.
+        # A speculative base refuses both, because the engine loads the
+        # drafter after this function, and the drafter gets no placement.
         if adapter_gguf is not None:
             raise NotImplementedError(
                 f"live GGUF LoRA on a {base_kind} base is not wired yet; "
                 f"adapter={adapter_gguf!r}")
-        if stream:
+        if stream and not (streamable and stream == "experts"):
             raise NotImplementedError(
                 f"stream placement on a {base_kind} base is not wired yet; "
                 f"stream={stream!r}")
@@ -563,6 +624,11 @@ def _reject_unwired(base_kind: str) -> None:
                 )
         moe_experts = moe_expert_mass = None
         moe_miss_shed = moe_layer_shed = moe_prestage = None
+    _levers = dict(
+        moe_experts=moe_experts, moe_expert_mass=moe_expert_mass,
+        moe_miss_shed=moe_miss_shed, moe_prestage=moe_prestage,
+        moe_layer_shed=moe_layer_shed)
+
     if mmproj_path is not None and speculative:
         # VLM x MTP: text-only requests speculate; image/audio requests prefill media
         # into the KV and decode normally (verify is token-only over that cache).
@@ -572,8 +638,14 @@ def _reject_unwired(base_kind: str) -> None:
             draft_gguf_path=draft_gguf_path, chat_template=chat_template)
 
     if mmproj_path is not None:
-        _reject_unwired("VLM")
-        return _load_serveable_vlm(gguf_path, mmproj_path, hf_source=hf_source)
+        _reject_unwired("VLM", streamable=True)
+        model, processor, config = _load_serveable_vlm(
+            gguf_path, mmproj_path, hf_source=hf_source)
+        _install_stream_placement(
+            getattr(model, "language_model", model), gguf_path=gguf_path,
+            stream=stream, feeder_prefill=feeder_prefill,
+            feeder_decode=feeder_decode, **_levers)
+        return model, processor, config
 
     if speculative:
         _reject_unwired("speculative/MTP")
@@ -598,38 +670,10 @@ def _reject_unwired(base_kind: str) -> None:
     if adapter_gguf is not None:
         _apply_gguf_adapter(raw_model, config, adapter_gguf,
                             base_gguf_path=gguf_path)
-    if stream == "cpu":
-        # Whole model on the CPU device (process-global). Intended for a single
-        # over-RAM positional model; mixing with GPU-resident models in one
-        # config-mode server is unsupported.
-        from .loader import configure_stream_cpu
-        configure_stream_cpu(
-            raw_model, gguf_path=gguf_path,
-            feeder_prefill=feeder_prefill, feeder_decode=feeder_decode)
-    elif stream:  # "experts": routed experts stream; rest of model + KV on GPU
-        from .loader import install_expert_streaming
-        install_expert_streaming(
-            raw_model, gguf_path=gguf_path,
-            feeder_prefill=feeder_prefill, feeder_decode=feeder_decode)
-    if moe_experts is not None:
-        from .loader import install_moe_experts_override
-        install_moe_experts_override(raw_model, moe_experts)
-    if moe_expert_mass is not None:
-        from .moe_experts import install_moe_expert_mass
-        install_moe_expert_mass(raw_model, moe_expert_mass)
-    if moe_miss_shed is not None:
-        from .moe_experts import install_moe_miss_shed
-        install_moe_miss_shed(raw_model, moe_miss_shed)
-    if moe_prestage == "keepers":
-        if moe_miss_shed is None:
-            print("[stream] moe_prestage: keepers ignored: it needs "
-                  "moe_miss_shed")
-        else:
-            from .moe_experts import install_moe_prestage_keepers
-            install_moe_prestage_keepers(raw_model)
-    if moe_layer_shed is not None:
-        from .moe_experts import install_moe_layer_shed
-        install_moe_layer_shed(raw_model, moe_layer_shed)
+    _install_stream_placement(
+        raw_model, gguf_path=gguf_path, stream=stream,
+        feeder_prefill=feeder_prefill, feeder_decode=feeder_decode,
+        **_levers)
     processor = _make_text_processor(tokenizer)
     # mlx-lm-arch caches must carry the vlm runtime's class identities or
     # apc/ar isinstance-gates (own classes since mlx-vlm 0.6.4) resolve

gmlx/vlm.py

@@ -1929,6 +1929,33 @@ def _synthesize_pixtral_processor(tokenizer, mm_meta: dict):
     return _attach_streaming_helpers(processor, tokenizer)
 
 
+def _kimi_k25_image_processor(**params):
+    """A ``KimiK25ImageProcessor`` whose outputs are evaluated, not lazy.
+
+    The stock processor resizes and patchifies in MLX, thus its outputs are
+    lazy graphs on the default stream of the thread that runs it. MLX default
+    streams are per-thread, and a thread cannot evaluate a lazy array from a
+    different thread. The server preprocesses a request on a caller thread
+    and evaluates on the engine thread, so the stock outputs fail there with
+    "There is no Stream(gpu, N) in current thread". This subclass evaluates
+    the outputs on the thread that builds them. The other supported families
+    preprocess in numpy, so they do not need this today, but mlx-vlm has many
+    MLX-native processors: give a family like this the same wrapper.
+    """
+    from mlx_vlm.models.kimi_k25.processing_kimi_k25 import (
+        KimiK25ImageProcessor,
+    )
+
+    class _Eager(KimiK25ImageProcessor):
+        def preprocess(self, images, return_tensors=None, **kwargs):
+            out = super().preprocess(
+                images, return_tensors=return_tensors, **kwargs)
+            mx.eval([v for v in out.values() if isinstance(v, mx.array)])
+            return out
+
+    return _Eager(**params)
+
+
 def _synthesize_kimi_k25_processor(tokenizer, mm_meta: dict):
     """Build the Kimi-K2.5/K2.7 processor from the GGUFs alone - no HF download.
 
@@ -1940,7 +1967,7 @@ def _synthesize_kimi_k25_processor(tokenizer, mm_meta: dict):
     for it. Its value is 512, the bound of MoonViT's 2-D RoPE table.
     """
     from mlx_vlm.models.kimi_k25.processing_kimi_k25 import (
-        KimiK25ImageProcessor, KimiK25Processor,
+        KimiK25Processor,
     )
 
     patch_size = _mm_int(mm_meta, "clip.vision.patch_size")
@@ -1951,7 +1978,7 @@ def _synthesize_kimi_k25_processor(tokenizer, mm_meta: dict):
     in_token_limit = (int(max_pixels) // (patch_size ** 2)
                       if max_pixels else 16384)
 
-    image_processor = KimiK25ImageProcessor(
+    image_processor = _kimi_k25_image_processor(
         patch_size=patch_size,
         image_mean=tuple(image_mean),
         image_std=tuple(image_std),

tests/test_ckpt_tier.py

@@ -1230,6 +1230,9 @@ def test_anchor_never_shadows_a_deeper_disk_skeleton(tmp_path):
         assert ckpt_store(man, ids[:32], shallow,
                           extra_hash=4, kind="anchor")
         assert ckpt_store(man, ids[:64], deep, extra_hash=4)
+        # The lookup below reads the deep skeleton off disk, so the async
+        # writer has to have published it first.
+        drain_disk(disk)
         # Drop the deep record from memory, keeping its disk skeleton:
         # exactly what strip-on-extend leaves behind as a chain deepens.
         idx = _ckpt_records(man)

tests/test_config.py

@@ -693,17 +693,6 @@ def test_float32_is_not_a_config_value():
         build_config(doc)
 
 
-def test_dtype_is_not_a_per_model_load_key():
-    """The reason to leave bfloat16 is a property of the GPU, not of a model,
-    so `load: {dtype: ...}` is an unknown key: it warns and is dropped rather
-    than forking one model onto a different activation width."""
-    assert "dtype" not in cfgmod.LOAD_ENV
-    doc = _doc()
-    doc["models"]["m-bare"]["overrides"] = {"load": {"dtype": "float16"}}
-    cfg = build_config(doc)
-    assert "GMLX_ACTIVATION_DTYPE" not in cfgmod.env_for(resolve_model("m-bare", cfg))
-
-
 def test_env_for_emits_speculative_flag():
     """The per-build speculative state rides the env window (the only signal that
     reaches the load bridge in the engine's worker thread). Always emitted - "0"

tests/test_kimi_k25_vision.py

@@ -217,3 +217,32 @@ def test_config_synth_without_the_placeholder_token():
     cfg = _synthesize_kimi_k25_vlm_config(
         {"vocab_size": 10}, MM_META, {"tokenizer.ggml.tokens": ["a"]})
     assert "media_placeholder_token_id" not in cfg
+
+
+# mlx-vlm's KimiK25ImageProcessor.to_mlx still calls Image.getdata, which
+# Pillow deprecates for removal in Pillow 14. Upstream's call, not ours.
+@pytest.mark.filterwarnings("ignore:Image.Image.getdata is deprecated")
+def test_image_processor_output_survives_a_thread_hop():
+    """The server preprocesses a request on a caller thread and evaluates on
+    the engine thread. The stock KimiK25ImageProcessor returns lazy MLX
+    graphs, and a different thread cannot evaluate them ("There is no
+    Stream(gpu, N) in current thread"). The synthesized processor evaluates
+    its outputs on the building thread."""
+    pytest.importorskip("mlx_vlm")
+    import threading
+
+    from PIL import Image
+
+    from gmlx.vlm import _kimi_k25_image_processor
+
+    proc = _kimi_k25_image_processor(
+        patch_size=14, image_mean=(0.5, 0.5, 0.5), image_std=(0.5, 0.5, 0.5),
+        in_token_limit=1024, merge_kernel_size=[2, 2],
+        patch_limit_on_one_side=64)
+    img = Image.new("RGB", (56, 42), (200, 30, 90))
+    box = {}
+    t = threading.Thread(target=lambda: box.update(proc.preprocess([img])))
+    t.start()
+    t.join()
+    mx.eval(box["pixel_values"], box["image_grid_hws"])
+    assert box["pixel_values"].shape[1:] == (3, 14, 14)