`mlx_lm.server` aborts with Metal OOM after prompt cache grows to ~23-26 GB

[agisilaos]

Summary

mlx_lm.server crashed twice while serving OpenAI-compatible chat completion requests for mlx-community/Qwen3.5-4B-MLX-8bit.

The immediate crash is an uncaught Metal out-of-memory exception:

libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
zsh: abort      mlx_lm.server --model mlx-community/Qwen3.5-4B-MLX-8bit

Before each abort, the server logs show the prompt cache growing substantially:

• First run: prompt cache reached 23.35 GB, then aborted shortly after.
• Second run: prompt cache reached 26.28 GB, then aborted shortly after.

There were also repeated BrokenPipeError: [Errno 32] Broken pipe traces when clients disconnected during streaming/progress callbacks. Those did not kill the server by themselves, but may be relevant because interrupted requests appear to leave cache state behind.

Environment

• macOS: 27.0, build 26A5353q
• Architecture: arm64
• RAM: 48 GB (51539607552 bytes)
• Python: 3.14.6
• mlx-lm: 0.31.3
• mlx: 0.31.2
• huggingface_hub: 1.18.0
• Model: mlx-community/Qwen3.5-4B-MLX-8bit

Command

mlx_lm.server --model mlx-community/Qwen3.5-4B-MLX-8bit

The server started on the default host/port:

Starting httpd at 127.0.0.1 on port 8080...

Observed Behavior

The server accepted multiple POST /v1/chat/completions requests. Some requests used very large prompts, for example:

Prompt processing progress: 2048/55817
...
Prompt processing progress: 55817/55817

and after restart:

Prompt processing progress: 2048/64523
...
Prompt processing progress: 22765/64523
BrokenPipeError: [Errno 32] Broken pipe

The prompt cache then grew over time:

Prompt Cache: 10 sequences, 20.50 GB
Prompt Cache: 10 sequences, 23.35 GB

and on the second run:

Prompt Cache: 10 sequences, 20.11 GB
Prompt Cache: 10 sequences, 22.07 GB
Prompt Cache: 10 sequences, 23.53 GB
Prompt Cache: 10 sequences, 24.93 GB
Prompt Cache: 10 sequences, 25.69 GB
Prompt Cache: 10 sequences, 26.28 GB

Then the process aborted:

libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
zsh: abort      mlx_lm.server --model mlx-community/Qwen3.5-4B-MLX-8bit

Expected Behavior

The server should avoid aborting the process when memory pressure is high. Ideally it would either:

• enforce a safer default prompt cache memory limit,
• evict prompt cache entries before the Metal command buffer fails,
• return an HTTP error for requests that cannot be served within available memory,
• or catch/report the OOM in a controlled way instead of terminating the process.

It would also be helpful if BrokenPipeError from disconnected clients during keepalive/final SSE writes were handled quietly or caused associated request/cache cleanup.

Notes

The help text shows cache limiting options exist:

--prompt-cache-size PROMPT_CACHE_SIZE
--prompt-cache-bytes PROMPT_CACHE_BYTES

No explicit cache limit was passed in this run, so this was default behavior.

[VitorLudke]

Hit the same abort on an M5 / 16 GB — can add the root-cause picture and a repro, since the byte limit being dead is what lets the cache grow like this.

Why --prompt-cache-bytes doesn't help: the server constructs the cache from --prompt-cache-size only (LRUPromptCache(model_provider.cli_args.prompt_cache_size), server.py:1743), so max_bytes stays at the default 1 << 63. The only byte enforcement is the trim_to() call in the batch-generator branch (server.py:795-798) — and batching is disabled entirely when a draft model is loaded (is_batchable = draft_model is None, server.py:371). So on the sequential path — which is every server started with --draft-model — the flag is parsed and silently ignored. With the default of 10 entries and no byte limit, chat-app-sized prompts (~7k tokens ≈ 1.9 GB of KV on an 8B model) walk straight past the GPU budget.

Repro (16 GB Mac, mlx-lm 0.31.3):

mlx_lm.server --model mlx-community/Qwen3-8B-4bit \
  --draft-model mlx-community/Qwen3-0.6B-4bit \
  --prompt-cache-bytes 2G --port 8081
# send 3-4 chat completions with distinct ~7k-token prompts

Prompt Cache: 1 sequences, 1.81 GB
Prompt Cache: 2 sequences, 3.62 GB
Prompt Cache: 3 sequences, 5.44 GB
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed: Insufficient Memory
(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Re: the fix — #1118 and #1392 both wire the flag into the LRUPromptCache constructor, which is the right layer: insert_cache already evicts by max_bytes (models/cache.py:1733), so the ceiling gets enforced at insert time on every serving path, and earlier than the batched path's post-insert trim (which only runs when the next request arrives, i.e. after the peak). We've been running exactly this constructor change in production via a wrapper: the cache pins at the configured ceiling and the SIGABRTs stopped.

One related amplifier, filed separately as #1395: fetch_nearest_cache deep-copies the matched entry's KV, so reusing a cached conversation transiently holds 2× its KV — which can abort the server even with the byte limit enforced.