Cache state regression causes step confusion/repetition after several turn

[lessu]

Issue Title

Regression Cache state regression causes step confusion/repetition after several turn

Description

Summary

Enabling prefix cache causes severe context state regression in long-context sessions (50k-100k tokens).
The model exhibits "step confusion": it forgets current progress and reverts to previous steps (e.g., executing Step 3 while claiming to be at Step 5), or fails to recognize file modifications it just made.
Crucially, this issue is NOT present in v0.1.12, where 90k context sessions complete successfully.
And not reproducible when turn off cache.

Environment

• Affected(Tested) Versions: v0.2.14, v0.2.21, v0.3.6
• Working Version: v0.1.12
• Model: Qwen3.6-35B-A3B-MLX-8bit
• Block Size: Auto-aligned to 2048
• Cache Config: Tiered/SSD Cache Enabled, Hot Cache Disabled (hot_cache_max_bytes=0)
• MacOs 26.4.1, M5 Max 128G
• Opencode / Claude code

Reproduction Scenario

1. Turn 1: Ask model to write a pinball game (Context ~20k).
2. Turn 2: Ask model to execute Open in browser ( some time, module repeatedly open the browser)
3. Turn 3: Ask model to change js to typescript 5 (Context ~60k+).
   • Bug: Model is trying to do step5 in TODO list, but suddenly claims it is doing Step 3 again or back to create TODO list.
   • Bug: Model modifies a file, but in the next turn claims the file is unmodified.

Logs & Evidence

Attachments

• Link to Log: With Cache (Failing - showing step regression), tested on tag 0.3.6, all the other version shows simular cases
• Link to Log: Without Cache (Working), tested on tag 0.3.6
• Link to Log: v0.1.12 (Working)

session-0.1.2-ok.md

session-0.3.6_cached.md

session-ses_0.3.6_nocache.md

[lessu]

I'm not sure if it is a common issue, or specific to my hardware sets.
I have being tracking this issue from 0.2.20, initially I thought it was the model issue (Qwen3.5 122b-a10b-q4), but now the model is absolutely good, Test also on LM Studio, which also performs good.

And Qwen3.6-27B-MLX-Q8 also has same issue.

[jundot]

Wanted to share what I'm seeing on my end. I gave 0.3.8.dev3 a try on a few model + cache combos (paged + SSD cache on, hot cache off, block size 2048, M3 Ultra) and couldn't reproduce the regression in my runs. To be clear, earlier dev builds did show similar issues on the same models for me, so this is not me saying "it never happened". It is me saying that on 0.3.8.dev3 the matrix below came back clean for me.

One reason I think it's worth a fresh test on your side: 0.3.8 changed the VLM inference path quite a bit, and several cache-related code paths that fed into the regression got reworked. So the relevant code on 0.3.8.dev3 is not the same code as in 0.3.6 where you originally hit it.

Method: send the same prompt twice with greedy decoding (temperature 0, max_tokens 128). Run 1 is cold prefill, run 2 is prefix cache hit. PASS = byte-identical outputs.

Coverage matrix (each model exercises a different cache type):

model	cache types	engine
Qwen3-4B-bf16	KVCache	mlx-lm
Qwen3.6-35B-A3B-oQ6	KVCache + ArraysCache (hybrid linear attn)	mlx-vlm
Qwen3.6-35B-A3B-oQ4	KVCache + ArraysCache (hybrid linear attn)	mlx-lm
gemma-4-26B-A4B-it-oQ8	KVCache + RotatingKVCache (sliding window)	mlx-vlm
gemma-4-26B-A4B-it-oQ5	KVCache + RotatingKVCache (sliding window)	mlx-lm
gpt-oss-120b-MXFP4-Q8	KVCache + RotatingKVCache (sliding window)	mlx-lm

Three scenarios per model: 2k single greedy, 64k single greedy, 8k 3-concurrent batched greedy. Plus a dedicated cached-prefix-length sweep on Qwen3.6-35B-A3B-oQ6 (the model in your report) at cached lengths 2048 / 4096 / 8192 / 16384 / 32768 / 65536, two attempts each. In my runs they all came back PASS.

SSD path was definitely firing (47s cold vs 4s cached at 65k cached tokens on the 35B-A3B model), so the cache was actually loading and replaying. I didn't see step regression, content drift, or "claims to be at step 3" behavior in my matrix.

If you want me to dig deeper, share your full test environment (model build, cache config, the exact session prompts that triggered it) and I can re-run on my side to see if I can hit the same regression.

[lessu]

Thanks for you reply,
I was also testing yesterday, I found out 0.2.0 was the first version that issue comes out.
I checked the commit, VLM cache support is the only feature that could affect, so I removed vision_config block out of Qwen3.6-35B-A3B-MLX-8bit config.json, ( exact same model I tested ) it will turn Qwen3.6-35B-A3B-MLX-8bit to be a pure TEXT model.

And amazingly the issue gone, I think it has something to do with it, then I switched to 0.3.6, seems ok for me now.

I will definitely try 0.3.8 with VLM settings.

[jundot]

@lessu

v0.3.8.dev3 has two fixes that look directly relevant here: Qwen3.5 / Qwen3.6 forward alignment with mlx-lm (which fixes output divergence between cold prefill and cache-hit prefill) and the sliding-window attention dilution coming from SSD-restored RotatingKVCache.

Could you re-run the same long-context scenario with prefix cache enabled on this build? https://github.com/jundot/omlx/releases

[lessu]

Quick report:
With the newest release build, cache issue is not reproduced in my first run. Seems working.
I will make sure there are no mistake in my test step, and test more models.

[lessu]

Tested with Qwen3.6-27B-MLX-Q8, and 35B-A3B Q4, the issue is not reproduced, I think the issue can be closed.