RoboTQuant MLX Experiments

Experimental extraction of the RoboTQuant integration work that was developed on top of mlx-vlm==0.4.4 for Apple Silicon.

This repository does not ship a full fork of MLX-VLM. It only contains:

• the standalone robotquant.py override used in the local experiments
• the generate.py patch needed to wire RoboTQuant into mlx_vlm
• a benchmark script for long-context quality checks
• benchmark result JSON files
• bilingual documentation for setup, benchmarks, and limitations

Korean documentation: README.ko.md

Why this repo exists

The local implementation lived directly under site-packages, which is hard to share and hard to reproduce. This repository isolates only the RoboTQuant-related experimental pieces and documents the current best-known operating configuration.

Current claim

Under the following conditions:

• Apple Silicon
• mlx-vlm==0.4.4
• gemma-4-26b-a4b-it-4bit
• long-context Korean/text-heavy prompts

the current best-known operating baseline is RoboTQuant + dynamic qstart guard(256), and it outperformed the local TurboQuant baseline on the sequential regression suite included here.

This is not a universal claim across all hardware, all models, all languages, or all multimodal workloads.

Repository layout

• overrides/mlx_vlm/robotquant.py
  • the experimental RoboTQuant implementation
• patches/mlx_vlm_0.4.4/generate.py.patch
  • the upstream integration patch against mlx-vlm==0.4.4
• scripts/benchmark-robotquant-quality.py
  • sequential quality benchmark runner
• scripts/benchmark-robotquant-regression.py
  • compact regression runner for the three promotion cases
• benchmarks/
  • selected benchmark result JSON files used in the evaluation
• docs/
  • bilingual benchmark, integration, and limitation notes
• configs/robotquant-live.env.example
  • example environment variables for the local stable setup

Benchmark summary

Current sequential regression 3x median, qstart=8192, guard=256:

Case	RoboTQuant wall_ms	TurboQuant wall_ms	RoboTQuant gen_tps	TurboQuant gen_tps
korean_long_summary	62165.41	64271.46	49.0524	45.6491
tail_instruction_following	24678.66	25009.89	56.4082	54.2991
needle_retrieval_tail	28533.90	29614.08	57.0029	53.7742

Historical full-size guard=512 results are still included as archived reference. See docs/BENCHMARKS.md for context and caveats.

What actually made RoboTQuant win

The most important change was not ParoQuant, TriAttention, or aggressive tier routing. The decisive improvement was:

• keep RoboTQuant as the quantizer
• do not quantize the prompt tail too early
• protect the last 256 prompt tokens with a dynamic qstart guard

In simple terms:

• old context can be compressed
• the most recent instructions should remain exact a bit longer

Reproduction outline

1. Install upstream mlx-vlm==0.4.4.
2. Copy overrides/mlx_vlm/robotquant.py into your installed mlx_vlm package.
3. Apply patches/mlx_vlm_0.4.4/generate.py.patch.
4. Add the server-side dynamic qstart handling described in docs/INTEGRATION.md.
5. Export the environment from configs/robotquant-live.env.example.
6. Run scripts/benchmark-robotquant-quality.py.

Limitations

• This is an experimental extraction, not a polished upstream PR.
• The repository reflects the current local best-known stable path, not every discarded experiment.
• Results are strongest on long Korean text prompts.
• Image/audio mixed workloads were not fully revalidated here.
• The server-side dynamic qstart logic is documented, but not packaged as a drop-in server patch in this repo because the local server file also contained unrelated work.

See docs/LIMITATIONS.md.