Context: Following up on the memory-residency thread (#4) — there's a complementary way to cut the MiniMax-H3 memory footprint that vpipe might evaluate: replace the Qwen3-VL-32B text/conditioner encoder with a much smaller one plus a learned linear projection.
What exists: nicolab28/ComfyUI-ClipProj (PoC, MIT) does exactly this for H3 in ComfyUI:
- Swaps H3's Qwen3-VL-32B (truncated to 50 layers, 15.7 GB NVFP4) for a Qwen3-VL-4B (4.5 GB int8)
- Fits a linear map (2560 → 5120 dims) via streaming ridge regression — no gradients/training
- Why it works: Qwen3-VL 2B/4B/8B/32B share the same tokenizer (151,936 tokens), so a prompt yields identical tokens at identical positions → position-wise mapping is well-posed
- Measured (author's own, on NVIDIA): cross-prompt CKA 0.92–0.95, test cosine 0.712, R² 0.507; holds up on simple/structured multi-shot prompts and fl2va first/last frame; honest trade-off: loses some 32B knowledge (non-English speech in the audio branch, some named identities)
Relevance to vpipe: the H3 path already streams the Qwen3-VL-32B conditioner to fit 16 GB. A projected 4B conditioner would shrink that component from ~15.7 GB (streamed) to ~4.5 GB (resident), freeing memory the DiT residency logic contends over — same spirit as the #4 reserve fix, attacking the other half of the equation. The same family-tokenizer trick applies to Klein's Qwen3-8B encoder (→ 4B/1.7B).
Questions / requests (any subset):
- Would you consider evaluating a projection-based small conditioner for H3 (even as an experiment)?
- Is the plugin mechanism (introduced 8-15) a viable home for community-contributed projection matrices, so this stays out of the core repo?
- At minimum, a note in docs/MINIMAX-H3.md pointing at this alternative path for memory-constrained users?
Happy to gather the projection matrices/calibration repo links or run anything on macOS if it helps — the approach is platform-agnostic (pure tensor math) though the reference implementation is ComfyUI/NVIDIA-tested.
Context: Following up on the memory-residency thread (#4) — there's a complementary way to cut the MiniMax-H3 memory footprint that vpipe might evaluate: replace the Qwen3-VL-32B text/conditioner encoder with a much smaller one plus a learned linear projection.
What exists: nicolab28/ComfyUI-ClipProj (PoC, MIT) does exactly this for H3 in ComfyUI:
Relevance to vpipe: the H3 path already streams the Qwen3-VL-32B conditioner to fit 16 GB. A projected 4B conditioner would shrink that component from ~15.7 GB (streamed) to ~4.5 GB (resident), freeing memory the DiT residency logic contends over — same spirit as the #4 reserve fix, attacking the other half of the equation. The same family-tokenizer trick applies to Klein's Qwen3-8B encoder (→ 4B/1.7B).
Questions / requests (any subset):
Happy to gather the projection matrices/calibration repo links or run anything on macOS if it helps — the approach is platform-agnostic (pure tensor math) though the reference implementation is ComfyUI/NVIDIA-tested.