Add H3_ATTENTION_CACHE: stream QKV/attention-out (and optionally MLP)… - #55
Add H3_ATTENTION_CACHE: stream QKV/attention-out (and optionally MLP)…#55JunzoKamahara wants to merge 3 commits into
Conversation
… int8 from a pre-quantized cache Parallels --ssd-streaming but for the int8-resident path: instead of loading BF16 and quantizing at startup, QKV/attention-output (and, with H3_INT8_STREAM_MLP=1, FC1/FC2 too) are streamed per block from a cache built once by the new build_attention_cache tool, using the same h3_gpu_quantize_weight_int8 routine as the existing resident path. This reaches full 50/50-block quality (no --layers skipping) at a DiT peak close to or below today's degraded-quality resident-int8 runs, since double-buffered streaming needs only two small slots instead of full block residency. With H3_INT8_STREAM_MLP additionally streaming the MLP, DiT peak drops further (~1.5 GiB at 22 frames, ~5.4 GiB at 362 frames), which is what makes a 24GB machine comfortable at ~15s/362-frame lengths. Verified bit-for-bit identical output (matching SHA-256) against the existing resident-int8 path at matched seed, for both attention-only streaming and full attention+MLP streaming.
|
Hi, |
Previously, having H3_ATTENTION_CACHE set in the environment while also passing --ssd-streaming was a hard error (int8_qkv/ int8_attention_out are unconditionally false under --ssd-streaming, which the cache setup rejected). That forced anyone who leaves H3_ATTENTION_CACHE set for their usual long runs to unset it by hand every time they want a quick --ssd-streaming test instead (reported against PR antirez#55 by monotected: short runs got slower after adopting H3_ATTENTION_CACHE, and --ssd-streaming was the easy way back, but the env var stayed in the way). --ssd-streaming is now treated as an explicit, intentional request for a different residency mode: when both are set, a one-line stderr notice explains H3_ATTENTION_CACHE is being ignored, and generation proceeds via --ssd-streaming instead of failing. The other conditions that disable int8_qkv/int8_attention_out (a short sequence, no int8 GPU support, --use-slower-bf16-qkv/-attention-output) stay hard errors, since those describe a configuration that cannot serve the cache at all rather than an alternate mode the user explicitly chose. Verified: H3_ATTENTION_CACHE=dit_int8_v2.cache with --ssd-streaming now prints the notice and completes generation instead of failing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hi, thanks for the report — and sorry, the original PR description didn't actually show how to enable this feature at all (just pushed a fix for that too, see the updated description above): you need to explicitly build a cache and point build_attention_cache MiniMax-H3/FL2VA dit_int8_v2.cache # once
H3_ATTENTION_CACHE=dit_int8_v2.cache H3_INT8_STREAM_MLP=1 ./h3 \
-d MiniMax-H3 -p "..." --width 512 --height 512 --frames 22 \
--steps 4 --layers 50 --reuse 2 -o outputs/fox-quick-test.mp4I've also pushed a separate fix ( |
|
Thanks!!! so i made about 12 vids 6 H3_ATTENTION_CACHE +6 standart h3.c --ssd-streaming in t2v - no visible degrade and its fater same on 0.3-0.5 mp cli args : **now with test short video with cash i got 1m 28sec ** (Memory lifespan is barely consumed.) ** and with --ssd-streaming. its 1m 50sec ** (Memory lifespan is on 15 to 18 gb in peak ) --ref-image /Users/icoidd/Desktop/minimax/start1.jpg |
…E_DIR A PR antirez#55 comment from monotected reported a visible quality drop (missing fine detail, "transparent feet") specifically in Ref2VA generations with H3_ATTENTION_CACHE, alongside a normal 25% speedup and no visible issue in T2V. The command they posted explains why: an FL2VA-quantized cache (fl2va_v2.cache) streamed into a generation that --ref-image switched to Ref2VA. attention_cache_validate() only ever checked block_count/hidden/inner/ffn, which are identical between FL2VA and Ref2VA (same DiT architecture) - nothing detected the mismatch, so the wrong weights streamed in without error or crash, just degraded output. That failure mode (works, but wrong) is worse than a hard error. Cache header v3 adds two fields where v2 had unused reserved[10] (40 bytes: 4 + 32 + 4, so the header stays the same total size): - model_kind (FL2VA=1/Ref2VA=2): read back from the transformer directory's own last two path components (".../FL2VA/transformer" or ".../Ref2VA/transformer" - the exact layout h3.c's own dit_path selection always produces, not a guess). H3_ATTENTION_CACHE now refuses a mismatched cache with a specific error ("Ref2VA generation cannot use a FL2VA attention cache") rather than silently streaming it. UNKNOWN (a non-standard directory on either side) skips the check entirely rather than risk a false mismatch. - model_id: h3_weight_store_fingerprint() (h3_weights.c/.h) - each shard's path/size/mtime hashed together (FNV-1a, non-cryptographic; a stat() per shard, not a read of the ~18GB of weight bytes) so it's cheap enough to recompute at every run. Catches a cache built against different/rewritten weights (e.g. a LoRA baked in after the cache was built). This is a warning, not a hard error, since mtimes can shift for benign reasons (a copied checkpoint) that model_kind never would. This bumps H3_ATTENTION_CACHE_VERSION to 3; existing v2 caches simply fail the version check and need rebuilding. build_attention_cache also gains a second mode: given a model root directory (detected by <root>/FL2VA/transformer/config.json existing) and an output directory, it builds fl2va.cache and, if a Ref2VA transformer is present, ref2va.cache in one pass - mirroring h3.c's own reference_count-driven FL2VA/Ref2VA selection. The prior single transformer-dir + single output-file usage is unchanged and still works. A new H3_ATTENTION_CACHE_DIR env var points at such a directory and lets h3_dit.c auto-select the matching cache by the same signal (reference presence) h3.c already uses to pick the transformer directory itself - the user no longer has to track that correspondence by hand, which is very likely what led to this bug in the first place. H3_ATTENTION_CACHE (a single file) still works for non-standard layouts; setting both is an error. Verified end to end: an old v2 cache is now rejected (version mismatch). A freshly built v3 FL2VA cache's header carries model_kind=1 and a non-zero model_id at the documented byte offsets. A matching FL2VA generation with it succeeds; the same cache with --ref-image (Ref2VA) is rejected with the new error, without ever mis-streaming. H3_ATTENTION_CACHE_DIR correctly auto-selects fl2va.cache for a plain run and (correctly) fails on a missing ref2va.cache for a --ref-image run. build_attention_cache's model-root-mode detection was checked against a synthetic directory tree without requiring a real build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks again — I found the cause of the Ref2VA degradation. The cache file was valid structurally, but an FL2VA cache could also be accepted during Ref2VA generation because both transformers use the same tensor dimensions. That meant Ref2VA could silently run with FL2VA cached weights. I changed the cache format and added automatic FL2VA / Ref2VA cache selection. New recommended usage Build both caches once from the model root: ./build_attention_cache This creates: attention-cache/ Then use the cache directory instead of selecting a cache file manually: H3_ATTENTION_CACHE_DIR=./attention-cache h3.c will now choose the correct cache automatically:
So for your previous Ref2VA command you should no longer use: H3_ATTENTION_CACHE=./fl2va_v2.cache Instead, build the two caches from your "MiniMax-DoubleLoRA-Master" directory and use "H3_ATTENTION_CACHE_DIR". This is especially important in your case because your two LoRAs are baked into the model: the caches should be generated from those exact baked transformer weights. The old single-file form is still supported: H3_ATTENTION_CACHE=./some.cache but the cache now records whether it is FL2VA or Ref2VA and h3.c rejects the wrong type instead of silently using it. The cache format has also changed from v2 to v3, so existing "fl2va_v2.cache" files need to be rebuilt. I tested:
Could you rebuild the caches from "MiniMax-DoubleLoRA-Master" with the command above and try the same Ref2VA seed again? That should tell us whether the missing-detail issue was entirely caused by the FL2VA/Ref2VA cache mismatch. |
|
Hi - sorry for late soo.... i rebuild h3.c also update and bake a new cashe ps icoidd@MacBook-Pro-icoidd h3.c % bash# 1. 2../build_attention_cache /Users/icoidd/h3.c/MiniMax-CustomLoRA-fl2va/FL2VA/transformer ./attention-cache/fl2va.cache 3.H3_ATTENTION_CACHE_DIR=./attention-cache H3_INT8_STREAM_MLP=1 ./h3 -d ./MiniMax-DoubleLoRA-Master -p "non_diegetic_music: N/A" |
|
i allready rebuild with original weights = same time 5-7 min |
|
Thanks for testing this. A couple of notes for the benchmark:
"/usr/bin/time ./h3 ..." That should make the comparison easier and more consistent. |
.....some kind a bud on my side tokenizer 1/1 |
|
thanx!! /issue was on my syde now its great no quality loss tokenizer 1/1 |
2 participants
Parallels
--ssd-streamingbut for the int8-resident path: instead of loading BF16 and quantizing at startup, QKV/attention-output (and, withH3_INT8_STREAM_MLP=1, FC1/FC2 too) are streamed per block from a cache built once by the newbuild_attention_cachetool, using the sameh3_gpu_quantize_weight_int8routine as the existing resident path.Usage:
This reaches full 50/50-block quality (no
--layersskipping) at a DiT peak close to or below today's degraded-quality resident-int8 runs, since double-buffered streaming needs only two small slots instead of full block residency. WithH3_INT8_STREAM_MLPadditionally streaming the MLP, DiT peak drops further (~1.5 GiB at 22 frames, ~5.4 GiB at 362 frames), which is what makes a 24GB machine comfortable at ~15s/362-frame lengths.In practice, this also makes full-depth generation substantially more usable on memory-constrained Macs: on a 24GB M5 Mac, a 5-second video can now be generated in under 10 minutes while still using all 50 DiT blocks.
The trade-off is additional SSD usage: the pre-quantized int8 cache requires approximately 18 GB of extra disk space in addition to the original model files. The cache is generated once and can then be reused across subsequent runs, avoiding repeated BF16 loading and runtime quantization.
Verified bit-for-bit identical output (matching SHA-256) against the existing resident-int8 path at matched seed, for both attention-only streaming and full attention+MLP streaming.