Qwen text encoder: sequence exceeds Metal threadgroup memory with reference video + ref images (≥150 frames)

[cuiyuxin-gif]

Error

h3: Qwen layer 0 causal GQA failed: causal attention sequence 10231 exceeds threadgroup memory

Generation aborted (exit 1) at the very beginning of the text encoder (layer 0, step 0/50), before denoising starts.

Trigger condition

Combining a reference video (--ref-silent-video, ~7-8s, 1080×1920) with multiple reference images (--ref-image × 3-5) and ≥150 frames (~6.25s at 24fps). Long prompt (~100 characters) further increases the sequence count.

The dominant factor is the reference video (~7500 tokens for 7.7s). Reference images contribute negligibly (~300 tokens total for 5 images). The apparent limit is ~7700 tokens (32KB threadgroup / 4 bytes per float).

Systematic single-variable experiments

Combination	Frames	Result	Sequence
5 images + video + long prompt	22	✅ pass	—
5 images + video + long prompt	90	✅ pass	—
5 images + video + long prompt	120	❌ fail	8199
5 images + video + long prompt	160	❌ fail	10231
video only + long prompt	160	❌ fail	8184
5 images only + long prompt	160	✅ pass	—
5 images + video + short prompt	160	❌ fail	8510

Downscaling all 5 images to 896px → sequence unchanged (10231). Vision token count from Qwen appears fixed-per-image, not resolution-dependent.

MPS fallback is not safe

The existing H3_MPS_GQA=1 fallback (h3_gpu_gqa_mps) fails differently: it causes a system-wide watchdog panic (machine freeze, 5 verified occurrences) during the denoise stage with these large sequences. This was tested on both M5 Max 128GB and M3 Ultra 96GB (macOS 26.4.1/26.5.2). I had to revert my MPS fallback patch because it was worse than the original error.

Current workaround (in our webapp)

• Auto-cap frames to 90 when a --ref-silent-video is present (sequence stays under ~7700)
• Auto-downscale all --ref-image to 896px before passing to h3 (solves the jetsam memory explosion: original 2400px images × 5 caused h3 to consume 168GB unified memory on a 96GB machine)
• Hard-block the "reference video + reference images" combination: even with the above guards, the denoise stage triggers watchdog panics with ≥2 images + 1 video (4 verified panics, including after a fresh reboot)

90-frame cap (~3.75s at 24fps) is quite limiting — users needing longer videos must generate multiple short clips and stitch them.

Suggested fix directions

The root cause is h3_gpu_gqa_causal_bf16 storing the entire causal attention score row (sequence × 4 bytes) in threadgroup memory in a single dispatch:

1. Tiled GQA: split the score computation across multiple dispatches with intermediate global memory, requiring only tile_size × sizeof(float) of threadgroup memory per dispatch.

2. Reference video token compression: the Qwen encoder produces ~1000 tokens per second of video — pooling/down-sampling these before feeding them into the text encoder would directly reduce sequence length without losing semantic content.

---

Note: PR #44 (16-byte alignment fix) is orthogonal — it addresses a Metal validation-layer assertion, not this capacity limit.

Environment

• h3.c: latest main (commit 8b7d5b9)
• macOS 26.5.2 / M5 Max 128GB; macOS 26.4.1 / Mac Studio M3 Ultra 96GB
• Model: MiniMax-AI/MiniMax-H3 (FL2VA checkpoint)
• Metal: maxThreadgroupMemoryLength = 32768 bytes