diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md new file mode 100644 index 0000000000..aa3a9d5c14 --- /dev/null +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -0,0 +1,409 @@ +# `LTX25-IC-LORA-REF-VIDEO` — the IC-LoRA reference clip and its attention mask + +Issue: [#3020](https://github.com/mudler/vllm.cpp/issues/3020). +Base: `fafb58ef6`. Oracle: `Lightricks/LTX-2 @ fd4ded7f` +(`.agents/oracles/ltx-2.md`), executed, never read alone. + +Gaps **A15** and **A16** of `.agents/specs/ltx25-completion-scope.md`, landed as +one row. §1 establishes why they are one row rather than two. + +## 0. What is claimed, and what is not + +CLAIMED. An IC-LoRA render can be conditioned on a reference clip through the +public video ABI, and that clip reaches the DiT as appended tokens carrying the +reference item's own RoPE geometry. An optional per-region attention mask +reaches the DiT as an additive log-space self-attention bias. + +NOT CLAIMED. No render quality claim, no speed claim, and no real-weights +measurement: this row has no GPU authority and no LTX checkpoint. The gates are +value gates against the executed pinned module plus an end-to-end reachability +gate on the reduced-dimension checkpoint fixture in +`tests/vllm/multimodal/ltx2_video_fixture.h`. + +NOT CLAIMED EITHER. The dtype. This engine's LTX host path materialises f32 +where upstream materialises bf16, which is the standing divergence the A24 wave +campaign owns; this row mirrors the tree's f32 convention at every new site and +records the width owed rather than pretending the bytes match. See §4 R4. + +## 1. Why A15 and A16 are one row + +Upstream applies `ConditioningItemAttentionStrengthWrapper` at exactly one site, +`iclora_utils.py:168-169`, and its argument is always the +`VideoConditionByReferenceLatent` built two lines above it. +`combined_image_conditionings` (`utils/helpers.py`) never wraps, and neither +appending VIDEO item passes a mask — `keyframe_cond.py:68-76` and +`reference_video_cond.py:88-96` both pass a literal `attention_mask=None`. + +In this tree `Ltx2ConditionVideoByReference` has zero production call sites at +`fafb58ef6`: its definition at `src/vllm/model_executor/models/ltx2_conditioning.cpp:565`, +prose in `ltx2_conditioning.h`, and two test call sites. The reference arm is +refused in `src/vllm/multimodal/ltx2_video.cpp`. So A16 alone has nowhere to +attach that upstream also attaches to: putting the mask on the keyframe or image +items that ARE reached would invent behaviour upstream does not have. + +`.agents/specs/ltx25-completion-scope.md` §8 order 8 already says this — +"A15, A16, A17 — the IC-LoRA family, one owed table", M-L total. A16's +standalone "M" sizing was wrong. A17 landed earlier in the campaign. + +## 2. The dead seam this row retires + +A16's consumption half is **already built, and nothing reaches it**. Verified on +`fafb58ef6` with `grep -rn 'attention_mask' src/ include/ tests/` and a second +pass for assignments: + +``` +$ grep -rn '\.attention_mask *=\|attention_mask_rows *=' src/ include/ tests/ +src/.../ltx2_device.cpp:1097: VT_CHECK(m.attention_mask_rows == 1 || ... <- a comparison +src/.../ltx2_dit.cpp:599: VT_CHECK(m.attention_mask_rows == 1 || ... <- a comparison +tests/vllm/models/test_ltx2.cpp:282-285 <- BuildModalities, by hand +tests/vllm/models/test_ltx2_device.cpp:282-285 <- BuildModalities, by hand +include/.../ltx2.h:558 <- the declaration's default +``` + +Four assignments, all inside a test helper that constructs `Ltx2ModalityInput` +by hand. Nothing in `src/` writes either field. The chain below those +assignments is complete and correct — `Ltx2PrepareSelfAttentionMask` +(`ltx2.cpp:846`) into `self_bias` on host (`ltx2_dit.cpp:598-604`) and device +(`ltx2_device.cpp:1096-1104`), into `vt::AttentionCross`'s additive bias — which +is exactly the failure `.agents/reachability.md` names: the class works, and no +capability reaches it. + +Landing this row is the production assignment. The reachability mutation in §5 +is deleting it. + +## 3. Upstream anchors + +Root `Lightricks/LTX-2 @ fd4ded7f`. + +| ours | upstream | +|---|---| +| `Ltx2IcLoraReferenceGeometry` | `ltx-pipelines/iclora_utils.py:111-117` | +| `Ltx2TemporalSubsample` | `ltx-pipelines/iclora_utils.py:87-90` (called `:143-144`) | +| the reference item's encode | `ltx-pipelines/iclora_utils.py:141-148` | +| `Ltx2ConditionVideoByReference` (already ported) | `ltx-core/conditioning/types/reference_video_cond.py:46-108`, built at `iclora_utils.py:162-167` | +| `Ltx2DownsampleMaskVideoToLatent` | `ltx-pipelines/iclora_utils.py:52-84` | +| the strength multiply | `ltx-pipelines/iclora_utils.py:151-156` | +| `Ltx2ResolveCrossMask` | `ltx-core/conditioning/mask_utils.py:13-73` | +| `Ltx2BuildAttentionMask` | `ltx-core/conditioning/mask_utils.py:170-243` | +| `Ltx2UpdateAttentionMask` | `ltx-core/conditioning/mask_utils.py:110-167` | +| the wrapper's apply order | `ltx-core/conditioning/types/attention_strength_wrapper.py:43-71` | +| the mask read (`Ltx2ReadFrameDirectory` + `Ltx2MaskVideoFromPixels`, inlined at the call site) | `ltx-pipelines/ic_lora.py:511-537` (`_load_mask_video`) | +| the stage split | `ltx-pipelines/ic_lora.py:269-281` (stage 1) vs `:314-321` (stage 2) | +| the CLI shape | `ltx-pipelines/ic_lora.py:415-441`, `:452-463`, `:481-498` | +| the consumption (already ported) | `ltx-core/model/transformer/transformer_args.py:208-237`, `:289` | + +## 4. Design + +### 4.1 A15 — the reference clip + +Upstream's pixel path is `decode_video_by_frame` into `video_preprocess` +(`iclora_utils.py:141-142`). `video_preprocess` (`media_io/decode.py:82-103`) is +per frame `resize_and_center_crop(f.float(), H, W)` then `normalize_images`, +concatenated on the frame axis. **That function is already ported and gated in +this tree** as `Ltx2ReadFrameDirectory` +(`src/vllm/model_executor/models/ltx2_retake.cpp:211`), which runs each +`frame_%06d.ppm` through `Ltx2LoadImageAndPreprocess` — the same +resize-and-centre-crop plus `/127.5 - 1` chain — and returns `[C, T, H, W]`, the +layout `Ltx2ConvVideoEncode` takes. The container-versus-frame-directory +substitution is the harness adaptation row `LTX25-RETAKE` already recorded and +`video_api.cpp` already documents; no demuxer is vendored here. + +So A15's genuinely new code is small and each piece is separately gateable: + +1. `Ltx2IcLoraReferenceGeometry(height, width, scale)` — refuse when + `scale != 1` and either axis is not divisible (`:112-115`), else return + `height / scale`, `width / scale` (`:116-117`). Integer division, and the + refusal carries upstream's own wording. +2. `Ltx2TemporalSubsample(clip, channels, frames, plane, factor)` — keep index + 0, then `range(1, frames, factor)` (`:89`). Note the shape: index 1 is + ALWAYS kept when it exists, so at `factor == 2` and 5 frames the kept set is + `{0, 1, 3}`, not `{0, 2, 4}`. Guarded on `factor > 1` exactly as `:143` is. +3. Encode the whole clip with the existing `Ltx2ConvVideoEncode`, which the + retake arm already drives multi-frame at `ltx2_video.cpp:3703`. +4. Apply `Ltx2ConditionVideoByReference` with `downscale_factor = scale` and + `temporal_scale_factor` from the adapter metadata (`im.dit.lora_reference`, + already read by row `LTX25-IC-LORA`), and `strength` from the request. + +**Stage 1 only.** `ICLoraPipeline` gives stage 1 `_create_conditionings`, which +appends the reference item (`ic_lora.py:269-281`), and stage 2 plain +`combined_image_conditionings` with no reference item (`:314-321`). The phase +loop therefore applies this block on `phase_index == 0` and on no other phase. +The predicate that routes and the predicate that refuses are the same +expression, bound once in a named local, so the two cannot drift. + +### 4.2 A16 — the attention mask + +The mask read mirrors `_load_mask_video`, and it is NOT a function of its own: +`ltx2_video.cpp` composes it at the call site out of the SAME +`Ltx2ReadFrameDirectory` the reference clip uses plus `Ltx2MaskVideoFromPixels`, +which means over the three channels, remaps `(x + 1) / 2` and clamps to +`[0, 1]`. It is read at the STAGE-1 +resolution, which upstream spells `args.height // 2` at `ic_lora.py:460-461` +and this engine reads from the phase's own grid — the same number, derived from +the phase rather than assumed from the CLI. + +`Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, latent_shape)` +mirrors `:52-84`: area-interpolate each pixel frame to `(h_lat, w_lat)`; keep +the first latent frame as the area-downsampled PIXEL frame 0 alone; mean-pool +the remaining `f_pix - 1` frames in groups of `t = (f_pix - 1) / (f_lat - 1)`; +flatten to `(f_lat * h_lat * w_lat)`. It refuses a non-divisible pair with +upstream's assertion text (`:74-77`), and it degenerates to the first frame +alone when `f_pix == 1` or `f_lat == 1` (`:81-82`). + +The **causal carve-out is the whole content of the temporal half**: latent frame +0 is pixel frame 0 alone, not a pooled group. A uniform pooling separates from +it by 0.168 on the fixture in §5, so the golden can see it. + +Then `attn_mask = latent_mask * conditioning_attention_strength` (`:156`). + +`Ltx2ResolveCrossMask` and `Ltx2BuildAttentionMask` port `mask_utils.py` at +batch 1. The block structure is the load-bearing part: + +``` + noisy(Nn) prev_ref(N-Nn) new_ref(M) + noisy existing existing cross + prev_ref existing existing 0 + new_ref cross 0 1 +``` + +`Ltx2UpdateAttentionMask` reproduces `:141-156`: a null mask on a state that +already carries one pads the new tokens with ones rather than returning null, +which is what keeps a second reference item from shrinking the mask below the +sequence. + +The wrapper's ORDER is ported as written (`attention_strength_wrapper.py:49-71`): +snapshot the pre-item state, apply the inner item, take `num_new_tokens` as the +difference, and build the mask against the ORIGINAL token count. Building it +against the post-append count would put the new block in the wrong place while +every shape check still passed. + +`Ltx2LatentState` grows an `attention_mask` field and an +`attention_mask_rows` count. The header comment at `ltx2_conditioning.h:57-74` +that says the field is deliberately absent is rewritten rather than deleted: it +was true, this row is what makes it false, and a reader needs to know which. + +### 4.3 The production assignment + +`StreamState` carries the mask through the phase loop; `ToLatentState` and +`FromLatentState` carry it in both directions. At the DiT call site +(`ltx2_video.cpp`, where `Ltx2ModalityInput vin` is built) the mask is handed +over with its row count, guarded exactly as the `keyframes_mask` handover beside +it is: the size is CHECKED against `video.tokens` before `data()` is taken, +because an empty vector's `data()` is a null pointer and therefore upstream's +legal "no mask" — a silent drop dressed as a supported path. + +### 4.4 Request surface + +Three per-request extras, the established family-knob mechanism in this ABI +(`kLtx2RetakeStartTimeExtra` and twenty others): + +| extra | upstream | +|---|---| +| `ref_video_strength` | the STRENGTH of `--video-conditioning PATH STRENGTH` (`ic_lora.py:416-425`), default 1.0 | +| `conditioning_attention_mask_dir` | the MASK_PATH of `--conditioning-attention-mask` (`:427-441`), a `frame_%06d.ppm` directory | +| `conditioning_attention_strength` | its STRENGTH (`:452-455`), default 1.0, refused outside `[0, 1]` exactly as `ic_lora.py:230-233` | + +`ref_video_dir` already exists on `VideoGenParams`. + +## 4a. Risks + +* **R1 — a degenerate golden.** A `build_attention_mask` fixture without a PRIOR + conditioning item has `num_existing == num_noisy`, and then the true block + structure and the plausible wrong one "cross on ALL existing rows" are + ELEMENTWISE EQUAL: measured 0 separating elements. Such a golden is a mute + switch. Mitigation: the generator asserts `separating > 0` for every emitted + case and REFUSES to write the file otherwise, and the fixture carries a prior + item so the count is 4. +* **R2 — the mask is too wide to notice being wrong.** A `[T, T]` mask of ones + is the identity, and a bug that produces all-ones renders correctly. Mitigation: + the reachability gate compares a masked render against an unmasked one through + the ABI and requires the pixels to MOVE, and a separate assertion requires the + built mask to contain a value strictly between 0 and 1. +* **R3 — the routing predicate and the refusal predicate drifting apart.** The + campaign has shipped one silently wrong answer this way. Mitigation: one named + local, used by both, and a mutation that changes only the refusal. +* **R4 — dtype.** Upstream runs this whole path at bf16 (`self.dtype`, + `ic_lora.py:271-281`; `torch.bfloat16` at `ic_lora.py:530`). This tree's LTX + host path is f32 throughout, which is the A24 campaign's standing divergence. + This row mirrors the tree, names the width beside each new buffer, and records + it owed. A token gate cannot see this and neither can these goldens. +* **R5 — anchor rot.** `ltx2_video.cpp`'s READER ANCHORS comment has rotted ten + times. This row's edit shifts lines in that file. The anchors are re-derived + with the test case's own rule after the edit, and re-derived AGAIN after any + later edit in the same file. + +## 5. Tests and gates + +Every gate below is RUNNABLE from the repository root and all of them were run. + +```sh +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=OFF +cmake --build build -j 2 --target \ + test_diffusion_device_seam test_ltx2 test_ltx2_device test_ltx2_dfr \ + test_ltx2_iclora_reference test_ltx2_image_cond test_ltx2_loader test_ltx2_lora \ + test_ltx2_pipeline test_ltx2_retake test_ltx2_text_encoder test_ltx2_tiling \ + test_ltx2_vae test_ltx2_video test_ltx2_video_device_forward ltx2-gen +ctest --test-dir build -j 2 --output-on-failure -R 'ltx2|diffusion' +python3 scripts/check-pr-size.py --base origin/main --head HEAD --branch row/LTX25-IC-LORA-REF-VIDEO +python3 scripts/agent-issue-index.py --refresh && python3 scripts/check-agent-record.py +scripts/agent-preflight.sh +``` + +**THE WHOLE-TREE GATE IS NO LONGER PENDING.** This spec recorded it as a +PENDING result on an external resource, because a `kill -9` during the row's +scheduling had corrupted `build/.ninja_deps`, ninja discarded its dependency +graph and scheduled all 1338 edges, and under `-j 2` on a box carrying four +other compiling agents at load 20-33 that did not finish — it reached 76 of +1338. **It has now been run twice on an idle box** and it is a pass, so the +PENDING is discharged locally and not only by CI: + +* Fresh review, on head `975bc0906`: full build rc 0, then + `ctest -j 2` — **100% tests passed, 0 tests failed out of 714**, 7 skipped. +* This repair round, on the merged head: full build rc 0 over all 1337 edges, + then `ctest --test-dir build -j 2 --output-on-failure` — **100% tests passed, + 0 tests failed out of 714**, rc 0, the same 7 skipped. Every skip is + GPU-gated or checkpoint-gated and none is in this row's reach. + +Per-suite on the merged head: `test_ltx2_iclora_reference` 102/102 over 14 +cases, `test_ltx2_video` 5237/5237 over 131, `test_ltx2` 4581/4581, +`test_ltx2_pipeline` 4726/4726, `test_ltx2_retake` 69/69. +`scripts/agent-preflight.sh` rc 0 with 155 `ok` and 5 `SKIP`, every skip being a +checker preflight cannot supply arguments for; `check-pr-size.py` was therefore +run explicitly, rc 0. `check-agent-record.py` rc 0 with the snapshot PRESENT, +refreshed by `agent-issue-index.py --refresh` (316 open issues). + +The fifteen suites the recipe above names are still the right focused set — they +are every target that links a translation unit this change touches, plus +`ltx2-gen`, the shipped CLI that reaches the new knobs — but they are no longer +the whole of what was run. + +**The goldens are generated by executing the pinned module**, never by reading +it: `scripts/gen-ltx2-iclora-reference-goldens.py` imports +`ltx_pipelines.iclora_utils` and `ltx_core.conditioning.mask_utils` from +`/home/mudler/_git/LTX-2` at `fd4ded7f` and writes +`tests/vllm/models/ltx2_iclora_reference_goldens.inc`. It emits, for every case, +upstream's answer AND the rejected hypothesis beside it, and asserts they +differ. + +Measured separations, which are what makes each case able to fail: + +| case | rejected hypothesis | separating | +|---|---|---| +| `build_attention_mask` with a prior ref | cross on ALL existing rows | 4 elements | +| the same with `num_existing == num_noisy` | the same | **0 — REFUSED by the generator** | +| `downsample_mask_video_to_latent` | bilinear spatial | 0.20678424835205078 | +| the same | uniform temporal pooling, no causal carve-out | 0.049749284982681274 | +| the same at 9 -> 2 | an integer-stride box filter | 0.08158010244369507 | +| `_prepare_self_attention_mask` on `[1, 0.5, 0, 1e-30]` | — | `[0, -0.6931471824645996, -3.4028234663852886e+38, -69.07755279541016]` | + +**Reachability.** `test_ltx2_video` renders through the public `Generate` on the +reduced-dimension checkpoint fixture, once without a reference clip and once +with one, and requires the pixels to differ; then once more with a mask and +requires all three to differ pairwise. The mutation that must red it is deleting +the production call site, not the helper. + +## 6. Mutations + +Twelve, each applied by `scripts`-external `mutate.py`, which REFUSES when its +`old` text is not found and asserts the edit took; each then COMPILED, both +suites run, and the tree restored with `git checkout --` plus a `git status +--porcelain` emptiness assertion. `unit` is `test_ltx2_iclora_reference` +(97 assertions green), `reach` is `test_ltx2_video --test-case='ltx2 ic-lora*'` +(169 assertions green). + +**The counts in this table are AS MEASURED, before fresh review.** The review +round that followed added the non-dividing pooling golden and the no-mask range +refusal, so `unit` is now 102 assertions over 14 cases and the full +`test_ltx2_video` is 5237 over 131. The table is not rewritten to those numbers, +because a mutation result is a measurement and restating it against a tree it +was not run on is exactly the drift these tables exist to prevent. Only M1's +count is corrected below, and it is corrected because it was RE-MEASURED. + +| # | mutation | unit | reach | verdict | +|---|---|---|---|---| +| M1 | delete the production `vin.attention_mask` assignment | 97/97 pass | **3 failed** | DETECTED — the reachability property | +| M2 | never apply `Ltx2ConditionVideoByReference` | 97/97 pass | **threw** | DETECTED | +| M3 | `temporal_subsample` steps from 0 | **21 failed** | 169/169 pass | DETECTED | +| M4 | drop the causal first-frame carve-out | **1 failed** | 169/169 pass | DETECTED | +| M5 | cross block on ALL existing rows | **3 failed** | 169/169 pass | DETECTED | +| M6 | drop `scale != 1` from the divisibility guard | 97/97 pass | 169/169 pass | **SURVIVED — IDENTITY** | +| M7 | read the reference at the phase's own grid | **4 failed** | **1 failed** | DETECTED | +| M8 | apply the reference on every phase | 97/97 pass | **1 failed** | DETECTED | +| M9 | never multiply the mask by the strength | 97/97 pass | **1 failed** | DETECTED | +| M10 | an append no longer extends the mask | **1 failed** | 169/169 pass | DETECTED | +| M11 | the refusal takes a second predicate | 97/97 pass | 169/169 pass | **SURVIVED — IDENTITY** | +| M12 | broadcast a short 1-D mask instead of refusing | **1 failed** | 169/169 pass | DETECTED | + +**Ten of twelve detected. Neither survivor is a blind spot, and both are +identities provable rather than argued.** + +**M1 reds THREE assertions, not four.** This spec recorded four; fresh review +re-ran it and measured three, at `test_ltx2_video.cpp:14097`, `:14109` and +`:14124`. The fourth is worth naming because of WHICH one it is: +`trace.ic_lora_attention_mask_rows == 16` still PASSED under the mutant. The +trace records that the mask was built and how many rows it has, and the deleted +line is the one that hands it to the model — so the counter is still correct +about a mask nothing consumes. **Only the pixel compares can see the drop.** A +future reader tempted to gate this seam on the trace alone, because a trace +assertion is cheaper than a render, would be gating the construction and not the +consumption. + +**M6.** The mutant drops `downscale_factor != 1` from +`downscale_factor != 1 && (h % downscale_factor || w % downscale_factor)`. For +`downscale_factor == 1` the right conjunct is `h % 1 || w % 1`, which is `0` +for every `h` and `w`. The two predicates therefore agree on every input, and no +fixture can separate them. Upstream writes the guard (`iclora_utils.py:112`) and +this port mirrors it; it is documentation of intent, not behaviour. + +**M11.** The mutant replaces `serves_reference` in the refusal with +`im.pipeline_kind != "ic_lora"`. `serves_reference` is +`im.recipe.ic_lora_reference`, and that field is set by `IcLoraRecipe` alone, +which `ResolveLtx2PipelineRecipe` reaches only for `pipeline_kind == "ic_lora"`. +So over the recipe table AS IT STANDS the two expressions are the same +predicate. **It stops being an identity the moment a second recipe sets the +flag**, and this row therefore adds the tripwire for that day: +`test_ltx2_pipeline`'s recipe-table case walks every kind against all four +versions and requires exactly ONE resolved pair to carry `ic_lora_reference`, +naming `ltx2_video.cpp`'s refusals as what must be re-read. **The kind list is +DERIVED**, from every `pipeline_kind == "..."` branch in +`ltx2_pipeline.cpp` — the file a new kind has to be written into to exist. It was +a hand-copied eleven-entry array until fresh review, which is a list that is +complete only until the next kind lands: a twelfth kind would have gone unwalked +while the case went on reporting "exactly one". Measured rather than argued — +adding a twelfth branch to the resolver now reds the case with +`declares 12 kinds, not the 11 this tripwire was written against`, and the +assertion count rises by two, which is the new kind actually being walked. That converts an +unmeasured residual into a guarded one; it does not make M11 detectable today, +because today it changes nothing. + +## Owed + +| owed | issue | +|---|---| +| the EXR / HDR reference arm (`iclora_utils.py:120-139`), which needs an OpenEXR reader and `--hdr` colour-space resolution this tree does not have | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the scalar-only `elif conditioning_attention_strength < 1.0` branch (`iclora_utils.py:157-158`). It is unreachable from upstream's CLI, because `conditioning_attention_strength` is assigned only inside `if args.conditioning_attention_mask is not None` (`ic_lora.py:454-455`) and is otherwise 1.0, so a strength below 1 always arrives with a mask. Python-API-only; refused by name | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| `tiled_encode` for the reference clip (`iclora_utils.py:145-146`), which this engine reaches for the target but not for a reference | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the reference-AUDIO arm, unchanged by this row and still refused by name | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the bf16 storage width of every buffer this row adds (R4) | the A24 dtype campaign | +| a real-weights IC-LoRA reference render; this row has no GPU authority and no checkpoint | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| `ref_video_strength` with no `ref_video_dir` is ACCEPTED and then never read. It is the same shape as the `conditioning_attention_strength` defect this round fixed, and it is left open rather than fixed because upstream cannot express it: `--video-conditioning` is `nargs=2` (`ic_lora.py:416-425`), so upstream has no reading of a strength without a clip to be mirrored, and inventing a refusal would be a product decision rather than a port | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | + +## 7. Stop conditions + +* Stop and report `NEEDS_DECISION` if the reference pixel path turns out to need + a media decoder, a checkpoint, or a GPU lease. **This was settled first and it + did not fire**: `Ltx2ReadFrameDirectory` is already `video_preprocess`, + `Ltx2ConvVideoEncode` already runs multi-frame, and + `tests/vllm/multimodal/ltx2_video_fixture.h` is a real reduced checkpoint that + `Generate` runs end to end. +* Stop if a golden's generator cannot produce a case with a non-zero separation + from its rejected hypothesis. An unfalsifiable golden is not a gate. +* Stop if the reachability mutation — deleting the production call site — leaves + the suite green. That measures a class, not a capability. +* No GPU, no lease, no downloads. One compiling agent at `-j 2`. + +## Now + +`DONE` — landed on `row/LTX25-IC-LORA-REF-VIDEO`. A15 and A16 are served and +reached from `Generate`; the dead `Ltx2ModalityInput::attention_mask` seam has a +production writer. What stays owed is in `## Owed` and is refused by name. diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md index 4e54dc39b6..ecb02ad7d4 100644 --- a/.agents/specs/ltx25-ic-lora.md +++ b/.agents/specs/ltx25-ic-lora.md @@ -353,8 +353,8 @@ Each is owed by this row and named in the commit and pull request bodies. | owed | issue | |---|---| -| serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter. §6 derives both. Token-append is NOT part of it any more — #930 landed in `c7cb59fbb` | [#975](https://github.com/mudler/vllm.cpp/issues/975) | -| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | [#932](https://github.com/mudler/vllm.cpp/issues/932) | +| ~~serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter~~ **DISCHARGED for the reference VIDEO arm by row `LTX25-IC-LORA-REF-VIDEO`** ([`ltx25-ic-lora-ref-video.md`](ltx25-ic-lora-ref-video.md), [#3020](https://github.com/mudler/vllm.cpp/issues/3020)), which found that the pixel path was already here — `Ltx2ReadFrameDirectory` IS `video_preprocess` over a frame directory and `Ltx2ConvVideoEncode` already runs multi-frame for retake — so what was missing was the reference item's own geometry and the stage split, both of which now land on a new `ic_lora` pipeline kind. The reference-IMAGE arm is NOT discharged and never will be: upstream has no such flag, and it is refused by name. #975 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| ~~the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`)~~ **DISCHARGED for the MASK arm by row `LTX25-IC-LORA-REF-VIDEO`** ([#3020](https://github.com/mudler/vllm.cpp/issues/3020)). `Ltx2LatentState` carries a mask, the block structure is ported and gated against the executed module, and the mask reaches the DiT — which also retired a seam that was built and dead: nothing in `src/` assigned `Ltx2ModalityInput::attention_mask` before that row. The `strength < 1.0` arm WITHOUT a mask stays owed and is refused by name, because upstream's own CLI cannot reach it either (`ic_lora.py:452-455`). #932 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | | ~~N-adapter fusion, which additionally needs upstream's SECOND rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`) that this row refuses rather than guesses~~ **DISCHARGED by row `LTX25-LORA-FUSION`** ([`ltx25-lora-fusion.md`](ltx25-lora-fusion.md)), which ported that second form and gated it as byte equality against the EXECUTED pinned module rather than against a transcription. The arity refusal is gone and `--lora` is repeatable. Lifting it made ONE new arm expressible — N adapters on a recipe with a `kNoAdapters` phase, whose stage 1 upstream gives a proper SUBSET — and that arm refuses by name and is owed in THAT row's `## Owed`, not here. **#932 itself stays OPEN**, because the row above is its other half and nothing has landed it | [#932](https://github.com/mudler/vllm.cpp/issues/932) | | GGUF k-quant LoRA fusion — **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | | a real-weights IC-LoRA fusion measurement | blocked on GPU authority; `dgx.casa` was under a long render for this row's duration and this row had no GPU authority | @@ -376,6 +376,10 @@ Each is owed by this row and named in the commit and pull request bodies. ## Now -`ACTIVE` — the adapter path is implemented and gated; the reference arm stays -refused on the two causes §6 derives, which are #975 and are neither of the two -this refusal has previously given. +`DONE` — the adapter path is implemented and gated. The reference arm that §6 +left refused was SERVED by row `LTX25-IC-LORA-REF-VIDEO` +([#3020](https://github.com/mudler/vllm.cpp/issues/3020)), together with the +conditioning attention mask; §6's derivation was correct about what was missing +and wrong about how much of it was, because the pixel path it named already +existed as `Ltx2ReadFrameDirectory`. What that row left owed is in its own +`## Owed`. diff --git a/CMakeLists.txt b/CMakeLists.txt index 57af967780..d91f9a495e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -972,6 +972,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp src/vllm/model_executor/models/ltx2_audio_vae_encoder_load.cpp src/vllm/model_executor/models/ltx2_audio_input.cpp + src/vllm/model_executor/models/ltx2_iclora_reference.cpp src/vllm/model_executor/models/ltx2_image_preprocess.cpp # LTX-2.5 RETAKE (row LTX25-RETAKE, #924): the temporal region mask, the # truncate-or-pad conform, and the frame-directory source. Its own TU for the diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 6659c5a3d8..18a2c3ef43 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -228,6 +228,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` (#1009). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 Conv VAE decode DEVICE arm | LTX-2.5 video VAE | `test_ops_conv3d` 4/4 vs an independent scalar reference; `test_diffusion_device_seam` runs a non-CPU dispatch and requires byte-identical pixels | **f32**, which is what `vt::Conv3d` and the `kLtx2Vae` CUDA table serve, and what `Ltx2VideoEngine::Load` asks for whenever the render resolved a device queue ([#2853](https://github.com/mudler/vllm.cpp/issues/2853)). So this arm decodes 776.6 MiB wider than the CPU arm and off upstream's SDR bf16 grid; the bf16 DEVICE arm is owed and needs #1007. Routes through the new `vt::Conv3d` op on the queue the engine resolved at load (#1007). NO GPU has run the CUDA arm (#1452); other stages stay host-side (#1451) | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | +| LTX-2.5 IC-LoRA reference conditioning (`ICLoraPipeline`, A15 + A16) | LTX-2.5 DiT + video VAE encoder + an IC-LoRA adapter | `test_ltx2_iclora_reference` 14/14 (102 assertions) against the EXECUTED pinned module, every case carrying its rejected hypothesis, including a NON-DIVIDING 9 -> 2 pooling shape that separates `area` from an integer-stride box filter; 4 `test_ltx2_video` cases entering through `Generate` | `--pipeline-kind ic_lora`. `--ref-video` is a `frame_%06d.ppm` DIRECTORY (no demuxer), read at `height // reference_downscale_factor` from the adapter's own metadata and temporally subsampled by its `reference_temporal_scale_factor`. The adapter rides STAGE 1 ONLY and stage 2 runs bare, which is the mirror image of every other two-stage arm here. `--conditioning-attention-mask` attenuates the reference per region and is the first production writer of `Ltx2ModalityInput::attention_mask`. REFUSED and owed: the EXR/HDR reference arm, `tiled_encode` for a reference, reference IMAGES (upstream has no such flag), and the scalar-only strength arm that upstream's own CLI cannot reach either | | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage` plus `--lora`, which is now REQUIRED (#1445, both stages, at ONE strength rather than upstream's 0.25/0.5; #1144). 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 (#1039) | diff --git a/docs/models/ltx-2-5.md b/docs/models/ltx-2-5.md index d4bc339367..a6f5773402 100644 --- a/docs/models/ltx-2-5.md +++ b/docs/models/ltx-2-5.md @@ -117,6 +117,71 @@ ltx2-gen \ --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4 ``` +## Condition a render on a reference clip (IC-LoRA) + +```sh +ltx2-gen \ + --dit "$LTX_ROOT/diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors" \ + --model-version 2.5 --checkpoint-class distilled \ + --video-vae "$LTX_ROOT/vae/ltx-2.5-video-vae-conv-bf16.safetensors" \ + --audio-vae "$LTX_ROOT/vae/ltx-2.5-audio-vae-bf16.safetensors" \ + --upsampler "$LTX_ROOT/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors" \ + --pipeline-kind ic_lora \ + --lora "$LTX_ROOT/loras/.safetensors" \ + --ref-video /tmp/depth_frames --ref-video-strength 1.0 \ + --prompt-embeds "$LTX_VIDEO_EMBEDS" --audio-prompt-embeds "$LTX_AUDIO_EMBEDS" \ + --frames 25 --width 320 --height 192 --seed 20260812 \ + --workdir /tmp/ltx25ic --out /tmp/ltx25ic/video.mp4 +``` + +`--lora` is a PLACEHOLDER above, and deliberately so. The pinned upstream +checkout names no reference-conditioning IC-LoRA for 2.5: its only 2.5 IC-LoRA is +[`ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler), +which is `DFRPipeline`'s detailing adapter and rides stage 2 (`README.md:95-96`). +The published control adapters that DO condition on a reference clip are 2.3-era +and 19b — `LTX-2.3-22b-IC-LoRA-Union-Control`, +`LTX-2.3-22b-IC-LoRA-Motion-Track-Control` and +`LTX-2-19b-IC-LoRA-Pose-Control` (`MODELS-LTX-2.3.md:32-35`) — so none of them +pairs with the 2.5 22b transformer this recipe loads. No adapter is named here +because naming one would claim a pairing nothing has run. The mechanism below is +what this row gated; the weights are not. + +`--ref-video` is a directory of `frame_%06d.ppm`, not a container: upstream opens +one with PyAV and no demuxer is vendored here. The clip is read at +`height // reference_downscale_factor` by `width // reference_downscale_factor`, +and both factors come from the adapter's own metadata rather than from a flag. A +load with no adapter reads both as 1, which is upstream's default. A target +either axis of which the factor does not divide is refused by name, with +upstream's own sentence. + +The adapter rides **stage 1 only** and stage 2 runs bare. That is the mirror +image of `ti2vid_two_stage`, `a2vid_two_stage` and `keyframe_interpolation`, +where the adapter rides stage 2, and it is why IC-LoRA has its own pipeline kind +rather than being a mode of `distilled_two_stage`. A reference clip supplied to +any other kind is refused, and the message names this one. + +`--conditioning-attention-mask` takes a directory of grayscale +`frame_%06d.ppm` whose pixels attenuate the reference per region: black ignores +the conditioning there, white takes it in full. The frames are read at the +stage's own resolution and downsampled to the reference latent's grid, so a mask +must describe the same moments as the clip. `--conditioning-attention-strength` +scales the whole mask and must be in `[0, 1]`; above 1 it would amplify +attention rather than attenuate it, and is refused. + +A strength below 1 with **no** mask is refused rather than served. Upstream has +that branch and its own CLI cannot reach it either: the strength is assigned only +alongside a mask, so a sub-1.0 value always arrives with one. + +Through the C ABI and the server these are per-generation extras rather than +flags: `ref_video_strength`, `conditioning_attention_mask_dir` and +`conditioning_attention_strength`, beside the existing `ref_video` field. + +Not served, and refused by name: the EXR/HDR reference arm, which needs an +OpenEXR reader and a declared colour space; a tiled reference encode; and +reference IMAGES, which are not an IC-LoRA shape at all — upstream's flag takes a +video or an EXR directory and `ltx-pipelines` has no reference-image +conditioning. + `--lora` is repeatable, and each repetition takes an adapter path with an optional strength, exactly as upstream's own flag does: `--lora first.safetensors 0.8 --lora second.safetensors`. An omitted strength is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 2a2d0e34a2..42f803046b 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -188,6 +188,17 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "how much is read (default: the clip\'s own duration). Either without\n" "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n" "input, not a VAE round trip, which is upstream\'s deliberate choice.\n\n" + "IC-LoRA REFERENCE CONDITIONING conditions the whole render on a reference clip\n" + "(depth, pose, edges) and needs --pipeline-kind ic_lora, which is upstream's own\n" + "ICLoraPipeline: the adapter rides stage 1 and stage 2 runs bare. --ref-video is\n" + "the clip, as a DIRECTORY of frame_%%06d.ppm, and --ref-video-strength is the\n" + "second half of upstream's --video-conditioning PATH STRENGTH. Supply the\n" + "IC-LoRA itself with --lora: its metadata carries the downscale and temporal\n" + "factors the reference is read at, and without one both default to 1.\n" + "--conditioning-attention-mask names a grayscale frame directory whose pixels\n" + "attenuate the reference per region, and --conditioning-attention-strength\n" + "scales it. A strength below 1 with no mask is REFUSED rather than served: it is\n" + "upstream's Python-API-only branch, which its own CLI cannot reach either.\n\n" "RETAKE regenerates a time window of an existing clip and keeps the rest.\n" "--ref-video names a DIRECTORY of frame_%%06d.ppm (the layout minimax-h3-gen\n" "writes), not a container file: upstream opens one with PyAV and no demuxer is\n" @@ -289,6 +300,9 @@ int main(int argc, char** argv) { // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container. std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio; + // IC-LoRA reference conditioning (#3020): the strength half of + // `--video-conditioning`, and both halves of `--conditioning-attention-mask`. + std::string ref_video_strength, cond_mask_dir, cond_mask_strength; // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120). std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; @@ -422,6 +436,17 @@ int main(int argc, char** argv) { retake_end = Need(argc, argv, ++i, "--retake-end-time"); else if (f == "--retake-frame-rate") retake_fps = Need(argc, argv, ++i, "--retake-frame-rate"); + // IC-LoRA REFERENCE CONDITIONING (#3020). `--ref-video` above names the + // clip; these three are the rest of upstream's two flags — + // `--video-conditioning PATH STRENGTH` and + // `--conditioning-attention-mask MASK_PATH STRENGTH` (ic_lora.py:416-441). + // Per-generation, so they ride vp.extra_* like the retake knobs. + else if (f == "--ref-video-strength") + ref_video_strength = Need(argc, argv, ++i, "--ref-video-strength"); + else if (f == "--conditioning-attention-mask") + cond_mask_dir = Need(argc, argv, ++i, "--conditioning-attention-mask"); + else if (f == "--conditioning-attention-strength") + cond_mask_strength = Need(argc, argv, ++i, "--conditioning-attention-strength"); // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which // is a LOAD extra; these six are per-generation and are refused by name on // any other pipeline rather than accepted and ignored. @@ -517,6 +542,18 @@ int main(int argc, char** argv) { gen_keys.emplace_back("audio_path"); gen_values.push_back(audio_path); } + if (!ref_video_strength.empty()) { + gen_keys.emplace_back("ref_video_strength"); + gen_values.push_back(ref_video_strength); + } + if (!cond_mask_dir.empty()) { + gen_keys.emplace_back("conditioning_attention_mask_dir"); + gen_values.push_back(cond_mask_dir); + } + if (!cond_mask_strength.empty()) { + gen_keys.emplace_back("conditioning_attention_strength"); + gen_values.push_back(cond_mask_strength); + } if (!audio_start_time.empty()) { gen_keys.emplace_back("audio_start_time"); gen_values.push_back(audio_start_time); diff --git a/include/vllm/model_executor/models/ltx2_conditioning.h b/include/vllm/model_executor/models/ltx2_conditioning.h index 69ac360c87..30603a2ca0 100644 --- a/include/vllm/model_executor/models/ltx2_conditioning.h +++ b/include/vllm/model_executor/models/ltx2_conditioning.h @@ -55,19 +55,27 @@ namespace vllm { // LatentState (types.py:251-287) at batch 1, carrying only the fields a // conditioning item touches. // -// `attention_mask` is deliberately absent, and the reason is a fact about the -// ported items rather than a simplification. Both appending VIDEO items pass a -// literal `attention_mask=None` (keyframe_cond.py:68-76, -// reference_video_cond.py:88-96), and `update_attention_mask` returns None when -// its argument is None and the state carries no mask -// (conditioning/mask_utils.py:110-143). The ONLY upstream route to a non-None -// mask is `ConditioningItemAttentionStrengthWrapper`, whose sole application -// site is the IC-LoRA path (ltx-pipelines/iclora_utils.py:169); +// `attention_mask` WAS deliberately absent until row LTX25-IC-LORA-REF-VIDEO +// (#3020), and the argument for its absence is kept rather than deleted because +// it was correct and this row is exactly what made it false. +// +// It read: both appending VIDEO items pass a literal `attention_mask=None` +// (keyframe_cond.py:68-76, reference_video_cond.py:88-96), and +// `update_attention_mask` returns None when its argument is None and the state +// carries no mask (conditioning/mask_utils.py:110-143). The ONLY upstream route +// to a non-None mask is `ConditioningItemAttentionStrengthWrapper`, whose sole +// application site is the IC-LoRA path (ltx-pipelines/iclora_utils.py:169); // `combined_image_conditionings` (ltx-pipelines/utils/helpers.py:272-308), which -// is the route this engine mirrors, never wraps. So a field here would be one no -// ported item can populate — the unpassed-parameter shape .agents/reachability.md -// enumerates. An item that DID need one would have to grow this struct rather -// than silently dropping it, which is why the omission is stated here. +// is the route this engine mirrored, never wraps. So a field here would have been +// one no ported item could populate — the unpassed-parameter shape +// .agents/reachability.md enumerates — and the note ended: an item that DID need +// one would have to grow this struct rather than silently dropping it. +// +// #3020 is that item. `iclora_utils.py:162-169` is now REACHED: the reference +// item is built and, when a conditioning attention mask was supplied, wrapped. +// The field below is what the wrapper writes, and `ltx2_video.cpp` hands it to +// `Ltx2ModalityInput::attention_mask` — whose whole consumption chain existed +// and was assigned only from a test helper before this row. // `GeneratedKeyframeLayout` (types.py:220-247): where a state's generated // keyframe slot tokens live, and what they represent. // @@ -111,6 +119,35 @@ struct Ltx2LatentState { // about the render's SHAPE can see the difference. std::vector keyframes_mask; + // `LatentState.attention_mask` (types.py:251-287), a DENSE [tokens, tokens] + // STRENGTH mask in [0, 1]. EMPTY is upstream's `None`, which every state + // carries until an item wraps. + // + // DENSE and not the key-only broadcast form. `build_attention_mask` returns + // (B, N+M, N+M) unconditionally (mask_utils.py:220), and the block structure + // it fills is not expressible as one row: the noisy rows and the prior-ref + // rows get DIFFERENT values in the same column (`:236` against `:242`). + // + // GROWS WITH THE SEQUENCE, and that is the trap. An item that appends tokens + // without extending this leaves a mask the DiT reads as the key-only form — + // legal, differently shaped, and silently masking the wrong axis. That is why + // `Ltx2PadAttentionMaskForUnmaskedTokens` exists and why it is upstream's own + // answer at mask_utils.py:141-156 rather than a defensive addition here. + std::vector attention_mask; + + // `latent_tools.target_shape.token_count()`, which every appending item hands + // `update_attention_mask` as `num_noisy_tokens` (keyframe_cond.py:71, + // reference_video_cond.py:89). It is the count BEFORE any item ran, and it is + // NOT `tokens` once one has: rows between `noisy_tokens` and `tokens` are + // PRIOR reference tokens, and the block structure gives them a different value + // from the noisy rows in the same column (mask_utils.py:236 against `:242`). + // + // Carried on the state because this engine's phase loop rebuilds an + // `Ltx2LatentState` per item from a running `StreamState`, so `tokens` has + // already grown by the time the second item is applied and cannot stand in. + // Zero means "no mask can be built", which is what an audio state carries. + int64_t noisy_tokens = 0; + // `LatentState.generated_keyframe_layout` (types.py:246). Default-constructed // is upstream's `None`, and `Ltx2ConditionVideoByGeneratedKeyframeSlots` // refuses a second application on a state that already carries one diff --git a/include/vllm/model_executor/models/ltx2_iclora_reference.h b/include/vllm/model_executor/models/ltx2_iclora_reference.h new file mode 100644 index 0000000000..91b088381a --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_iclora_reference.h @@ -0,0 +1,144 @@ +// LTX-2.5 IC-LoRA REFERENCE CONDITIONING — the reference clip's own geometry, +// and the conditioning attention mask that rides it. +// +// Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +// spec .agents/specs/ltx25-ic-lora-ref-video.md (gaps A15 and A16). +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f +// OURS <- UPSTREAM +// Ltx2IcLoraReferenceGeometry <- ltx-pipelines/iclora_utils.py:111-117 +// Ltx2TemporalSubsample <- ltx-pipelines/iclora_utils.py:87-90 +// Ltx2MaskVideoFromPixels <- ltx-pipelines/ic_lora.py:511-537 +// Ltx2DownsampleMaskVideoToLatent <- ltx-pipelines/iclora_utils.py:52-84 +// Ltx2ResolveCrossMask <- ltx-core/conditioning/mask_utils.py:13-73 +// Ltx2BuildAttentionMask <- ltx-core/conditioning/mask_utils.py:170-243 +// Ltx2UpdateAttentionMask <- ltx-core/conditioning/mask_utils.py:110-167 +// +// The PIXEL path is not re-implemented here. `video_preprocess` +// (media_io/decode.py:82-103) is per frame `resize_and_center_crop` then +// `normalize_images`, concatenated on the frame axis, and this tree already has +// that as `Ltx2ReadFrameDirectory` (ltx2_retake.h) — which row LTX25-RETAKE +// ported and gated. This header only says at WHAT GEOMETRY to call it. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * `temporal_subsample` KEEPS INDEX 1. It is `[0, *range(1, F, N)]` (`:89`), +// not `range(0, F, N)`. At N=2 over 5 frames the kept set is {0, 1, 3} and +// the plausible reading gives {0, 2, 4} — the same COUNT, so every shape +// check agrees and the reference simply describes different moments. +// * THE MASK'S FIRST LATENT FRAME IS PIXEL FRAME 0 ALONE (`:70`, `:80`), a +// causal carve-out that mirrors the VAE's own first-frame rule. Pooling all +// `f_pix` frames uniformly produces a correctly shaped mask of plausible +// values. +// * `build_attention_mask` PUTS THE CROSS BLOCK ON THE NOISY ROWS ONLY +// (`:236`, `:240`), and leaves the prev-ref-to-new-ref blocks at ZERO +// (`:242`). On a fixture with no PRIOR conditioning item that is elementwise +// equal to "cross on all existing rows", so the wrong reading is invisible +// until a second item exists. +// * AN ALL-ONES MASK IS THE IDENTITY. A downsample that lost its values, a +// strength that was never multiplied in, and a correct unmasked render are +// all the same bytes. Nothing about the render's shape can see it. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// Upstream runs this whole path at bf16 (`self.dtype`, ic_lora.py:271-281, and +// `torch.bfloat16` at ic_lora.py:530). Every buffer here is f32, which is this +// tree's standing LTX host-path width and the A24 wave campaign's open +// divergence — NOT a decision this row made. Recorded in +// .agents/specs/ltx25-ic-lora-ref-video.md §4 R4 and in its `## Owed`, because a +// token gate and these goldens are both blind to a dtype that is too wide. +#pragma once + +#include +#include +#include + +namespace vllm { + +// `iclora_utils.py:111-117`. The resolution at which the reference clip is READ, +// which is the target's divided by the adapter's `reference_downscale_factor`. +// +// THE REFUSAL IS UPSTREAM'S AND IT IS GUARDED ON `scale != 1` (`:112`), not on +// divisibility alone: at scale 1 no dimension is ever refused, whatever it is. +// Refusing there would reject every render that supplies no adapter, since an +// absent `reference_downscale_factor` reads as 1 (`:35`). +struct Ltx2IcLoraReferenceGeometry { + int64_t height = 0; + int64_t width = 0; +}; +Ltx2IcLoraReferenceGeometry Ltx2ResolveIcLoraReferenceGeometry(int64_t height, int64_t width, + int64_t downscale_factor); + +// `temporal_subsample` (`iclora_utils.py:87-90`) over a channel-major +// `[channels, frames, height, width]` volume — the layout `Ltx2ReadFrameDirectory` +// returns and `Ltx2ConvVideoEncode` takes. +// +// Returns the KEPT FRAME INDICES, and the caller gathers. The indices are the +// whole content of the function and a vector of them is the only thing a gate +// can compare against upstream without also re-gating the copy. +std::vector Ltx2TemporalSubsampleIndices(int64_t frames, int64_t temporal_scale_factor); + +// Gather those indices out of a `[channels, frames, plane]` volume. +std::vector Ltx2TemporalSubsample(const std::vector& clip, int64_t channels, + int64_t frames, int64_t plane, + int64_t temporal_scale_factor); + +// `_load_mask_video`'s ARITHMETIC half (`ic_lora.py:530-536`): the three-channel +// pixel volume in [-1, 1] that `Ltx2ReadFrameDirectory` returns becomes a +// one-channel mask in [0, 1] — mean over channels, then `(x + 1) / 2`, then +// clamp. The READ half is `Ltx2ReadFrameDirectory` itself, at the stage's own +// height and width, which is what `ic_lora.py:460-461` spells `args.height // 2`. +std::vector Ltx2MaskVideoFromPixels(const std::vector& pixels, int64_t channels, + int64_t frames, int64_t plane); + +// `downsample_mask_video_to_latent` (`iclora_utils.py:52-84`). A pixel-space +// mask `[frames, height, width]` becomes flattened latent token weights +// `[f_lat * h_lat * w_lat]`, in the token order the video patchifier uses. +// +// AREA interpolation spatially (`:63-67`) — for the shapes this engine reaches +// it is exact box averaging — then the causal split: latent frame 0 is pixel +// frame 0 alone (`:70`, `:80`) and the remaining `f_pix - 1` frames mean-pool in +// groups of `t = (f_pix - 1) / (f_lat - 1)` (`:73`, `:78-79`). Refuses a +// non-divisible pair with upstream's own assertion (`:74-77`), and degenerates +// to the first frame alone when either axis has one frame (`:81-82`). +std::vector Ltx2DownsampleMaskVideoToLatent(const std::vector& mask, int64_t f_pix, + int64_t h_pix, int64_t w_pix, int64_t f_lat, + int64_t h_lat, int64_t w_lat); + +// `resolve_cross_mask` (`mask_utils.py:13-73`) at batch 1. `values` empty is the +// SCALAR arm (`:31-37`): `scalar` fills all `num_new_tokens`. Otherwise `values` +// is the 1-D `(M,)` form (`:49-54`) and its length must be `num_new_tokens`, +// refused by name exactly as `:50-53` refuses it. +std::vector Ltx2ResolveCrossMask(const std::vector& values, double scalar, + int64_t num_new_tokens); + +// `build_attention_mask` (`mask_utils.py:170-243`) at batch 1. Returns a dense +// `[(N+M) * (N+M)]` row-major mask in [0, 1]. +// +// noisy(Nn) prev_ref(N-Nn) new_ref(M) +// noisy existing existing cross +// prev_ref existing existing 0 +// new_ref cross 0 1 +// +// `existing` EMPTY is upstream's `existing_mask=None` and fills the top-left +// `N x N` block with ones (`:225-226`); otherwise it is preserved (`:224`), +// which is what keeps an earlier item's attenuation alive under a later one. +std::vector Ltx2BuildAttentionMask(const std::vector& existing, + int64_t num_noisy_tokens, int64_t num_new_tokens, + int64_t num_existing_tokens, + const std::vector& cross_mask); + +// `update_attention_mask` (`mask_utils.py:110-167`) at batch 1, for the +// `attention_mask is None` arm alone (`:141-156`). +// +// `existing` empty returns empty — upstream's "no mask requested and none +// present, return None" (`:142-143`). `existing` NON-empty pads the new tokens +// with FULL attention (`:147`) rather than returning the mask unchanged, which +// is what keeps the mask's dimensions equal to the growing sequence. A mask left +// short of the sequence is not a shape error anywhere downstream: it is read as +// a broadcastable key-only form and silently masks the wrong axis. +std::vector Ltx2PadAttentionMaskForUnmaskedTokens(const std::vector& existing, + int64_t num_noisy_tokens, + int64_t num_new_tokens, + int64_t num_existing_tokens); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 2d4ba645cf..b12a5b884a 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -988,6 +988,28 @@ struct Ltx2PipelineRecipe { // takes the keyframe item under BOTH of them. Ltx2ImageConditioningBuilder image_conditioning = Ltx2ImageConditioningBuilder::kCombined; + // WHETHER THIS PIPELINE CONDITIONS ON A REFERENCE CLIP. Row + // LTX25-IC-LORA-REF-VIDEO, issue #3020 (gaps A15 and A16). + // + // A FLAG ON THE RECIPE, for the reason `checkpoint_class` gives above, and + // because exactly one upstream pipeline class has this: `ICLoraPipeline` + // (ic_lora.py:60) calls `append_ic_lora_reference_video_conditionings` + // (`:381-402`) and no other pipeline in `ltx-pipelines` does. Serving the arm + // on `distilled_two_stage` would be inventing conditioning `distilled.py` has + // no call to. + // + // AND IT IS THE SAME PREDICATE THAT REFUSES. The phase loop reads this field + // once, binds it to a named local, and both the routing branch and the refusal + // beside it use that local. A refusal and its route predicate that are two + // expressions is how this campaign shipped a silently wrong answer. + // + // STAGE 1 ONLY is NOT this flag: it is the phase index, because upstream gives + // stage 1 `_create_conditionings` (which appends the reference item, + // ic_lora.py:269-281) and stage 2 plain `combined_image_conditionings` with no + // reference item at all (`:314-321`). A recipe-level "which phase" field would + // have exactly one value on the one recipe that sets this. + bool ic_lora_reference = false; + int64_t max_spatial_downscale() const; }; diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 5a8671fb93..69af709a5b 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -465,6 +465,48 @@ inline constexpr char kLtx2RetakeFrameRateExtra[] = "retake_frame_rate"; inline constexpr char kLtx2RegenerateVideoExtra[] = "regenerate_video"; inline constexpr char kLtx2RegenerateAudioExtra[] = "regenerate_audio"; +// ── IC-LoRA REFERENCE CONDITIONING. Row LTX25-IC-LORA-REF-VIDEO (#3020) ───── +// +// Read ONLY on an engine whose recipe declares `ic_lora_reference`, which is +// `pipeline_kind = ic_lora` alone. `ICLoraPipeline` is a pipeline CLASS upstream +// (ic_lora.py:60) and no other pipeline in `ltx-pipelines` calls +// `append_ic_lora_reference_video_conditionings`, so serving the arm elsewhere +// would be inventing conditioning the reference does not have. +// +// The reference CLIP itself is `VideoGenParams::ref_video_dir` — a directory of +// `frame_%06d.ppm`, this ABI's standing spelling for a video, because no demuxer +// is vendored here. Upstream reads a container (`decode_video_by_frame`, +// iclora_utils.py:141) and its folder arm is the adaptation row LTX25-RETAKE +// already recorded. + +// The STRENGTH half of `--video-conditioning PATH STRENGTH` +// (ic_lora.py:416-425), which becomes `VideoConditionByReferenceLatent.strength` +// (iclora_utils.py:166) and therefore the reference tokens' denoise mask +// `1 - strength`. Default 1.0. Upstream's flag is REQUIRED and takes both halves +// together; here the path is a request field that already exists, so only the +// strength needs a name. +inline constexpr char kLtx2RefVideoStrengthExtra[] = "ref_video_strength"; + +// The MASK_PATH half of `--conditioning-attention-mask MASK_PATH STRENGTH` +// (ic_lora.py:427-441), as a directory of `frame_%06d.ppm` for the reason above. +// Upstream loads it at the STAGE-1 resolution (`args.height // 2`, `:460-461`); +// this engine reads it at the stage's own grid, which is the same number derived +// rather than assumed. +inline constexpr char kLtx2CondAttentionMaskDirExtra[] = "conditioning_attention_mask_dir"; + +// Its STRENGTH half (`:454-455`), multiplied into the downsampled mask +// (iclora_utils.py:156). Default 1.0, and refused outside [0, 1] exactly as +// `ICLoraPipeline.__call__` refuses it. +// +// A STRENGTH BELOW 1 WITHOUT A MASK IS REFUSED BY NAME, and that is a mirror +// rather than a limitation. Upstream's scalar-only arm (`iclora_utils.py:157-158`) +// cannot be reached from the CLI: `conditioning_attention_strength` is assigned +// only inside `if args.conditioning_attention_mask is not None` +// (ic_lora.py:452-455) and is 1.0 otherwise, so a sub-1.0 strength always +// arrives with a mask. It is a Python-API-only branch, MEASURED as such by +// `kLtx2RefWrapScalarBelowOne`, and it is recorded owed rather than guessed at. +inline constexpr char kLtx2CondAttentionStrengthExtra[] = "conditioning_attention_strength"; + // ── TEXT-TO-AUDIO. Row LTX25-T2A-ONE-STAGE (#1005) ───────────────────────── // // These are read ONLY on a `pipeline_kind = t2a_one_stage` engine — that is a @@ -1072,6 +1114,37 @@ struct Ltx2ConditioningTrace { uint64_t retake_latent_digest = 0; double retake_latent_absmax = 0.0; + // ── IC-LoRA REFERENCE VIDEO and its ATTENTION MASK ──────────────────────── + // (row LTX25-IC-LORA-REF-VIDEO, #3020 — gaps A15 and A16) + // + // Observed for the reason the retake block above is: the whole mechanism is + // APPENDED TOKENS plus an attention BIAS, and neither is visible in a frame + // count, an output resolution or a finished clip. A build that read the + // reference clip, encoded it, and then appended nothing renders a video of + // exactly the right length with exactly the right soundtrack. + // + // `ic_lora_reference_tokens` is how many tokens the reference item appended, + // measured as the sequence GROWTH across the item rather than recomputed from + // the latent shape — a count derived from the shape would agree with itself on + // a build that computed the shape and appended nothing. + int64_t ic_lora_reference_tokens = 0; + // The encoded reference latent, before any denoising. The lower bound a token + // count cannot make: a zeroed latent has the right size and the right count. + uint64_t ic_lora_reference_digest = 0; + double ic_lora_reference_absmax = 0.0; + // The self-attention STRENGTH mask actually handed to the DiT, and NOT the one + // that was built. `rows` is `Ltx2ModalityInput::attention_mask_rows` at the + // forward — 0 when no mask was handed over — so a build that constructed the + // mask and then dropped the pointer is separated from one that passed it. + // + // `min` and `max` are both recorded because an ALL-ONES mask is the identity: + // it renders correctly, it has the right shape, and it is exactly what a + // downsample that lost its values produces. A mask whose min equals its max is + // not attenuating anything. + int64_t ic_lora_attention_mask_rows = 0; + double ic_lora_attention_mask_min = 0.0; + double ic_lora_attention_mask_max = 0.0; + // ── THE SAMPLER (row LTX25-RES2S-LOOP, #921) ────────────────────────────── // // TWO COUNTERS, BECAUSE THERE ARE TWO QUESTIONS AND ONE NUMBER CANNOT ANSWER diff --git a/scripts/gen-ltx2-iclora-reference-goldens.py b/scripts/gen-ltx2-iclora-reference-goldens.py new file mode 100755 index 0000000000..7bda56ac1f --- /dev/null +++ b/scripts/gen-ltx2-iclora-reference-goldens.py @@ -0,0 +1,540 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_iclora_reference_goldens.inc — the IC-LoRA +reference-clip and conditioning-attention-mask oracle. + +Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +spec .agents/specs/ltx25-ic-lora-ref-video.md. + +Every number below is what upstream's own code RETURNED. Nothing is transcribed. + +WHAT IS IMPORTED AND WHAT IS SLICED. `ltx_core.conditioning.mask_utils` and +`ltx_core.model.transformer.transformer_args` import cleanly and are imported. +`ltx_pipelines.iclora_utils` cannot be: it reaches `utils.media_io`, which +imports PyAV, a codec binding this project deliberately does not vendor. So its +three functions are lifted from the real file BY AST and their own source text is +compiled verbatim, the same technique scripts/gen-ltx2-dfr-goldens.py uses and +for the same reason. Each name is asserted present, so an upstream rename raises +here instead of leaving this generator running a local copy of the thing under +test. + +`append_ic_lora_reference_video_conditionings` is EXECUTED, with only its I/O and +its VAE stubbed: the frame decode, the pixel preprocess and the encoder are +recording stubs, and every branch — the divisibility refusal, the reference +geometry, the `if reference_temporal_scale_factor > 1` guard, the three-way mask +selection and the `if attn_mask is not None` wrap — is upstream's own control +flow running. That is what these goldens are about; the numerics of the resize +are already gated by ltx2_image_cond_goldens.inc. + +EVERY CASE CARRIES ITS REJECTED HYPOTHESIS AND THIS SCRIPT REFUSES TO EMIT A +GOLDEN THAT CANNOT FAIL. `build_attention_mask` on a fixture with no PRIOR +conditioning item is elementwise equal to the plausible wrong reading "cross on +all existing rows" — measured 0 separating elements — so such a fixture is a mute +switch. `require_separation` is called on every case and raises rather than +writing the file. + +Regenerate with: + python3 scripts/gen-ltx2-iclora-reference-goldens.py --ltx2 \ + --out tests/vllm/models/ltx2_iclora_reference_goldens.inc +""" + +from __future__ import annotations + +import argparse +import ast +import pathlib +import subprocess +import sys + + +def git_revision(root: pathlib.Path) -> str: + """The checkout's SHA, and REFUSE a dirty tree.""" + head = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, capture_output=True, text=True, + ).stdout.strip() + dirty = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, capture_output=True, text=True, + ).stdout.strip() + if dirty: + raise SystemExit( + f"REFUSING: {root} is dirty, so {head} would not describe the executed code:\n{dirty}" + ) + return head + + +def slice_functions(path: pathlib.Path, wanted: tuple[str, ...], namespace: dict) -> None: + """Compile the named top-level functions out of `path` verbatim into `namespace`.""" + source = path.read_text() + tree = ast.parse(source) + found: dict[str, ast.stmt] = {} + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in wanted: + found[node.name] = node + missing = [n for n in wanted if n not in found] + if missing: + raise SystemExit(f"REFUSING: {path} no longer defines {missing}") + lines = source.splitlines() + for name in wanted: + node = found[name] + text = "\n".join(lines[node.lineno - 1:node.end_lineno]) + exec(compile(ast.parse(text), str(path), "exec"), namespace) # noqa: S102 + + +SEPARATIONS: list[tuple[str, str, float]] = [] + + +def require_separation(case: str, rejected: str, value: float) -> None: + """A golden that agrees with the hypothesis it claims to reject is a mute switch.""" + if not (value > 0.0): + raise SystemExit( + f"REFUSING to emit '{case}': it does not separate from '{rejected}' " + f"(separation {value!r}). A golden that cannot fail is not a gate." + ) + SEPARATIONS.append((case, rejected, value)) + + +def fmt(x: float) -> str: + # `.9g` alone emits `0` for zero, which is an int literal in C++ and makes the + # array's element type depend on where the zero sits. Force a decimal point. + text = f"{float(x):.9g}" + if "." not in text and "e" not in text and "n" not in text: + text += ".0" + return text + "f" + + +def carray(name: str, values) -> str: + body = ", ".join(fmt(v) for v in values) + return f"inline constexpr float {name}[] = {{{body}}};" + + +def iarray(name: str, values) -> str: + body = ", ".join(str(int(v)) for v in values) + return f"inline constexpr int64_t {name}[] = {{{body}}};" + + +def deterministic(n: int, seed: int) -> list[float]: + """A splitmix64-derived stream in [0, 1). The C++ side reads the SAME bytes out + of this file, so the generator is free to choose any reproducible stream.""" + out = [] + state = seed & 0xFFFFFFFFFFFFFFFF + for _ in range(n): + state = (state + 0x9E3779B97F4A7C15) & 0xFFFFFFFFFFFFFFFF + z = state + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & 0xFFFFFFFFFFFFFFFF + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & 0xFFFFFFFFFFFFFFFF + z ^= z >> 31 + # 24 bits, so the value is exact in f32 and survives the round trip. + out.append((z >> 40) / float(1 << 24)) + return out + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--ltx2", required=True, type=pathlib.Path) + ap.add_argument("--out", required=True, type=pathlib.Path) + args = ap.parse_args() + + root = args.ltx2 + sys.path.insert(0, str(root / "packages" / "ltx-core" / "src")) + sys.path.insert(0, str(root / "packages" / "ltx-pipelines" / "src")) + revision = git_revision(root) + + import torch + from einops import rearrange + import torch.nn.functional as F + + from ltx_core.conditioning import ( + ConditioningItemAttentionStrengthWrapper, + VideoConditionByReferenceLatent, + ) + from ltx_core.conditioning.mask_utils import build_attention_mask, resolve_cross_mask + from ltx_core.types import VideoLatentShape + + iclora_path = root / "packages" / "ltx-pipelines" / "src" / "ltx_pipelines" / "iclora_utils.py" + ns: dict = { + "torch": torch, + "rearrange": rearrange, + "VideoLatentShape": VideoLatentShape, + "ConditioningItem": object, + "ConditioningItemAttentionStrengthWrapper": ConditioningItemAttentionStrengthWrapper, + "VideoConditionByReferenceLatent": VideoConditionByReferenceLatent, + "VideoEncoder": object, + "TilingConfig": object, + "HDRColorSpace": object, + "ResizeMode": object, + "is_exr_dir": lambda p: False, + "load_exr_folder_conditioning_hdr": None, + "annotations": None, + } + slice_functions( + iclora_path, + ("temporal_subsample", "downsample_mask_video_to_latent", + "append_ic_lora_reference_video_conditionings"), + ns, + ) + temporal_subsample = ns["temporal_subsample"] + downsample = ns["downsample_mask_video_to_latent"] + append_refs = ns["append_ic_lora_reference_video_conditionings"] + + out: list[str] = [] + E = out.append + + E("// GENERATED by scripts/gen-ltx2-iclora-reference-goldens.py — DO NOT EDIT BY HAND.") + E("//") + E("// LTX-2.5 IC-LoRA REFERENCE-CLIP and CONDITIONING-ATTENTION-MASK goldens") + E("// (row LTX25-IC-LORA-REF-VIDEO, issue #3020), produced by IMPORTING AND") + E("// EXECUTING upstream's `ltx_core.conditioning.mask_utils`, its") + E("// `_prepare_self_attention_mask`, and the three AST-sliced functions of") + E("// `ltx_pipelines.iclora_utils` (which cannot be imported: it reaches PyAV).") + E("//") + E("// Every case carries the REJECTED hypothesis beside upstream's answer, and") + E("// the generator REFUSES to write a case whose separation is zero. See the") + E("// separation table at the bottom of this file.") + E("//") + E(f"// Upstream revision (Lightricks/LTX-2): {revision}") + E("#pragma once") + E("") + E("#include ") + E("") + E("namespace vllm_test {") + E("") + E("// The suite asserts this equals the SHA it pins, so regenerating against a") + E("// DIFFERENT checkout fails the gate instead of silently replacing the oracle.") + E(f'inline constexpr const char* kLtx2IcLoraRefUpstreamRevision = "{revision}";') + E("") + + # ── 1. temporal_subsample (iclora_utils.py:87-90) ──────────────────────── + E("// --- 1. temporal_subsample (iclora_utils.py:87-90) -----------------------") + E("//") + E("// `[0, *range(1, F, N)]`. INDEX 1 IS ALWAYS KEPT when it exists, so at N=2") + E("// over 5 frames the kept set is {0, 1, 3} and NOT the plausible {0, 2, 4}.") + E("// That is the whole content of this function and the rejected hypothesis below") + E("// is exactly the plausible reading.") + ts_cases = [(5, 2), (9, 4), (7, 3), (6, 1), (1, 4)] + E(f"inline constexpr int kLtx2TemporalSubsampleCases = {len(ts_cases)};") + E(iarray("kLtx2TemporalSubsampleFrames", [f for f, _ in ts_cases])) + E(iarray("kLtx2TemporalSubsampleFactor", [n for _, n in ts_cases])) + counts, flat = [], [] + total_sep = 0 + for frames, factor in ts_cases: + # Upstream indexes a (B, C, F, H, W) tensor on dim 2. One channel, 1x1 + # pixels: the value at each kept frame IS its index, so what comes back is + # the index vector itself rather than a restatement of it. + video = torch.arange(frames, dtype=torch.float32).reshape(1, 1, frames, 1, 1) + got = temporal_subsample(video, factor).flatten().tolist() + rejected = list(range(0, frames, factor)) # "every Nth from 0" + sep = sum(1 for i in range(max(len(got), len(rejected))) + if i >= len(got) or i >= len(rejected) or got[i] != rejected[i]) + total_sep += sep + counts.append(len(got)) + flat.extend(int(v) for v in got) + require_separation("temporal_subsample", "every Nth frame from index 0 (range(0, F, N))", + float(total_sep)) + E(iarray("kLtx2TemporalSubsampleCount", counts)) + E(iarray("kLtx2TemporalSubsampleKept", flat)) + E("") + + # ── 2. the reference geometry and the branch map (iclora_utils.py:93-170) ── + E("// --- 2. append_ic_lora_reference_video_conditionings (iclora_utils.py:93-170)") + E("//") + E("// UPSTREAM'S OWN CONTROL FLOW, executed with the frame decode, the pixel") + E("// preprocess and the VAE replaced by recording stubs. What is gated here is") + E("// which geometry it asks the preprocess for, whether it subsamples, which of") + E("// the three mask branches it takes, and whether it wraps.") + + class RecordingEncoder: + def __init__(self, rec): + self.rec = rec + + def __call__(self, video): + self.rec["encoded_shape"] = tuple(video.shape) + # A latent volume with the shape the caller will read back through + # VideoLatentShape.from_torch_shape: (B, C, F, H, W). + return torch.zeros(1, 4, 2, 2, 2) + + def run_append(height, width, scale, temporal, strength, mask, cond_strength): + rec: dict = {} + ns2 = dict(ns) + ns2["decode_video_by_frame"] = lambda path, frame_cap, device: iter(()) + def preprocess(frames, h, w, dtype, device): + rec["ref_height"], rec["ref_width"] = h, w + # 5 pixel frames of 1x1, value == frame index, so a subsample is visible. + return torch.arange(5, dtype=torch.float32).reshape(1, 1, 5, 1, 1) + ns2["video_preprocess"] = preprocess + slice_functions(iclora_path, ("append_ic_lora_reference_video_conditionings",), ns2) + conds: list = [] + ns2["append_ic_lora_reference_video_conditionings"]( + conds, [("ref.mp4", strength)], + height=height, width=width, num_frames=5, + video_encoder=RecordingEncoder(rec), dtype=torch.float32, + device=torch.device("cpu"), + reference_downscale_factor=scale, + reference_temporal_scale_factor=temporal, + conditioning_attention_strength=cond_strength, + conditioning_attention_mask=mask, + ) + rec["wrapped"] = isinstance(conds[0], ConditioningItemAttentionStrengthWrapper) + inner = conds[0].conditioning if rec["wrapped"] else conds[0] + rec["downscale"] = inner.downscale_factor + rec["temporal"] = inner.temporal_scale_factor + rec["strength"] = inner.strength + return rec + + # (a) the geometry: 448x768 at scale 4 -> the preprocess is asked for 112x192. + geo = run_append(448, 768, 4, 1, 0.75, None, 1.0) + E(f"inline constexpr int64_t kLtx2RefGeomHeight = {geo['ref_height']};") + E(f"inline constexpr int64_t kLtx2RefGeomWidth = {geo['ref_width']};") + require_separation("reference geometry at scale 4", + "the phase's own height and width (no downscale)", + float(abs(geo["ref_height"] - 448) + abs(geo["ref_width"] - 768))) + E("// The item carries the factors, and the request's strength, unchanged.") + E(f"inline constexpr int64_t kLtx2RefItemDownscale = {geo['downscale']};") + E(f"inline constexpr float kLtx2RefItemStrength = {fmt(geo['strength'])};") + E(f"inline constexpr bool kLtx2RefGeomWrapped = {'true' if geo['wrapped'] else 'false'};") + + # (b) the temporal-subsample guard: it fires at N>1 and NOT at N==1. + sub = run_append(64, 64, 1, 2, 1.0, None, 1.0) + nosub = run_append(64, 64, 1, 1, 1.0, None, 1.0) + E("// The subsample guard (`if reference_temporal_scale_factor > 1`, :143). The") + E("// encoder sees 3 of 5 frames at N=2 and all 5 at N=1 — the encoded SHAPE is") + E("// what records it, so a port that subsampled unconditionally is separated.") + E(f"inline constexpr int64_t kLtx2RefEncodedFramesN2 = {sub['encoded_shape'][2]};") + E(f"inline constexpr int64_t kLtx2RefEncodedFramesN1 = {nosub['encoded_shape'][2]};") + require_separation("the temporal-subsample guard", + "subsampling unconditionally", + float(abs(sub["encoded_shape"][2] - nosub["encoded_shape"][2]))) + + # (c) the three-way mask branch (iclora_utils.py:151-160), and the wrap. + scalar_lt1 = run_append(64, 64, 1, 1, 1.0, None, 0.5) + scalar_eq1 = run_append(64, 64, 1, 1, 1.0, None, 1.0) + E("// The three-way mask selection (:151-160). With NO mask and strength < 1 the") + E("// scalar arm wraps; with NO mask and strength == 1 nothing wraps. The CLI") + E("// cannot reach the first (a strength is only ever set alongside a mask,") + E("// ic_lora.py:452-455), and this engine refuses it by name — the golden is") + E("// here so the refusal is pinned to a MEASURED branch rather than a reading.") + E(f"inline constexpr bool kLtx2RefWrapScalarBelowOne = {'true' if scalar_lt1['wrapped'] else 'false'};") + E(f"inline constexpr bool kLtx2RefWrapScalarAtOne = {'true' if scalar_eq1['wrapped'] else 'false'};") + require_separation("the strength<1 wrap branch", "wrapping unconditionally", + float(scalar_lt1["wrapped"] != scalar_eq1["wrapped"])) + + # (d) the divisibility refusal (:112-115). + try: + run_append(448, 770, 4, 1, 1.0, None, 1.0) + raise SystemExit("REFUSING: upstream accepted an indivisible target; the refusal moved") + except ValueError as e: + refusal = str(e) + E("// The divisibility refusal (:112-115), verbatim from the raised ValueError.") + E(f'inline constexpr const char* kLtx2RefDivisibilityRefusal = "{refusal}";') + # scale == 1 never refuses, whatever the dimensions. + ok = run_append(449, 771, 1, 1, 1.0, None, 1.0) + E(f"inline constexpr int64_t kLtx2RefScaleOneHeight = {ok['ref_height']};") + require_separation("the divisibility refusal", "refusing whenever height % scale != 0, scale==1 included", + float(ok["ref_height"] == 449)) + E("") + + # ── 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ──────────── + E("// --- 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ----------") + E("//") + E("// AREA spatial interpolation, then the CAUSAL carve-out: latent frame 0 is") + E("// pixel frame 0 ALONE, and the remaining f_pix-1 frames mean-pool in groups") + E("// of t = (f_pix-1)/(f_lat-1). Both rejected hypotheses below are measured.") + f_pix, h_pix, w_pix = 9, 8, 8 + f_lat, h_lat, w_lat = 3, 2, 2 + vals = deterministic(f_pix * h_pix * w_pix, 0x5EED1C10) + mask = torch.tensor(vals, dtype=torch.float32).reshape(1, 1, f_pix, h_pix, w_pix) + shape = VideoLatentShape.from_torch_shape((1, 1, f_lat, h_lat, w_lat)) + got = downsample(mask, shape) + t = (f_pix - 1) // (f_lat - 1) + bil = F.interpolate(rearrange(mask, "b 1 f h w -> (b f) 1 h w"), + size=(h_lat, w_lat), mode="bilinear", align_corners=False) + bil = rearrange(bil, "(b f) 1 h w -> b 1 f h w", b=1) + bil_rest = rearrange(bil[:, :, 1:], "b 1 (f t) h w -> b 1 f t h w", t=t).mean(3) + bil_flat = rearrange(torch.cat([bil[:, :, :1], bil_rest], 2), "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent", "bilinear spatial interpolation", + float((got - bil_flat).abs().max())) + area = rearrange(F.interpolate(rearrange(mask, "b 1 f h w -> (b f) 1 h w"), + size=(h_lat, w_lat), mode="area"), + "(b f) 1 h w -> b 1 f h w", b=1) + # The plausible wrong reading: pool ALL f_pix frames uniformly into f_lat + # groups, with no first-frame carve-out. f_pix must divide by f_lat for the + # hypothesis to be expressible at all, which the fixture arranges. + assert f_pix % f_lat == 0, "the no-carve-out hypothesis needs f_pix % f_lat == 0" + unif = rearrange(rearrange(area, "b 1 (f t) h w -> b 1 f t h w", t=f_pix // f_lat).mean(3), + "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent", + "uniform temporal pooling with no causal first-frame carve-out", + float((got[:, : unif.shape[1]] - unif).abs().max())) + E(f"inline constexpr int64_t kLtx2MaskPixFrames = {f_pix};") + E(f"inline constexpr int64_t kLtx2MaskPixHeight = {h_pix};") + E(f"inline constexpr int64_t kLtx2MaskPixWidth = {w_pix};") + E(f"inline constexpr int64_t kLtx2MaskLatFrames = {f_lat};") + E(f"inline constexpr int64_t kLtx2MaskLatHeight = {h_lat};") + E(f"inline constexpr int64_t kLtx2MaskLatWidth = {w_lat};") + E("// The INPUT, so both sides read identical bytes rather than agreeing about a") + E("// random stream. Every value is exact in f32 (24 significant bits).") + E(carray("kLtx2MaskPixels", vals)) + E(carray("kLtx2MaskLatentWeights", got.flatten().tolist())) + E("// The two rejected answers, emitted so the suite asserts it is NOT them.") + E(carray("kLtx2MaskLatentBilinear", bil_flat.flatten().tolist())) + E(carray("kLtx2MaskLatentUniformPool", unif.flatten().tolist())) + + # A SECOND fixture whose spatial output does NOT divide its input. `area` + # pools output index i over `[floor(i*I/O), ceil((i+1)*I/O))`, which is only + # the integer-stride window `[i*(I//O), (i+1)*(I//O))` when O divides I. The + # 8 -> 2 case above is divisible, so it is elementwise equal to the stride + # reading and cannot see the difference; 9 -> 2 can. + nd_f_pix, nd_h_pix, nd_w_pix = 9, 9, 9 + nd_f_lat, nd_h_lat, nd_w_lat = 3, 2, 2 + nd_vals = deterministic(nd_f_pix * nd_h_pix * nd_w_pix, 0x9D19151B) + nd_mask = torch.tensor(nd_vals, dtype=torch.float32).reshape( + 1, 1, nd_f_pix, nd_h_pix, nd_w_pix) + nd_shape = VideoLatentShape.from_torch_shape((1, 1, nd_f_lat, nd_h_lat, nd_w_lat)) + nd_got = downsample(nd_mask, nd_shape) + + def stride_box_spatial(volume, h_out, w_out): + """The plausible wrong reading: an integer-stride box filter.""" + frames, h_in, w_in = volume.shape[2], volume.shape[3], volume.shape[4] + sh, sw = h_in // h_out, w_in // w_out + out_v = torch.empty(1, 1, frames, h_out, w_out, dtype=torch.float32) + for fi in range(frames): + for oh in range(h_out): + for ow in range(w_out): + out_v[0, 0, fi, oh, ow] = volume[ + 0, 0, fi, oh * sh:(oh + 1) * sh, ow * sw:(ow + 1) * sw].mean() + return out_v + + nd_t = (nd_f_pix - 1) // (nd_f_lat - 1) + box = stride_box_spatial(nd_mask, nd_h_lat, nd_w_lat) + box_rest = rearrange(box[:, :, 1:], "b 1 (f t) h w -> b 1 f t h w", t=nd_t).mean(3) + box_flat = rearrange(torch.cat([box[:, :, :1], box_rest], 2), "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent at a NON-DIVIDING spatial pair (9 -> 2)", + "an integer-stride box filter, which `area` equals only when O divides I", + float((nd_got - box_flat).abs().max())) + E("// A NON-DIVIDING spatial pair, 9 -> 2. `area` pools output index i over") + E("// `[floor(i*I/O), ceil((i+1)*I/O))`; the integer-stride window") + E("// `[i*(I//O), (i+1)*(I//O))` agrees with it on every divisible shape and") + E("// only there, so the 8 -> 2 case above cannot see the difference and this") + E("// one can.") + E(f"inline constexpr int64_t kLtx2MaskNdPixFrames = {nd_f_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdPixHeight = {nd_h_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdPixWidth = {nd_w_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdLatFrames = {nd_f_lat};") + E(f"inline constexpr int64_t kLtx2MaskNdLatHeight = {nd_h_lat};") + E(f"inline constexpr int64_t kLtx2MaskNdLatWidth = {nd_w_lat};") + E(carray("kLtx2MaskNdPixels", nd_vals)) + E(carray("kLtx2MaskNdLatentWeights", nd_got.flatten().tolist())) + E("// The rejected answer, emitted so the suite asserts it is NOT this one.") + E(carray("kLtx2MaskNdLatentStrideBox", box_flat.flatten().tolist())) + E("") + + # ── 4. resolve_cross_mask + build_attention_mask (mask_utils.py) ────────── + E("// --- 4. build_attention_mask (mask_utils.py:170-243) ---------------------") + E("//") + E("// THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN ON PURPOSE. With") + E("// num_existing == num_noisy the true block structure and the plausible wrong") + E("// reading `cross on ALL existing rows` are ELEMENTWISE EQUAL — measured 0") + E("// separating elements — and the golden would be a mute switch. The generator") + E("// refuses that case; this one separates on 4 elements.") + n_noisy, n_new, n_existing = 2, 2, 3 + cross_vals = [0.25, 0.75] + cross = torch.tensor([cross_vals], dtype=torch.float32) + got_mask = build_attention_mask(None, n_noisy, n_new, n_existing, cross, + torch.device("cpu"), torch.float32) + alt = got_mask.clone() + alt[:, n_noisy:n_existing, n_existing:] = cross + alt[:, n_existing:, n_noisy:n_existing] = cross.T + require_separation("build_attention_mask", "cross on ALL existing rows, not only the noisy ones", + float((got_mask != alt).sum())) + E(f"inline constexpr int64_t kLtx2AmNoisy = {n_noisy};") + E(f"inline constexpr int64_t kLtx2AmNew = {n_new};") + E(f"inline constexpr int64_t kLtx2AmExisting = {n_existing};") + E(carray("kLtx2AmCross", cross_vals)) + E(carray("kLtx2AmMask", got_mask.flatten().tolist())) + E(carray("kLtx2AmMaskCrossOnAllRows", alt.flatten().tolist())) + + E("// A SECOND item on top of the first: `existing_mask` is preserved in the") + E("// top-left block rather than refilled with ones (`:223-226`), which is what") + E("// keeps the first item's attenuation alive after a second one is applied.") + cross2_vals = [0.5] + cross2 = torch.tensor([cross2_vals], dtype=torch.float32) + got2 = build_attention_mask(got_mask, n_noisy, 1, n_existing + n_new, cross2, + torch.device("cpu"), torch.float32) + alt2 = build_attention_mask(None, n_noisy, 1, n_existing + n_new, cross2, + torch.device("cpu"), torch.float32) + require_separation("build_attention_mask with an existing mask", + "discarding the existing mask and refilling the block with ones", + float((got2 - alt2).abs().max())) + E(carray("kLtx2AmCross2", cross2_vals)) + E(carray("kLtx2AmMask2", got2.flatten().tolist())) + E(carray("kLtx2AmMask2Refilled", alt2.flatten().tolist())) + + E("// resolve_cross_mask's scalar arm (`:31-37`) and its 1-D broadcast (`:49-54`).") + sc = resolve_cross_mask(0.5, 4, 1, torch.device("cpu"), torch.float32) + E(carray("kLtx2CrossScalar", sc.flatten().tolist())) + oned = resolve_cross_mask(torch.tensor([0.1, 0.2, 0.3, 0.4]), 4, 1, + torch.device("cpu"), torch.float32) + E(carray("kLtx2CrossOneD", oned.flatten().tolist())) + require_separation("resolve_cross_mask's 1-D arm", "the scalar fill", + float((sc - oned).abs().max())) + + E("// update_attention_mask's PAD-WITH-ONES arm (`:141-156`): a null mask on a") + E("// state that already carries one grows the mask with full attention rather") + E("// than returning None and leaving it short of the sequence.") + ones = torch.ones(1, 1, dtype=torch.float32) + padded = build_attention_mask(got_mask, n_noisy, 1, n_existing + n_new, ones, + torch.device("cpu"), torch.float32) + E(carray("kLtx2AmMaskPaddedOnes", padded.flatten().tolist())) + require_separation("update_attention_mask's pad-with-ones arm", + "returning the mask unchanged (short of the sequence)", + float(padded.numel() - got_mask.numel())) + E("") + + # ── 5. _prepare_self_attention_mask (transformer_args.py:208-237) ───────── + E("// --- 5. _prepare_self_attention_mask (transformer_args.py:208-237) -------") + E("//") + E("// The CONSUMPTION seam this row makes reachable. Already ported as") + E("// Ltx2PrepareSelfAttentionMask; pinned here because nothing in src/ assigned") + E("// Ltx2ModalityInput::attention_mask before this row, so these numbers were") + E("// gated only through a hand-built test struct.") + import inspect + import ltx_core.model.transformer.transformer_args as ta + owner = None + for _name, obj in vars(ta).items(): + if inspect.isclass(obj) and "_prepare_self_attention_mask" in vars(obj): + owner = obj + break + if owner is None: + raise SystemExit("REFUSING: no class in transformer_args defines _prepare_self_attention_mask") + prep = owner._prepare_self_attention_mask + probe = [1.0, 0.5, 0.0, 1e-30] + bias = prep(None, torch.tensor([[probe]], dtype=torch.float32), torch.float32) + vals_b = [float(x) for x in bias.flatten()] + E(f'inline constexpr const char* kLtx2SelfMaskOwner = "{owner.__name__}";') + E(carray("kLtx2SelfMaskProbe", probe)) + E("// f32. The third entry is finfo(f32).min, NOT -inf and NOT 0; the fourth is") + E("// log(tiny) after the clamp, NOT log(1e-30).") + E(carray("kLtx2SelfMaskBias", vals_b)) + naive = [float(torch.log(torch.tensor(max(v, 0.0)))) if v > 0 else float("-inf") for v in probe] + require_separation("_prepare_self_attention_mask", "-inf for a zero and an unclamped log", + float(max(abs(a - b) for a, b in zip(vals_b, naive) if b != float("-inf")) + + (1.0 if any(b == float("-inf") for b in naive) else 0.0))) + E("") + + # ── the separation table ───────────────────────────────────────────────── + E("// --- separation table ----------------------------------------------------") + E("//") + E("// Every case above, its rejected hypothesis, and the measured separation. A") + E("// zero here is a mute switch and the generator raises rather than writing it.") + for case, rejected, value in SEPARATIONS: + E(f"// {case} vs {rejected} -> {value!r}") + E("") + E("} // namespace vllm_test") + + args.out.write_text("\n".join(out) + "\n") + print(f"wrote {args.out} ({len(out)} lines), {len(SEPARATIONS)} separated cases") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/model_executor/models/ltx2_conditioning.cpp b/src/vllm/model_executor/models/ltx2_conditioning.cpp index dd733d6393..4c3f822067 100644 --- a/src/vllm/model_executor/models/ltx2_conditioning.cpp +++ b/src/vllm/model_executor/models/ltx2_conditioning.cpp @@ -9,6 +9,8 @@ // (tools.py:158-176, keyframe_cond.py:57-65), so this one is not a widening. #include "vllm/model_executor/models/ltx2_conditioning.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + #include #include #include @@ -74,6 +76,27 @@ void AppendTokens(Ltx2LatentState* state, const std::vector& tokens, int6 // term to the wrong tokens. Ltx2ExtendKeyframesMask(state, token_count, /*marked=*/false); + // AND THE ATTENTION MASK, for the same reason and from the same pre-append + // state. Every appending item upstream calls `update_attention_mask` with a + // literal `attention_mask=None` (keyframe_cond.py:68-76, + // reference_video_cond.py:86-94), which returns None when the state carries no + // mask and otherwise pads the new tokens with FULL attention + // (mask_utils.py:141-156). Row LTX25-IC-LORA-REF-VIDEO (#3020) is what made + // the second branch reachable: before it no state could carry a mask at all. + // + // A stale mask is NOT a shape error downstream. `Ltx2ModalityInput` accepts + // both the dense [tokens, tokens] form and the key-only [1, tokens] broadcast, + // so a mask left at its pre-append size is read as a legal mask over a + // different axis, and the render comes out the right size with a trained term + // applied to the wrong tokens. + if (!state->attention_mask.empty()) { + VT_CHECK(state->noisy_tokens > 0, + "ltx2 conditioning: a state carrying an attention mask must know its own noisy-token " + "count, which is `latent_tools.target_shape.token_count()` upstream"); + state->attention_mask = Ltx2PadAttentionMaskForUnmaskedTokens( + state->attention_mask, state->noisy_tokens, token_count, before); + } + state->latent.resize(static_cast(after * state->width), 0.0f); state->clean.insert(state->clean.end(), tokens.begin(), tokens.end()); state->mask.insert(state->mask.end(), static_cast(token_count), @@ -126,6 +149,13 @@ Ltx2LatentState Ltx2CreateVideoLatentState(const Ltx2VideoLatentShape& shape, in if (out_keyframes_mask != nullptr) { *out_keyframes_mask = state.keyframes_mask; } + // `latent_tools.target_shape.token_count()` — what an appending item hands + // `update_attention_mask` as `num_noisy_tokens`. Recorded HERE, at the one + // moment it is knowable without searching: after the first append `tokens` is + // no longer it, and the rows between the two are prior REFERENCE tokens whose + // block in the mask differs from the noisy one (mask_utils.py:236 vs :242). + state.noisy_tokens = state.tokens; + return state; } @@ -487,6 +517,13 @@ Ltx2LatentState Ltx2CreateAudioLatentState(const Ltx2AudioLatentShape& shape, // The audio positions ARE the patch grid bounds, in seconds — there is no // get_pixel_coords and no fps division on this side (tools.py:271-279). state.positions = Ltx2AudioPatchTimings(shape, params); + // `latent_tools.target_shape.token_count()` — what an appending item hands + // `update_attention_mask` as `num_noisy_tokens`. Recorded HERE, at the one + // moment it is knowable without searching: after the first append `tokens` is + // no longer it, and the rows between the two are prior REFERENCE tokens whose + // block in the mask differs from the noisy one (mask_utils.py:236 vs :242). + state.noisy_tokens = state.tokens; + return state; } diff --git a/src/vllm/model_executor/models/ltx2_iclora_reference.cpp b/src/vllm/model_executor/models/ltx2_iclora_reference.cpp new file mode 100644 index 0000000000..9fc12e763d --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_iclora_reference.cpp @@ -0,0 +1,319 @@ +// LTX-2.5 IC-LoRA reference conditioning — see ltx2_iclora_reference.h for the +// port map and for the four failure modes this file exists to make impossible. +// +// Row LTX25-IC-LORA-REF-VIDEO (#3020). Upstream: Lightricks/LTX-2 @ fd4ded7f. +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + +#include +#include +#include +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { + throw std::runtime_error("ltx2 ic-lora reference: " + why); +} + +// `torch.nn.functional.interpolate(..., mode="area")` dispatches to +// `adaptive_avg_pool2d`, whose window for output index `i` over input extent `I` +// and output extent `O` is `[floor(i*I/O), ceil((i+1)*I/O))`. That is NOT the +// same as an integer-stride box filter whenever `O` does not divide `I`, and the +// difference is invisible on the divisible shapes a reduced fixture reaches — so +// the general form is written here rather than the special case. +int64_t PoolStart(int64_t index, int64_t in_extent, int64_t out_extent) { + return (index * in_extent) / out_extent; +} +int64_t PoolEnd(int64_t index, int64_t in_extent, int64_t out_extent) { + return ((index + 1) * in_extent + out_extent - 1) / out_extent; +} + +} // namespace + +Ltx2IcLoraReferenceGeometry Ltx2ResolveIcLoraReferenceGeometry(int64_t height, int64_t width, + int64_t downscale_factor) { + if (downscale_factor < 1) { + Fail("reference_downscale_factor must be at least 1, got " + + std::to_string(downscale_factor) + + ". Upstream's absent-metadata default is 1 (iclora_utils.py:35)"); + } + // `:112` — the guard is `scale != 1 AND (h % scale or w % scale)`. At scale 1 + // NOTHING is ever refused, which is what keeps every adapter-less render + // working, and a port that dropped the `scale != 1` half would refuse none of + // them differently (1 divides everything) while a port that dropped the + // divisibility half would refuse all of them. + if (downscale_factor != 1 && (height % downscale_factor != 0 || width % downscale_factor != 0)) { + Fail("Output dimensions (" + std::to_string(height) + "x" + std::to_string(width) + + ") must be divisible by reference_downscale_factor (" + + std::to_string(downscale_factor) + + "). This is upstream's own refusal (iclora_utils.py:112-115): the reference clip is read " + "at height // factor by width // factor and a truncating division would place the " + "reference tokens on a grid the target does not have"); + } + Ltx2IcLoraReferenceGeometry out; + out.height = height / downscale_factor; // :116 + out.width = width / downscale_factor; // :117 + return out; +} + +std::vector Ltx2TemporalSubsampleIndices(int64_t frames, int64_t temporal_scale_factor) { + if (frames < 1) Fail("a reference clip needs at least one frame"); + if (temporal_scale_factor < 1) { + Fail("reference_temporal_scale_factor must be at least 1, got " + + std::to_string(temporal_scale_factor)); + } + // `indices = [0, *list(range(1, video.shape[2], temporal_scale_factor))]` (:89). + // + // INDEX 1 IS ALWAYS KEPT when it exists, because the range starts AT 1 rather + // than stepping from 0. At factor 2 over 5 frames this is {0, 1, 3}; the + // plausible `range(0, F, N)` gives {0, 2, 4} — the same COUNT, so no shape + // check and no token count can tell them apart, and the reference simply + // describes different moments of the clip. Gated by `kLtx2TemporalSubsampleKept` + // against exactly that hypothesis. + std::vector indices; + indices.push_back(0); + for (int64_t i = 1; i < frames; i += temporal_scale_factor) indices.push_back(i); + return indices; +} + +std::vector Ltx2TemporalSubsample(const std::vector& clip, int64_t channels, + int64_t frames, int64_t plane, + int64_t temporal_scale_factor) { + const std::vector keep = Ltx2TemporalSubsampleIndices(frames, temporal_scale_factor); + const size_t expect = static_cast(channels) * static_cast(frames) * + static_cast(plane); + if (clip.size() != expect) { + Fail("the reference clip holds " + std::to_string(clip.size()) + " values but " + + std::to_string(channels) + " x " + std::to_string(frames) + " x " + + std::to_string(plane) + " is " + std::to_string(expect)); + } + const int64_t kept = static_cast(keep.size()); + std::vector out(static_cast(channels) * static_cast(kept) * + static_cast(plane)); + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < kept; ++t) { + const size_t src = (static_cast(c) * static_cast(frames) + + static_cast(keep[static_cast(t)])) * + static_cast(plane); + const size_t dst = (static_cast(c) * static_cast(kept) + + static_cast(t)) * + static_cast(plane); + std::copy(clip.begin() + static_cast(src), + clip.begin() + static_cast(src + static_cast(plane)), + out.begin() + static_cast(dst)); + } + } + return out; +} + +std::vector Ltx2MaskVideoFromPixels(const std::vector& pixels, int64_t channels, + int64_t frames, int64_t plane) { + if (channels < 1) Fail("a mask video needs at least one channel"); + const size_t expect = static_cast(channels) * static_cast(frames) * + static_cast(plane); + if (pixels.size() != expect) { + Fail("the mask video holds " + std::to_string(pixels.size()) + " values but " + + std::to_string(channels) + " x " + std::to_string(frames) + " x " + + std::to_string(plane) + " is " + std::to_string(expect)); + } + std::vector out(static_cast(frames) * static_cast(plane)); + for (int64_t t = 0; t < frames; ++t) { + for (int64_t p = 0; p < plane; ++p) { + // `mask_video.mean(dim=1, keepdim=True)` (:531). The pixels arrive + // CHANNEL-major, which is the layout the encoder takes, so the three + // samples of one pixel are a plane apart rather than adjacent. + double sum = 0.0; + for (int64_t c = 0; c < channels; ++c) { + const size_t at = (static_cast(c) * static_cast(frames) + + static_cast(t)) * + static_cast(plane) + + static_cast(p); + sum += static_cast(pixels[at]); + } + const double mean = sum / static_cast(channels); + // `(mask + 1.0) / 2.0` then `.clamp(0.0, 1.0)` (:534-536). The remap + // undoes `normalize_images`' `/127.5 - 1`, and the clamp is upstream's and + // not defensive: a mask frame that decoded above the range would otherwise + // become an attention weight above 1, which `_prepare_self_attention_mask` + // turns into a POSITIVE log-space bias — an amplifier, not an attenuator. + out[static_cast(t) * static_cast(plane) + static_cast(p)] = + static_cast(std::min(1.0, std::max(0.0, (mean + 1.0) / 2.0))); + } + } + return out; +} + +std::vector Ltx2DownsampleMaskVideoToLatent(const std::vector& mask, int64_t f_pix, + int64_t h_pix, int64_t w_pix, int64_t f_lat, + int64_t h_lat, int64_t w_lat) { + if (f_pix < 1 || h_pix < 1 || w_pix < 1 || f_lat < 1 || h_lat < 1 || w_lat < 1) { + Fail("every mask dimension must be positive"); + } + const size_t expect = + static_cast(f_pix) * static_cast(h_pix) * static_cast(w_pix); + if (mask.size() != expect) { + Fail("the mask holds " + std::to_string(mask.size()) + " values but " + + std::to_string(f_pix) + " x " + std::to_string(h_pix) + " x " + std::to_string(w_pix) + + " is " + std::to_string(expect)); + } + + // :63-67 — AREA interpolation, per pixel frame, to the LATENT spatial grid. + const size_t lat_plane = static_cast(h_lat) * static_cast(w_lat); + std::vector spatial(static_cast(f_pix) * lat_plane); + for (int64_t t = 0; t < f_pix; ++t) { + for (int64_t oh = 0; oh < h_lat; ++oh) { + const int64_t h0 = PoolStart(oh, h_pix, h_lat), h1 = PoolEnd(oh, h_pix, h_lat); + for (int64_t ow = 0; ow < w_lat; ++ow) { + const int64_t w0 = PoolStart(ow, w_pix, w_lat), w1 = PoolEnd(ow, w_pix, w_lat); + double sum = 0.0; + for (int64_t ih = h0; ih < h1; ++ih) { + for (int64_t iw = w0; iw < w1; ++iw) { + sum += static_cast( + mask[(static_cast(t) * static_cast(h_pix) + + static_cast(ih)) * + static_cast(w_pix) + + static_cast(iw)]); + } + } + spatial[static_cast(t) * lat_plane + static_cast(oh) * + static_cast(w_lat) + + static_cast(ow)] = sum / static_cast((h1 - h0) * (w1 - w0)); + } + } + } + + // :70 — the FIRST latent frame is pixel frame 0 ALONE. This is the causal + // carve-out, and it is the whole content of the temporal half: pooling all + // `f_pix` frames uniformly into `f_lat` groups produces a correctly shaped + // mask of entirely plausible values. Gated against exactly that hypothesis by + // `kLtx2MaskLatentUniformPool`. + std::vector out; + out.reserve(static_cast(f_lat) * lat_plane); + for (size_t i = 0; i < lat_plane; ++i) { + out.push_back(static_cast(spatial[i])); + } + // :72 — and the rest ONLY when both axes have more than one frame; otherwise + // the first frame alone is the whole answer (:81-82). + if (f_pix > 1 && f_lat > 1) { + const int64_t t = (f_pix - 1) / (f_lat - 1); // :73 + if ((f_pix - 1) % (f_lat - 1) != 0) { // :74-77, upstream's assertion + Fail("Pixel frames (" + std::to_string(f_pix) + ") not compatible with latent frames (" + + std::to_string(f_lat) + "): (f_pix - 1) must be divisible by (f_lat - 1). The mask " + "video and the reference clip must describe the same moments, and a truncating " + "group size would silently drop the tail of the mask"); + } + for (int64_t lf = 1; lf < f_lat; ++lf) { + for (size_t p = 0; p < lat_plane; ++p) { + double sum = 0.0; + for (int64_t k = 0; k < t; ++k) { + // :78-79 — the remaining frames reshape to [f_lat - 1, t] and mean + // over the group axis. Frame `1 + (lf - 1) * t + k`, because the + // reshape happens AFTER frame 0 has been taken out. + const int64_t src = 1 + (lf - 1) * t + k; + sum += spatial[static_cast(src) * lat_plane + p]; + } + out.push_back(static_cast(sum / static_cast(t))); + } + } + } + return out; +} + +std::vector Ltx2ResolveCrossMask(const std::vector& values, double scalar, + int64_t num_new_tokens) { + if (num_new_tokens < 1) Fail("a cross mask needs at least one new token"); + if (values.empty()) { + // :31-37 — the scalar arm fills every new token with the same weight. + return std::vector(static_cast(num_new_tokens), static_cast(scalar)); + } + // :49-53 — the 1-D `(M,)` form, whose length upstream refuses to broadcast. + if (static_cast(values.size()) != num_new_tokens) { + Fail("1-D attention_mask length must equal num_new_tokens (" + + std::to_string(num_new_tokens) + "), got " + std::to_string(values.size()) + + ". Upstream refuses rather than broadcasting (mask_utils.py:50-53), because a mask one " + "token short of the sequence would attenuate the wrong tokens and still render"); + } + return values; +} + +std::vector Ltx2BuildAttentionMask(const std::vector& existing, + int64_t num_noisy_tokens, int64_t num_new_tokens, + int64_t num_existing_tokens, + const std::vector& cross_mask) { + if (num_noisy_tokens < 1 || num_new_tokens < 1 || num_existing_tokens < num_noisy_tokens) { + Fail("the attention mask's block structure needs 1 <= num_noisy <= num_existing and at least " + "one new token; got noisy=" + + std::to_string(num_noisy_tokens) + " new=" + std::to_string(num_new_tokens) + + " existing=" + std::to_string(num_existing_tokens)); + } + if (static_cast(cross_mask.size()) != num_new_tokens) { + Fail("the cross mask must hold one weight per new token (" + std::to_string(num_new_tokens) + + "), got " + std::to_string(cross_mask.size())); + } + const int64_t total = num_existing_tokens + num_new_tokens; // :217 + if (!existing.empty() && + static_cast(existing.size()) != num_existing_tokens * num_existing_tokens) { + Fail("the existing mask must be [" + std::to_string(num_existing_tokens) + ", " + + std::to_string(num_existing_tokens) + "], got " + std::to_string(existing.size()) + + " values"); + } + + std::vector out(static_cast(total) * static_cast(total), 0.0F); // :220 + const auto At = [&out, total](int64_t r, int64_t c) -> float& { + return out[static_cast(r) * static_cast(total) + static_cast(c)]; + }; + + // :223-226 — the top-left N x N block is the existing mask, or ones when there + // was none. NOT ones-plus-existing and not zeros: an absent mask means every + // existing token attends to every other, INCLUDING prior reference tokens that + // carried no mask of their own (`:204-205`). + for (int64_t r = 0; r < num_existing_tokens; ++r) { + for (int64_t c = 0; c < num_existing_tokens; ++c) { + At(r, c) = existing.empty() + ? 1.0F + : existing[static_cast(r) * static_cast(num_existing_tokens) + + static_cast(c)]; + } + } + // :229 — the new reference tokens fully attend to THEMSELVES. + for (int64_t r = num_existing_tokens; r < total; ++r) { + for (int64_t c = num_existing_tokens; c < total; ++c) At(r, c) = 1.0F; + } + // :236 — noisy rows attend to the new reference tokens at the cross weight, + // one weight per COLUMN. `num_noisy_tokens` and NOT `num_existing_tokens`: + // rows between the two are PRIOR reference tokens and they stay at the zero + // `:242` leaves them at. On a fixture with no prior item the two readings are + // elementwise equal, which is why the golden's fixture carries one. + for (int64_t r = 0; r < num_noisy_tokens; ++r) { + for (int64_t j = 0; j < num_new_tokens; ++j) { + At(r, num_existing_tokens + j) = cross_mask[static_cast(j)]; + } + } + // :240 — and the transpose: new reference ROWS attend to the noisy tokens, one + // weight per ROW. + for (int64_t i = 0; i < num_new_tokens; ++i) { + for (int64_t c = 0; c < num_noisy_tokens; ++c) { + At(num_existing_tokens + i, c) = cross_mask[static_cast(i)]; + } + } + return out; +} + +std::vector Ltx2PadAttentionMaskForUnmaskedTokens(const std::vector& existing, + int64_t num_noisy_tokens, + int64_t num_new_tokens, + int64_t num_existing_tokens) { + // :142-143 — no mask requested and none present is upstream's `None`, and an + // empty vector is how this engine spells it. + if (existing.empty()) return {}; + // :147 — otherwise the new tokens get FULL attention, so the mask keeps + // describing the whole sequence. + const std::vector ones(static_cast(num_new_tokens), 1.0F); + return Ltx2BuildAttentionMask(existing, num_noisy_tokens, num_new_tokens, num_existing_tokens, + ones); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index d4d4927407..cdf91f58ff 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -2121,6 +2121,48 @@ Ltx2PipelineRecipe KeyframeInterpolationRecipe(const Ltx2PipelineParams& params, return recipe; } +// ICLoraPipeline (ltx-pipelines/ic_lora.py:60-402). Row LTX25-IC-LORA-REF-VIDEO, +// issue #3020. +// +// THE SCHEDULE IS THE DISTILLED TWO-STAGE ONE, EXACTLY, and that is a finding +// rather than a shortcut. `ICLoraPipeline.__call__` takes `stage_1_sigmas` +// defaulting to `DISTILLED_SIGMAS` and `stage_2_sigmas` to +// `STAGE_2_DISTILLED_SIGMAS` (the same two constants `distilled.py` imports), +// runs stage 1 at `width // 2, height // 2` (`:260-266`), and re-noises stage 2 +// to `stage_2_sigmas[0]` (`:333-343`). Every one of those is what +// `DistilledTwoStageRecipe` already carries. IC-LoRA differs from the distilled +// two-stage pipeline in its CONDITIONING and in where its adapter rides, not in +// its schedule — recorded here so a reader does not go looking for an IC-LoRA +// sigma set upstream does not have. +// +// TWO FIELDS SEPARATE IT, and they are the two `ICLoraPipeline` was written for. +Ltx2PipelineRecipe IcLoraRecipe(const std::string& version) { + Ltx2PipelineRecipe recipe = DistilledTwoStageRecipe(version); + if (recipe.phases.size() != 2) { + Refuse("The IC-LoRA recipe is the distilled two-stage recipe with IC-LoRA's adapter split, " + "and that recipe just returned " + + std::to_string(recipe.phases.size()) + + " phases. Stage 1 and stage 2 are addressed by INDEX below, so a changed phase count " + "would scope the adapter to the wrong one."); + } + recipe.phases[0].name = "ic_lora_lowres"; + recipe.phases[1].name = "ic_lora_refine"; + // `ic_lora.py:108` against `:119` — the MIRROR IMAGE of every other two-stage + // pipeline in this table. `A2VidTwoStageRecipe` and its siblings give stage 1 + // `kNoAdapters` and stage 2 the adapters; here the adapter rides stage 1 and + // stage 2 runs BARE, because the IC-LoRA is what teaches the model to read the + // reference and stage 2 only refines an already-conditioned latent. + // + // Getting this backwards renders a video. Stage 2 would run a 3-step + // refinement under an adapter it was never given, and the clip comes out the + // right length at the right resolution. + recipe.phases[1].loras = Ltx2PhaseLoraScope::kNoAdapters; + // `_create_conditionings` (`:269-281`, calling `:381-402`) against stage 2's + // plain `combined_image_conditionings` (`:314-321`). + recipe.ic_lora_reference = true; + return recipe; +} + } // namespace Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, @@ -2149,6 +2191,14 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // resolving DFR onto it would build a recipe whose first stage the engine // must then refuse at load. Refusing at the recipe table names the version. if (model_version == "2.5") return DfrRecipe(model_version); + } else if (pipeline_kind == "ic_lora") { + // 2.5 only, and 2.0 is refused rather than resolved. `ICLoraPipeline`'s + // reference item carries the adapter's `reference_downscale_factor` and + // `reference_temporal_scale_factor` (iclora_utils.py:30-49), which are + // LTX-2.5 IC-LoRA metadata keys; the 2.0 distilled row here exists for a + // generation that predates them, and resolving onto it would build a + // pipeline whose reference geometry is always the unscaled one. + if (model_version == "2.5") return IcLoraRecipe(model_version); } else if (pipeline_kind == "res2s_two_stage") { // 2.5 only — see `Res2sTwoStageRecipe`: `LTX_2_3_HQ_PARAMS` is a plain // constant with no `detect_params` lineage, so there is no second version to diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index a36beb85a9..8981580a88 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -46,6 +46,7 @@ #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" #include "vllm/model_executor/models/ltx2_samplers.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" #include "vllm/model_executor/models/ltx2_retake.h" #include "vllm/model_executor/models/ltx2_t2a.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" @@ -191,6 +192,16 @@ struct StreamState { // was supplied. Empty on the audio stream, whose args preprocessor upstream // builds with no keyframes_embedding_provider (model.py:333). std::vector keyframes_mask; + // `LatentState.attention_mask` (types.py:251-287), DENSE [tokens, tokens] in + // [0, 1], and the target token count every appending item measures the mask's + // block structure against. Row LTX25-IC-LORA-REF-VIDEO (#3020). + // + // EMPTY is upstream's `None` and is what every render carries unless an + // IC-LoRA conditioning attention mask was supplied. `noisy_tokens` is set once + // beside `tokens` on the target grid and never moves, which is exactly what + // `latent_tools.target_shape.token_count()` is upstream. + std::vector attention_mask; + int64_t noisy_tokens = 0; }; // ── StreamState <-> Ltx2LatentState (row LTX25-TOKEN-APPEND, issue #930) ──── @@ -223,6 +234,8 @@ Ltx2LatentState ToLatentState(const StreamState& s, int64_t pos_dims) { out.positions[i] = static_cast(s.positions[i]); } out.keyframes_mask = s.keyframes_mask; + out.attention_mask = s.attention_mask; + out.noisy_tokens = s.noisy_tokens; return out; } @@ -237,6 +250,8 @@ void FromLatentState(const Ltx2LatentState& in, StreamState* s) { s->positions[i] = static_cast(in.positions[i]); } s->keyframes_mask = in.keyframes_mask; + s->attention_mask = in.attention_mask; + s->noisy_tokens = in.noisy_tokens; } // `post_process_latent` (utils/helpers.py:462-464): @@ -482,7 +497,7 @@ constexpr char kLtx2AutoDurationExtra[] = "auto_duration"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 680 682 1324 1420 1516 1532 1667 1671 1829 1865 2017 2135 2177 2219 2221 +// 695 697 1339 1435 1531 1547 1682 1686 1844 1880 2032 2150 2192 2234 2236 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, @@ -2713,7 +2728,15 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2VideoSkipStepExtra || kv.first == kLtx2VideoStgBlocksExtra || kv.first == kLtx2A2vGuidanceScaleExtra || - kv.first == kLtx2V2aGuidanceScaleExtra; + kv.first == kLtx2V2aGuidanceScaleExtra || + // The IC-LoRA reference row (row LTX25-IC-LORA-REF-VIDEO, + // #3020). DEFINED here on every kind so that the refusal + // below can name the pipeline that serves them; a knob + // dropped from this list is refused as a TYPO, which + // tells a caller nothing about which pipeline to load. + kv.first == kLtx2RefVideoStrengthExtra || + kv.first == kLtx2CondAttentionMaskDirExtra || + kv.first == kLtx2CondAttentionStrengthExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -2727,7 +2750,9 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2AudioStgBlocksExtra + ", " + kLtx2VideoCfgScaleExtra + ", " + kLtx2VideoStgScaleExtra + ", " + kLtx2VideoRescaleScaleExtra + ", " + kLtx2VideoSkipStepExtra + ", " + kLtx2VideoStgBlocksExtra + ", " + - kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra); + kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra + ", " + + kLtx2RefVideoStrengthExtra + ", " + kLtx2CondAttentionMaskDirExtra + ", " + + kLtx2CondAttentionStrengthExtra); } } // ── the knobs that belong to ONE pipeline (#1005, corrected by #1092) ───── @@ -2767,7 +2792,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, kLtx2VideoRescaleScaleExtra, kLtx2VideoSkipStepExtra, kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, - kLtx2V2aGuidanceScaleExtra}; + kLtx2V2aGuidanceScaleExtra, + // A reference CLIP is a picture (#3020), + // and so is the mask that attenuates it. + kLtx2RefVideoStrengthExtra, + kLtx2CondAttentionMaskDirExtra, + kLtx2CondAttentionStrengthExtra}; for (const char* key : kNotOnT2a) { if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { Fail("the '" + std::string(key) + @@ -3208,120 +3238,125 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { const bool wants_first_frame = !gen.first_frame_path.empty() || !gen.first_frame_ppm.empty(); const bool wants_last_frame = !gen.last_frame_path.empty(); const bool wants_image = wants_first_frame || wants_last_frame; - // `!wants_retake` IS LOAD-BEARING, and it narrows this refusal rather than - // weakening it. Retake and IC-LoRA reference conditioning both arrive as - // `ref_video_dir`, and they consume it in completely different ways: retake - // encodes the clip at its OWN resolution and seeds the video stream's initial - // latent with it (retake.py:238-247, :273), while the reference item is - // downscaled by the adapter's factor, temporally subsampled, and APPENDED as - // extra tokens to a stage-1-only adapter (iclora_utils.py:112-117, :87-89, - // :144-148). Serving the first says nothing about the second, so the second - // stays refused and #975 stays open. - if (!wants_retake && (!gen.ref_image_paths.empty() || !gen.ref_video_dir.empty())) { - // TWO CAUSES REMAIN, AND NEITHER IS ONE THIS MESSAGE HAS EVER GIVEN. The - // message names both, and then names the three ruled-out reasons with what - // ruled each one out, because a reader who arrives here in a month should - // re-check the claim rather than re-derive the refutation for a third time. - // - // 1. THE REFERENCE CLIP HAS NO PIXEL PATH. Upstream resolves the reference - // at `height // scale` by `width // scale` (iclora_utils.py:116-117), - // refuses a target either axis of which the factor does not divide - // (:112-115), keeps frame 0 and then every Nth frame (`temporal_subsample`, - // :87-89, called at :144), and encodes the whole clip (:145-148). This - // engine's only pixel-to-latent route for a REFERENCE item is - // `Ltx2LoadImageAndPreprocess` followed by `Ltx2ConvVideoEncode` at - // `frame_count = 1` and the phase's OWN height and width, and it refuses - // an encode that returns more than one latent frame. - // - // THIS USED TO SAY "nothing anywhere reads `ref_video_dir`", and that was - // false about the tree even when it was written (#987): MiniMax-H3 has - // always consumed the directory in full — `ReadReferenceClipChw`, - // `minimax_h3_video.cpp:135`, called at `:650`. Since row LTX25-RETAKE - // (#924) the LTX-2.5 side reads it too, through - // `Ltx2ReadFrameDirectory`. So the missing piece is NOT a reader. It is - // the reference item's own geometry: the downscale-factor resize and the - // temporal subsample, neither of which retake performs and neither of - // which any reader supplies. - // - // THE SECOND REASON THIS MESSAGE GAVE IS NOW FALSE, and it is recorded - // here rather than deleted because it is the third reason in this block - // to come true and a reader needs to know which. It said: "the reference - // item belongs to stage 1 and stage 2 must run unfused — - // `ICLoraPipeline` gives stage 1 `loras=tuple(loras)` (ic_lora.py:108) - // and stage 2 `loras=()` (:119), and this engine holds ONE `Ltx2Dit`, - // fused at load, that every phase of the recipe runs. Serving the arm - // needs a second unfused DiT or a phase-scoped adapter." - // - // Row LTX25-PHASE-LORA (#1118) landed the phase-scoped adapter. - // `Ltx2PhaseRecipe::loras` (ltx2_pipeline.h) carries upstream's per-stage - // set and the phase loop in this file honours it through - // `Ltx2RebindDitLoras`, which re-materializes only the tensors an adapter - // targets — so a two-phase recipe CAN now give stage 1 the adapter and - // stage 2 none, which is exactly `ic_lora.py:108` against `:119`, and it - // does so without a second resident weight set. `A2VidTwoStageRecipe` is - // the executable proof it exists: it gives stage 1 `kNoAdapters` and the - // gate "the distilled adapter rides stage 2 ALONE" renders both states - // through this ABI and compares the pixels. - // - // What that leaves is reason 1 ALONE, and reason 1 is unrelated to - // weights: it is the reference clip's own geometry. The conditioning - // split is also still upstream's — stage 1 takes `_create_conditionings`, - // which appends the reference item (:269-278, :377-402), and stage 2 - // takes plain `combined_image_conditionings` with no reference item - // (:314-321) — but that is a conditioning question, not a fused-weight - // one, and serving the arm on one phase only is upstream's `skip_stage_2` - // (:302-308), a different request. - std::string factors = "no adapter was supplied, so none were read"; + // ── IC-LoRA REFERENCE CONDITIONING (row LTX25-IC-LORA-REF-VIDEO, #3020) ── + // + // ONE PREDICATE, READ ONCE, and it is what makes this block correct rather + // than merely present. `serves_reference` is bound here and used by BOTH the + // routing branch in the phase loop and every refusal below it. A refusal and + // its route predicate written as two expressions is how this campaign shipped + // a silently wrong answer, and the two are one local for exactly that reason. + // + // WHY IT IS A RECIPE FLAG. `ICLoraPipeline` (ic_lora.py:60) is the only + // pipeline in `ltx-pipelines` that calls + // `append_ic_lora_reference_video_conditionings`; `distilled.py`, `dfr`, + // `retake` and the four `*_two_stage` arms call none. Serving the reference on + // any of them would be inventing conditioning the reference does not have. + const bool serves_reference = im.recipe.ic_lora_reference; + const std::string cond_mask_dir = VideoExtra(gen.extras, kLtx2CondAttentionMaskDirExtra); + const double cond_attention_strength = + ExtraDouble(gen.extras, kLtx2CondAttentionStrengthExtra, 1.0); + const double ref_video_strength = ExtraDouble(gen.extras, kLtx2RefVideoStrengthExtra, 1.0); + + // `!wants_retake` IS LOAD-BEARING, and it narrows rather than weakens. Retake + // and IC-LoRA reference conditioning both arrive as `ref_video_dir`, and they + // consume it in completely different ways: retake encodes the clip at its OWN + // resolution and seeds the video stream's initial latent with it + // (retake.py:238-247, :273), while the reference item is downscaled by the + // adapter's factor, temporally subsampled, and APPENDED as extra tokens to a + // stage-1-only adapter (iclora_utils.py:112-117, :87-89, :144-148). + const bool wants_reference_video = !wants_retake && !gen.ref_video_dir.empty(); + + if (!wants_retake && !gen.ref_image_paths.empty()) { + // REFERENCE IMAGES ARE NOT UPSTREAM'S IC-LoRA SHAPE, and this stays refused + // whatever pipeline is loaded. `--video-conditioning` takes a video file or + // a directory of scene-linear EXR frames (ic_lora.py:416-425) and there is + // no `--reference-image` anywhere in `ltx-pipelines`. Serving a still here + // by encoding it at `frame_count = 1` would be a conditioning item upstream + // does not build, and it would silently answer a request for a different + // feature. Recorded owed in .agents/specs/ltx25-ic-lora-ref-video.md. + Fail( + "reference-IMAGE conditioning is not served, and it is not what an IC-LoRA reference is. " + "Upstream's IC-LoRA flag is `--video-conditioning PATH STRENGTH`, whose PATH is a video " + "file or a directory of scene-linear .exr frames (ic_lora.py:416-425); there is no " + "reference-image conditioning anywhere in `ltx-pipelines`, so serving a still here would " + "build a conditioning item the reference does not have. Use `ref_video_dir` with " + "'pipeline_kind' 'ic_lora' for a reference CLIP, first_frame_ppm / first_frame_path for " + "image-to-video, or last_frame_path for a closing keyframe. Recorded owed (#3020)."); + } + if (wants_reference_video && !serves_reference) { + // The SAME predicate the route uses, negated. A reference clip on a pipeline + // that has no reference conditioning is a request for a different pipeline, + // and saying so is more useful than saying the feature is missing — it is + // not missing, it is elsewhere. + std::string factors_note = "no adapter was supplied, so none were read"; if (im.dit.lora_fused_tensors > 0) { - factors = "the supplied adapter declares downscale=" + - std::to_string(im.dit.lora_reference.downscale) + - " temporal=" + std::to_string(im.dit.lora_reference.temporal) + - ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; + factors_note = "the supplied adapter declares downscale=" + + std::to_string(im.dit.lora_reference.downscale) + + " temporal=" + std::to_string(im.dit.lora_reference.temporal) + + ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; } - Fail( - "reference-image / reference-video conditioning is not served. TWO things are " - "missing. FIRST, the reference CLIP has no pixel path: upstream reads it at " - "`height // reference_downscale_factor` by `width // reference_downscale_factor` " - "(iclora_utils.py:116-117), refuses a target the factor does not divide (:112-115), " - "keeps frame 0 and then every Nth frame (`temporal_subsample`, :87-89, called at " - ":144) and encodes the whole clip (:145-148), while this engine's only " - "pixel-to-latent route for a REFERENCE item encodes exactly ONE frame at the phase's " - "own resolution. WHAT IS *NOT* THE REASON here: the READER. This message used to say " - "\"nothing reads `ref_video_dir` at all\", which was false about the tree when it was " - "written (#987) — MiniMax-H3 consumes the directory in full at " - "`minimax_h3_video.cpp:650` — and is doubly false now that row LTX25-RETAKE (#924) " - "reads it on this side through `Ltx2ReadFrameDirectory`. What is missing is the " - "reference item's own geometry, the downscale resize and the temporal subsample, " - "which no reader supplies. SECOND, the reference item is a STAGE-1 item and stage 2 " - "takes `combined_image_conditionings` with no reference item at all: `ICLoraPipeline` " - "gives stage 1 the reference conditioning (ic_lora.py:269-278) and stage 2 none " - "(:314-321), and this phase loop appends the same conditioning set to every phase. " - "That is a CONDITIONING gap and not a weights one. WHAT IS *NOT* THE REASON, because " - "this refusal has now given THREE reasons that later became false: (a) the IC-LoRA " - "METADATA. Row LTX25-IC-LORA (#923) " - "closed that; supply `lora_path` and the factors are read at load " - "(iclora_utils.py:30-49) — right now, " + factors + - ". (b) the TOKEN-APPEND machinery. This message blamed it on 2026-08-15 and row " - "LTX25-TOKEN-APPEND (#930) landed it in `c7cb59fbb` the next day: the phase loop " - "now binds a `target_tokens` local, grows `video.tokens` past it on an appending " - "item, carries the grown count through denoise, and trims back through " - "`Ltx2ClearConditioning` (ltx_core/tools.py:88-117) before unpatchify. The " - "last-frame keyframe arm is SERVED on exactly that machinery, which is the " - "executable proof it exists. (c) `Ltx2LatentState` carrying no attention-mask " - "field. On the DEFAULT arm upstream builds no mask: at " - "`conditioning_attention_strength >= 1.0` with no latent mask `attn_mask` is None " - "(iclora_utils.py:159-160) and `ConditioningItemAttentionStrengthWrapper` is " - "applied only `if attn_mask is not None` (:168-169). The sub-1.0 arm is owed by " - "#932, and it is not what blocks this one. (d) the FUSED-AT-LOAD adapter. This " - "message said until 2026-08-17 that stage 2 must run with no adapter while \"this " - "engine holds one DiT, fused at load, that every phase runs\", and row " - "LTX25-PHASE-LORA (#1118) closed it: `Ltx2PhaseRecipe::loras` carries upstream's " - "per-stage set and the phase loop rebinds the DiT through `Ltx2RebindDitLoras`, so " - "`loras=tuple(loras)` on stage 1 against `loras=()` on stage 2 (ic_lora.py:108, " - ":119) is now expressible with no second weight set. `a2vid_two_stage`'s stage 1 " - "runs `kNoAdapters` on exactly that machinery, which is the executable proof it " - "exists. Use first_frame_ppm / first_frame_path " - "for image-to-video, and last_frame_path for a closing keyframe."); + Fail("the '" + im.pipeline_kind + + "' pipeline does not condition on a reference clip, so `ref_video_dir` has no meaning on " + "it. `ICLoraPipeline` is the only pipeline in `ltx-pipelines` that calls " + "`append_ic_lora_reference_video_conditionings` (ic_lora.py:381-402); `distilled.py`, " + "`dfr_pipeline.py`, the four `*_two_stage` arms and `t2a_one_stage.py` call none of it. " + "Load with 'pipeline_kind' 'ic_lora' to serve a reference clip, or supply " + "'retake_start_time' and 'retake_end_time' to RETAKE a window of this directory instead " + "— which is a different operation: retake encodes the clip at its own resolution and " + "SEEDS the video latent with it (retake.py:238-247), while the reference item downscales " + "it, temporally subsamples it and APPENDS it as extra tokens (iclora_utils.py:112-117, " + ":87-89, :144-148). Right now, " + factors_note + "."); + } + if (!cond_mask_dir.empty() && !serves_reference) { + Fail("the '" + std::string(kLtx2CondAttentionMaskDirExtra) + "' extra attenuates the IC-LoRA " + "REFERENCE tokens' attention (iclora_utils.py:151-156, :168-169) and the '" + + im.pipeline_kind + + "' pipeline appends none, so there would be nothing for it to attenuate. Upstream applies " + "`ConditioningItemAttentionStrengthWrapper` at exactly one site and its argument is " + "always the `VideoConditionByReferenceLatent` built two lines above it; " + "`combined_image_conditionings` never wraps. Load with 'pipeline_kind' 'ic_lora'."); + } + if (serves_reference && !wants_reference_video) { + // `--video-conditioning` is `required=True` on this parser + // (ic_lora.py:416-425). An IC-LoRA render with no reference is upstream's + // distilled two-stage render, and defaulting to it silently would answer a + // request for a conditioned clip with an unconditioned one. + Fail("the 'ic_lora' pipeline conditions on a reference clip and this request supplied none. " + "`--video-conditioning PATH STRENGTH` is `required=True` upstream " + "(ic_lora.py:416-425), because `ICLoraPipeline` without it is exactly the distilled " + "two-stage pipeline. Set `ref_video_dir` to a directory of frame_%06d.ppm, or load with " + "'pipeline_kind' 'distilled_two_stage' for an unconditioned render."); + } + // UNCONDITIONAL, exactly as upstream's is. `ic_lora.py:230-233` sits in the + // method body, before anything about the mask and outside every `if + // args.conditioning_attention_mask is not None`. Guarding it on a mask made + // this engine ACCEPT an out-of-range strength whenever none was supplied and + // then never read it, because its only reader is inside the mask branch: a + // knob that takes a value, refuses nothing, and does nothing. + if (cond_attention_strength < 0.0 || cond_attention_strength > 1.0) { + Fail("'" + std::string(kLtx2CondAttentionStrengthExtra) + "' must be in [0.0, 1.0], got " + + std::to_string(cond_attention_strength) + + ". This is upstream's own refusal (ic_lora.py:230-233): the value multiplies a mask in " + "[0, 1] and the product becomes a log-space attention bias, so a value above 1 would " + "AMPLIFY attention rather than attenuate it."); + } + if (cond_mask_dir.empty() && cond_attention_strength < 1.0) { + // UPSTREAM HAS THIS BRANCH AND ITS CLI CANNOT REACH IT + // (`elif conditioning_attention_strength < 1.0`, iclora_utils.py:157-158). + // `conditioning_attention_strength` is assigned only inside + // `if args.conditioning_attention_mask is not None` (ic_lora.py:452-455) and + // is 1.0 otherwise, so a sub-1.0 strength always arrives WITH a mask. + // MEASURED rather than read: `kLtx2RefWrapScalarBelowOne` is what the pinned + // module returned for the no-mask, strength-0.5 call, and it is `true` — so + // the branch is real and this refusal is naming a Python-API-only arm rather + // than a branch that does not exist. Recorded owed (#3020). + Fail("'" + std::string(kLtx2CondAttentionStrengthExtra) + "' below 1.0 without '" + + std::string(kLtx2CondAttentionMaskDirExtra) + + "' is upstream's scalar-only attention arm (iclora_utils.py:157-158), which its own CLI " + "cannot reach: the strength is assigned only alongside a mask (ic_lora.py:452-455) and " + "is 1.0 otherwise. It is a Python-API-only branch, it is not ported, and it is recorded " + "owed (#3020) rather than guessed at. Supply a mask directory, or leave the strength at " + "1.0."); } if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { Fail( @@ -4169,6 +4204,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { StreamState video; video.width = vshape.channels; // patch_size 1 (VideoLatentPatchifier(1)) video.tokens = target_tokens; + // ...and remembers it, because every appending item measures the attention + // mask's block structure against the count BEFORE any item ran + // (mask_utils.py:236 gives the noisy rows the cross weight and `:242` leaves + // the prior-reference rows at zero). Bound to the SAME local the schedule + // reads, so a change to one moves both. + video.noisy_tokens = target_tokens; { std::vector volume(static_cast(vshape.channels) * static_cast(vshape.frames) * @@ -4753,6 +4794,175 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "size while omitting the trained embedding."); } + // ── THE IC-LoRA REFERENCE CLIP (row LTX25-IC-LORA-REF-VIDEO, #3020) ───── + // + // AFTER every image and keyframe item, because upstream appends it after + // `combined_image_conditionings` (ic_lora.py:377-402 runs at the END of + // `_create_conditionings`) and conditioning items are applied in list order + // (`state_with_conditionings`, helpers.py:448-458). Order is not cosmetic + // here: appended tokens land at the END of the sequence and + // `Ltx2ClearConditioning` trims from the end, so two appends in the wrong + // order swap their positions while every count still agrees. + // + // PHASE 0 ONLY, and that is upstream's split rather than an optimisation. + // `ICLoraPipeline` gives stage 1 `_create_conditionings`, which appends the + // reference item (ic_lora.py:269-281), and stage 2 plain + // `combined_image_conditionings` with NO reference item (`:314-321`). + // Appending it to stage 2 as well would put the reference tokens into a + // 3-step refinement that upstream runs unconditioned — and the clip would + // come out the right length at the right resolution. + // + // THE SAME `serves_reference` LOCAL the refusals above read. Two expressions + // for one predicate is how a refusal and its route drift apart. + if (serves_reference && phase_index == 0) { + // `scale = reference_downscale_factor` and the divisibility refusal + // (iclora_utils.py:111-115), then `height // scale` by `width // scale` + // (`:116-117`). Measured against the phase's OWN grid, which is stage 1's + // half resolution — the same number `ic_lora.py:460-461` spells + // `args.height // 2`, derived here rather than assumed. + const Ltx2IcLoraReferenceGeometry ref_geom = Ltx2ResolveIcLoraReferenceGeometry( + phase_h, phase_w, im.dit.lora_reference.downscale); + + // `decode_video_by_frame(path, frame_cap=num_frames)` then + // `video_preprocess(frames, ref_height, ref_width, ...)` + // (iclora_utils.py:141-142). `Ltx2ReadFrameDirectory` IS `video_preprocess` + // over the folder arm — row LTX25-RETAKE ported and gated it — so what is + // new here is only the GEOMETRY it is called at. + const Ltx2RetakeSourceGeometry ref_probe = Ltx2ProbeFrameDirectory(gen.ref_video_dir); + // `frame_cap` (`:141`). The reference may be longer than the render, and a + // clip read past the cap would append tokens for moments the target grid + // has no time for. + const int64_t ref_frames = std::min(ref_probe.frames, frames); + const std::vector ref_pixels = + Ltx2ReadFrameDirectory(gen.ref_video_dir, ref_geom.height, ref_geom.width); + const int64_t ref_channels = im.video_encoder_cfg.in_channels; + const int64_t ref_plane = ref_geom.height * ref_geom.width; + std::vector ref_clip; + if (ref_frames == ref_probe.frames) { + ref_clip = ref_pixels; + } else { + // The cap, applied on the FRAME axis of a channel-major volume. A plain + // prefix of the buffer would keep whole channels and drop others. + ref_clip.resize(static_cast(ref_channels * ref_frames * ref_plane)); + for (int64_t c = 0; c < ref_channels; ++c) { + const size_t src = static_cast(c * ref_probe.frames * ref_plane); + const size_t dst = static_cast(c * ref_frames * ref_plane); + std::copy(ref_pixels.begin() + static_cast(src), + ref_pixels.begin() + + static_cast(src + static_cast(ref_frames * ref_plane)), + ref_clip.begin() + static_cast(dst)); + } + } + + // `if reference_temporal_scale_factor > 1: video = temporal_subsample(...)` + // (`:143-144`). GUARDED, and the guard is measured: at factor 1 upstream + // hands the encoder all 5 frames of the probe fixture and at factor 2 it + // hands it 3, which `kLtx2RefEncodedFramesN1` and `...N2` pin. + int64_t ref_kept = ref_frames; + if (im.dit.lora_reference.temporal > 1) { + ref_clip = Ltx2TemporalSubsample(ref_clip, ref_channels, ref_frames, ref_plane, + im.dit.lora_reference.temporal); + ref_kept = static_cast( + Ltx2TemporalSubsampleIndices(ref_frames, im.dit.lora_reference.temporal).size()); + } + + // `encoded_video = video_encoder(video)` (`:148`) — the WHOLE clip, which + // is the same multi-frame encode the retake arm already drives. The + // `tiled_encode` arm (`:145-146`) is not reached: this engine tiles the + // target's decode and not a reference's encode, and it is recorded owed. + int64_t ref_cropped = 0; + const Ltx2LatentVolume ref_encoded = Ltx2ConvVideoEncode( + im.video_encoder_cfg, im.video_encoder_weights, ref_clip, ref_channels, ref_kept, + ref_geom.height, ref_geom.width, &ref_cropped); + im.trace.ic_lora_reference_digest = DigestF32(ref_encoded.data); + im.trace.ic_lora_reference_absmax = AbsMax(ref_encoded.data); + + // ── the MASK half (gap A16) ───────────────────────────────────────── + // + // Built BEFORE the item is applied, because it needs the reference + // LATENT's shape (`reference_video_shape`, `:149`) and because the + // wrapper measures its block structure against the PRE-append state + // (attention_strength_wrapper.py:49-64). + std::vector cross_mask; + if (!cond_mask_dir.empty()) { + // `_load_mask_video` (ic_lora.py:511-537): the same read, at the STAGE's + // resolution rather than the reference's — upstream passes + // `args.height // 2` and not `ref_height` (`:460-461`) — then mean over + // channels, `(x + 1) / 2`, clamp. + const Ltx2RetakeSourceGeometry mask_probe = Ltx2ProbeFrameDirectory(cond_mask_dir); + const int64_t mask_frames = std::min(mask_probe.frames, frames); + const std::vector mask_pixels = + Ltx2ReadFrameDirectory(cond_mask_dir, phase_h, phase_w); + const std::vector mask_video = Ltx2MaskVideoFromPixels( + mask_pixels, ref_channels, mask_probe.frames, phase_h * phase_w); + std::vector capped( + mask_video.begin(), + mask_video.begin() + static_cast(mask_frames * phase_h * phase_w)); + // `downsample_mask_video_to_latent(mask, target_latent_shape= + // reference_video_shape)` (`:151-155`) — the REFERENCE's latent shape and + // not the target's, because these weights attenuate the reference's own + // tokens. + std::vector latent_mask = Ltx2DownsampleMaskVideoToLatent( + capped, mask_frames, phase_h, phase_w, ref_encoded.frames, ref_encoded.height, + ref_encoded.width); + // `attn_mask = latent_mask * conditioning_attention_strength` (`:156`). + for (float& v : latent_mask) v *= static_cast(cond_attention_strength); + // `resolve_cross_mask`'s 1-D arm (mask_utils.py:49-54), which refuses a + // length that is not the new-token count rather than broadcasting. + cross_mask = Ltx2ResolveCrossMask(latent_mask, /*scalar=*/1.0, + static_cast(latent_mask.size())); + } + + // THE SEQUENCE LENGTH AND THE MASK BEFORE THIS ITEM. The wrapper snapshots + // the ORIGINAL state (attention_strength_wrapper.py:50) and builds against + // it; building against the post-append count would place the new block + // over the tokens it just added while every dimension still agreed. + const int64_t before_reference = video.tokens; + const std::vector mask_before = video.attention_mask; + + Ltx2LatentState state = ToLatentState(video, /*pos_dims=*/3); + Ltx2ConditionVideoByReference(&state, ref_encoded, /*patch_size=*/1, factors, fps, + im.dit.lora_reference.downscale, + im.dit.lora_reference.temporal, ref_video_strength, + /*causal_fix=*/true); + FromLatentState(state, &video); + + const int64_t ref_tokens = video.tokens - before_reference; + VT_CHECK(ref_tokens > 0, + "ltx2 video: the reference item must APPEND tokens " + "(reference_video_cond.py:96-108) and this one left the sequence unchanged"); + VT_CHECK(static_cast(video.latent.size()) == video.tokens * video.width && + static_cast(video.clean.size()) == video.tokens * video.width && + static_cast(video.mask.size()) == video.tokens && + static_cast(video.keyframes_mask.size()) == video.tokens && + static_cast(video.positions.size()) == 3 * video.tokens * 2, + "ltx2 video: after the reference append every per-token buffer must have one entry " + "per token. A buffer that did not grow with the others is invisible to the " + "render's SHAPE — the clip comes out the right size and describes the wrong " + "tokens."); + // MEASURED GROWTH, not a count recomputed from the latent's shape. A + // derived number agrees with itself on a build that computed the shape and + // appended nothing. + im.trace.ic_lora_reference_tokens = ref_tokens; + + // `ConditioningItemAttentionStrengthWrapper.apply_to` (`:55-71`): the mask + // is built from the ORIGINAL state and its `num_new_tokens` is the + // measured difference, then it REPLACES whatever the inner item's own + // `update_attention_mask(None, ...)` left (which is the pad-with-ones + // form, or nothing at all on a first item). + if (!cross_mask.empty()) { + VT_CHECK(static_cast(cross_mask.size()) == ref_tokens, + "ltx2 video: the conditioning attention mask resolved to " + + std::to_string(cross_mask.size()) + " weights but the reference item " + "appended " + std::to_string(ref_tokens) + + " tokens. The mask is downsampled to the REFERENCE latent's own shape " + "(iclora_utils.py:151-155), so these cannot differ unless one of the two " + "read a different grid."); + video.attention_mask = Ltx2BuildAttentionMask(mask_before, video.noisy_tokens, ref_tokens, + before_reference, cross_mask); + } + } + // The noiser draws VIDEO first, AUDIO second, from one generator // (blocks.py:554-563 builds the video state before the audio one; :576-580, // which this used to cite, is the TEARDOWN and proves nothing about order). @@ -5006,6 +5216,47 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "trained term and look exactly like a working render."); vin.keyframes_mask = video.keyframes_mask.data(); } + + // ── THE PRODUCTION ASSIGNMENT OF `attention_mask` (#3020) ──────────── + // + // THIS LINE IS WHAT THIS ROW EXISTS FOR. `Ltx2ModalityInput::attention_mask`, + // `Ltx2PrepareSelfAttentionMask`, `self_bias` on host and device, and the + // additive bias inside `vt::AttentionCross` were all built and all + // correct, and until this row NOTHING IN `src/` ASSIGNED EITHER FIELD: + // the only assignments in the tree were four lines of a `BuildModalities` + // helper in `test_ltx2.cpp` and `test_ltx2_device.cpp`. That is the shape + // .agents/reachability.md names — a unit test that constructs the type + // proves the class works, never that anything reaches it. Deleting this + // assignment is the mutation the reachability case must red on. + // + // THE DENSE FORM, `rows == tokens`. `build_attention_mask` returns + // (B, N+M, N+M) unconditionally (mask_utils.py:220) and its block + // structure is not expressible as one row. + // + // SIZE CHECKED BEFORE `data()` IS TAKEN, exactly as the keyframes mask + // above is and for the identical reason: an empty vector's `data()` is a + // null pointer, which the forward reads as upstream's legal "no mask". A + // mask built and then dropped renders a finite, correctly shaped, plausible + // clip with a trained term silently omitted. + if (!video.attention_mask.empty()) { + VT_CHECK(static_cast(video.attention_mask.size()) == + video.tokens * video.tokens, + "ltx2 video: a self-attention strength mask must be dense [tokens, tokens] " + "(mask_utils.py:220). One that is short of the sequence is NOT a shape error " + "downstream — it is read as the key-only broadcast form and masks the wrong " + "axis."); + vin.attention_mask = video.attention_mask.data(); + vin.attention_mask_rows = video.tokens; + // Observed off the buffer HANDED OVER, not off the one built. `min` and + // `max` both, because an all-ones mask is the identity: it renders + // correctly, it has the right shape, and it is exactly what a downsample + // that lost its values produces. + const auto range = + std::minmax_element(video.attention_mask.begin(), video.attention_mask.end()); + im.trace.ic_lora_attention_mask_rows = vin.attention_mask_rows; + im.trace.ic_lora_attention_mask_min = static_cast(*range.first); + im.trace.ic_lora_attention_mask_max = static_cast(*range.second); + } // AND THE HANDOVER IS CHECKED SEPARATELY FROM THE CONSTRUCTION, because the // check above cannot see the handover. It reads `video.keyframes_mask` — the // VECTOR — so it fires when the mask is built conditionally and stays silent diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f14de29231..daf464fba1 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -461,6 +461,7 @@ target_compile_definitions(test_ltx2_pipeline PRIVATE # CLI-stage validations. Its own target rather than cases inside # test_ltx2_pipeline, which several concurrent rows of this campaign edit. The # REACHABILITY proof for the same row lives in test_ltx2_video. +vllm_cpp_add_test(test_ltx2_iclora_reference vllm/models/test_ltx2_iclora_reference.cpp) vllm_cpp_add_test(test_ltx2_retake vllm/models/test_ltx2_retake.cpp) vllm_cpp_add_test(test_ltx2_lora vllm/models/test_ltx2_lora.cpp) vllm_cpp_add_test(test_ltx2_loader vllm/models/test_ltx2_loader.cpp) diff --git a/tests/vllm/models/ltx2_iclora_reference_goldens.inc b/tests/vllm/models/ltx2_iclora_reference_goldens.inc new file mode 100644 index 0000000000..f014288ec1 --- /dev/null +++ b/tests/vllm/models/ltx2_iclora_reference_goldens.inc @@ -0,0 +1,155 @@ +// GENERATED by scripts/gen-ltx2-iclora-reference-goldens.py — DO NOT EDIT BY HAND. +// +// LTX-2.5 IC-LoRA REFERENCE-CLIP and CONDITIONING-ATTENTION-MASK goldens +// (row LTX25-IC-LORA-REF-VIDEO, issue #3020), produced by IMPORTING AND +// EXECUTING upstream's `ltx_core.conditioning.mask_utils`, its +// `_prepare_self_attention_mask`, and the three AST-sliced functions of +// `ltx_pipelines.iclora_utils` (which cannot be imported: it reaches PyAV). +// +// Every case carries the REJECTED hypothesis beside upstream's answer, and +// the generator REFUSES to write a case whose separation is zero. See the +// separation table at the bottom of this file. +// +// Upstream revision (Lightricks/LTX-2): fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca +#pragma once + +#include + +namespace vllm_test { + +// The suite asserts this equals the SHA it pins, so regenerating against a +// DIFFERENT checkout fails the gate instead of silently replacing the oracle. +inline constexpr const char* kLtx2IcLoraRefUpstreamRevision = "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +// --- 1. temporal_subsample (iclora_utils.py:87-90) ----------------------- +// +// `[0, *range(1, F, N)]`. INDEX 1 IS ALWAYS KEPT when it exists, so at N=2 +// over 5 frames the kept set is {0, 1, 3} and NOT the plausible {0, 2, 4}. +// That is the whole content of this function and the rejected hypothesis below +// is exactly the plausible reading. +inline constexpr int kLtx2TemporalSubsampleCases = 5; +inline constexpr int64_t kLtx2TemporalSubsampleFrames[] = {5, 9, 7, 6, 1}; +inline constexpr int64_t kLtx2TemporalSubsampleFactor[] = {2, 4, 3, 1, 4}; +inline constexpr int64_t kLtx2TemporalSubsampleCount[] = {3, 3, 3, 6, 1}; +inline constexpr int64_t kLtx2TemporalSubsampleKept[] = {0, 1, 3, 0, 1, 5, 0, 1, 4, 0, 1, 2, 3, 4, 5, 0}; + +// --- 2. append_ic_lora_reference_video_conditionings (iclora_utils.py:93-170) +// +// UPSTREAM'S OWN CONTROL FLOW, executed with the frame decode, the pixel +// preprocess and the VAE replaced by recording stubs. What is gated here is +// which geometry it asks the preprocess for, whether it subsamples, which of +// the three mask branches it takes, and whether it wraps. +inline constexpr int64_t kLtx2RefGeomHeight = 112; +inline constexpr int64_t kLtx2RefGeomWidth = 192; +// The item carries the factors, and the request's strength, unchanged. +inline constexpr int64_t kLtx2RefItemDownscale = 4; +inline constexpr float kLtx2RefItemStrength = 0.75f; +inline constexpr bool kLtx2RefGeomWrapped = false; +// The subsample guard (`if reference_temporal_scale_factor > 1`, :143). The +// encoder sees 3 of 5 frames at N=2 and all 5 at N=1 — the encoded SHAPE is +// what records it, so a port that subsampled unconditionally is separated. +inline constexpr int64_t kLtx2RefEncodedFramesN2 = 3; +inline constexpr int64_t kLtx2RefEncodedFramesN1 = 5; +// The three-way mask selection (:151-160). With NO mask and strength < 1 the +// scalar arm wraps; with NO mask and strength == 1 nothing wraps. The CLI +// cannot reach the first (a strength is only ever set alongside a mask, +// ic_lora.py:452-455), and this engine refuses it by name — the golden is +// here so the refusal is pinned to a MEASURED branch rather than a reading. +inline constexpr bool kLtx2RefWrapScalarBelowOne = true; +inline constexpr bool kLtx2RefWrapScalarAtOne = false; +// The divisibility refusal (:112-115), verbatim from the raised ValueError. +inline constexpr const char* kLtx2RefDivisibilityRefusal = "Output dimensions (448x770) must be divisible by reference_downscale_factor (4)"; +inline constexpr int64_t kLtx2RefScaleOneHeight = 449; + +// --- 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ---------- +// +// AREA spatial interpolation, then the CAUSAL carve-out: latent frame 0 is +// pixel frame 0 ALONE, and the remaining f_pix-1 frames mean-pool in groups +// of t = (f_pix-1)/(f_lat-1). Both rejected hypotheses below are measured. +inline constexpr int64_t kLtx2MaskPixFrames = 9; +inline constexpr int64_t kLtx2MaskPixHeight = 8; +inline constexpr int64_t kLtx2MaskPixWidth = 8; +inline constexpr int64_t kLtx2MaskLatFrames = 3; +inline constexpr int64_t kLtx2MaskLatHeight = 2; +inline constexpr int64_t kLtx2MaskLatWidth = 2; +// The INPUT, so both sides read identical bytes rather than agreeing about a +// random stream. Every value is exact in f32 (24 significant bits). +inline constexpr float kLtx2MaskPixels[] = {0.224978805f, 0.738994956f, 0.247904778f, 0.856395721f, 0.161342263f, 0.304656029f, 0.226417065f, 0.629713774f, 0.621277452f, 0.243836701f, 0.929327548f, 0.254041076f, 0.541148961f, 0.463537216f, 0.940397978f, 0.709314048f, 0.693672299f, 0.542568564f, 0.269681811f, 0.96947515f, 0.621686339f, 0.700837612f, 0.620844126f, 0.845988333f, 0.941008925f, 0.100188732f, 0.835158408f, 0.0902274251f, 0.157826722f, 0.0348206162f, 0.0689323545f, 0.566456556f, 0.455129385f, 0.770264328f, 0.082477808f, 0.414557874f, 0.892665267f, 0.825091422f, 0.233238816f, 0.891862512f, 0.755941391f, 0.525208294f, 0.812666357f, 0.408799589f, 0.933392882f, 0.328110039f, 0.123371542f, 0.54055661f, 0.112813234f, 0.00529891253f, 0.10176909f, 0.0247758627f, 0.0249340534f, 0.303246439f, 0.804710567f, 0.538730443f, 0.394230783f, 0.116368294f, 0.654279828f, 0.797493339f, 0.992072105f, 0.222010791f, 0.323121369f, 0.684363902f, 0.456148565f, 0.329604149f, 0.150325954f, 0.211824954f, 0.857371747f, 0.218813181f, 0.627006412f, 0.716362476f, 0.411796451f, 0.733595669f, 0.942894995f, 0.146117151f, 0.794231236f, 0.246105671f, 0.453785658f, 0.402912438f, 0.291079164f, 0.745548785f, 0.166508615f, 0.189101517f, 0.801723778f, 0.589329779f, 0.502525091f, 0.237636983f, 0.757415771f, 0.040210247f, 0.483881652f, 0.6307953f, 0.0602877736f, 0.667216182f, 0.588570833f, 0.560185254f, 0.975480795f, 0.113663495f, 0.428635895f, 0.132393956f, 0.0268669724f, 0.453999043f, 0.458613455f, 0.694886446f, 0.754171073f, 0.454550207f, 0.18075633f, 0.112146139f, 0.675278902f, 0.268378139f, 0.861036241f, 0.692431509f, 0.287955284f, 0.283611059f, 0.437588453f, 0.275979996f, 0.627004325f, 0.630954862f, 0.924760938f, 0.0718417764f, 0.355810046f, 0.67786628f, 0.880366683f, 0.30253154f, 0.786205053f, 0.745979965f, 0.576954007f, 0.0308310986f, 0.130777419f, 0.597290516f, 0.5143255f, 0.746656358f, 0.0455746055f, 0.672296584f, 0.713463485f, 0.191987753f, 0.540562212f, 0.21183306f, 0.912392437f, 0.867345572f, 0.215253115f, 0.353586614f, 0.892430186f, 0.885174572f, 0.390232682f, 0.157626808f, 0.0664351583f, 0.47427243f, 0.0192090273f, 0.0610715151f, 0.856101871f, 0.202588677f, 0.844750762f, 0.437463284f, 0.615182161f, 0.535516262f, 0.65188539f, 0.386852443f, 0.440248549f, 0.140840232f, 0.751649737f, 0.69611603f, 0.116815269f, 0.634107947f, 0.0645769238f, 0.242201626f, 0.191012144f, 0.362300277f, 0.510971785f, 0.577358842f, 0.345588863f, 0.389454961f, 0.720236659f, 0.852962494f, 0.375399411f, 0.593073785f, 0.196979284f, 0.273119688f, 0.133840621f, 0.159698129f, 0.635126412f, 0.242724895f, 0.678991795f, 0.199027658f, 0.196128547f, 0.00642746687f, 0.677461445f, 0.536804438f, 0.499773204f, 0.718390286f, 0.51330483f, 0.837298334f, 0.770378292f, 0.492825508f, 0.0426640511f, 0.91784507f, 0.583577335f, 0.577757061f, 0.546435297f, 0.162060916f, 0.0517340899f, 0.797736764f, 0.0987977386f, 0.0808457732f, 0.371301949f, 0.204327643f, 0.461616695f, 0.74985528f, 0.604997396f, 0.739948332f, 0.761415303f, 0.489027202f, 0.524555445f, 0.535394847f, 0.745264113f, 0.327727675f, 0.494373322f, 0.106823564f, 0.18626678f, 0.946658432f, 0.758277535f, 0.121624708f, 0.0415190458f, 0.347359002f, 0.276930034f, 0.394188046f, 0.581585288f, 0.380344629f, 0.893188417f, 0.580474913f, 0.309349656f, 0.178186297f, 0.242652476f, 0.305642009f, 0.0238131285f, 0.159330785f, 0.150969028f, 0.460921586f, 0.194989204f, 0.870569885f, 0.0330894589f, 0.0111231804f, 0.683476746f, 0.326002181f, 0.245743096f, 0.064491272f, 0.644147575f, 0.000202894211f, 0.275570333f, 0.963738084f, 0.366046369f, 0.618974566f, 0.170288444f, 0.774100304f, 0.924566567f, 0.909448624f, 0.0803639889f, 0.306199133f, 0.606006145f, 0.985461771f, 0.841015458f, 0.416215777f, 0.697441697f, 0.574185193f, 0.298144281f, 0.239260674f, 0.80518645f, 0.351780355f, 0.119580925f, 0.0518564582f, 0.897045553f, 0.349761903f, 0.583748102f, 0.382675707f, 0.0559566021f, 0.672386229f, 0.290330529f, 0.860345006f, 0.593835175f, 0.0300983191f, 0.296267509f, 0.646880746f, 0.519856811f, 0.543841064f, 0.994504869f, 0.0849536061f, 0.755463958f, 0.993644476f, 0.209379613f, 0.594450593f, 0.484336495f, 0.170750082f, 0.993458867f, 0.925972521f, 0.266003489f, 0.660282254f, 0.586643994f, 0.778126061f, 0.0565010309f, 0.584053874f, 0.627667725f, 0.223118246f, 0.235853374f, 0.988708973f, 0.24592489f, 0.892781913f, 0.117504597f, 0.5365839f, 0.405869424f, 0.673124969f, 0.702657282f, 0.360488355f, 0.245491028f, 0.159264743f, 0.858900845f, 0.061032176f, 0.519282579f, 0.363428295f, 0.373078167f, 0.94595331f, 0.175679445f, 0.718593955f, 0.234594822f, 0.457112908f, 0.816635609f, 0.130899251f, 0.503757238f, 0.105866075f, 0.0278913975f, 0.882681727f, 0.777195275f, 0.0241299868f, 0.836323261f, 0.00590604544f, 0.733849347f, 0.87148279f, 0.475699008f, 0.363211811f, 0.407663643f, 0.787238002f, 0.604611456f, 0.468820512f, 0.969316483f, 0.995938003f, 0.272820413f, 0.461982131f, 0.0572282672f, 0.08695364f, 0.765583396f, 0.0331178904f, 0.444453776f, 0.745676398f, 0.207386196f, 0.741251171f, 0.979560256f, 0.875282645f, 0.425274253f, 0.100177765f, 0.314046502f, 0.686280549f, 0.112682164f, 0.694772184f, 0.761486769f, 0.307273686f, 0.0175738931f, 0.558968961f, 0.567652822f, 0.832921207f, 0.0997376442f, 0.451044917f, 0.212615252f, 0.195685387f, 0.499477386f, 0.347878397f, 0.577536881f, 0.955278218f, 0.912000775f, 0.667354465f, 0.906242669f, 0.249278307f, 0.404088557f, 0.709416687f, 0.577703059f, 0.616618574f, 0.774920166f, 0.751746535f, 0.117762864f, 0.513761044f, 0.878428102f, 0.551457703f, 0.762865603f, 0.663778841f, 0.6750741f, 0.482571781f, 0.311710536f, 0.0577930808f, 0.942345262f, 0.430845559f, 0.0900357366f, 0.818745911f, 0.591225147f, 0.448996902f, 0.0356302261f, 0.346290588f, 0.980599225f, 0.972895503f, 0.84816581f, 0.930217624f, 0.0287508368f, 0.0362402797f, 0.305136859f, 0.500912428f, 0.23494029f, 0.603680074f, 0.938094497f, 0.625056803f, 0.509535789f, 0.342275202f, 0.367460787f, 0.504711986f, 0.143217921f, 0.871813238f, 0.149221659f, 0.697380841f, 0.325163662f, 0.905375302f, 0.90978688f, 0.327758372f, 0.735852122f, 0.908246279f, 0.616322279f, 0.447793245f, 0.90318352f, 0.800132334f, 0.797131181f, 0.296567619f, 0.493443847f, 0.485734046f, 0.217474163f, 0.709294379f, 0.708687544f, 0.669715464f, 0.706661522f, 0.699035525f, 0.189122796f, 0.449640393f, 0.409109354f, 0.0466797948f, 0.480667889f, 0.237748444f, 0.886014342f, 0.204600096f, 0.818930268f, 0.173705816f, 0.622837245f, 0.103790164f, 0.904882073f, 0.743250608f, 0.227131367f, 0.0243018866f, 0.929496706f, 0.99599576f, 0.850975215f, 0.381912231f, 0.661735356f, 0.720816493f, 0.370992064f, 0.0936429501f, 0.870625913f, 0.399675965f, 0.0414492488f, 0.78073287f, 0.977315605f, 0.481281281f, 0.398207009f, 0.628759801f, 0.882363558f, 0.47795701f, 0.324389875f, 0.600753963f, 0.17456162f, 0.477541447f, 0.849557698f, 0.502795577f, 0.653657317f, 0.92385453f, 0.246045411f, 0.648090661f, 0.0513747931f, 0.209317505f, 0.427504122f, 0.23591435f, 0.699629843f, 0.970116138f, 0.287622392f, 0.726968825f, 0.343225002f, 0.958356082f, 0.962094307f, 0.71631515f, 0.504042327f, 0.0353083611f, 0.287226319f, 0.0955541134f, 0.689278305f, 0.609330833f, 0.26156348f, 0.909175575f, 0.604726076f, 0.910757482f, 0.881617069f, 0.454441786f, 0.623414814f, 0.0372434258f, 0.515186727f, 0.85244137f, 0.209577382f, 0.365545928f, 0.0413528085f, 0.133180678f, 0.821608067f, 0.181242228f, 0.0989111066f, 0.0549529195f, 0.0250368714f, 0.981382191f, 0.828009486f, 0.176285863f, 0.404071748f, 0.730219722f, 0.72375077f, 0.337819993f, 0.583175123f, 0.00325196981f, 0.103610337f, 0.137881756f, 0.454132259f, 0.638733625f, 0.973367333f, 0.390789509f, 0.460909188f, 0.0850528479f, 0.142819703f, 0.831980586f, 0.827947915f, 0.23010689f, 0.396763444f, 0.205432773f, 0.691007018f, 0.663131058f, 0.907645226f, 0.234843791f, 0.946535945f, 0.0815069079f, 0.297047138f, 0.54718852f, 0.599617898f, 0.672928989f, 0.310182512f, 0.945918143f, 0.260028422f, 0.626050293f, 0.722653747f, 0.291521311f, 0.0546925068f, 0.370481193f, 0.828670382f, 0.181400359f, 0.856415391f, 0.311807752f, 0.211450398f, 0.231483996f, 0.175142884f, 0.101020813f, 0.671769917f, 0.967861652f}; +inline constexpr float kLtx2MaskLatentWeights[] = {0.534921169f, 0.474619985f, 0.402004659f, 0.541342437f, 0.463841647f, 0.479086936f, 0.391879141f, 0.513890743f, 0.438569248f, 0.541703641f, 0.524510682f, 0.516675234f}; +// The two rejected answers, emitted so the suite asserts it is NOT them. +inline constexpr float kLtx2MaskLatentBilinear[] = {0.496353656f, 0.681404233f, 0.361235648f, 0.389859647f, 0.488613337f, 0.519038856f, 0.347211421f, 0.502157032f, 0.469123542f, 0.545261562f, 0.613554657f, 0.502646863f}; +inline constexpr float kLtx2MaskLatentUniformPool[] = {0.485171884f, 0.471803099f, 0.401835531f, 0.519039631f, 0.440979809f, 0.513357699f, 0.429122686f, 0.517239153f, 0.45536992f, 0.534099936f, 0.524896443f, 0.518256605f}; +// A NON-DIVIDING spatial pair, 9 -> 2. `area` pools output index i over +// `[floor(i*I/O), ceil((i+1)*I/O))`; the integer-stride window +// `[i*(I//O), (i+1)*(I//O))` agrees with it on every divisible shape and +// only there, so the 8 -> 2 case above cannot see the difference and this +// one can. +inline constexpr int64_t kLtx2MaskNdPixFrames = 9; +inline constexpr int64_t kLtx2MaskNdPixHeight = 9; +inline constexpr int64_t kLtx2MaskNdPixWidth = 9; +inline constexpr int64_t kLtx2MaskNdLatFrames = 3; +inline constexpr int64_t kLtx2MaskNdLatHeight = 2; +inline constexpr int64_t kLtx2MaskNdLatWidth = 2; +inline constexpr float kLtx2MaskNdPixels[] = {0.176431715f, 0.414751172f, 0.428075731f, 0.396973491f, 0.993891835f, 0.363071442f, 0.108931065f, 0.0180123448f, 0.528322339f, 0.528225362f, 0.898173034f, 0.225970566f, 0.284193099f, 0.122897267f, 0.871926129f, 0.905635357f, 0.975744545f, 0.339984655f, 0.347717524f, 0.107571602f, 0.216426015f, 0.891348183f, 0.687187612f, 0.361214876f, 0.122908354f, 0.398467839f, 0.859053314f, 0.552588582f, 0.0223608613f, 0.381933153f, 0.133411825f, 0.611968935f, 0.25096339f, 0.615133226f, 0.93135798f, 0.185379148f, 0.997653782f, 0.891151607f, 0.205703974f, 0.275035024f, 0.63247329f, 0.52415204f, 0.965521038f, 0.271311641f, 0.941007137f, 0.689656556f, 0.563211441f, 0.112410963f, 0.663916469f, 0.515940726f, 0.884365559f, 0.129287601f, 0.287406147f, 0.00508952141f, 0.829226613f, 0.913085282f, 0.0933603644f, 0.363948286f, 0.169794202f, 0.449740171f, 0.683066249f, 0.0454002619f, 0.887509406f, 0.352954507f, 0.0286763906f, 0.570546269f, 0.496683061f, 0.361666799f, 0.399156332f, 0.796284735f, 0.498372436f, 0.445922732f, 0.257750392f, 0.804583311f, 0.274433672f, 0.329919934f, 0.176333547f, 0.564143956f, 0.902864873f, 0.747000337f, 0.413248956f, 0.294984937f, 0.184212923f, 0.165899992f, 0.553884089f, 0.492762089f, 0.424729645f, 0.253381908f, 0.252146065f, 0.951010942f, 0.611447692f, 0.66240108f, 0.261108398f, 0.755251467f, 0.329187453f, 0.187325478f, 0.877708912f, 0.95155549f, 0.633138597f, 0.0419675708f, 0.232635796f, 0.66833657f, 0.273359537f, 0.412642837f, 0.525232077f, 0.810980916f, 0.261211991f, 0.947197556f, 0.9000718f, 0.368062198f, 0.0773826838f, 0.131558359f, 0.608510911f, 0.976616681f, 0.876119256f, 0.42066896f, 0.388753116f, 0.481222034f, 0.83724457f, 0.638977587f, 0.255988657f, 0.429601073f, 0.0447826385f, 0.397245526f, 0.654212952f, 0.982474685f, 0.0029489398f, 0.713418663f, 0.193713427f, 0.848715007f, 0.914697945f, 0.518172383f, 0.890321732f, 0.820030034f, 0.499699891f, 0.990263462f, 0.363884985f, 0.850643575f, 0.525403023f, 0.910626292f, 0.622525096f, 0.338403821f, 0.752064824f, 0.510951936f, 0.619227588f, 0.770158052f, 0.774864852f, 0.365872204f, 0.70408833f, 0.310323238f, 0.902955055f, 0.467824221f, 0.067301631f, 0.901615739f, 0.397632003f, 0.145964682f, 0.340275347f, 0.0559407473f, 0.55675292f, 0.983822882f, 0.787080526f, 0.174605727f, 0.65383178f, 0.442226827f, 0.453761578f, 0.097636044f, 0.671319366f, 0.145175278f, 0.901205122f, 0.937905014f, 0.620723784f, 0.23314029f, 0.0584473014f, 0.311875403f, 0.0517009497f, 0.0715442896f, 0.401783466f, 0.312211573f, 0.982579291f, 0.45889771f, 0.106410503f, 0.13699156f, 0.568376482f, 0.621710718f, 0.788453102f, 0.565417409f, 0.843536675f, 0.0152804852f, 0.161386311f, 0.499435306f, 0.133455098f, 0.287302732f, 0.649116874f, 0.747392416f, 0.0525169373f, 0.149493515f, 0.485217988f, 0.299057841f, 0.0585591197f, 0.800601184f, 0.705874681f, 0.760249972f, 0.255273879f, 0.450765669f, 0.854833901f, 0.269396067f, 0.204905152f, 0.123510599f, 0.994540036f, 0.251888454f, 0.313174903f, 0.722878397f, 0.125090957f, 0.300181627f, 0.319087744f, 0.470823646f, 0.0245870948f, 0.819541156f, 0.0128717422f, 0.167695045f, 0.656143904f, 0.432101429f, 0.402993858f, 0.739083767f, 0.00925147533f, 0.951627731f, 0.218733788f, 0.471463203f, 0.0320233107f, 0.900513947f, 0.335975945f, 0.325684547f, 0.724274635f, 0.813946486f, 0.00837951899f, 0.587917566f, 0.593036115f, 0.7367419f, 0.509701729f, 0.119342268f, 0.764099121f, 0.786913455f, 0.839243591f, 0.647685528f, 0.746626496f, 0.293163419f, 0.343819261f, 0.563849211f, 0.805668294f, 0.66591531f, 0.769356966f, 0.803707838f, 0.58803916f, 0.54597795f, 0.168208063f, 0.0741783381f, 0.229800701f, 0.494519413f, 0.362460494f, 0.637498617f, 0.161039829f, 0.489256442f, 0.092312634f, 0.823154628f, 0.832165182f, 0.247525454f, 0.294949114f, 0.748880327f, 0.451600671f, 0.40436697f, 0.51445049f, 0.640046299f, 0.356705248f, 0.432429552f, 0.825973094f, 0.623867393f, 0.922766864f, 0.0799279213f, 0.626308441f, 0.622054577f, 0.271120965f, 0.202672243f, 0.575166166f, 0.402850449f, 0.7335006f, 0.663229048f, 0.488371372f, 0.336939275f, 0.128312945f, 0.282105863f, 0.52510792f, 0.685502052f, 0.39796555f, 0.382492661f, 0.918198705f, 0.0393769741f, 0.746138871f, 0.476073146f, 0.446476877f, 0.99091053f, 0.178707838f, 0.565140069f, 0.348153234f, 0.187597871f, 0.109171689f, 0.623592556f, 0.778553843f, 0.00752681494f, 0.386353374f, 0.282530725f, 0.290333092f, 0.840117931f, 0.924435139f, 0.812919915f, 0.845062375f, 0.445622146f, 0.11066848f, 0.485649109f, 0.288371027f, 0.0302713513f, 0.903456211f, 0.318011343f, 0.108384132f, 0.861936212f, 0.503161728f, 0.52669543f, 0.857944191f, 0.945909142f, 0.792777896f, 0.672234178f, 0.133958578f, 0.690484643f, 0.557196558f, 0.177605987f, 0.550531685f, 0.856805444f, 0.118242562f, 0.704421759f, 0.229511619f, 0.0119316578f, 0.482146025f, 0.530855834f, 0.450839043f, 0.498370945f, 0.502073348f, 0.610073924f, 0.563346505f, 0.213598371f, 0.671187341f, 0.324515104f, 0.969206989f, 0.11610949f, 0.387870967f, 0.197868168f, 0.433820367f, 0.241572917f, 0.16652894f, 0.778164625f, 0.672078907f, 0.870177925f, 0.185845196f, 0.26016748f, 0.259678304f, 0.00730335712f, 0.817301929f, 0.308321118f, 0.142065346f, 0.602425873f, 0.0963254571f, 0.44181931f, 0.564264774f, 0.785137177f, 0.577842534f, 0.494561434f, 0.109093845f, 0.272757351f, 0.129696846f, 0.704673588f, 0.899068415f, 0.5506953f, 0.677172124f, 0.385102749f, 0.915118158f, 0.685579658f, 0.684499741f, 0.917760491f, 0.0294851065f, 0.160535574f, 0.138951302f, 0.747179031f, 0.453814983f, 0.993482947f, 0.493778586f, 0.368394077f, 0.759645939f, 0.270795047f, 0.281280041f, 0.440939605f, 0.377233982f, 0.627582848f, 0.0953028202f, 0.770352066f, 0.311038017f, 0.652339518f, 0.953617692f, 0.938524067f, 0.248702705f, 0.849620342f, 0.473485649f, 0.178462207f, 0.481127799f, 0.359116018f, 0.996653497f, 0.460334182f, 0.766413271f, 0.947203696f, 0.38217622f, 0.472835422f, 0.548464417f, 0.994021535f, 0.505486369f, 0.112265825f, 0.649771154f, 0.484351277f, 0.433071434f, 0.178781092f, 0.34736073f, 0.926333845f, 0.447269499f, 0.0620956421f, 0.322753251f, 0.104726017f, 0.402469754f, 0.111355186f, 0.858210564f, 0.102687776f, 0.0320823789f, 0.36260885f, 0.613993347f, 0.553087652f, 0.764120519f, 0.876651525f, 0.805870414f, 0.784868956f, 0.377298594f, 0.250272334f, 0.339644015f, 0.379174411f, 0.547709107f, 0.859425783f, 0.365451097f, 0.28319329f, 0.437577188f, 0.417913139f, 0.763182938f, 0.794263363f, 0.214469671f, 0.845055819f, 0.878148019f, 0.968808532f, 0.645396233f, 0.433998883f, 0.924810052f, 0.197890162f, 0.551473022f, 0.153582096f, 0.579355478f, 0.0588880777f, 0.398751438f, 0.397648275f, 0.343063891f, 0.220753551f, 0.836437583f, 0.78373307f, 0.876475275f, 0.251742601f, 0.131393731f, 0.00924360752f, 0.0378209352f, 0.920103848f, 0.239557207f, 0.785963535f, 0.458949089f, 0.917492151f, 0.82664746f, 0.824898958f, 0.748798013f, 0.0437062383f, 0.275460839f, 0.802760303f, 0.562115431f, 0.253444493f, 0.721078157f, 0.270546675f, 0.289948046f, 0.264674067f, 0.988374114f, 0.921821356f, 0.639254689f, 0.357419848f, 0.0164500475f, 0.987020433f, 0.134222806f, 0.470599413f, 0.214927495f, 0.804601073f, 0.00276684761f, 0.108661234f, 0.1645329f, 0.930413604f, 0.156331539f, 0.826165974f, 0.992276132f, 0.788320959f, 0.839853644f, 0.86417073f, 0.00915694237f, 0.359093964f, 0.153696716f, 0.950254858f, 0.110191941f, 0.0497874618f, 0.756811023f, 0.407120287f, 0.833793044f, 0.768642783f, 0.129186273f, 0.603067219f, 0.391770661f, 0.459852576f, 0.544464171f, 0.412319064f, 0.261648715f, 0.0876111388f, 0.913850784f, 0.621434808f, 0.907304585f, 0.0707091093f, 0.96202302f, 0.148205638f, 0.329700232f, 0.164689302f, 0.999581754f, 0.549011528f, 0.836014986f, 0.1472615f, 0.821966588f, 0.080082655f, 0.472028852f, 0.149907827f, 0.470775306f, 0.178379297f, 0.252197027f, 0.486711025f, 0.138778448f, 0.355213642f, 0.0720499754f, 0.524939477f, 0.06805861f, 0.678370833f, 0.811598957f, 0.386808276f, 0.546884239f, 0.939647734f, 0.124761701f, 0.983676314f, 0.53366524f, 0.407992423f, 0.748240411f, 0.186389923f, 0.456374049f, 0.264949262f, 0.101999223f, 0.840950489f, 0.935143232f, 0.737413049f, 0.572279096f, 0.453462601f, 0.424460053f, 0.0912488699f, 0.661141574f, 0.352765918f, 0.14255327f, 0.0666944981f, 0.512379944f, 0.040235579f, 0.0467900038f, 0.563910425f, 0.0949526429f, 0.824949086f, 0.786893725f, 0.480244696f, 0.512091756f, 0.253281355f, 0.371683776f, 0.750189245f, 0.236807227f, 0.250991404f, 0.148420811f, 0.760418773f, 0.706954002f, 0.483227968f, 0.860947609f, 0.958144605f, 0.204591811f, 0.346984088f, 0.330379963f, 0.260562778f, 0.454497993f, 0.175507128f, 0.874245822f, 0.655669034f, 0.133049369f, 0.681265354f, 0.0751605034f, 0.381592751f, 0.65002501f, 0.332112014f, 0.664795637f, 0.0514030457f, 0.520705462f, 0.428389311f, 0.695400119f, 0.903649747f, 0.591303825f, 0.103616059f, 0.83450526f, 0.955841482f, 0.135760486f, 0.620482624f, 0.523535073f, 0.502091646f, 0.189656317f, 0.936189651f, 0.590592086f, 0.583584547f, 0.610268474f, 0.443874002f, 0.130639553f, 0.61188519f, 0.877523184f, 0.420557976f, 0.567815006f, 0.255108118f, 0.364960551f, 0.185195923f, 0.476062536f, 0.600627542f, 0.558772326f, 0.957621634f, 0.3928231f, 0.455191553f, 0.00377166271f, 0.62539798f, 0.225390792f, 0.177161932f, 0.169970393f, 0.661524951f, 0.412112474f, 0.132826149f, 0.0998225212f, 0.661162794f, 0.0239869952f, 0.944482744f, 0.544507325f, 0.9350034f, 0.575270712f, 0.519179463f, 0.451092482f, 0.25096643f, 0.54721272f, 0.374691486f, 0.774538517f, 0.722382426f, 0.564935863f, 0.707167804f, 0.114523411f, 0.201274157f, 0.161640704f, 0.597701728f, 0.378583431f, 0.567182422f, 0.617218018f, 0.420417905f, 0.0773469806f, 0.00652354956f, 0.650327325f, 0.171233237f, 0.856792927f, 0.207099199f, 0.348393142f, 0.928106427f, 0.729782224f, 0.825324774f, 0.0101621747f, 0.731117189f, 0.597788632f, 0.566959321f, 0.672528088f, 0.413965285f, 0.682504356f, 0.509997785f, 0.150990427f, 0.0506671667f, 0.739574969f, 0.57869482f, 0.853643f, 0.209993958f, 0.966246784f, 0.978174984f, 0.852503657f, 0.941322565f, 0.538044572f, 0.509447157f, 0.230799377f, 0.555043757f, 0.62337774f, 0.479802608f, 0.826905191f, 0.681678236f, 0.559722781f, 0.143384337f, 0.509182394f, 0.288674951f, 0.293766856f, 0.00271993876f, 0.570420086f, 0.619341314f, 0.831060112f}; +inline constexpr float kLtx2MaskNdLatentWeights[] = {0.456964582f, 0.543460608f, 0.462804705f, 0.507882357f, 0.439289629f, 0.513226748f, 0.477039456f, 0.520850718f, 0.453535885f, 0.485347092f, 0.484792441f, 0.527994156f}; +// The rejected answer, emitted so the suite asserts it is NOT this one. +inline constexpr float kLtx2MaskNdLatentStrideBox[] = {0.37538448f, 0.521206975f, 0.502951264f, 0.475871205f, 0.418908179f, 0.512575507f, 0.466769427f, 0.535788894f, 0.447849631f, 0.473493338f, 0.482950687f, 0.520915866f}; + +// --- 4. build_attention_mask (mask_utils.py:170-243) --------------------- +// +// THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN ON PURPOSE. With +// num_existing == num_noisy the true block structure and the plausible wrong +// reading `cross on ALL existing rows` are ELEMENTWISE EQUAL — measured 0 +// separating elements — and the golden would be a mute switch. The generator +// refuses that case; this one separates on 4 elements. +inline constexpr int64_t kLtx2AmNoisy = 2; +inline constexpr int64_t kLtx2AmNew = 2; +inline constexpr int64_t kLtx2AmExisting = 3; +inline constexpr float kLtx2AmCross[] = {0.25f, 0.75f}; +inline constexpr float kLtx2AmMask[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f}; +inline constexpr float kLtx2AmMaskCrossOnAllRows[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.25f, 0.25f, 0.25f, 1.0f, 1.0f, 0.75f, 0.75f, 0.75f, 1.0f, 1.0f}; +// A SECOND item on top of the first: `existing_mask` is preserved in the +// top-left block rather than refilled with ones (`:223-226`), which is what +// keeps the first item's attenuation alive after a second one is applied. +inline constexpr float kLtx2AmCross2[] = {0.5f}; +inline constexpr float kLtx2AmMask2[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.5f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.5f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f, 0.0f, 0.5f, 0.5f, 0.0f, 0.0f, 0.0f, 1.0f}; +inline constexpr float kLtx2AmMask2Refilled[] = {1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.5f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.5f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.5f, 0.5f, 0.0f, 0.0f, 0.0f, 1.0f}; +// resolve_cross_mask's scalar arm (`:31-37`) and its 1-D broadcast (`:49-54`). +inline constexpr float kLtx2CrossScalar[] = {0.5f, 0.5f, 0.5f, 0.5f}; +inline constexpr float kLtx2CrossOneD[] = {0.100000001f, 0.200000003f, 0.300000012f, 0.400000006f}; +// update_attention_mask's PAD-WITH-ONES arm (`:141-156`): a null mask on a +// state that already carries one grows the mask with full attention rather +// than returning None and leaving it short of the sequence. +inline constexpr float kLtx2AmMaskPaddedOnes[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f}; + +// --- 5. _prepare_self_attention_mask (transformer_args.py:208-237) ------- +// +// The CONSUMPTION seam this row makes reachable. Already ported as +// Ltx2PrepareSelfAttentionMask; pinned here because nothing in src/ assigned +// Ltx2ModalityInput::attention_mask before this row, so these numbers were +// gated only through a hand-built test struct. +inline constexpr const char* kLtx2SelfMaskOwner = "TransformerArgsPreprocessor"; +inline constexpr float kLtx2SelfMaskProbe[] = {1.0f, 0.5f, 0.0f, 1e-30f}; +// f32. The third entry is finfo(f32).min, NOT -inf and NOT 0; the fourth is +// log(tiny) after the clamp, NOT log(1e-30). +inline constexpr float kLtx2SelfMaskBias[] = {0.0f, -0.693147182f, -3.40282347e+38f, -69.0775528f}; + +// --- separation table ---------------------------------------------------- +// +// Every case above, its rejected hypothesis, and the measured separation. A +// zero here is a mute switch and the generator raises rather than writing it. +// temporal_subsample vs every Nth frame from index 0 (range(0, F, N)) -> 6.0 +// reference geometry at scale 4 vs the phase's own height and width (no downscale) -> 912.0 +// the temporal-subsample guard vs subsampling unconditionally -> 2.0 +// the strength<1 wrap branch vs wrapping unconditionally -> 1.0 +// the divisibility refusal vs refusing whenever height % scale != 0, scale==1 included -> 1.0 +// downsample_mask_video_to_latent vs bilinear spatial interpolation -> 0.20678424835205078 +// downsample_mask_video_to_latent vs uniform temporal pooling with no causal first-frame carve-out -> 0.049749284982681274 +// downsample_mask_video_to_latent at a NON-DIVIDING spatial pair (9 -> 2) vs an integer-stride box filter, which `area` equals only when O divides I -> 0.08158010244369507 +// build_attention_mask vs cross on ALL existing rows, not only the noisy ones -> 4.0 +// build_attention_mask with an existing mask vs discarding the existing mask and refilling the block with ones -> 1.0 +// resolve_cross_mask's 1-D arm vs the scalar fill -> 0.4000000059604645 +// update_attention_mask's pad-with-ones arm vs returning the mask unchanged (short of the sequence) -> 11.0 +// _prepare_self_attention_mask vs -inf for a zero and an unclamped log -> 1.0 + +} // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2_iclora_reference.cpp b/tests/vllm/models/test_ltx2_iclora_reference.cpp new file mode 100644 index 0000000000..770b2449ed --- /dev/null +++ b/tests/vllm/models/test_ltx2_iclora_reference.cpp @@ -0,0 +1,449 @@ +// LTX-2.5 IC-LoRA reference conditioning — VALUE gates against the EXECUTED +// pinned module. Row LTX25-IC-LORA-REF-VIDEO (#3020), +// spec .agents/specs/ltx25-ic-lora-ref-video.md. +// +// THESE ARE NOT THE PROOF THAT ANYTHING IS REACHED. Per .agents/reachability.md +// they localize a failure: every case here stays green when the production call +// site in `ltx2_video.cpp` is deleted. The reachability cases are +// `ltx2 ic-lora: ...` in `test_ltx2_video`, which render through `Generate`. +// +// Every golden is what upstream's own code RETURNED, and every case carries the +// hypothesis it rejects. `scripts/gen-ltx2-iclora-reference-goldens.py` refuses +// to emit a case whose separation is zero, and the separation table it writes at +// the bottom of the .inc is the record of what each case can actually see. +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_conditioning.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + +#include "ltx2_iclora_reference_goldens.inc" + +namespace { + +// The pin the goldens were generated against. Asserted rather than assumed: a +// regeneration against a different checkout must fail this gate instead of +// silently replacing the oracle. +constexpr const char* kExpectedUpstream = "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +std::string Message(void (*run)()) { + try { + run(); + } catch (const std::exception& e) { + return std::string(e.what()); + } + return std::string(); +} + +} // namespace + +TEST_CASE("ltx2 ic-lora goldens: the oracle pin is the one this suite claims") { + CHECK(std::string(vllm_test::kLtx2IcLoraRefUpstreamRevision) == kExpectedUpstream); +} + +TEST_CASE("ltx2 ic-lora: temporal_subsample keeps index 1, not every Nth from 0") { + // The whole content of `temporal_subsample` (iclora_utils.py:87-90) is WHICH + // indices it keeps, and the plausible wrong reading `range(0, F, N)` returns + // the SAME COUNT — so no shape check, no token count and no rendered frame + // count can tell them apart. The reference simply describes different moments. + size_t at = 0; + for (int c = 0; c < vllm_test::kLtx2TemporalSubsampleCases; ++c) { + const int64_t frames = vllm_test::kLtx2TemporalSubsampleFrames[c]; + const int64_t factor = vllm_test::kLtx2TemporalSubsampleFactor[c]; + const int64_t count = vllm_test::kLtx2TemporalSubsampleCount[c]; + INFO("frames = " << frames << ", factor = " << factor); + const std::vector got = vllm::Ltx2TemporalSubsampleIndices(frames, factor); + REQUIRE(static_cast(got.size()) == count); + for (int64_t i = 0; i < count; ++i) { + CHECK(got[static_cast(i)] == vllm_test::kLtx2TemporalSubsampleKept[at + i]); + } + // ...and it is NOT the rejected hypothesis, asserted case by case rather + // than once over the whole set: the two agree at factor 1 and on a + // single-frame clip, and a golden that only checked the aggregate would pass + // on a build that got the two live cases wrong and the degenerate ones right. + std::vector rejected; + for (int64_t i = 0; i < frames; i += factor) rejected.push_back(i); + if (frames > 1 && factor > 1) { + CHECK_MESSAGE(got != rejected, + "at frames = " << frames << " factor = " << factor + << " the port agrees with `range(0, F, N)`, which upstream is " + "not"); + } + at += static_cast(count); + } +} + +TEST_CASE("ltx2 ic-lora: the gather follows the indices on the FRAME axis") { + // `Ltx2TemporalSubsampleIndices` says which frames; this says the copy takes + // them out of a CHANNEL-MAJOR volume. A prefix of the buffer would keep whole + // channels and drop others, and the result is the right size either way. + const int64_t channels = 3, frames = 5, plane = 2; + std::vector clip(static_cast(channels * frames * plane)); + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < frames; ++t) { + for (int64_t p = 0; p < plane; ++p) { + clip[static_cast((c * frames + t) * plane + p)] = + static_cast(c * 100 + t * 10 + p); + } + } + } + const std::vector got = vllm::Ltx2TemporalSubsample(clip, channels, frames, plane, 2); + // Kept indices are {0, 1, 3}. + REQUIRE(got.size() == static_cast(channels * 3 * plane)); + const int64_t keep[] = {0, 1, 3}; + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < 3; ++t) { + for (int64_t p = 0; p < plane; ++p) { + CHECK(got[static_cast((c * 3 + t) * plane + p)] == + doctest::Approx(static_cast(c * 100 + keep[t] * 10 + p))); + } + } + } +} + +TEST_CASE("ltx2 ic-lora: the reference geometry divides, and refuses only when scale != 1") { + // `iclora_utils.py:111-117`, executed. The refusal's guard is `scale != 1` + // AND indivisible; dropping the first half refuses nothing extra (1 divides + // everything) while dropping the second refuses every render. + const vllm::Ltx2IcLoraReferenceGeometry geom = + vllm::Ltx2ResolveIcLoraReferenceGeometry(448, 768, 4); + CHECK(geom.height == vllm_test::kLtx2RefGeomHeight); + CHECK(geom.width == vllm_test::kLtx2RefGeomWidth); + // ...and NOT the phase's own grid, which is the plausible wrong answer. + CHECK(geom.height != 448); + CHECK(geom.width != 768); + + // Scale 1 never refuses, whatever the dimensions. Measured on the oracle: + // 449 at scale 1 comes back as 449. + CHECK(vllm::Ltx2ResolveIcLoraReferenceGeometry(449, 771, 1).height == + vllm_test::kLtx2RefScaleOneHeight); + + const std::string refusal = + Message([] { (void)vllm::Ltx2ResolveIcLoraReferenceGeometry(448, 770, 4); }); + INFO(refusal); + // Upstream's own sentence, from the raised ValueError rather than from a + // reading of the f-string. + CHECK(refusal.find(vllm_test::kLtx2RefDivisibilityRefusal) != std::string::npos); +} + +TEST_CASE("ltx2 ic-lora: the mask downsample carves out the first latent frame") { + // `downsample_mask_video_to_latent` (iclora_utils.py:52-84). The INPUT comes + // out of the .inc so both sides read identical bytes rather than agreeing + // about a random stream. + const int64_t f_pix = vllm_test::kLtx2MaskPixFrames; + const int64_t h_pix = vllm_test::kLtx2MaskPixHeight; + const int64_t w_pix = vllm_test::kLtx2MaskPixWidth; + const int64_t f_lat = vllm_test::kLtx2MaskLatFrames; + const int64_t h_lat = vllm_test::kLtx2MaskLatHeight; + const int64_t w_lat = vllm_test::kLtx2MaskLatWidth; + const std::vector mask( + vllm_test::kLtx2MaskPixels, + vllm_test::kLtx2MaskPixels + static_cast(f_pix * h_pix * w_pix)); + + const std::vector got = + vllm::Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, f_lat, h_lat, w_lat); + REQUIRE(got.size() == static_cast(f_lat * h_lat * w_lat)); + + double max_diff = 0.0, bilinear_diff = 0.0, uniform_diff = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + max_diff = std::max(max_diff, std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentWeights[i]))); + bilinear_diff = std::max(bilinear_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentBilinear[i]))); + uniform_diff = std::max(uniform_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentUniformPool[i]))); + } + // f32 area pooling accumulated in f64 against torch's own f32 reduction. + CHECK_MESSAGE(max_diff < 1e-6, "max |diff| against the executed oracle is " << max_diff); + // AND IT IS NOT EITHER REJECTED ANSWER. Both produce a correctly shaped mask + // of entirely plausible values, and nothing about a render's shape can see the + // difference — which is why they are asserted rather than described. + CHECK_MESSAGE(bilinear_diff > 1e-3, + "the port agrees with BILINEAR spatial interpolation; `mode=\"area\"` is what " + "upstream passes (iclora_utils.py:63-67)"); + CHECK_MESSAGE(uniform_diff > 1e-3, + "the port agrees with uniform temporal pooling, so the causal first-frame " + "carve-out (iclora_utils.py:70, :80) is not happening"); +} + +TEST_CASE("ltx2 ic-lora: the area window is the general form, not an integer stride") { + // `mode="area"` dispatches to `adaptive_avg_pool2d`, whose window for output + // index `i` is `[floor(i*I/O), ceil((i+1)*I/O))`. An integer-stride box filter + // `[i*(I//O), (i+1)*(I//O))` is elementwise EQUAL to it on every divisible + // shape, so the 8 -> 2 fixture above cannot see the difference and neither can + // the e2e case, which reads a 64x64 mask and pools to 2x2. This fixture is + // 9 -> 2, where the two readings separate. + const int64_t f_pix = vllm_test::kLtx2MaskNdPixFrames; + const int64_t h_pix = vllm_test::kLtx2MaskNdPixHeight; + const int64_t w_pix = vllm_test::kLtx2MaskNdPixWidth; + const int64_t f_lat = vllm_test::kLtx2MaskNdLatFrames; + const int64_t h_lat = vllm_test::kLtx2MaskNdLatHeight; + const int64_t w_lat = vllm_test::kLtx2MaskNdLatWidth; + REQUIRE(h_pix % h_lat != 0); // the fixture is only a gate while it stays indivisible + REQUIRE(w_pix % w_lat != 0); + const std::vector mask( + vllm_test::kLtx2MaskNdPixels, + vllm_test::kLtx2MaskNdPixels + static_cast(f_pix * h_pix * w_pix)); + + const std::vector got = + vllm::Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, f_lat, h_lat, w_lat); + REQUIRE(got.size() == static_cast(f_lat * h_lat * w_lat)); + + double max_diff = 0.0, stride_diff = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + max_diff = std::max(max_diff, std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskNdLatentWeights[i]))); + stride_diff = std::max(stride_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskNdLatentStrideBox[i]))); + } + CHECK_MESSAGE(max_diff < 1e-6, "max |diff| against the executed oracle is " << max_diff); + CHECK_MESSAGE(stride_diff > 1e-3, + "the port agrees with an integer-stride box filter, so the `area` window is " + "the divisible special case rather than the general form " + "(iclora_utils.py:63-67, ltx2_iclora_reference.cpp PoolStart/PoolEnd)"); +} + +TEST_CASE("ltx2 ic-lora: the mask downsample refuses an incompatible frame pair") { + // Upstream's own assertion (`:74-77`). A truncating group size would silently + // drop the tail of the mask and still produce the right number of weights. + const std::string refusal = Message([] { + const std::vector mask(8 * 1 * 1, 0.5F); + (void)vllm::Ltx2DownsampleMaskVideoToLatent(mask, 8, 1, 1, 4, 1, 1); + }); + INFO(refusal); + CHECK(refusal.find("must be divisible by") != std::string::npos); + + // ...and the DEGENERATE arms return the first frame alone (`:81-82`) rather + // than refusing, which is what keeps a single-frame reference working. + const std::vector single(1 * 2 * 2, 0.25F); + CHECK(vllm::Ltx2DownsampleMaskVideoToLatent(single, 1, 2, 2, 1, 2, 2).size() == 4U); +} + +TEST_CASE("ltx2 ic-lora: the mask video's arithmetic remaps [-1, 1] to [0, 1]") { + // `_load_mask_video` (ic_lora.py:530-536) after the read: mean over channels, + // `(x + 1) / 2`, clamp. The pixels arrive CHANNEL-major, so the three samples + // of one pixel are a plane apart — reading them as adjacent would average + // three different pixels and still produce a plausible mask. + const int64_t channels = 3, frames = 1, plane = 2; + const std::vector pixels = { + -1.0F, 1.0F, // channel 0 + -1.0F, 1.0F, // channel 1 + -1.0F, 1.0F}; // channel 2 + const std::vector got = vllm::Ltx2MaskVideoFromPixels(pixels, channels, frames, plane); + REQUIRE(got.size() == 2U); + CHECK(got[0] == doctest::Approx(0.0)); + CHECK(got[1] == doctest::Approx(1.0)); + + // A pixel whose channels DISAGREE separates the channel-major read from the + // interleaved one: the mean of {-1, 0, 1} is 0 and maps to 0.5, while reading + // the first three values as one pixel would average {-1, 1, -1}. + const std::vector mixed = {-1.0F, 0.0F, 0.0F, 0.0F, 1.0F, 0.0F}; + const std::vector mixed_got = vllm::Ltx2MaskVideoFromPixels(mixed, channels, frames, plane); + CHECK(mixed_got[0] == doctest::Approx(0.5)); + + // The clamp is upstream's and it is not defensive: a value above 1 becomes a + // POSITIVE log-space bias, which amplifies attention rather than attenuating. + const std::vector hot(6, 4.0F); + CHECK(vllm::Ltx2MaskVideoFromPixels(hot, channels, frames, plane)[0] == doctest::Approx(1.0)); +} + +TEST_CASE("ltx2 ic-lora: build_attention_mask puts the cross block on the NOISY rows only") { + // THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN, and that is the whole reason + // this case can fail. With `num_existing == num_noisy` the true block + // structure and the plausible reading `cross on ALL existing rows` are + // ELEMENTWISE EQUAL — measured 0 separating elements on the oracle — and the + // golden would be a mute switch. The generator refuses that fixture. + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t added = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(added)); + const std::vector got = + vllm::Ltx2BuildAttentionMask({}, noisy, added, existing, cross); + const int64_t total = existing + added; + REQUIRE(got.size() == static_cast(total * total)); + + size_t oracle_diff = 0, rejected_same = 0; + for (size_t i = 0; i < got.size(); ++i) { + if (got[i] != vllm_test::kLtx2AmMask[i]) ++oracle_diff; + if (got[i] == vllm_test::kLtx2AmMaskCrossOnAllRows[i]) ++rejected_same; + } + CHECK_MESSAGE(oracle_diff == 0, oracle_diff << " elements differ from the executed oracle"); + CHECK_MESSAGE(rejected_same < got.size(), + "the port is elementwise equal to `cross on ALL existing rows`, which is the " + "reading this fixture exists to separate"); +} + +TEST_CASE("ltx2 ic-lora: an existing mask is PRESERVED under a second item") { + // `:223-226`. Refilling the top-left block with ones would discard the first + // item's attenuation entirely, and the result is still a legal mask of the + // right shape whose values are all in [0, 1]. + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t first_new = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(first_new)); + const std::vector first = + vllm::Ltx2BuildAttentionMask({}, noisy, first_new, existing, cross); + const std::vector cross2(vllm_test::kLtx2AmCross2, vllm_test::kLtx2AmCross2 + 1); + const std::vector second = + vllm::Ltx2BuildAttentionMask(first, noisy, 1, existing + first_new, cross2); + + size_t diff = 0, matches_refilled = 0; + for (size_t i = 0; i < second.size(); ++i) { + if (second[i] != vllm_test::kLtx2AmMask2[i]) ++diff; + if (second[i] == vllm_test::kLtx2AmMask2Refilled[i]) ++matches_refilled; + } + CHECK_MESSAGE(diff == 0, diff << " elements differ from the executed oracle"); + CHECK_MESSAGE(matches_refilled < second.size(), + "the port discarded the existing mask and refilled the block with ones"); +} + +TEST_CASE("ltx2 ic-lora: resolve_cross_mask's scalar and 1-D arms, and its refusal") { + const std::vector scalar = vllm::Ltx2ResolveCrossMask({}, 0.5, 4); + REQUIRE(scalar.size() == 4U); + for (size_t i = 0; i < scalar.size(); ++i) CHECK(scalar[i] == vllm_test::kLtx2CrossScalar[i]); + + const std::vector values = {0.1F, 0.2F, 0.3F, 0.4F}; + const std::vector oned = vllm::Ltx2ResolveCrossMask(values, 1.0, 4); + REQUIRE(oned.size() == 4U); + for (size_t i = 0; i < oned.size(); ++i) CHECK(oned[i] == vllm_test::kLtx2CrossOneD[i]); + // ...and the 1-D arm is not the scalar fill, which is the failure a per-token + // mask that silently collapsed to a constant would produce. + CHECK(oned[0] != oned[3]); + + // `:50-53` refuses rather than broadcasting: a mask one token short of the + // sequence would attenuate the wrong tokens and still render. + const std::string refusal = Message([] { + const std::vector three = {0.1F, 0.2F, 0.3F}; + (void)vllm::Ltx2ResolveCrossMask(three, 1.0, 4); + }); + INFO(refusal); + CHECK(refusal.find("must equal num_new_tokens") != std::string::npos); +} + +TEST_CASE("ltx2 ic-lora: an unmasked append PADS the mask instead of leaving it short") { + // `update_attention_mask`'s `attention_mask is None` arm (`:141-156`). + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t first_new = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(first_new)); + const std::vector first = + vllm::Ltx2BuildAttentionMask({}, noisy, first_new, existing, cross); + + // No mask and none present is upstream's None. + CHECK(vllm::Ltx2PadAttentionMaskForUnmaskedTokens({}, noisy, 1, existing).empty()); + + const std::vector padded = + vllm::Ltx2PadAttentionMaskForUnmaskedTokens(first, noisy, 1, existing + first_new); + REQUIRE(padded.size() == static_cast((existing + first_new + 1) * + (existing + first_new + 1))); + size_t diff = 0; + for (size_t i = 0; i < padded.size(); ++i) { + if (padded[i] != vllm_test::kLtx2AmMaskPaddedOnes[i]) ++diff; + } + CHECK_MESSAGE(diff == 0, diff << " elements differ from the executed oracle"); + // GREW WITH THE SEQUENCE. A mask left at its pre-append size is not a shape + // error downstream — `Ltx2ModalityInput` accepts the key-only broadcast form + // too, so it is read as a legal mask over a different axis. + CHECK(padded.size() > first.size()); +} + +TEST_CASE("ltx2 ic-lora: an appending item extends the state's attention mask") { + // THE OBLIGATION LIVES IN `AppendTokens`, which is what upstream's docstring + // makes it (mask_utils.py:83-85 for the keyframes mask; every appending item + // calls `update_attention_mask` with a literal None at keyframe_cond.py:68-76 + // and reference_video_cond.py:86-94). Before this row no state could carry a + // mask at all, so this branch was unreachable. + vllm::Ltx2VideoLatentShape shape; + shape.batch = 1; + shape.channels = 2; + shape.frames = 1; + shape.height = 2; + shape.width = 2; + const vllm::Ltx2ScaleFactors factors; + vllm::Ltx2LatentState state = + vllm::Ltx2CreateVideoLatentState(shape, /*patch_size=*/1, factors, /*fps=*/24.0, + /*causal_fix=*/true, nullptr, nullptr); + REQUIRE(state.tokens == 4); + CHECK_MESSAGE(state.noisy_tokens == 4, + "the state must record `latent_tools.target_shape.token_count()` at construction; " + "after the first append `tokens` is no longer it"); + vllm::Ltx2LatentVolume ref; + ref.batch = 1; + ref.channels = 2; + ref.frames = 1; + ref.height = 2; + ref.width = 2; + ref.data.assign(8, 0.5F); + const auto apply = [&] { + vllm::Ltx2ConditionVideoByReference(&state, ref, /*patch_size=*/1, factors, /*fps=*/24.0, + /*downscale_factor=*/1, /*temporal_scale_factor=*/1, + /*strength=*/1.0, /*causal_fix=*/true); + }; + + // FIRST ITEM, no mask anywhere. Upstream's `update_attention_mask` returns + // None when its argument is None and the state carries none (`:142-143`), so + // an append on an unmasked state must leave the field empty rather than + // inventing an all-ones mask — which would be the identity and would make + // every later assertion vacuous. + apply(); + REQUIRE(state.tokens == 8); + CHECK_MESSAGE(state.attention_mask.empty(), + "an append on an unmasked state invented a mask; upstream returns None"); + + // Now the wrapper writes one, over the 8 tokens that exist. + state.attention_mask = vllm::Ltx2BuildAttentionMask({}, state.noisy_tokens, 4, 4, + {0.25F, 0.75F, 0.5F, 1.0F}); + REQUIRE(state.attention_mask.size() == 64U); + + // SECOND ITEM, and this is the branch this row made reachable. + const int64_t before = state.tokens; + apply(); + const int64_t added = state.tokens - before; + REQUIRE(added == 4); + // The mask covered 8 tokens and the state is now 12. A build that left it at + // 64 entries would hand the DiT a legal, differently shaped mask. + CHECK_MESSAGE(static_cast(state.attention_mask.size()) == 12 * 12, + "the appending item did not extend the attention mask; it holds " + << state.attention_mask.size() << " entries for " << state.tokens + << " tokens"); + // ...and it PADDED with full attention rather than refilling with ones: the + // first item's 0.25 is still there. + bool has_quarter = false; + for (const float v : state.attention_mask) { + if (v == 0.25F) has_quarter = true; + } + CHECK_MESSAGE(has_quarter, + "the earlier item's attenuation was discarded when the mask grew " + "(mask_utils.py:224 preserves the existing block)"); +} + +TEST_CASE("ltx2 ic-lora: the consumption seam converts [0,1] to an additive log-space bias") { + // `_prepare_self_attention_mask` (transformer_args.py:208-237), already ported + // as `Ltx2PrepareSelfAttentionMask`. Pinned HERE because nothing in `src/` + // assigned `Ltx2ModalityInput::attention_mask` before this row, so these + // numbers were previously gated only through a hand-built test struct. + const std::vector probe(vllm_test::kLtx2SelfMaskProbe, + vllm_test::kLtx2SelfMaskProbe + 4); + const std::vector bias = vllm::Ltx2PrepareSelfAttentionMask(probe.data(), 4); + REQUIRE(bias.size() == 4U); + for (size_t i = 0; i < bias.size(); ++i) { + INFO("i = " << i); + CHECK(bias[i] == vllm_test::kLtx2SelfMaskBias[i]); + } + // The two values a naive `log(x)` gets wrong, named rather than left to the + // element compare: a zero is `finfo.min` and NOT -inf, and a subnormal is + // clamped to `finfo.tiny` before the log rather than logged as-is. + CHECK(std::isfinite(bias[2])); + CHECK(bias[3] > -100.0F); +} diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 3032e9b243..ad8022643b 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1173,6 +1173,86 @@ TEST_CASE("ltx2 the recipe table mirrors vLLM-Omni's, and refuses everything els CHECK(std::string(vllm_test::kLtx2OmniRecipeVersions[i]) != "2.5"); } + // `ic_lora_reference` IS TRUE FOR EXACTLY ONE KIND, and this walk is what + // keeps that true. Row LTX25-IC-LORA-REF-VIDEO (#3020). + // + // WHY IT EXISTS, stated as a MEASUREMENT rather than a worry. The phase loop + // routes on `im.recipe.ic_lora_reference` and its refusals read the SAME local, + // because a refusal and its route predicate written as two expressions is how + // this campaign shipped a silently wrong answer. Mutation M11 replaced the + // refusal's half with `im.pipeline_kind != "ic_lora"` and BOTH suites stayed + // green — 97/97 and 169/169 — because over the recipe table as it stands the + // two expressions are the same predicate. That is an IDENTITY, not a blind + // spot, and it stops being one the moment a second recipe sets this flag. + // + // So this case is the tripwire for that day: a second recipe carrying + // `ic_lora_reference` reds HERE, where the reader is told to reconcile the + // refusal, rather than in a render that silently refuses a supported request. + { + // THE KIND LIST IS DERIVED, not hand-copied. A hand-copied list is complete + // only until the next kind is added, and this tripwire's whole job is to + // fire on that day — a list that silently omits the twelfth kind would go on + // reporting "exactly one" while the new recipe went unwalked. So the kinds + // come out of `ResolveLtx2PipelineRecipe`'s own source: every + // `pipeline_kind == "..."` branch in ltx2_pipeline.cpp, which is where a new + // kind has to be written down to exist at all. + const std::filesystem::path resolver_source = + std::filesystem::path(VLLM_CPP_SOURCE_ROOT) / + "src/vllm/model_executor/models/ltx2_pipeline.cpp"; + std::ifstream resolver_in(resolver_source); + REQUIRE_MESSAGE(resolver_in.good(), + "cannot read " << resolver_source.string() + << "; VLLM_CPP_SOURCE_ROOT is wrong"); + const std::string resolver((std::istreambuf_iterator(resolver_in)), + std::istreambuf_iterator()); + std::vector kAllKinds; + const std::string needle = "pipeline_kind == \""; + for (size_t at = resolver.find(needle); at != std::string::npos; + at = resolver.find(needle, at + 1)) { + const size_t start = at + needle.size(); + const size_t end = resolver.find('"', start); + REQUIRE(end != std::string::npos); + kAllKinds.push_back(resolver.substr(start, end - start)); + } + // A derivation that found nothing would pass every assertion below without + // walking anything, so the count is asserted before it is used. Eleven is + // what the table carries today; a twelfth kind reds HERE, which is the point. + CHECK_MESSAGE(kAllKinds.size() == 11, + "ResolveLtx2PipelineRecipe declares " << kAllKinds.size() + << " kinds, not the 11 this tripwire was " + "written against. Re-read the " + "ic_lora_reference refusals in " + "ltx2_video.cpp, then update this " + "count."); + REQUIRE(kAllKinds.size() >= 11); + const char* const kAllVersions[] = {"2", "2.3", "2.4", "2.5"}; + int resolved_with_flag = 0; + for (const std::string& kind_s : kAllKinds) { + const char* const kind = kind_s.c_str(); + for (const char* version : kAllVersions) { + vllm::Ltx2PipelineRecipe recipe; + try { + recipe = vllm::ResolveLtx2PipelineRecipe(kind, version); + } catch (const std::exception&) { + continue; // an unkeyed pair; the table refuses it by name + } + INFO("kind = ", kind, " version = ", version); + if (recipe.ic_lora_reference) { + ++resolved_with_flag; + CHECK_MESSAGE(std::string(kind) == "ic_lora", + "a recipe other than 'ic_lora' declares ic_lora_reference. The phase " + "loop's refusals in ltx2_video.cpp must be re-read: they name 'ic_lora' " + "in their text, and that text is now wrong."); + } else { + CHECK(std::string(kind) != "ic_lora"); + } + } + } + CHECK_MESSAGE(resolved_with_flag == 1, + "exactly one (kind, version) pair carries ic_lora_reference; got " + << resolved_with_flag); + } + // The 2.4 and 2.5 rows this port adds, sourced from Lightricks. CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("one_stage", "2.4")); CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5")); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 83072d99ee..e64473441c 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1774,96 +1774,57 @@ TEST_CASE("ltx2 video: keyframe and reference conditioning is refused BY WHAT IS const vllm::multimodal::VideoResult result = engine->Generate(gen); CHECK(result.frame_count == 9); } - SUBCASE("a reference video may not blame a seam THIS ENGINE demonstrably has") { - // WHAT THIS CASE USED TO DO, AND WHY THAT WAS THE DEFECT. It asserted five - // SUBSTRINGS of the refusal: `reference_video_cond.py`, `clear_conditioning`, - // `TOKEN-APPEND`, `NOT the IC-LoRA metadata`, and the absence of one retired - // phrase. Two of those five are UPSTREAM symbol names, which are present in - // the pinned checkout whatever this engine can do, and the other three are - // literals the message declares about itself. So not one of them could go - // red when the ENGINE changed — and the engine did change, twice, in two - // days: #923 made the metadata readable and #930 (`c7cb59fbb`) built the - // token-append seam. A reviewer replaced the local-cause sentence with a - // self-declared falsehood, kept all five substrings, and the whole suite - // stayed green. + SUBCASE("a reference video on a pipeline that does not condition on one names where it lives") { + // THIS SUBCASE USED TO ASSERT A REFUSAL OF THE FEATURE ITSELF, and row + // LTX25-IC-LORA-REF-VIDEO (#3020) SERVED it. What it required — that the + // refusal not blame the token-append seam this engine demonstrably has, and + // that every closed cause appear only under a `WHAT IS *NOT* THE REASON` + // marker — was the right constraint on a message whose job was to explain a + // MISSING capability. The capability is no longer missing. // - // SO THIS CASE MEASURES THE ENGINE FIRST and only then constrains the - // message. The measurement is the same instrument the token-append row - // gates itself with: `video_tokens` is written INSIDE the phase loop, so it - // can observe what the loop does, unlike every field filled before denoise. - const vllm::multimodal::VideoModelParams cond_params = ConditioningParams(ws.paths); - const std::string kf = ws.root + "/append_witness.ppm"; - WriteBytes(kf, ConditioningPpm(20, 28, 31)); - - auto tokens_of = [&](const std::string& tag, const std::string& keyframe) { - const std::unique_ptr own = - vllm::multimodal::LoadVideoEngine(cond_params); - auto* ltx2 = dynamic_cast(own.get()); - REQUIRE(ltx2 != nullptr); - vllm::multimodal::VideoGenParams g = FixtureGen(ws.root + "/" + tag); - if (!keyframe.empty()) { - g.last_frame_path = keyframe; - g.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; - } - const vllm::multimodal::VideoResult result = own->Generate(g); - // THE TRIM, observed from outside: the volume handed to unpatchify is the - // target grid, so the clip comes back at the requested length whether or - // not anything was appended. - CHECK(result.frame_count == 9); - return ltx2->last_conditioning().video_tokens; - }; - - const int64_t plain = tokens_of("ref_witness_plain", ""); - const int64_t grown = tokens_of("ref_witness_grown", kf); - // THE GROWTH. Both numbers are measured; pinning either to a literal would - // pass on a build that never grew anything. - REQUIRE_MESSAGE(grown > plain, - "this engine's phase loop did not grow its token sequence for an appending " - "conditioning item (" - << grown << " against a target of " << plain - << "), so the rest of this case cannot say what the refusal may claim"); - + // So the constraint is replaced by the stronger one the new shape allows: + // the message may not describe a missing feature AT ALL, because there is + // none. It has to name the pipeline that serves it. That is a property a + // reader can act on in one step, where the old message asked them to + // re-derive which of four causes still held. + // + // The token-append measurement the old subcase made is not dropped: it moved + // to `ltx2 ic-lora: a reference clip REACHES the render through the public + // ABI`, which measures the same growth by rendering through it. const std::string msg = refusal("a reference video", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { g.ref_video_dir = w.root; }); INFO(msg); - // BECAUSE THE TWO MEASUREMENTS ABOVE HOLD, the refusal may not CLAIM the - // phase loop. It may still MENTION it — the message's own convention is to - // record a ruled-out cause under `WHAT IS *NOT* THE REASON` so the next - // reader re-checks rather than re-derives — so the property asserted here is - // positional: every occurrence of a closed cause sits after that marker. - // - // That is what makes this case red for the mutation that motivated it. - // Restoring the pre-repair message leaves no marker at all AND puts - // `TOKEN-APPEND` in the first sentence, so both halves fire. - const size_t ruled_out = msg.find("WHAT IS *NOT* THE REASON"); - REQUIRE_MESSAGE(ruled_out != std::string::npos, - "the refusal carries no `WHAT IS *NOT* THE REASON` section, so a cause this " - "engine has already closed cannot be told apart from one it still has"); - for (const char* closed : {"TOKEN-APPEND", "fixed at the target grid's token count", - "nowhere to go", "nothing to trim"}) { - const size_t at = msg.find(closed); - const bool only_as_ruled_out = (at == std::string::npos) || (at > ruled_out); - CHECK_MESSAGE(only_as_ruled_out, + // IT NAMES WHERE THE FEATURE IS, and does not describe it as absent. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("ic_lora.py:381-402") != std::string::npos); + CHECK(msg.find("ref_video_dir") != std::string::npos); + // ...and it distinguishes the OTHER thing this directory can mean, which is + // the whole reason `!wants_retake` guards the route. + CHECK(msg.find("retake_start_time") != std::string::npos); + + // AND IT CLAIMS NO MISSING SEAM. Every phrase below named a cause this + // engine has since built, and each one appeared in this refusal at some + // point in the campaign; a message that reintroduces any of them is + // describing a build that no longer exists. There is no `WHAT IS *NOT* THE + // REASON` escape hatch here, because a refusal that only says "load a + // different pipeline" has no reason to mention a seam at all. + for (const char* closed : {"TOKEN-APPEND", "is not served", "has no pixel path", + "no reader", "second resident", "fused at load"}) { + CHECK_MESSAGE(msg.find(closed) == std::string::npos, "the refusal states '" << std::string(closed) - << "' as a cause rather than as a ruled-out one, and this case has just " - "MEASURED that the loop grows (" - << plain << " -> " << grown << ") and trims back to the target grid"); + << "', which describes a MISSING capability. The reference arm is served " + "on 'pipeline_kind' 'ic_lora' (#3020) and this message's only job is " + "to say so."); } - // The metadata half, same shape: the factors printed are READ from the - // adapter at load, so this asserts the read happened rather than asserting - // a sentence about it. `factors` says "no adapter was supplied" here. + // The adapter factors are still READ and reported, which is what says the + // metadata half happened rather than asserting a sentence about it. CHECK(msg.find("no adapter was supplied") != std::string::npos); - CHECK(msg.find("which this project does not read") == std::string::npos); - // And the two causes that DO remain are named, by the upstream anchors a - // reader can go and check. - CHECK(msg.find("iclora_utils.py:116-117") != std::string::npos); - CHECK(msg.find("ic_lora.py:108") != std::string::npos); - CHECK(msg.find("ref_video_dir") != std::string::npos); } + SUBCASE("reference audio names the AUDIO encoder, which this row did not build") { const std::string msg = refusal("reference audio", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { @@ -7379,9 +7340,13 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { // // IT IS STILL A REFUSAL AFTER ROW LTX25-IMAGE-COND (#644), which served the // first-frame arm and would have made a `first_frame_ppm` probe stop - // refusing. The reference arm stays refused for a reason this row did not - // touch (the IC-LoRA scale factors), so the probe was moved to it rather than - // to whatever happened to still throw. + // refusing, and after row LTX25-IC-LORA-REF-VIDEO (#3020), which served the + // reference CLIP. THE PROBE IS NOW A REFERENCE IMAGE and that is the durable + // choice rather than the convenient one: `--video-conditioning` takes a video + // file or an EXR directory (ic_lora.py:416-425) and there is no + // reference-IMAGE conditioning anywhere in `ltx-pipelines`, so this arm is + // refused because upstream has nothing to mirror — not because a seam here is + // missing. It is the one refusal in this file that no later row can serve. Workspace ws; const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); const std::unique_ptr engine = @@ -7398,7 +7363,7 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - CHECK(msg.find("reference-image / reference-video conditioning") != std::string::npos); + CHECK(msg.find("reference-IMAGE conditioning is not served") != std::string::npos); } const vllm::multimodal::Ltx2ConditioningTrace trace = ltx->last_conditioning(); @@ -8591,29 +8556,36 @@ TEST_CASE("ltx2 retake: the wrong recipe refuses, and the reference arm still do } } - SUBCASE("a reference clip WITHOUT the retake knobs is still refused, and #975 stays open") { - // THE LOCAL FACT this row's refusal edit turns on: the LTX side now reads - // `ref_video_dir`, so the sentence claiming nothing did is gone. A case that - // asserted only upstream symbol names could not see that going stale — which - // is exactly how the sentence survived (#987). + SUBCASE("a reference clip WITHOUT the retake knobs points at the pipeline that serves it") { + // THE LOCAL FACT this subcase turns on has CHANGED TWICE and the history is + // kept because that is the point of the case. It first asserted that the + // refusal must not claim nothing reads `ref_video_dir`, because the LTX side + // reads it (#987). Row LTX25-IC-LORA-REF-VIDEO (#3020) then SERVED the + // reference arm, on `pipeline_kind` `ic_lora`, so there is no missing seam + // left for a message to misdescribe. + // + // What survives is the shape: at least one assertion tied to a fact about + // THIS tree, so a message whose claim goes stale is visible. That fact is + // now that the engine HAS the feature and this pipeline is not it. vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refstill"); gen.ref_video_dir = clip; try { (void)engine->Generate(gen); - FAIL_CHECK("reference-video conditioning must still be refused"); + FAIL_CHECK("a reference clip on a distilled_two_stage engine must be refused"); } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - const size_t ruled_out = msg.find("WHAT IS *NOT* THE REASON"); - REQUIRE(ruled_out != std::string::npos); - const size_t stale = msg.find("nothing reads `ref_video_dir` at all"); - const bool only_as_ruled_out = (stale == std::string::npos) || (stale > ruled_out); - CHECK_MESSAGE(only_as_ruled_out, - "the refusal states as a CAUSE that nothing reads ref_video_dir, and this " - "suite has just rendered a retake that reads it"); - // And it names the reader that DOES exist, so the next reader goes looking - // for the reference item's geometry rather than for a file walker. - CHECK(msg.find("Ltx2ReadFrameDirectory") != std::string::npos); + // It names the pipeline that serves it, and the two operations this same + // directory can mean — which is what a caller can act on. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("retake_start_time") != std::string::npos); + CHECK(msg.find("SEEDS the video latent") != std::string::npos); + // AND IT CLAIMS NOTHING IS MISSING. This suite has just rendered a retake + // that reads this very directory, and the row above serves the reference + // arm from it, so a message that described either as absent would be + // describing a build that no longer exists. + CHECK(msg.find("nothing reads `ref_video_dir` at all") == std::string::npos); + CHECK(msg.find("is not served") == std::string::npos); } } } @@ -13866,3 +13838,478 @@ TEST_CASE("ltx2 video: auto_duration refuses MIN > MAX and refuses doubling an e } } } + + +// ───────────────────────────────────────────────────────────────────────────── +// IC-LoRA REFERENCE VIDEO and its CONDITIONING ATTENTION MASK +// Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +// spec .agents/specs/ltx25-ic-lora-ref-video.md (gaps A15 and A16). +// +// These are the REACHABILITY cases and they are the only proof this row has. +// They enter through `LoadVideoEngine` and `Generate` — where +// `vllm_video_generate` arrives — and not through the ported helpers. The +// value-level cases live in `test_ltx2_iclora_reference` and per +// .agents/reachability.md they localize a failure rather than prove one: they +// stay green when the production call site is deleted. +// +// THE SEAM THIS ROW RETIRES. Before it, `Ltx2ModalityInput::attention_mask` and +// its whole consumption chain — `Ltx2PrepareSelfAttentionMask`, `self_bias` on +// host and device, `vt::AttentionCross`'s additive bias — were assigned at +// exactly four sites, all of them inside a `BuildModalities` helper in +// `test_ltx2.cpp` and `test_ltx2_device.cpp`. Nothing in `src/` wrote either +// field. The mask case below is the production assignment. +// ───────────────────────────────────────────────────────────────────────────── + +namespace { + +// An `ic_lora` engine. The kind is a LOAD extra, because `ICLoraPipeline` is a +// pipeline CLASS upstream (ic_lora.py:60) and not a per-request mode: stage 2 +// runs `loras=()` against stage 1's `loras=tuple(loras)` (`:108` against +// `:119`), which is a property of how the weights are held. +// +// `skip_stage_2` IS UPSTREAM'S OWN FLAG (`--skip-stage-2`, ic_lora.py:442-448), +// spelled here as the `max_phase` load extra this engine already has. It is not +// a test convenience, and the case below records exactly why one leg needs it. +vllm::multimodal::VideoModelParams IcLoraParams(const ltx2_fixture::Paths& paths, + const std::string& upsampler, + bool skip_stage_2 = false) { + vllm::multimodal::VideoModelParams mp = FixtureParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "ic_lora"; + mp.extras[vllm::multimodal::kLtx2CheckpointClassExtra] = FixtureCheckpointClass("ic_lora"); + // Stage 2 spatially upsamples, exactly as the distilled two-stage recipe this + // one derives from does (`ICLoraPipeline` calls `self.upsampler` at + // ic_lora.py:307). + mp.extras["upsampler_path"] = upsampler; + if (skip_stage_2) mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + return mp; +} + +// The spatial x2 upsampler stage 2 needs, written at the fixture's own reduced +// channel count so the phase's shapes follow the DiT contract rather than a +// literal. +std::string WriteIcLoraUpsampler(const std::string& path) { + vllm::Ltx2UpsamplerConfig cfg = + ltx2_fixture::ReducedUpsamplerConfig(ltx2_fixture::ReducedDitParams().in_channels); + cfg.dims = 2; + ltx2_fixture::WriteReducedUpsampler(cfg, path); + return path; +} + +// A clip of CONSTANT frames, and NOT `ConditioningPpm`'s seeded ramp. +// +// MEASURED, and it is the reduced-fixture trap this campaign has recorded twice. +// `ConditioningPpm` fills a frame with `(i * 37 + seed * 101) % 251` — a dense +// high-frequency ramp that a seed only PHASE-SHIFTS. The video VAE downsamples +// 32x spatially, so two seeds average to nearly the same latent: measured absmax +// 0.539 against 0.464, with the two renders identical to the byte. A fixture +// built on it cannot see whether the reference reached the DiT at all. Constant +// frames survive the downsample, and 20 against 235 encode to latents whose +// absmax is 0.434 against 0.637. +// +// The retake cases use the ramp and are right to: retake SEEDS the video latent +// with the clip, so a small latent difference propagates one-for-one. A +// reference's whole effect arrives through ATTENTION, which attenuates. +std::string WriteIcLoraClip(const std::string& dir, int frames, int height, int width, + unsigned char value) { + ::mkdir(dir.c_str(), 0755); + for (int i = 0; i < frames; ++i) { + std::string ppm = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + ppm.append(static_cast(height) * static_cast(width) * 3, + static_cast(value)); + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06d.ppm", i); + WriteBytes(dir + name, ppm); + } + return dir; +} + +// A grayscale mask clip: the LEFT half at `left` and the right half at `right`, +// so the mask is SPATIALLY structured rather than uniform. A uniform mask cannot +// separate "the mask reached the DiT" from "the strength scalar did", and an +// all-ones mask is the identity. +std::string WriteIcLoraMask(const std::string& dir, int frames, int height, int width, + unsigned char left, unsigned char right) { + ::mkdir(dir.c_str(), 0755); + for (int i = 0; i < frames; ++i) { + std::string ppm = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + for (int y = 0; y < height; ++y) { + for (int x = 0; x < width; ++x) { + const char v = static_cast(x < width / 2 ? left : right); + ppm.push_back(v); + ppm.push_back(v); + ppm.push_back(v); + } + } + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06d.ppm", i); + WriteBytes(dir + name, ppm); + } + return dir; +} + +// 128x128 and NOT the 64x64 every other case uses, deliberately. The distilled +// two-stage recipe runs stage 1 at half resolution, so the reference clip is +// read at 64x64, and the fixture VAE's (8, 32, 32) factors then give a 2 x 2 x 2 +// reference latent — 8 tokens. At 64x64 the same arithmetic gives a 1 x 1 +// SPATIAL grid, where a row-major and a column-major patchify are identical and +// a downscale factor of 2 collapses to the same single token as a factor of 1. +// A reduced fixture that degenerates the axis it gates is this campaign's +// recorded failure, so the geometry is chosen against it. +vllm::multimodal::VideoGenParams IcLoraGen(const std::string& out_dir, const std::string& clip) { + vllm::multimodal::VideoGenParams gen; + gen.num_frames = 9; + gen.height = 128; + gen.width = 128; + gen.has_seed = true; + gen.seed = 7; + gen.output_dir = out_dir; + gen.ref_video_dir = clip; + return gen; +} + +std::string FirstFrame(const vllm::multimodal::VideoResult& result) { + return ReadAll(std::string(result.frame_dir) + "/frame_000000.ppm"); +} + +} // namespace + +TEST_CASE("ltx2 ic-lora: a reference clip REACHES the render through the public ABI") { + // THE RED-FIRST CASE. Before this row `pipeline_kind=ic_lora` resolved to no + // recipe at all and `ref_video_dir` was refused outright on every kind but + // `retake`, so nothing here could reach the new code. The literal red was + // ERROR: test case THREW exception: Unsupported LTX pipeline kind/version: + // 'ic_lora'/'2.5'. + Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/ic_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + // Stage 1 renders at 64x64 and the fixture's factors are (8, 32, 32), so the + // reference clip encodes to 2 x 2 x 2 = 8 tokens. + const std::string clip = WriteIcLoraClip(ws.root + "/ic_clip", 9, 64, 64, /*value=*/20); + const vllm::multimodal::VideoResult result = + engine->Generate(IcLoraGen(ws.root + "/ic_out", clip)); + + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + CHECK(trace.completed); + CHECK_MESSAGE(trace.ic_lora_reference_tokens == 8, + "the reference clip did not append one latent frame's worth of tokens per latent " + "frame; got " + << trace.ic_lora_reference_tokens); + // The lower bound a token count cannot make: a clip that encoded to zeros has + // the right shape, the right count, and renders. + CHECK(trace.ic_lora_reference_absmax > 0.0); + CHECK(trace.ic_lora_reference_digest != 0); + // The output geometry comes from the REQUEST, not from the reference — which + // is where it differs from retake, whose geometry comes from the clip. + CHECK(result.frame_count == 9); + CHECK(result.width == 128); + CHECK(result.height == 128); + // NO MASK WAS SUPPLIED, so no mask was handed over. `attention_mask` stays a + // null pointer and the DiT takes upstream's `attention_mask is None` branch. + // Asserted, because a build that handed over an all-ones mask would render + // identically and would be measuring nothing at every case below. + CHECK(trace.ic_lora_attention_mask_rows == 0); + + // THE RENDER DEPENDS ON THE REFERENCE CLIP'S PIXELS, and this is the assertion + // the trace above cannot make. `ic_lora_reference_absmax` observes the ENCODE; + // it says nothing about whether the encoded latent was ever handed to the + // phase. A build that read the clip, encoded it, recorded the trace and then + // appended zeros satisfies every check above. + // + // ON THE SKIP-STAGE-2 ARM, AND THAT IS MEASURED RATHER THAN CONVENIENT. + // Upstream applies the reference to stage 1 ONLY (ic_lora.py:269-281 against + // `:314-321`) and stage 2 re-noises the upsampled latent to + // `STAGE_2_DISTILLED_SIGMAS[0]` = 0.909375, so 91% of stage 2's input is fresh + // noise. MEASURED on this reduced fixture: stage 1's latents for the two clips + // DO differ (clean digests 7109293506366579875 against 12136535927875385707) + // and stage 2's inputs still differ, but the surviving difference falls below + // 1/255 and the two-stage renders come out byte-identical. That is a property + // of a 2-block random-weight DiT, not of the port, and the honest answer is to + // make the pixel claim where the conditioning is applied. `--skip-stage-2` is + // upstream's own flag for exactly this configuration (ic_lora.py:442-448), so + // this leg is a supported request shape and not a test hook. + const std::unique_ptr stage1 = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + const std::string other = WriteIcLoraClip(ws.root + "/ic_clip_b", 9, 64, 64, /*value=*/235); + REQUIRE(ReadAll(clip + "/frame_000000.ppm") != ReadAll(other + "/frame_000000.ppm")); + const std::string frame_a = FirstFrame(stage1->Generate(IcLoraGen(ws.root + "/s1_a", clip))); + const std::string frame_b = FirstFrame(stage1->Generate(IcLoraGen(ws.root + "/s1_b", other))); + REQUIRE(frame_a.size() == frame_b.size()); + CHECK_MESSAGE(frame_a != frame_b, + "two different reference clips rendered byte-identical frames at the same seed, " + "so the encoded reference latent never reached the phase"); +} + +TEST_CASE("ltx2 ic-lora: the conditioning attention MASK reaches the DiT and moves the render") { + // THE CASE THAT RETIRES THE DEAD SEAM. `Ltx2ModalityInput::attention_mask` had + // no assignment in `src/` before this row; the only four in the tree were in a + // `BuildModalities` test helper. This is the production assignment, reached + // from `Generate`. + Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/mask_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + const std::string clip = WriteIcLoraClip(ws.root + "/mask_clip", 9, 64, 64, /*value=*/20); + // The mask is read at the STAGE's resolution (64x64), which is what upstream + // spells `args.height // 2` (ic_lora.py:460-461), and downsampled to the + // REFERENCE latent's own 2 x 2 x 2 shape (iclora_utils.py:151-155). + const std::string mask = + WriteIcLoraMask(ws.root + "/mask_dir", 9, 64, 64, /*left=*/0, /*right=*/255); + + const std::string plain = FirstFrame(engine->Generate(IcLoraGen(ws.root + "/m_plain", clip))); + const vllm::multimodal::Ltx2ConditioningTrace no_mask = ltx2->last_conditioning(); + CHECK(no_mask.ic_lora_attention_mask_rows == 0); + + vllm::multimodal::VideoGenParams masked = IcLoraGen(ws.root + "/m_masked", clip); + masked.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = mask; + const std::string with_mask = FirstFrame(engine->Generate(masked)); + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + + // THE MASK WAS HANDED OVER, and the row count says it was the DENSE form. A + // build that constructed the mask and dropped the pointer leaves this at 0 + // while every other check in this case still passes. + // + // 8 target tokens + 8 reference tokens = 16. + CHECK_MESSAGE(trace.ic_lora_attention_mask_rows == 16, + "the self-attention strength mask did not reach the DiT as the dense " + "[tokens, tokens] form; rows = " + << trace.ic_lora_attention_mask_rows); + // AND IT IS NOT THE IDENTITY. An all-ones mask renders correctly, has the + // right shape, and is exactly what a downsample that lost its values produces. + // Both ends are asserted because one of them alone cannot see a constant mask. + CHECK_MESSAGE(trace.ic_lora_attention_mask_min < 1.0, + "every weight in the mask is 1.0, which is the identity: this case would be " + "measuring nothing"); + CHECK(trace.ic_lora_attention_mask_max == 1.0); + // The left half of the mask is 0 and the right half is 255, so after + // `(x + 1) / 2` the latent weights straddle 0 and 1 — a mask whose minimum is + // 0 is a token that is FULLY masked, which is `finfo.min` in the log-space + // bias and the one value a smooth attenuation cannot produce. + CHECK(trace.ic_lora_attention_mask_min == 0.0); + + REQUIRE(plain.size() == with_mask.size()); + CHECK_MESSAGE(plain != with_mask, + "a conditioning attention mask that fully masks half the reference's tokens " + "rendered byte-identical frames, so the mask never reached the attention"); + + // AND THE MASK'S VALUES MATTER, not merely its presence. Two DIFFERENT masks + // must render differently; a build that read the directory and then handed + // over a constant satisfies every check above. + const std::string other_mask = + WriteIcLoraMask(ws.root + "/mask_dir_b", 9, 64, 64, /*left=*/255, /*right=*/0); + vllm::multimodal::VideoGenParams flipped = IcLoraGen(ws.root + "/m_flipped", clip); + flipped.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = other_mask; + const std::string with_flipped = FirstFrame(engine->Generate(flipped)); + CHECK_MESSAGE(with_flipped != with_mask, + "a mask and its left-right mirror rendered identically, so the mask's VALUES " + "never reached the attention — only its presence did"); + + // AND THE STRENGTH SCALES IT (iclora_utils.py:156). Halving the strength + // halves every latent weight, so the maximum drops from 1.0 to 0.5 and the + // render moves again. + vllm::multimodal::VideoGenParams half = IcLoraGen(ws.root + "/m_half", clip); + half.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = mask; + half.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "0.5"; + const std::string with_half = FirstFrame(engine->Generate(half)); + const vllm::multimodal::Ltx2ConditioningTrace half_trace = ltx2->last_conditioning(); + CHECK_MESSAGE(half_trace.ic_lora_attention_mask_max == 1.0, + "the mask's own block still attends fully to itself (mask_utils.py:229); the " + "strength scales the CROSS weights, not the whole matrix"); + CHECK(with_half != with_mask); +} + +TEST_CASE("ltx2 ic-lora: the reference GEOMETRY follows the adapter, and refuses what it cannot divide") { + Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/geom_ups.safetensors"); + const std::string clip = WriteIcLoraClip(ws.root + "/geom_clip", 9, 64, 64, /*value=*/20); + + SUBCASE("downscale 2 reads the clip at HALF the stage's grid, and appends fewer tokens") { + // The adapter's own metadata (iclora_utils.py:30-49), read at load by row + // LTX25-IC-LORA. Stage 1 runs at 64x64, so `reference_downscale_factor = 2` + // reads the clip at 32x32 and the (8, 32, 32) factors give a 2 x 1 x 1 + // reference latent — 2 tokens against the unscaled arm's 8. + vllm::multimodal::VideoModelParams mp = IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = WriteFixtureLora( + ws.root + "/ref_lora.safetensors", kFixtureLoraTarget, 1.0F, + {{"reference_downscale_factor", "2"}}); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + (void)engine->Generate(IcLoraGen(ws.root + "/geom_out", clip)); + CHECK_MESSAGE(ltx2->last_conditioning().ic_lora_reference_tokens == 2, + "the reference was not read at `height // reference_downscale_factor` " + "(iclora_utils.py:116-117); it appended " + << ltx2->last_conditioning().ic_lora_reference_tokens << " tokens"); + } + + SUBCASE("a target the factor does not divide is refused by upstream's own message") { + vllm::multimodal::VideoModelParams mp = IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = WriteFixtureLora( + ws.root + "/ref_lora3.safetensors", kFixtureLoraTarget, 1.0F, + {{"reference_downscale_factor", "3"}}); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + try { + // Stage 1 is 64x64 and 64 % 3 != 0. + (void)engine->Generate(IcLoraGen(ws.root + "/geom_bad", clip)); + FAIL("a target the reference_downscale_factor does not divide must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // Upstream's own wording, pinned by `kLtx2RefDivisibilityRefusal`. + CHECK(msg.find("must be divisible by reference_downscale_factor") != std::string::npos); + CHECK(msg.find("iclora_utils.py:112-115") != std::string::npos); + } + } +} + +TEST_CASE("ltx2 ic-lora: every refusal names what is wrong and where the feature lives") { + Workspace ws; + const std::string clip = WriteIcLoraClip(ws.root + "/ref_clip", 9, 64, 64, /*value=*/20); + + SUBCASE("a reference clip on a pipeline that has no reference conditioning") { + // THE SAME `serves_reference` PREDICATE the route uses, negated. A refusal + // and its route predicate written as two expressions is how this campaign + // shipped a silently wrong answer, and this case is what holds them together. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refuse_ref"); + gen.ref_video_dir = clip; + try { + (void)engine->Generate(gen); + FAIL("a reference clip on distilled_two_stage must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // It names the pipeline that DOES serve it, rather than saying the feature + // is missing — it is not missing, it is elsewhere. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("ic_lora.py:381-402") != std::string::npos); + // ...and it names retake as the other thing this directory can mean. + CHECK(msg.find("retake_start_time") != std::string::npos); + // The adapter factors are READ, so the message reports what was read + // rather than a sentence about it. + CHECK(msg.find("no adapter was supplied") != std::string::npos); + } + } + + SUBCASE("an ic_lora render with NO reference clip") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/none_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/no_ref", ""); + try { + (void)engine->Generate(gen); + FAIL("`--video-conditioning` is required=True upstream; an ic_lora render without one " + "must be refused rather than silently becoming a distilled render"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("required=True") != std::string::npos); + CHECK(msg.find("distilled_two_stage") != std::string::npos); + } + } + + SUBCASE("a mask on a pipeline that appends no reference tokens") { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refuse_mask"); + gen.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = ws.root; + try { + (void)engine->Generate(gen); + FAIL("a conditioning attention mask with no reference tokens to attenuate must refuse"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("combined_image_conditionings` never wraps") != std::string::npos); + CHECK(msg.find("'ic_lora'") != std::string::npos); + } + } + + SUBCASE("the scalar-only strength arm, which upstream's CLI cannot reach either") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/scalar_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/scalar", clip); + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "0.5"; + try { + (void)engine->Generate(gen); + FAIL("a sub-1.0 strength with no mask is upstream's Python-API-only arm and must be " + "refused by name rather than served"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("iclora_utils.py:157-158") != std::string::npos); + CHECK(msg.find("ic_lora.py:452-455") != std::string::npos); + CHECK(msg.find("owed") != std::string::npos); + } + } + + SUBCASE("a strength outside [0, 1]") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/range_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/range", clip); + gen.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = ws.root; + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "1.5"; + try { + (void)engine->Generate(gen); + FAIL("a strength above 1 would AMPLIFY attention and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("[0.0, 1.0]") != std::string::npos); + CHECK(msg.find("AMPLIFY") != std::string::npos); + } + } + + SUBCASE("a strength outside [0, 1] with NO mask, which upstream refuses just the same") { + // `ic_lora.py:230-233` is UNCONDITIONAL: it is in the method body, not + // inside `if args.conditioning_attention_mask is not None`. Guarding this + // engine's copy on a mask directory made an out-of-range strength with no + // mask pass the range check AND pass the sub-1.0 scalar-arm refusal below + // it, leaving a value that is accepted, never read (its only reader is + // inside the mask branch) and silently ignored on every pipeline. + const std::string ups = WriteIcLoraUpsampler(ws.root + "/nomask_range_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/nomask_range", clip); + // NO kLtx2CondAttentionMaskDirExtra. That is the whole point of the case. + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "1.5"; + try { + (void)engine->Generate(gen); + FAIL("an out-of-range strength must be refused whether or not a mask was supplied; " + "upstream's check does not consult the mask"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("[0.0, 1.0]") != std::string::npos); + CHECK(msg.find("AMPLIFY") != std::string::npos); + CHECK(msg.find("ic_lora.py:230-233") != std::string::npos); + } + } + + SUBCASE("a reference IMAGE, which is not what an IC-LoRA reference is") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/img_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/refimg", clip); + gen.ref_image_paths.push_back(ws.paths.audio_embeds); + try { + (void)engine->Generate(gen); + FAIL("reference images are not an IC-LoRA shape and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("--video-conditioning") != std::string::npos); + CHECK(msg.find("first_frame_ppm") != std::string::npos); + } + } +}