From 4ad8b45527b9dba6b007cd7d0a9fbe6e79413177 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:21:12 +0000 Subject: [PATCH 01/11] spec(LTX25-IC-LORA-REF-VIDEO): the IC-LoRA reference clip and its attention mask A15 and A16 of the LTX-2.5 completion scope, specced as one row because upstream applies ConditioningItemAttentionStrengthWrapper at exactly one site and its argument is always the VideoConditionByReferenceLatent built two lines above it (iclora_utils.py:162-169). combined_image_conditionings never wraps, so A16 alone has nowhere to attach that upstream also attaches to. The spec records what settling gateability found first: the reference pixel path needs no media decoder, no checkpoint and no GPU. Ltx2ReadFrameDirectory is already video_preprocess over a frame directory, Ltx2ConvVideoEncode already runs multi-frame for retake, and ltx2_video_fixture.h is a real reduced checkpoint that Generate runs end to end. What is missing is the reference item's own geometry, the temporal subsample, and the whole mask half. It also records the defect this row retires. Ltx2ModalityInput::attention_mask and its whole consumption chain are built and dead: the only assignments in the tree are four lines in a BuildModalities test helper, and nothing in src/ writes either field. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora-ref-video.md | 308 +++++++++++++++++++++++ 1 file changed, 308 insertions(+) create mode 100644 .agents/specs/ltx25-ic-lora-ref-video.md diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md new file mode 100644 index 000000000..06bd5ba3d --- /dev/null +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -0,0 +1,308 @@ +# `LTX25-IC-LORA-REF-VIDEO` — the IC-LoRA reference clip and its attention mask + +Issue: [#3020](https://github.com/mudler/vllm.cpp/issues/3020). +Base: `fafb58ef6`. Oracle: `Lightricks/LTX-2 @ fd4ded7f` +(`.agents/oracles/ltx-2.md`), executed, never read alone. + +Gaps **A15** and **A16** of `.agents/specs/ltx25-completion-scope.md`, landed as +one row. §1 establishes why they are one row rather than two. + +## 0. What is claimed, and what is not + +CLAIMED. An IC-LoRA render can be conditioned on a reference clip through the +public video ABI, and that clip reaches the DiT as appended tokens carrying the +reference item's own RoPE geometry. An optional per-region attention mask +reaches the DiT as an additive log-space self-attention bias. + +NOT CLAIMED. No render quality claim, no speed claim, and no real-weights +measurement: this row has no GPU authority and no LTX checkpoint. The gates are +value gates against the executed pinned module plus an end-to-end reachability +gate on the reduced-dimension checkpoint fixture in +`tests/vllm/multimodal/ltx2_video_fixture.h`. + +NOT CLAIMED EITHER. The dtype. This engine's LTX host path materialises f32 +where upstream materialises bf16, which is the standing divergence the A24 wave +campaign owns; this row mirrors the tree's f32 convention at every new site and +records the width owed rather than pretending the bytes match. See §4 R4. + +## 1. Why A15 and A16 are one row + +Upstream applies `ConditioningItemAttentionStrengthWrapper` at exactly one site, +`iclora_utils.py:168-169`, and its argument is always the +`VideoConditionByReferenceLatent` built two lines above it. +`combined_image_conditionings` (`utils/helpers.py`) never wraps, and neither +appending VIDEO item passes a mask — `keyframe_cond.py:68-76` and +`reference_video_cond.py:88-96` both pass a literal `attention_mask=None`. + +In this tree `Ltx2ConditionVideoByReference` has zero production call sites at +`fafb58ef6`: its definition at `src/vllm/model_executor/models/ltx2_conditioning.cpp:565`, +prose in `ltx2_conditioning.h`, and two test call sites. The reference arm is +refused in `src/vllm/multimodal/ltx2_video.cpp`. So A16 alone has nowhere to +attach that upstream also attaches to: putting the mask on the keyframe or image +items that ARE reached would invent behaviour upstream does not have. + +`.agents/specs/ltx25-completion-scope.md` §8 order 8 already says this — +"A15, A16, A17 — the IC-LoRA family, one owed table", M-L total. A16's +standalone "M" sizing was wrong. A17 landed earlier in the campaign. + +## 2. The dead seam this row retires + +A16's consumption half is **already built, and nothing reaches it**. Verified on +`fafb58ef6` with `grep -rn 'attention_mask' src/ include/ tests/` and a second +pass for assignments: + +``` +$ grep -rn '\.attention_mask *=\|attention_mask_rows *=' src/ include/ tests/ +src/.../ltx2_device.cpp:1097: VT_CHECK(m.attention_mask_rows == 1 || ... <- a comparison +src/.../ltx2_dit.cpp:599: VT_CHECK(m.attention_mask_rows == 1 || ... <- a comparison +tests/vllm/models/test_ltx2.cpp:282-285 <- BuildModalities, by hand +tests/vllm/models/test_ltx2_device.cpp:282-285 <- BuildModalities, by hand +include/.../ltx2.h:558 <- the declaration's default +``` + +Four assignments, all inside a test helper that constructs `Ltx2ModalityInput` +by hand. Nothing in `src/` writes either field. The chain below those +assignments is complete and correct — `Ltx2PrepareSelfAttentionMask` +(`ltx2.cpp:846`) into `self_bias` on host (`ltx2_dit.cpp:598-604`) and device +(`ltx2_device.cpp:1096-1104`), into `vt::AttentionCross`'s additive bias — which +is exactly the failure `.agents/reachability.md` names: the class works, and no +capability reaches it. + +Landing this row is the production assignment. The reachability mutation in §5 +is deleting it. + +## 3. Upstream anchors + +Root `Lightricks/LTX-2 @ fd4ded7f`. + +| ours | upstream | +|---|---| +| `Ltx2IcLoraReferenceGeometry` | `ltx-pipelines/iclora_utils.py:111-117` | +| `Ltx2TemporalSubsample` | `ltx-pipelines/iclora_utils.py:87-90` (called `:143-144`) | +| the reference item's encode | `ltx-pipelines/iclora_utils.py:141-148` | +| `Ltx2ConditionVideoByReference` (already ported) | `ltx-core/conditioning/types/reference_video_cond.py:46-108`, built at `iclora_utils.py:162-167` | +| `Ltx2DownsampleMaskVideoToLatent` | `ltx-pipelines/iclora_utils.py:52-84` | +| the strength multiply | `ltx-pipelines/iclora_utils.py:151-156` | +| `Ltx2ResolveCrossMask` | `ltx-core/conditioning/mask_utils.py:13-73` | +| `Ltx2BuildAttentionMask` | `ltx-core/conditioning/mask_utils.py:170-243` | +| `Ltx2UpdateAttentionMask` | `ltx-core/conditioning/mask_utils.py:110-167` | +| the wrapper's apply order | `ltx-core/conditioning/types/attention_strength_wrapper.py:43-71` | +| `Ltx2ReadMaskFrameDirectory` | `ltx-pipelines/ic_lora.py:511-537` (`_load_mask_video`) | +| the stage split | `ltx-pipelines/ic_lora.py:269-281` (stage 1) vs `:314-321` (stage 2) | +| the CLI shape | `ltx-pipelines/ic_lora.py:415-441`, `:452-463`, `:481-498` | +| the consumption (already ported) | `ltx-core/model/transformer/transformer_args.py:208-237`, `:289` | + +## 4. Design + +### 4.1 A15 — the reference clip + +Upstream's pixel path is `decode_video_by_frame` into `video_preprocess` +(`iclora_utils.py:141-142`). `video_preprocess` (`media_io/decode.py:82-103`) is +per frame `resize_and_center_crop(f.float(), H, W)` then `normalize_images`, +concatenated on the frame axis. **That function is already ported and gated in +this tree** as `Ltx2ReadFrameDirectory` +(`src/vllm/model_executor/models/ltx2_retake.cpp:211`), which runs each +`frame_%06d.ppm` through `Ltx2LoadImageAndPreprocess` — the same +resize-and-centre-crop plus `/127.5 - 1` chain — and returns `[C, T, H, W]`, the +layout `Ltx2ConvVideoEncode` takes. The container-versus-frame-directory +substitution is the harness adaptation row `LTX25-RETAKE` already recorded and +`video_api.cpp` already documents; no demuxer is vendored here. + +So A15's genuinely new code is small and each piece is separately gateable: + +1. `Ltx2IcLoraReferenceGeometry(height, width, scale)` — refuse when + `scale != 1` and either axis is not divisible (`:112-115`), else return + `height / scale`, `width / scale` (`:116-117`). Integer division, and the + refusal carries upstream's own wording. +2. `Ltx2TemporalSubsample(clip, channels, frames, plane, factor)` — keep index + 0, then `range(1, frames, factor)` (`:89`). Note the shape: index 1 is + ALWAYS kept when it exists, so at `factor == 2` and 5 frames the kept set is + `{0, 1, 3}`, not `{0, 2, 4}`. Guarded on `factor > 1` exactly as `:143` is. +3. Encode the whole clip with the existing `Ltx2ConvVideoEncode`, which the + retake arm already drives multi-frame at `ltx2_video.cpp:3703`. +4. Apply `Ltx2ConditionVideoByReference` with `downscale_factor = scale` and + `temporal_scale_factor` from the adapter metadata (`im.dit.lora_reference`, + already read by row `LTX25-IC-LORA`), and `strength` from the request. + +**Stage 1 only.** `ICLoraPipeline` gives stage 1 `_create_conditionings`, which +appends the reference item (`ic_lora.py:269-281`), and stage 2 plain +`combined_image_conditionings` with no reference item (`:314-321`). The phase +loop therefore applies this block on `phase_index == 0` and on no other phase. +The predicate that routes and the predicate that refuses are the same +expression, bound once in a named local, so the two cannot drift. + +### 4.2 A16 — the attention mask + +`Ltx2ReadMaskFrameDirectory(dir, height, width)` mirrors `_load_mask_video`: +read the frames through the SAME `Ltx2ReadFrameDirectory`, mean over the three +channels, remap `(x + 1) / 2`, clamp to `[0, 1]`. It is read at the STAGE-1 +resolution, which upstream spells `args.height // 2` at `ic_lora.py:460-461` +and this engine reads from the phase's own grid — the same number, derived from +the phase rather than assumed from the CLI. + +`Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, latent_shape)` +mirrors `:52-84`: area-interpolate each pixel frame to `(h_lat, w_lat)`; keep +the first latent frame as the area-downsampled PIXEL frame 0 alone; mean-pool +the remaining `f_pix - 1` frames in groups of `t = (f_pix - 1) / (f_lat - 1)`; +flatten to `(f_lat * h_lat * w_lat)`. It refuses a non-divisible pair with +upstream's assertion text (`:74-77`), and it degenerates to the first frame +alone when `f_pix == 1` or `f_lat == 1` (`:81-82`). + +The **causal carve-out is the whole content of the temporal half**: latent frame +0 is pixel frame 0 alone, not a pooled group. A uniform pooling separates from +it by 0.168 on the fixture in §5, so the golden can see it. + +Then `attn_mask = latent_mask * conditioning_attention_strength` (`:156`). + +`Ltx2ResolveCrossMask` and `Ltx2BuildAttentionMask` port `mask_utils.py` at +batch 1. The block structure is the load-bearing part: + +``` + noisy(Nn) prev_ref(N-Nn) new_ref(M) + noisy existing existing cross + prev_ref existing existing 0 + new_ref cross 0 1 +``` + +`Ltx2UpdateAttentionMask` reproduces `:141-156`: a null mask on a state that +already carries one pads the new tokens with ones rather than returning null, +which is what keeps a second reference item from shrinking the mask below the +sequence. + +The wrapper's ORDER is ported as written (`attention_strength_wrapper.py:49-71`): +snapshot the pre-item state, apply the inner item, take `num_new_tokens` as the +difference, and build the mask against the ORIGINAL token count. Building it +against the post-append count would put the new block in the wrong place while +every shape check still passed. + +`Ltx2LatentState` grows an `attention_mask` field and an +`attention_mask_rows` count. The header comment at `ltx2_conditioning.h:57-74` +that says the field is deliberately absent is rewritten rather than deleted: it +was true, this row is what makes it false, and a reader needs to know which. + +### 4.3 The production assignment + +`StreamState` carries the mask through the phase loop; `ToLatentState` and +`FromLatentState` carry it in both directions. At the DiT call site +(`ltx2_video.cpp`, where `Ltx2ModalityInput vin` is built) the mask is handed +over with its row count, guarded exactly as the `keyframes_mask` handover beside +it is: the size is CHECKED against `video.tokens` before `data()` is taken, +because an empty vector's `data()` is a null pointer and therefore upstream's +legal "no mask" — a silent drop dressed as a supported path. + +### 4.4 Request surface + +Three per-request extras, the established family-knob mechanism in this ABI +(`kLtx2RetakeStartTimeExtra` and twenty others): + +| extra | upstream | +|---|---| +| `ref_video_strength` | the STRENGTH of `--video-conditioning PATH STRENGTH` (`ic_lora.py:416-425`), default 1.0 | +| `conditioning_attention_mask_dir` | the MASK_PATH of `--conditioning-attention-mask` (`:427-441`), a `frame_%06d.ppm` directory | +| `conditioning_attention_strength` | its STRENGTH (`:452-455`), default 1.0, refused outside `[0, 1]` exactly as `ic_lora.py:230-233` | + +`ref_video_dir` already exists on `VideoGenParams`. + +## 4a. Risks + +* **R1 — a degenerate golden.** A `build_attention_mask` fixture without a PRIOR + conditioning item has `num_existing == num_noisy`, and then the true block + structure and the plausible wrong one "cross on ALL existing rows" are + ELEMENTWISE EQUAL: measured 0 separating elements. Such a golden is a mute + switch. Mitigation: the generator asserts `separating > 0` for every emitted + case and REFUSES to write the file otherwise, and the fixture carries a prior + item so the count is 4. +* **R2 — the mask is too wide to notice being wrong.** A `[T, T]` mask of ones + is the identity, and a bug that produces all-ones renders correctly. Mitigation: + the reachability gate compares a masked render against an unmasked one through + the ABI and requires the pixels to MOVE, and a separate assertion requires the + built mask to contain a value strictly between 0 and 1. +* **R3 — the routing predicate and the refusal predicate drifting apart.** The + campaign has shipped one silently wrong answer this way. Mitigation: one named + local, used by both, and a mutation that changes only the refusal. +* **R4 — dtype.** Upstream runs this whole path at bf16 (`self.dtype`, + `ic_lora.py:271-281`; `torch.bfloat16` at `ic_lora.py:530`). This tree's LTX + host path is f32 throughout, which is the A24 campaign's standing divergence. + This row mirrors the tree, names the width beside each new buffer, and records + it owed. A token gate cannot see this and neither can these goldens. +* **R5 — anchor rot.** `ltx2_video.cpp`'s READER ANCHORS comment has rotted ten + times. This row's edit shifts lines in that file. The anchors are re-derived + with the test case's own rule after the edit, and re-derived AGAIN after any + later edit in the same file. + +## 5. Tests and gates + +Every gate below is RUNNABLE from the repository root and all of them were run. + +```sh +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=OFF +cmake --build build -j 2 --target test_ltx2_iclora_reference test_ltx2_video test_ltx2 test_ltx2_pipeline +./build/tests/test_ltx2_iclora_reference +./build/tests/test_ltx2_video +./build/tests/test_ltx2 +./build/tests/test_ltx2_pipeline +ctest --test-dir build -j 2 --output-on-failure +python3 scripts/check-pr-size.py --base origin/main +python3 scripts/agent-issue-index.py --refresh && python3 scripts/check-agent-record.py +scripts/agent-preflight.sh +``` + +**The goldens are generated by executing the pinned module**, never by reading +it: `tools/ltx2/gen_iclora_reference_goldens.py` imports +`ltx_pipelines.iclora_utils` and `ltx_core.conditioning.mask_utils` from +`/home/mudler/_git/LTX-2` at `fd4ded7f` and writes +`tests/vllm/models/ltx2_iclora_reference_goldens.inc`. It emits, for every case, +upstream's answer AND the rejected hypothesis beside it, and asserts they +differ. + +Measured separations, which are what makes each case able to fail: + +| case | rejected hypothesis | separating | +|---|---|---| +| `build_attention_mask` with a prior ref | cross on ALL existing rows | 4 elements | +| the same with `num_existing == num_noisy` | the same | **0 — REFUSED by the generator** | +| `downsample_mask_video_to_latent` | bilinear spatial | 0.1890125870704651 | +| the same | uniform temporal pooling, no causal carve-out | 0.16791561245918274 | +| `_prepare_self_attention_mask` on `[1, 0.5, 0, 1e-30]` | — | `[0, -0.6931471824645996, -3.4028234663852886e+38, -69.07755279541016]` | + +**Reachability.** `test_ltx2_video` renders through the public `Generate` on the +reduced-dimension checkpoint fixture, once without a reference clip and once +with one, and requires the pixels to differ; then once more with a mask and +requires all three to differ pairwise. The mutation that must red it is deleting +the production call site, not the helper. + +## 6. Mutations + +Every claimed guarantee is mutated in a scratch copy, each mutation confirmed +APPLIED and COMPILED before its result is recorded. Eight of eight A24 waves +shipped a guarantee that mutation showed nothing measured; this row assumes the +same until each one has fired. + +## Owed + +| owed | issue | +|---|---| +| the EXR / HDR reference arm (`iclora_utils.py:120-139`), which needs an OpenEXR reader and `--hdr` colour-space resolution this tree does not have | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the scalar-only `elif conditioning_attention_strength < 1.0` branch (`iclora_utils.py:157-158`). It is unreachable from upstream's CLI, because `conditioning_attention_strength` is assigned only inside `if args.conditioning_attention_mask is not None` (`ic_lora.py:454-455`) and is otherwise 1.0, so a strength below 1 always arrives with a mask. Python-API-only; refused by name | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| `tiled_encode` for the reference clip (`iclora_utils.py:145-146`), which this engine reaches for the target but not for a reference | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the reference-AUDIO arm, unchanged by this row and still refused by name | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| the bf16 storage width of every buffer this row adds (R4) | the A24 dtype campaign | +| a real-weights IC-LoRA reference render; this row has no GPU authority and no checkpoint | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | + +## 7. Stop conditions + +* Stop and report `NEEDS_DECISION` if the reference pixel path turns out to need + a media decoder, a checkpoint, or a GPU lease. **This was settled first and it + did not fire**: `Ltx2ReadFrameDirectory` is already `video_preprocess`, + `Ltx2ConvVideoEncode` already runs multi-frame, and + `tests/vllm/multimodal/ltx2_video_fixture.h` is a real reduced checkpoint that + `Generate` runs end to end. +* Stop if a golden's generator cannot produce a case with a non-zero separation + from its rejected hypothesis. An unfalsifiable golden is not a gate. +* Stop if the reachability mutation — deleting the production call site — leaves + the suite green. That measures a class, not a capability. +* No GPU, no lease, no downloads. One compiling agent at `-j 2`. + +## Now + +`ACTIVE` — implementation on `row/LTX25-IC-LORA-REF-VIDEO`. From c121e496505c5d8192807e3b3e48ce15c20853e0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:28:32 +0000 Subject: [PATCH 02/11] test(LTX25-IC-LORA-REF-VIDEO): the red-first reachability case, and the oracle it will be gated on The case enters through LoadVideoEngine and Generate, which is where vllm_video_generate arrives, and asks for a render conditioned on a reference clip. It is red for the intended reason and the red is literal: ERROR: test case THREW exception: Unsupported LTX pipeline kind/version: 'ic_lora'/'2.5'. Recipes are resolved from an EXACT (kind, version) table The goldens beside it are produced by EXECUTING the pinned module, never by reading it. iclora_utils cannot be imported here because it reaches PyAV, so its three functions are lifted by AST and their own source text is compiled verbatim, and append_ic_lora_reference_video_conditionings is run with only its frame decode, its pixel preprocess and its VAE replaced by recording stubs, so every branch under test is upstream's own control flow. Every case carries the hypothesis it rejects and the generator REFUSES to write one whose separation is zero. That guard is not decorative: build_attention_mask on a fixture with no PRIOR conditioning item is elementwise equal to the plausible wrong reading, measured 0 separating elements, and such a golden is a mute switch. The fixture carries a prior item and separates on 4. The fixture renders at 128x128 rather than the 64x64 every other case uses, because the distilled recipe halves it for stage 1 and 64x64 would give a 1x1 spatial reference grid where row-major and column-major patchify are identical and a downscale factor of 2 collapses onto a factor of 1. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- include/vllm/multimodal/ltx2_video.h | 31 ++ scripts/gen-ltx2-iclora-reference-goldens.py | 492 ++++++++++++++++++ .../models/ltx2_iclora_reference_goldens.inc | 139 +++++ tests/vllm/multimodal/test_ltx2_video.cpp | 101 ++++ 4 files changed, 763 insertions(+) create mode 100755 scripts/gen-ltx2-iclora-reference-goldens.py create mode 100644 tests/vllm/models/ltx2_iclora_reference_goldens.inc diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 5a8671fb9..c1c6ee39d 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -1072,6 +1072,37 @@ struct Ltx2ConditioningTrace { uint64_t retake_latent_digest = 0; double retake_latent_absmax = 0.0; + // ── IC-LoRA REFERENCE VIDEO and its ATTENTION MASK ──────────────────────── + // (row LTX25-IC-LORA-REF-VIDEO, #3020 — gaps A15 and A16) + // + // Observed for the reason the retake block above is: the whole mechanism is + // APPENDED TOKENS plus an attention BIAS, and neither is visible in a frame + // count, an output resolution or a finished clip. A build that read the + // reference clip, encoded it, and then appended nothing renders a video of + // exactly the right length with exactly the right soundtrack. + // + // `ic_lora_reference_tokens` is how many tokens the reference item appended, + // measured as the sequence GROWTH across the item rather than recomputed from + // the latent shape — a count derived from the shape would agree with itself on + // a build that computed the shape and appended nothing. + int64_t ic_lora_reference_tokens = 0; + // The encoded reference latent, before any denoising. The lower bound a token + // count cannot make: a zeroed latent has the right size and the right count. + uint64_t ic_lora_reference_digest = 0; + double ic_lora_reference_absmax = 0.0; + // The self-attention STRENGTH mask actually handed to the DiT, and NOT the one + // that was built. `rows` is `Ltx2ModalityInput::attention_mask_rows` at the + // forward — 0 when no mask was handed over — so a build that constructed the + // mask and then dropped the pointer is separated from one that passed it. + // + // `min` and `max` are both recorded because an ALL-ONES mask is the identity: + // it renders correctly, it has the right shape, and it is exactly what a + // downsample that lost its values produces. A mask whose min equals its max is + // not attenuating anything. + int64_t ic_lora_attention_mask_rows = 0; + double ic_lora_attention_mask_min = 0.0; + double ic_lora_attention_mask_max = 0.0; + // ── THE SAMPLER (row LTX25-RES2S-LOOP, #921) ────────────────────────────── // // TWO COUNTERS, BECAUSE THERE ARE TWO QUESTIONS AND ONE NUMBER CANNOT ANSWER diff --git a/scripts/gen-ltx2-iclora-reference-goldens.py b/scripts/gen-ltx2-iclora-reference-goldens.py new file mode 100755 index 000000000..094a32427 --- /dev/null +++ b/scripts/gen-ltx2-iclora-reference-goldens.py @@ -0,0 +1,492 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/models/ltx2_iclora_reference_goldens.inc — the IC-LoRA +reference-clip and conditioning-attention-mask oracle. + +Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +spec .agents/specs/ltx25-ic-lora-ref-video.md. + +Every number below is what upstream's own code RETURNED. Nothing is transcribed. + +WHAT IS IMPORTED AND WHAT IS SLICED. `ltx_core.conditioning.mask_utils` and +`ltx_core.model.transformer.transformer_args` import cleanly and are imported. +`ltx_pipelines.iclora_utils` cannot be: it reaches `utils.media_io`, which +imports PyAV, a codec binding this project deliberately does not vendor. So its +three functions are lifted from the real file BY AST and their own source text is +compiled verbatim, the same technique scripts/gen-ltx2-dfr-goldens.py uses and +for the same reason. Each name is asserted present, so an upstream rename raises +here instead of leaving this generator running a local copy of the thing under +test. + +`append_ic_lora_reference_video_conditionings` is EXECUTED, with only its I/O and +its VAE stubbed: the frame decode, the pixel preprocess and the encoder are +recording stubs, and every branch — the divisibility refusal, the reference +geometry, the `if reference_temporal_scale_factor > 1` guard, the three-way mask +selection and the `if attn_mask is not None` wrap — is upstream's own control +flow running. That is what these goldens are about; the numerics of the resize +are already gated by ltx2_image_cond_goldens.inc. + +EVERY CASE CARRIES ITS REJECTED HYPOTHESIS AND THIS SCRIPT REFUSES TO EMIT A +GOLDEN THAT CANNOT FAIL. `build_attention_mask` on a fixture with no PRIOR +conditioning item is elementwise equal to the plausible wrong reading "cross on +all existing rows" — measured 0 separating elements — so such a fixture is a mute +switch. `require_separation` is called on every case and raises rather than +writing the file. + +Regenerate with: + python3 scripts/gen-ltx2-iclora-reference-goldens.py --ltx2 \ + --out tests/vllm/models/ltx2_iclora_reference_goldens.inc +""" + +from __future__ import annotations + +import argparse +import ast +import pathlib +import subprocess +import sys + + +def git_revision(root: pathlib.Path) -> str: + """The checkout's SHA, and REFUSE a dirty tree.""" + head = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, capture_output=True, text=True, + ).stdout.strip() + dirty = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, capture_output=True, text=True, + ).stdout.strip() + if dirty: + raise SystemExit( + f"REFUSING: {root} is dirty, so {head} would not describe the executed code:\n{dirty}" + ) + return head + + +def slice_functions(path: pathlib.Path, wanted: tuple[str, ...], namespace: dict) -> None: + """Compile the named top-level functions out of `path` verbatim into `namespace`.""" + source = path.read_text() + tree = ast.parse(source) + found: dict[str, ast.stmt] = {} + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in wanted: + found[node.name] = node + missing = [n for n in wanted if n not in found] + if missing: + raise SystemExit(f"REFUSING: {path} no longer defines {missing}") + lines = source.splitlines() + for name in wanted: + node = found[name] + text = "\n".join(lines[node.lineno - 1:node.end_lineno]) + exec(compile(ast.parse(text), str(path), "exec"), namespace) # noqa: S102 + + +SEPARATIONS: list[tuple[str, str, float]] = [] + + +def require_separation(case: str, rejected: str, value: float) -> None: + """A golden that agrees with the hypothesis it claims to reject is a mute switch.""" + if not (value > 0.0): + raise SystemExit( + f"REFUSING to emit '{case}': it does not separate from '{rejected}' " + f"(separation {value!r}). A golden that cannot fail is not a gate." + ) + SEPARATIONS.append((case, rejected, value)) + + +def fmt(x: float) -> str: + # `.9g` alone emits `0` for zero, which is an int literal in C++ and makes the + # array's element type depend on where the zero sits. Force a decimal point. + text = f"{float(x):.9g}" + if "." not in text and "e" not in text and "n" not in text: + text += ".0" + return text + "f" + + +def carray(name: str, values) -> str: + body = ", ".join(fmt(v) for v in values) + return f"inline constexpr float {name}[] = {{{body}}};" + + +def iarray(name: str, values) -> str: + body = ", ".join(str(int(v)) for v in values) + return f"inline constexpr int64_t {name}[] = {{{body}}};" + + +def deterministic(n: int, seed: int) -> list[float]: + """A splitmix64-derived stream in [0, 1). The C++ side reads the SAME bytes out + of this file, so the generator is free to choose any reproducible stream.""" + out = [] + state = seed & 0xFFFFFFFFFFFFFFFF + for _ in range(n): + state = (state + 0x9E3779B97F4A7C15) & 0xFFFFFFFFFFFFFFFF + z = state + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & 0xFFFFFFFFFFFFFFFF + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & 0xFFFFFFFFFFFFFFFF + z ^= z >> 31 + # 24 bits, so the value is exact in f32 and survives the round trip. + out.append((z >> 40) / float(1 << 24)) + return out + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--ltx2", required=True, type=pathlib.Path) + ap.add_argument("--out", required=True, type=pathlib.Path) + args = ap.parse_args() + + root = args.ltx2 + sys.path.insert(0, str(root / "packages" / "ltx-core" / "src")) + sys.path.insert(0, str(root / "packages" / "ltx-pipelines" / "src")) + revision = git_revision(root) + + import torch + from einops import rearrange + import torch.nn.functional as F + + from ltx_core.conditioning import ( + ConditioningItemAttentionStrengthWrapper, + VideoConditionByReferenceLatent, + ) + from ltx_core.conditioning.mask_utils import build_attention_mask, resolve_cross_mask + from ltx_core.types import VideoLatentShape + + iclora_path = root / "packages" / "ltx-pipelines" / "src" / "ltx_pipelines" / "iclora_utils.py" + ns: dict = { + "torch": torch, + "rearrange": rearrange, + "VideoLatentShape": VideoLatentShape, + "ConditioningItem": object, + "ConditioningItemAttentionStrengthWrapper": ConditioningItemAttentionStrengthWrapper, + "VideoConditionByReferenceLatent": VideoConditionByReferenceLatent, + "VideoEncoder": object, + "TilingConfig": object, + "HDRColorSpace": object, + "ResizeMode": object, + "is_exr_dir": lambda p: False, + "load_exr_folder_conditioning_hdr": None, + "annotations": None, + } + slice_functions( + iclora_path, + ("temporal_subsample", "downsample_mask_video_to_latent", + "append_ic_lora_reference_video_conditionings"), + ns, + ) + temporal_subsample = ns["temporal_subsample"] + downsample = ns["downsample_mask_video_to_latent"] + append_refs = ns["append_ic_lora_reference_video_conditionings"] + + out: list[str] = [] + E = out.append + + E("// GENERATED by scripts/gen-ltx2-iclora-reference-goldens.py — DO NOT EDIT BY HAND.") + E("//") + E("// LTX-2.5 IC-LoRA REFERENCE-CLIP and CONDITIONING-ATTENTION-MASK goldens") + E("// (row LTX25-IC-LORA-REF-VIDEO, issue #3020), produced by IMPORTING AND") + E("// EXECUTING upstream's `ltx_core.conditioning.mask_utils`, its") + E("// `_prepare_self_attention_mask`, and the three AST-sliced functions of") + E("// `ltx_pipelines.iclora_utils` (which cannot be imported: it reaches PyAV).") + E("//") + E("// Every case carries the REJECTED hypothesis beside upstream's answer, and") + E("// the generator REFUSES to write a case whose separation is zero. See the") + E("// separation table at the bottom of this file.") + E("//") + E(f"// Upstream revision (Lightricks/LTX-2): {revision}") + E("#pragma once") + E("") + E("#include ") + E("") + E("namespace vllm_test {") + E("") + E("// The suite asserts this equals the SHA it pins, so regenerating against a") + E("// DIFFERENT checkout fails the gate instead of silently replacing the oracle.") + E(f'inline constexpr const char* kLtx2IcLoraRefUpstreamRevision = "{revision}";') + E("") + + # ── 1. temporal_subsample (iclora_utils.py:87-90) ──────────────────────── + E("// --- 1. temporal_subsample (iclora_utils.py:87-90) -----------------------") + E("//") + E("// `[0, *range(1, F, N)]`. INDEX 1 IS ALWAYS KEPT when it exists, so at N=2") + E("// over 5 frames the kept set is {0, 1, 3} and NOT the plausible {0, 2, 4}.") + E("// That is the whole content of this function and the rejected hypothesis below") + E("// is exactly the plausible reading.") + ts_cases = [(5, 2), (9, 4), (7, 3), (6, 1), (1, 4)] + E(f"inline constexpr int kLtx2TemporalSubsampleCases = {len(ts_cases)};") + E(iarray("kLtx2TemporalSubsampleFrames", [f for f, _ in ts_cases])) + E(iarray("kLtx2TemporalSubsampleFactor", [n for _, n in ts_cases])) + counts, flat = [], [] + total_sep = 0 + for frames, factor in ts_cases: + # Upstream indexes a (B, C, F, H, W) tensor on dim 2. One channel, 1x1 + # pixels: the value at each kept frame IS its index, so what comes back is + # the index vector itself rather than a restatement of it. + video = torch.arange(frames, dtype=torch.float32).reshape(1, 1, frames, 1, 1) + got = temporal_subsample(video, factor).flatten().tolist() + rejected = list(range(0, frames, factor)) # "every Nth from 0" + sep = sum(1 for i in range(max(len(got), len(rejected))) + if i >= len(got) or i >= len(rejected) or got[i] != rejected[i]) + total_sep += sep + counts.append(len(got)) + flat.extend(int(v) for v in got) + require_separation("temporal_subsample", "every Nth frame from index 0 (range(0, F, N))", + float(total_sep)) + E(iarray("kLtx2TemporalSubsampleCount", counts)) + E(iarray("kLtx2TemporalSubsampleKept", flat)) + E("") + + # ── 2. the reference geometry and the branch map (iclora_utils.py:93-170) ── + E("// --- 2. append_ic_lora_reference_video_conditionings (iclora_utils.py:93-170)") + E("//") + E("// UPSTREAM'S OWN CONTROL FLOW, executed with the frame decode, the pixel") + E("// preprocess and the VAE replaced by recording stubs. What is gated here is") + E("// which geometry it asks the preprocess for, whether it subsamples, which of") + E("// the three mask branches it takes, and whether it wraps.") + + class RecordingEncoder: + def __init__(self, rec): + self.rec = rec + + def __call__(self, video): + self.rec["encoded_shape"] = tuple(video.shape) + # A latent volume with the shape the caller will read back through + # VideoLatentShape.from_torch_shape: (B, C, F, H, W). + return torch.zeros(1, 4, 2, 2, 2) + + def run_append(height, width, scale, temporal, strength, mask, cond_strength): + rec: dict = {} + ns2 = dict(ns) + ns2["decode_video_by_frame"] = lambda path, frame_cap, device: iter(()) + def preprocess(frames, h, w, dtype, device): + rec["ref_height"], rec["ref_width"] = h, w + # 5 pixel frames of 1x1, value == frame index, so a subsample is visible. + return torch.arange(5, dtype=torch.float32).reshape(1, 1, 5, 1, 1) + ns2["video_preprocess"] = preprocess + slice_functions(iclora_path, ("append_ic_lora_reference_video_conditionings",), ns2) + conds: list = [] + ns2["append_ic_lora_reference_video_conditionings"]( + conds, [("ref.mp4", strength)], + height=height, width=width, num_frames=5, + video_encoder=RecordingEncoder(rec), dtype=torch.float32, + device=torch.device("cpu"), + reference_downscale_factor=scale, + reference_temporal_scale_factor=temporal, + conditioning_attention_strength=cond_strength, + conditioning_attention_mask=mask, + ) + rec["wrapped"] = isinstance(conds[0], ConditioningItemAttentionStrengthWrapper) + inner = conds[0].conditioning if rec["wrapped"] else conds[0] + rec["downscale"] = inner.downscale_factor + rec["temporal"] = inner.temporal_scale_factor + rec["strength"] = inner.strength + return rec + + # (a) the geometry: 448x768 at scale 4 -> the preprocess is asked for 112x192. + geo = run_append(448, 768, 4, 1, 0.75, None, 1.0) + E(f"inline constexpr int64_t kLtx2RefGeomHeight = {geo['ref_height']};") + E(f"inline constexpr int64_t kLtx2RefGeomWidth = {geo['ref_width']};") + require_separation("reference geometry at scale 4", + "the phase's own height and width (no downscale)", + float(abs(geo["ref_height"] - 448) + abs(geo["ref_width"] - 768))) + E("// The item carries the factors, and the request's strength, unchanged.") + E(f"inline constexpr int64_t kLtx2RefItemDownscale = {geo['downscale']};") + E(f"inline constexpr float kLtx2RefItemStrength = {fmt(geo['strength'])};") + E(f"inline constexpr bool kLtx2RefGeomWrapped = {'true' if geo['wrapped'] else 'false'};") + + # (b) the temporal-subsample guard: it fires at N>1 and NOT at N==1. + sub = run_append(64, 64, 1, 2, 1.0, None, 1.0) + nosub = run_append(64, 64, 1, 1, 1.0, None, 1.0) + E("// The subsample guard (`if reference_temporal_scale_factor > 1`, :143). The") + E("// encoder sees 3 of 5 frames at N=2 and all 5 at N=1 — the encoded SHAPE is") + E("// what records it, so a port that subsampled unconditionally is separated.") + E(f"inline constexpr int64_t kLtx2RefEncodedFramesN2 = {sub['encoded_shape'][2]};") + E(f"inline constexpr int64_t kLtx2RefEncodedFramesN1 = {nosub['encoded_shape'][2]};") + require_separation("the temporal-subsample guard", + "subsampling unconditionally", + float(abs(sub["encoded_shape"][2] - nosub["encoded_shape"][2]))) + + # (c) the three-way mask branch (iclora_utils.py:151-160), and the wrap. + scalar_lt1 = run_append(64, 64, 1, 1, 1.0, None, 0.5) + scalar_eq1 = run_append(64, 64, 1, 1, 1.0, None, 1.0) + E("// The three-way mask selection (:151-160). With NO mask and strength < 1 the") + E("// scalar arm wraps; with NO mask and strength == 1 nothing wraps. The CLI") + E("// cannot reach the first (a strength is only ever set alongside a mask,") + E("// ic_lora.py:452-455), and this engine refuses it by name — the golden is") + E("// here so the refusal is pinned to a MEASURED branch rather than a reading.") + E(f"inline constexpr bool kLtx2RefWrapScalarBelowOne = {'true' if scalar_lt1['wrapped'] else 'false'};") + E(f"inline constexpr bool kLtx2RefWrapScalarAtOne = {'true' if scalar_eq1['wrapped'] else 'false'};") + require_separation("the strength<1 wrap branch", "wrapping unconditionally", + float(scalar_lt1["wrapped"] != scalar_eq1["wrapped"])) + + # (d) the divisibility refusal (:112-115). + try: + run_append(448, 770, 4, 1, 1.0, None, 1.0) + raise SystemExit("REFUSING: upstream accepted an indivisible target; the refusal moved") + except ValueError as e: + refusal = str(e) + E("// The divisibility refusal (:112-115), verbatim from the raised ValueError.") + E(f'inline constexpr const char* kLtx2RefDivisibilityRefusal = "{refusal}";') + # scale == 1 never refuses, whatever the dimensions. + ok = run_append(449, 771, 1, 1, 1.0, None, 1.0) + E(f"inline constexpr int64_t kLtx2RefScaleOneHeight = {ok['ref_height']};") + require_separation("the divisibility refusal", "refusing whenever height % scale != 0, scale==1 included", + float(ok["ref_height"] == 449)) + E("") + + # ── 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ──────────── + E("// --- 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ----------") + E("//") + E("// AREA spatial interpolation, then the CAUSAL carve-out: latent frame 0 is") + E("// pixel frame 0 ALONE, and the remaining f_pix-1 frames mean-pool in groups") + E("// of t = (f_pix-1)/(f_lat-1). Both rejected hypotheses below are measured.") + f_pix, h_pix, w_pix = 9, 8, 8 + f_lat, h_lat, w_lat = 3, 2, 2 + vals = deterministic(f_pix * h_pix * w_pix, 0x5EED1C10) + mask = torch.tensor(vals, dtype=torch.float32).reshape(1, 1, f_pix, h_pix, w_pix) + shape = VideoLatentShape.from_torch_shape((1, 1, f_lat, h_lat, w_lat)) + got = downsample(mask, shape) + t = (f_pix - 1) // (f_lat - 1) + bil = F.interpolate(rearrange(mask, "b 1 f h w -> (b f) 1 h w"), + size=(h_lat, w_lat), mode="bilinear", align_corners=False) + bil = rearrange(bil, "(b f) 1 h w -> b 1 f h w", b=1) + bil_rest = rearrange(bil[:, :, 1:], "b 1 (f t) h w -> b 1 f t h w", t=t).mean(3) + bil_flat = rearrange(torch.cat([bil[:, :, :1], bil_rest], 2), "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent", "bilinear spatial interpolation", + float((got - bil_flat).abs().max())) + area = rearrange(F.interpolate(rearrange(mask, "b 1 f h w -> (b f) 1 h w"), + size=(h_lat, w_lat), mode="area"), + "(b f) 1 h w -> b 1 f h w", b=1) + # The plausible wrong reading: pool ALL f_pix frames uniformly into f_lat + # groups, with no first-frame carve-out. f_pix must divide by f_lat for the + # hypothesis to be expressible at all, which the fixture arranges. + assert f_pix % f_lat == 0, "the no-carve-out hypothesis needs f_pix % f_lat == 0" + unif = rearrange(rearrange(area, "b 1 (f t) h w -> b 1 f t h w", t=f_pix // f_lat).mean(3), + "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent", + "uniform temporal pooling with no causal first-frame carve-out", + float((got[:, : unif.shape[1]] - unif).abs().max())) + E(f"inline constexpr int64_t kLtx2MaskPixFrames = {f_pix};") + E(f"inline constexpr int64_t kLtx2MaskPixHeight = {h_pix};") + E(f"inline constexpr int64_t kLtx2MaskPixWidth = {w_pix};") + E(f"inline constexpr int64_t kLtx2MaskLatFrames = {f_lat};") + E(f"inline constexpr int64_t kLtx2MaskLatHeight = {h_lat};") + E(f"inline constexpr int64_t kLtx2MaskLatWidth = {w_lat};") + E("// The INPUT, so both sides read identical bytes rather than agreeing about a") + E("// random stream. Every value is exact in f32 (24 significant bits).") + E(carray("kLtx2MaskPixels", vals)) + E(carray("kLtx2MaskLatentWeights", got.flatten().tolist())) + E("// The two rejected answers, emitted so the suite asserts it is NOT them.") + E(carray("kLtx2MaskLatentBilinear", bil_flat.flatten().tolist())) + E(carray("kLtx2MaskLatentUniformPool", unif.flatten().tolist())) + E("") + + # ── 4. resolve_cross_mask + build_attention_mask (mask_utils.py) ────────── + E("// --- 4. build_attention_mask (mask_utils.py:170-243) ---------------------") + E("//") + E("// THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN ON PURPOSE. With") + E("// num_existing == num_noisy the true block structure and the plausible wrong") + E("// reading `cross on ALL existing rows` are ELEMENTWISE EQUAL — measured 0") + E("// separating elements — and the golden would be a mute switch. The generator") + E("// refuses that case; this one separates on 4 elements.") + n_noisy, n_new, n_existing = 2, 2, 3 + cross_vals = [0.25, 0.75] + cross = torch.tensor([cross_vals], dtype=torch.float32) + got_mask = build_attention_mask(None, n_noisy, n_new, n_existing, cross, + torch.device("cpu"), torch.float32) + alt = got_mask.clone() + alt[:, n_noisy:n_existing, n_existing:] = cross + alt[:, n_existing:, n_noisy:n_existing] = cross.T + require_separation("build_attention_mask", "cross on ALL existing rows, not only the noisy ones", + float((got_mask != alt).sum())) + E(f"inline constexpr int64_t kLtx2AmNoisy = {n_noisy};") + E(f"inline constexpr int64_t kLtx2AmNew = {n_new};") + E(f"inline constexpr int64_t kLtx2AmExisting = {n_existing};") + E(carray("kLtx2AmCross", cross_vals)) + E(carray("kLtx2AmMask", got_mask.flatten().tolist())) + E(carray("kLtx2AmMaskCrossOnAllRows", alt.flatten().tolist())) + + E("// A SECOND item on top of the first: `existing_mask` is preserved in the") + E("// top-left block rather than refilled with ones (`:223-226`), which is what") + E("// keeps the first item's attenuation alive after a second one is applied.") + cross2_vals = [0.5] + cross2 = torch.tensor([cross2_vals], dtype=torch.float32) + got2 = build_attention_mask(got_mask, n_noisy, 1, n_existing + n_new, cross2, + torch.device("cpu"), torch.float32) + alt2 = build_attention_mask(None, n_noisy, 1, n_existing + n_new, cross2, + torch.device("cpu"), torch.float32) + require_separation("build_attention_mask with an existing mask", + "discarding the existing mask and refilling the block with ones", + float((got2 - alt2).abs().max())) + E(carray("kLtx2AmCross2", cross2_vals)) + E(carray("kLtx2AmMask2", got2.flatten().tolist())) + E(carray("kLtx2AmMask2Refilled", alt2.flatten().tolist())) + + E("// resolve_cross_mask's scalar arm (`:31-37`) and its 1-D broadcast (`:49-54`).") + sc = resolve_cross_mask(0.5, 4, 1, torch.device("cpu"), torch.float32) + E(carray("kLtx2CrossScalar", sc.flatten().tolist())) + oned = resolve_cross_mask(torch.tensor([0.1, 0.2, 0.3, 0.4]), 4, 1, + torch.device("cpu"), torch.float32) + E(carray("kLtx2CrossOneD", oned.flatten().tolist())) + require_separation("resolve_cross_mask's 1-D arm", "the scalar fill", + float((sc - oned).abs().max())) + + E("// update_attention_mask's PAD-WITH-ONES arm (`:141-156`): a null mask on a") + E("// state that already carries one grows the mask with full attention rather") + E("// than returning None and leaving it short of the sequence.") + ones = torch.ones(1, 1, dtype=torch.float32) + padded = build_attention_mask(got_mask, n_noisy, 1, n_existing + n_new, ones, + torch.device("cpu"), torch.float32) + E(carray("kLtx2AmMaskPaddedOnes", padded.flatten().tolist())) + require_separation("update_attention_mask's pad-with-ones arm", + "returning the mask unchanged (short of the sequence)", + float(padded.numel() - got_mask.numel())) + E("") + + # ── 5. _prepare_self_attention_mask (transformer_args.py:208-237) ───────── + E("// --- 5. _prepare_self_attention_mask (transformer_args.py:208-237) -------") + E("//") + E("// The CONSUMPTION seam this row makes reachable. Already ported as") + E("// Ltx2PrepareSelfAttentionMask; pinned here because nothing in src/ assigned") + E("// Ltx2ModalityInput::attention_mask before this row, so these numbers were") + E("// gated only through a hand-built test struct.") + import inspect + import ltx_core.model.transformer.transformer_args as ta + owner = None + for _name, obj in vars(ta).items(): + if inspect.isclass(obj) and "_prepare_self_attention_mask" in vars(obj): + owner = obj + break + if owner is None: + raise SystemExit("REFUSING: no class in transformer_args defines _prepare_self_attention_mask") + prep = owner._prepare_self_attention_mask + probe = [1.0, 0.5, 0.0, 1e-30] + bias = prep(None, torch.tensor([[probe]], dtype=torch.float32), torch.float32) + vals_b = [float(x) for x in bias.flatten()] + E(f'inline constexpr const char* kLtx2SelfMaskOwner = "{owner.__name__}";') + E(carray("kLtx2SelfMaskProbe", probe)) + E("// f32. The third entry is finfo(f32).min, NOT -inf and NOT 0; the fourth is") + E("// log(tiny) after the clamp, NOT log(1e-30).") + E(carray("kLtx2SelfMaskBias", vals_b)) + naive = [float(torch.log(torch.tensor(max(v, 0.0)))) if v > 0 else float("-inf") for v in probe] + require_separation("_prepare_self_attention_mask", "-inf for a zero and an unclamped log", + float(max(abs(a - b) for a, b in zip(vals_b, naive) if b != float("-inf")) + + (1.0 if any(b == float("-inf") for b in naive) else 0.0))) + E("") + + # ── the separation table ───────────────────────────────────────────────── + E("// --- separation table ----------------------------------------------------") + E("//") + E("// Every case above, its rejected hypothesis, and the measured separation. A") + E("// zero here is a mute switch and the generator raises rather than writing it.") + for case, rejected, value in SEPARATIONS: + E(f"// {case} vs {rejected} -> {value!r}") + E("") + E("} // namespace vllm_test") + + args.out.write_text("\n".join(out) + "\n") + print(f"wrote {args.out} ({len(out)} lines), {len(SEPARATIONS)} separated cases") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/vllm/models/ltx2_iclora_reference_goldens.inc b/tests/vllm/models/ltx2_iclora_reference_goldens.inc new file mode 100644 index 000000000..2379a0d1b --- /dev/null +++ b/tests/vllm/models/ltx2_iclora_reference_goldens.inc @@ -0,0 +1,139 @@ +// GENERATED by scripts/gen-ltx2-iclora-reference-goldens.py — DO NOT EDIT BY HAND. +// +// LTX-2.5 IC-LoRA REFERENCE-CLIP and CONDITIONING-ATTENTION-MASK goldens +// (row LTX25-IC-LORA-REF-VIDEO, issue #3020), produced by IMPORTING AND +// EXECUTING upstream's `ltx_core.conditioning.mask_utils`, its +// `_prepare_self_attention_mask`, and the three AST-sliced functions of +// `ltx_pipelines.iclora_utils` (which cannot be imported: it reaches PyAV). +// +// Every case carries the REJECTED hypothesis beside upstream's answer, and +// the generator REFUSES to write a case whose separation is zero. See the +// separation table at the bottom of this file. +// +// Upstream revision (Lightricks/LTX-2): fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca +#pragma once + +#include + +namespace vllm_test { + +// The suite asserts this equals the SHA it pins, so regenerating against a +// DIFFERENT checkout fails the gate instead of silently replacing the oracle. +inline constexpr const char* kLtx2IcLoraRefUpstreamRevision = "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +// --- 1. temporal_subsample (iclora_utils.py:87-90) ----------------------- +// +// `[0, *range(1, F, N)]`. INDEX 1 IS ALWAYS KEPT when it exists, so at N=2 +// over 5 frames the kept set is {0, 1, 3} and NOT the plausible {0, 2, 4}. +// That is the whole content of this function and the rejected hypothesis below +// is exactly the plausible reading. +inline constexpr int kLtx2TemporalSubsampleCases = 5; +inline constexpr int64_t kLtx2TemporalSubsampleFrames[] = {5, 9, 7, 6, 1}; +inline constexpr int64_t kLtx2TemporalSubsampleFactor[] = {2, 4, 3, 1, 4}; +inline constexpr int64_t kLtx2TemporalSubsampleCount[] = {3, 3, 3, 6, 1}; +inline constexpr int64_t kLtx2TemporalSubsampleKept[] = {0, 1, 3, 0, 1, 5, 0, 1, 4, 0, 1, 2, 3, 4, 5, 0}; + +// --- 2. append_ic_lora_reference_video_conditionings (iclora_utils.py:93-170) +// +// UPSTREAM'S OWN CONTROL FLOW, executed with the frame decode, the pixel +// preprocess and the VAE replaced by recording stubs. What is gated here is +// which geometry it asks the preprocess for, whether it subsamples, which of +// the three mask branches it takes, and whether it wraps. +inline constexpr int64_t kLtx2RefGeomHeight = 112; +inline constexpr int64_t kLtx2RefGeomWidth = 192; +// The item carries the factors, and the request's strength, unchanged. +inline constexpr int64_t kLtx2RefItemDownscale = 4; +inline constexpr float kLtx2RefItemStrength = 0.75f; +inline constexpr bool kLtx2RefGeomWrapped = false; +// The subsample guard (`if reference_temporal_scale_factor > 1`, :143). The +// encoder sees 3 of 5 frames at N=2 and all 5 at N=1 — the encoded SHAPE is +// what records it, so a port that subsampled unconditionally is separated. +inline constexpr int64_t kLtx2RefEncodedFramesN2 = 3; +inline constexpr int64_t kLtx2RefEncodedFramesN1 = 5; +// The three-way mask selection (:151-160). With NO mask and strength < 1 the +// scalar arm wraps; with NO mask and strength == 1 nothing wraps. The CLI +// cannot reach the first (a strength is only ever set alongside a mask, +// ic_lora.py:452-455), and this engine refuses it by name — the golden is +// here so the refusal is pinned to a MEASURED branch rather than a reading. +inline constexpr bool kLtx2RefWrapScalarBelowOne = true; +inline constexpr bool kLtx2RefWrapScalarAtOne = false; +// The divisibility refusal (:112-115), verbatim from the raised ValueError. +inline constexpr const char* kLtx2RefDivisibilityRefusal = "Output dimensions (448x770) must be divisible by reference_downscale_factor (4)"; +inline constexpr int64_t kLtx2RefScaleOneHeight = 449; + +// --- 3. downsample_mask_video_to_latent (iclora_utils.py:52-84) ---------- +// +// AREA spatial interpolation, then the CAUSAL carve-out: latent frame 0 is +// pixel frame 0 ALONE, and the remaining f_pix-1 frames mean-pool in groups +// of t = (f_pix-1)/(f_lat-1). Both rejected hypotheses below are measured. +inline constexpr int64_t kLtx2MaskPixFrames = 9; +inline constexpr int64_t kLtx2MaskPixHeight = 8; +inline constexpr int64_t kLtx2MaskPixWidth = 8; +inline constexpr int64_t kLtx2MaskLatFrames = 3; +inline constexpr int64_t kLtx2MaskLatHeight = 2; +inline constexpr int64_t kLtx2MaskLatWidth = 2; +// The INPUT, so both sides read identical bytes rather than agreeing about a +// random stream. Every value is exact in f32 (24 significant bits). +inline constexpr float kLtx2MaskPixels[] = {0.224978805f, 0.738994956f, 0.247904778f, 0.856395721f, 0.161342263f, 0.304656029f, 0.226417065f, 0.629713774f, 0.621277452f, 0.243836701f, 0.929327548f, 0.254041076f, 0.541148961f, 0.463537216f, 0.940397978f, 0.709314048f, 0.693672299f, 0.542568564f, 0.269681811f, 0.96947515f, 0.621686339f, 0.700837612f, 0.620844126f, 0.845988333f, 0.941008925f, 0.100188732f, 0.835158408f, 0.0902274251f, 0.157826722f, 0.0348206162f, 0.0689323545f, 0.566456556f, 0.455129385f, 0.770264328f, 0.082477808f, 0.414557874f, 0.892665267f, 0.825091422f, 0.233238816f, 0.891862512f, 0.755941391f, 0.525208294f, 0.812666357f, 0.408799589f, 0.933392882f, 0.328110039f, 0.123371542f, 0.54055661f, 0.112813234f, 0.00529891253f, 0.10176909f, 0.0247758627f, 0.0249340534f, 0.303246439f, 0.804710567f, 0.538730443f, 0.394230783f, 0.116368294f, 0.654279828f, 0.797493339f, 0.992072105f, 0.222010791f, 0.323121369f, 0.684363902f, 0.456148565f, 0.329604149f, 0.150325954f, 0.211824954f, 0.857371747f, 0.218813181f, 0.627006412f, 0.716362476f, 0.411796451f, 0.733595669f, 0.942894995f, 0.146117151f, 0.794231236f, 0.246105671f, 0.453785658f, 0.402912438f, 0.291079164f, 0.745548785f, 0.166508615f, 0.189101517f, 0.801723778f, 0.589329779f, 0.502525091f, 0.237636983f, 0.757415771f, 0.040210247f, 0.483881652f, 0.6307953f, 0.0602877736f, 0.667216182f, 0.588570833f, 0.560185254f, 0.975480795f, 0.113663495f, 0.428635895f, 0.132393956f, 0.0268669724f, 0.453999043f, 0.458613455f, 0.694886446f, 0.754171073f, 0.454550207f, 0.18075633f, 0.112146139f, 0.675278902f, 0.268378139f, 0.861036241f, 0.692431509f, 0.287955284f, 0.283611059f, 0.437588453f, 0.275979996f, 0.627004325f, 0.630954862f, 0.924760938f, 0.0718417764f, 0.355810046f, 0.67786628f, 0.880366683f, 0.30253154f, 0.786205053f, 0.745979965f, 0.576954007f, 0.0308310986f, 0.130777419f, 0.597290516f, 0.5143255f, 0.746656358f, 0.0455746055f, 0.672296584f, 0.713463485f, 0.191987753f, 0.540562212f, 0.21183306f, 0.912392437f, 0.867345572f, 0.215253115f, 0.353586614f, 0.892430186f, 0.885174572f, 0.390232682f, 0.157626808f, 0.0664351583f, 0.47427243f, 0.0192090273f, 0.0610715151f, 0.856101871f, 0.202588677f, 0.844750762f, 0.437463284f, 0.615182161f, 0.535516262f, 0.65188539f, 0.386852443f, 0.440248549f, 0.140840232f, 0.751649737f, 0.69611603f, 0.116815269f, 0.634107947f, 0.0645769238f, 0.242201626f, 0.191012144f, 0.362300277f, 0.510971785f, 0.577358842f, 0.345588863f, 0.389454961f, 0.720236659f, 0.852962494f, 0.375399411f, 0.593073785f, 0.196979284f, 0.273119688f, 0.133840621f, 0.159698129f, 0.635126412f, 0.242724895f, 0.678991795f, 0.199027658f, 0.196128547f, 0.00642746687f, 0.677461445f, 0.536804438f, 0.499773204f, 0.718390286f, 0.51330483f, 0.837298334f, 0.770378292f, 0.492825508f, 0.0426640511f, 0.91784507f, 0.583577335f, 0.577757061f, 0.546435297f, 0.162060916f, 0.0517340899f, 0.797736764f, 0.0987977386f, 0.0808457732f, 0.371301949f, 0.204327643f, 0.461616695f, 0.74985528f, 0.604997396f, 0.739948332f, 0.761415303f, 0.489027202f, 0.524555445f, 0.535394847f, 0.745264113f, 0.327727675f, 0.494373322f, 0.106823564f, 0.18626678f, 0.946658432f, 0.758277535f, 0.121624708f, 0.0415190458f, 0.347359002f, 0.276930034f, 0.394188046f, 0.581585288f, 0.380344629f, 0.893188417f, 0.580474913f, 0.309349656f, 0.178186297f, 0.242652476f, 0.305642009f, 0.0238131285f, 0.159330785f, 0.150969028f, 0.460921586f, 0.194989204f, 0.870569885f, 0.0330894589f, 0.0111231804f, 0.683476746f, 0.326002181f, 0.245743096f, 0.064491272f, 0.644147575f, 0.000202894211f, 0.275570333f, 0.963738084f, 0.366046369f, 0.618974566f, 0.170288444f, 0.774100304f, 0.924566567f, 0.909448624f, 0.0803639889f, 0.306199133f, 0.606006145f, 0.985461771f, 0.841015458f, 0.416215777f, 0.697441697f, 0.574185193f, 0.298144281f, 0.239260674f, 0.80518645f, 0.351780355f, 0.119580925f, 0.0518564582f, 0.897045553f, 0.349761903f, 0.583748102f, 0.382675707f, 0.0559566021f, 0.672386229f, 0.290330529f, 0.860345006f, 0.593835175f, 0.0300983191f, 0.296267509f, 0.646880746f, 0.519856811f, 0.543841064f, 0.994504869f, 0.0849536061f, 0.755463958f, 0.993644476f, 0.209379613f, 0.594450593f, 0.484336495f, 0.170750082f, 0.993458867f, 0.925972521f, 0.266003489f, 0.660282254f, 0.586643994f, 0.778126061f, 0.0565010309f, 0.584053874f, 0.627667725f, 0.223118246f, 0.235853374f, 0.988708973f, 0.24592489f, 0.892781913f, 0.117504597f, 0.5365839f, 0.405869424f, 0.673124969f, 0.702657282f, 0.360488355f, 0.245491028f, 0.159264743f, 0.858900845f, 0.061032176f, 0.519282579f, 0.363428295f, 0.373078167f, 0.94595331f, 0.175679445f, 0.718593955f, 0.234594822f, 0.457112908f, 0.816635609f, 0.130899251f, 0.503757238f, 0.105866075f, 0.0278913975f, 0.882681727f, 0.777195275f, 0.0241299868f, 0.836323261f, 0.00590604544f, 0.733849347f, 0.87148279f, 0.475699008f, 0.363211811f, 0.407663643f, 0.787238002f, 0.604611456f, 0.468820512f, 0.969316483f, 0.995938003f, 0.272820413f, 0.461982131f, 0.0572282672f, 0.08695364f, 0.765583396f, 0.0331178904f, 0.444453776f, 0.745676398f, 0.207386196f, 0.741251171f, 0.979560256f, 0.875282645f, 0.425274253f, 0.100177765f, 0.314046502f, 0.686280549f, 0.112682164f, 0.694772184f, 0.761486769f, 0.307273686f, 0.0175738931f, 0.558968961f, 0.567652822f, 0.832921207f, 0.0997376442f, 0.451044917f, 0.212615252f, 0.195685387f, 0.499477386f, 0.347878397f, 0.577536881f, 0.955278218f, 0.912000775f, 0.667354465f, 0.906242669f, 0.249278307f, 0.404088557f, 0.709416687f, 0.577703059f, 0.616618574f, 0.774920166f, 0.751746535f, 0.117762864f, 0.513761044f, 0.878428102f, 0.551457703f, 0.762865603f, 0.663778841f, 0.6750741f, 0.482571781f, 0.311710536f, 0.0577930808f, 0.942345262f, 0.430845559f, 0.0900357366f, 0.818745911f, 0.591225147f, 0.448996902f, 0.0356302261f, 0.346290588f, 0.980599225f, 0.972895503f, 0.84816581f, 0.930217624f, 0.0287508368f, 0.0362402797f, 0.305136859f, 0.500912428f, 0.23494029f, 0.603680074f, 0.938094497f, 0.625056803f, 0.509535789f, 0.342275202f, 0.367460787f, 0.504711986f, 0.143217921f, 0.871813238f, 0.149221659f, 0.697380841f, 0.325163662f, 0.905375302f, 0.90978688f, 0.327758372f, 0.735852122f, 0.908246279f, 0.616322279f, 0.447793245f, 0.90318352f, 0.800132334f, 0.797131181f, 0.296567619f, 0.493443847f, 0.485734046f, 0.217474163f, 0.709294379f, 0.708687544f, 0.669715464f, 0.706661522f, 0.699035525f, 0.189122796f, 0.449640393f, 0.409109354f, 0.0466797948f, 0.480667889f, 0.237748444f, 0.886014342f, 0.204600096f, 0.818930268f, 0.173705816f, 0.622837245f, 0.103790164f, 0.904882073f, 0.743250608f, 0.227131367f, 0.0243018866f, 0.929496706f, 0.99599576f, 0.850975215f, 0.381912231f, 0.661735356f, 0.720816493f, 0.370992064f, 0.0936429501f, 0.870625913f, 0.399675965f, 0.0414492488f, 0.78073287f, 0.977315605f, 0.481281281f, 0.398207009f, 0.628759801f, 0.882363558f, 0.47795701f, 0.324389875f, 0.600753963f, 0.17456162f, 0.477541447f, 0.849557698f, 0.502795577f, 0.653657317f, 0.92385453f, 0.246045411f, 0.648090661f, 0.0513747931f, 0.209317505f, 0.427504122f, 0.23591435f, 0.699629843f, 0.970116138f, 0.287622392f, 0.726968825f, 0.343225002f, 0.958356082f, 0.962094307f, 0.71631515f, 0.504042327f, 0.0353083611f, 0.287226319f, 0.0955541134f, 0.689278305f, 0.609330833f, 0.26156348f, 0.909175575f, 0.604726076f, 0.910757482f, 0.881617069f, 0.454441786f, 0.623414814f, 0.0372434258f, 0.515186727f, 0.85244137f, 0.209577382f, 0.365545928f, 0.0413528085f, 0.133180678f, 0.821608067f, 0.181242228f, 0.0989111066f, 0.0549529195f, 0.0250368714f, 0.981382191f, 0.828009486f, 0.176285863f, 0.404071748f, 0.730219722f, 0.72375077f, 0.337819993f, 0.583175123f, 0.00325196981f, 0.103610337f, 0.137881756f, 0.454132259f, 0.638733625f, 0.973367333f, 0.390789509f, 0.460909188f, 0.0850528479f, 0.142819703f, 0.831980586f, 0.827947915f, 0.23010689f, 0.396763444f, 0.205432773f, 0.691007018f, 0.663131058f, 0.907645226f, 0.234843791f, 0.946535945f, 0.0815069079f, 0.297047138f, 0.54718852f, 0.599617898f, 0.672928989f, 0.310182512f, 0.945918143f, 0.260028422f, 0.626050293f, 0.722653747f, 0.291521311f, 0.0546925068f, 0.370481193f, 0.828670382f, 0.181400359f, 0.856415391f, 0.311807752f, 0.211450398f, 0.231483996f, 0.175142884f, 0.101020813f, 0.671769917f, 0.967861652f}; +inline constexpr float kLtx2MaskLatentWeights[] = {0.534921169f, 0.474619985f, 0.402004659f, 0.541342437f, 0.463841647f, 0.479086936f, 0.391879141f, 0.513890743f, 0.438569248f, 0.541703641f, 0.524510682f, 0.516675234f}; +// The two rejected answers, emitted so the suite asserts it is NOT them. +inline constexpr float kLtx2MaskLatentBilinear[] = {0.496353656f, 0.681404233f, 0.361235648f, 0.389859647f, 0.488613337f, 0.519038856f, 0.347211421f, 0.502157032f, 0.469123542f, 0.545261562f, 0.613554657f, 0.502646863f}; +inline constexpr float kLtx2MaskLatentUniformPool[] = {0.485171884f, 0.471803099f, 0.401835531f, 0.519039631f, 0.440979809f, 0.513357699f, 0.429122686f, 0.517239153f, 0.45536992f, 0.534099936f, 0.524896443f, 0.518256605f}; + +// --- 4. build_attention_mask (mask_utils.py:170-243) --------------------- +// +// THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN ON PURPOSE. With +// num_existing == num_noisy the true block structure and the plausible wrong +// reading `cross on ALL existing rows` are ELEMENTWISE EQUAL — measured 0 +// separating elements — and the golden would be a mute switch. The generator +// refuses that case; this one separates on 4 elements. +inline constexpr int64_t kLtx2AmNoisy = 2; +inline constexpr int64_t kLtx2AmNew = 2; +inline constexpr int64_t kLtx2AmExisting = 3; +inline constexpr float kLtx2AmCross[] = {0.25f, 0.75f}; +inline constexpr float kLtx2AmMask[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f}; +inline constexpr float kLtx2AmMaskCrossOnAllRows[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.25f, 0.25f, 0.25f, 1.0f, 1.0f, 0.75f, 0.75f, 0.75f, 1.0f, 1.0f}; +// A SECOND item on top of the first: `existing_mask` is preserved in the +// top-left block rather than refilled with ones (`:223-226`), which is what +// keeps the first item's attenuation alive after a second one is applied. +inline constexpr float kLtx2AmCross2[] = {0.5f}; +inline constexpr float kLtx2AmMask2[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.5f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 0.5f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f, 0.0f, 0.5f, 0.5f, 0.0f, 0.0f, 0.0f, 1.0f}; +inline constexpr float kLtx2AmMask2Refilled[] = {1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.5f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.5f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.5f, 0.5f, 0.0f, 0.0f, 0.0f, 1.0f}; +// resolve_cross_mask's scalar arm (`:31-37`) and its 1-D broadcast (`:49-54`). +inline constexpr float kLtx2CrossScalar[] = {0.5f, 0.5f, 0.5f, 0.5f}; +inline constexpr float kLtx2CrossOneD[] = {0.100000001f, 0.200000003f, 0.300000012f, 0.400000006f}; +// update_attention_mask's PAD-WITH-ONES arm (`:141-156`): a null mask on a +// state that already carries one grows the mask with full attention rather +// than returning None and leaving it short of the sequence. +inline constexpr float kLtx2AmMaskPaddedOnes[] = {1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 1.0f, 0.25f, 0.75f, 1.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.25f, 0.25f, 0.0f, 1.0f, 1.0f, 0.0f, 0.75f, 0.75f, 0.0f, 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f}; + +// --- 5. _prepare_self_attention_mask (transformer_args.py:208-237) ------- +// +// The CONSUMPTION seam this row makes reachable. Already ported as +// Ltx2PrepareSelfAttentionMask; pinned here because nothing in src/ assigned +// Ltx2ModalityInput::attention_mask before this row, so these numbers were +// gated only through a hand-built test struct. +inline constexpr const char* kLtx2SelfMaskOwner = "TransformerArgsPreprocessor"; +inline constexpr float kLtx2SelfMaskProbe[] = {1.0f, 0.5f, 0.0f, 1e-30f}; +// f32. The third entry is finfo(f32).min, NOT -inf and NOT 0; the fourth is +// log(tiny) after the clamp, NOT log(1e-30). +inline constexpr float kLtx2SelfMaskBias[] = {0.0f, -0.693147182f, -3.40282347e+38f, -69.0775528f}; + +// --- separation table ---------------------------------------------------- +// +// Every case above, its rejected hypothesis, and the measured separation. A +// zero here is a mute switch and the generator raises rather than writing it. +// temporal_subsample vs every Nth frame from index 0 (range(0, F, N)) -> 6.0 +// reference geometry at scale 4 vs the phase's own height and width (no downscale) -> 912.0 +// the temporal-subsample guard vs subsampling unconditionally -> 2.0 +// the strength<1 wrap branch vs wrapping unconditionally -> 1.0 +// the divisibility refusal vs refusing whenever height % scale != 0, scale==1 included -> 1.0 +// downsample_mask_video_to_latent vs bilinear spatial interpolation -> 0.20678424835205078 +// downsample_mask_video_to_latent vs uniform temporal pooling with no causal first-frame carve-out -> 0.049749284982681274 +// build_attention_mask vs cross on ALL existing rows, not only the noisy ones -> 4.0 +// build_attention_mask with an existing mask vs discarding the existing mask and refilling the block with ones -> 1.0 +// resolve_cross_mask's 1-D arm vs the scalar fill -> 0.4000000059604645 +// update_attention_mask's pad-with-ones arm vs returning the mask unchanged (short of the sequence) -> 11.0 +// _prepare_self_attention_mask vs -inf for a zero and an unclamped log -> 1.0 + +} // namespace vllm_test diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 83072d99e..5e75f8210 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -13866,3 +13866,104 @@ TEST_CASE("ltx2 video: auto_duration refuses MIN > MAX and refuses doubling an e } } } + +// ───────────────────────────────────────────────────────────────────────────── +// IC-LoRA REFERENCE VIDEO and its CONDITIONING ATTENTION MASK +// Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +// spec .agents/specs/ltx25-ic-lora-ref-video.md (gaps A15 and A16). +// +// These are the REACHABILITY cases, and they are the only proof this row has. +// They enter through `LoadVideoEngine` and `Generate` — where +// `vllm_video_generate` arrives — and not through +// `Ltx2AppendIcLoraReferenceConditionings`. The value-level cases for the ported +// pieces live in `test_ltx2_iclora_reference` and per .agents/reachability.md +// they localize a failure rather than prove one: they stay green when the +// production call site is deleted. +// +// THE SEAM THIS ROW RETIRES. Before it, `Ltx2ModalityInput::attention_mask` and +// its whole consumption chain — `Ltx2PrepareSelfAttentionMask`, `self_bias` on +// host and device, `vt::AttentionCross`'s additive bias — were assigned at +// exactly four sites, all of them inside a `BuildModalities` helper in +// `test_ltx2.cpp` and `test_ltx2_device.cpp`. Nothing in `src/` wrote either +// field. The mask case below is the production assignment. +// ───────────────────────────────────────────────────────────────────────────── + +namespace { + +// An `ic_lora` engine. The kind is a LOAD extra, because `ICLoraPipeline` is a +// pipeline CLASS upstream (ic_lora.py:60) and not a per-request mode: stage 2 +// runs `loras=()` against stage 1's `loras=tuple(loras)` (`:108` against `:119`), +// which is a property of how the weights are held. +vllm::multimodal::VideoModelParams IcLoraParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp = FixtureParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "ic_lora"; + mp.extras[vllm::multimodal::kLtx2CheckpointClassExtra] = FixtureCheckpointClass("ic_lora"); + return mp; +} + +// 128x128 and NOT the 64x64 every other case uses, deliberately. The distilled +// two-stage recipe runs stage 1 at half resolution, so the reference clip is +// read at 64x64, and the fixture VAE's (8, 32, 32) factors then give a 2 x 2 x 2 +// reference latent — 8 tokens. At 64x64 the same arithmetic gives a 1 x 1 +// SPATIAL grid, where a row-major and a column-major patchify are identical and +// a downscale factor of 2 collapses to the same single token as a factor of 1. +// A reduced fixture that degenerates the axis it gates is this campaign's +// recorded failure, so the geometry is chosen against it. +vllm::multimodal::VideoGenParams IcLoraGen(const std::string& out_dir, const std::string& clip) { + vllm::multimodal::VideoGenParams gen; + gen.num_frames = 9; + gen.height = 128; + gen.width = 128; + gen.has_seed = true; + gen.seed = 7; + gen.output_dir = out_dir; + gen.ref_video_dir = clip; + return gen; +} + +} // namespace + +TEST_CASE("ltx2 ic-lora: a reference clip REACHES the render through the public ABI") { + // THE RED-FIRST CASE. Before this row `pipeline_kind=ic_lora` resolved to no + // recipe and `ref_video_dir` was refused outright on every kind but `retake`, + // so nothing here could reach the new code at all. + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + // Stage 1 renders at 64x64 and the fixture's factors are (8, 32, 32), so the + // reference clip encodes to 2 x 2 x 2 = 8 tokens. + const std::string clip = WriteRetakeClip(ws.root + "/ic_clip", 9, 64, 64); + const vllm::multimodal::VideoResult result = + engine->Generate(IcLoraGen(ws.root + "/ic_out", clip)); + + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + CHECK(trace.completed); + CHECK_MESSAGE(trace.ic_lora_reference_tokens == 8, + "the reference clip did not append one latent frame's worth of tokens per latent " + "frame; got " << trace.ic_lora_reference_tokens); + // The lower bound: a clip that encoded to zeros has the right shape, the right + // token count and renders. + CHECK(trace.ic_lora_reference_absmax > 0.0); + // The output geometry comes from the request, not from the reference. + CHECK(result.frame_count == 9); + + // THE RENDER DEPENDS ON THE REFERENCE CLIP'S PIXELS. `ic_lora_reference_absmax` + // observes the ENCODE and says nothing about whether the encoded latent was + // ever handed to the phase. A build that read the clip, encoded it, recorded + // the trace and then appended zeros satisfies every check above and renders a + // clip of the right length. Two DIFFERENT references at the same seed must + // therefore produce different pixels. + const std::string other = WriteRetakeClip(ws.root + "/ic_clip_b", 9, 64, 64, /*seed_base=*/113); + REQUIRE(ReadAll(clip + "/frame_000000.ppm") != ReadAll(other + "/frame_000000.ppm")); + const vllm::multimodal::VideoResult from_other = + engine->Generate(IcLoraGen(ws.root + "/ic_out_b", other)); + const std::string frame_a = ReadAll(std::string(result.frame_dir) + "/frame_000000.ppm"); + const std::string frame_b = ReadAll(std::string(from_other.frame_dir) + "/frame_000000.ppm"); + REQUIRE(frame_a.size() == frame_b.size()); + CHECK_MESSAGE(frame_a != frame_b, + "two different reference clips rendered byte-identical frames at the same seed, " + "so the encoded reference latent never reached the phase"); +} From 721f245ff70c6ca7432d80dbe972c55df4420ed0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:56:04 +0000 Subject: [PATCH 03/11] feat(LTX25-IC-LORA-REF-VIDEO): serve the IC-LoRA reference clip and its attention mask, and retire a dead seam MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A15 and A16. An `ic_lora` render can now be conditioned on a reference clip through the public video ABI, and an optional per-region mask attenuates that clip's attention. WHAT WAS ACTUALLY MISSING, once gateability was settled, was small. Ltx2ReadFrameDirectory already IS upstream's video_preprocess over a frame directory, and Ltx2ConvVideoEncode already runs multi-frame for retake. What no code supplied was the reference item's own geometry: the divisibility refusal, `height // scale`, the temporal subsample that keeps index 1 rather than stepping from 0, and the stage-1-only placement. Those are ported here, together with the whole mask half — the area downsample with its causal first-frame carve-out, resolve_cross_mask, and build_attention_mask's block structure. THIS RETIRES A DEFECT THAT WAS ON MAIN. Ltx2ModalityInput::attention_mask, Ltx2PrepareSelfAttentionMask, self_bias on host and device and the additive bias inside vt::AttentionCross were all built and all correct, and nothing in src/ assigned either field: the only four assignments in the tree were in a BuildModalities test helper. ltx2_video.cpp now makes that assignment on a production path, and the reachability case renders through it. ONE PREDICATE ROUTES AND REFUSES. `serves_reference` is bound once from the recipe and read by the routing branch and by every refusal beside it, because a refusal and its route predicate written as two expressions is how this campaign shipped a silently wrong answer. TWO MEASUREMENTS SHAPED THE FIXTURE RATHER THAN THE OTHER WAY ROUND. The render is 128x128 because at 64x64 the distilled recipe's half-resolution stage 1 gives a 1x1 spatial reference grid, where row-major and column-major patchify agree and a downscale factor of 2 collapses onto 1. And the reference clips are CONSTANT frames, not ConditioningPpm's seeded ramp: the VAE downsamples 32x, two seeds of that ramp average to nearly the same latent (absmax 0.539 against 0.464) and the two renders came out byte-identical. The pixel claim is made on the skip-stage-2 arm for a measured reason recorded in the case: stage 2 re-noises to 0.909375, and on a 2-block random-weight DiT the surviving difference falls below 1/255. Four refusals this change had to reconcile, each rewritten rather than relaxed: the reference arm is served, so a message may no longer describe it as missing, and the case that held it to naming a live cause now holds it to naming the pipeline that serves it. The reference-IMAGE arm stays refused and is the one refusal in the file no later row can serve, because upstream has no such flag. test_ltx2_video: 131 cases, 5225 assertions, 0 failed. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- CMakeLists.txt | 1 + .../model_executor/models/ltx2_conditioning.h | 61 +- .../models/ltx2_iclora_reference.h | 144 +++++ .../model_executor/models/ltx2_pipeline.h | 22 + include/vllm/multimodal/ltx2_video.h | 42 ++ .../models/ltx2_conditioning.cpp | 37 ++ .../models/ltx2_iclora_reference.cpp | 319 ++++++++++ .../model_executor/models/ltx2_pipeline.cpp | 50 ++ src/vllm/multimodal/ltx2_video.cpp | 468 ++++++++++---- tests/vllm/multimodal/test_ltx2_video.cpp | 574 ++++++++++++++---- 10 files changed, 1472 insertions(+), 246 deletions(-) create mode 100644 include/vllm/model_executor/models/ltx2_iclora_reference.h create mode 100644 src/vllm/model_executor/models/ltx2_iclora_reference.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 57af96778..d91f9a495 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -972,6 +972,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp src/vllm/model_executor/models/ltx2_audio_vae_encoder_load.cpp src/vllm/model_executor/models/ltx2_audio_input.cpp + src/vllm/model_executor/models/ltx2_iclora_reference.cpp src/vllm/model_executor/models/ltx2_image_preprocess.cpp # LTX-2.5 RETAKE (row LTX25-RETAKE, #924): the temporal region mask, the # truncate-or-pad conform, and the frame-directory source. Its own TU for the diff --git a/include/vllm/model_executor/models/ltx2_conditioning.h b/include/vllm/model_executor/models/ltx2_conditioning.h index 69ac360c8..30603a2ca 100644 --- a/include/vllm/model_executor/models/ltx2_conditioning.h +++ b/include/vllm/model_executor/models/ltx2_conditioning.h @@ -55,19 +55,27 @@ namespace vllm { // LatentState (types.py:251-287) at batch 1, carrying only the fields a // conditioning item touches. // -// `attention_mask` is deliberately absent, and the reason is a fact about the -// ported items rather than a simplification. Both appending VIDEO items pass a -// literal `attention_mask=None` (keyframe_cond.py:68-76, -// reference_video_cond.py:88-96), and `update_attention_mask` returns None when -// its argument is None and the state carries no mask -// (conditioning/mask_utils.py:110-143). The ONLY upstream route to a non-None -// mask is `ConditioningItemAttentionStrengthWrapper`, whose sole application -// site is the IC-LoRA path (ltx-pipelines/iclora_utils.py:169); +// `attention_mask` WAS deliberately absent until row LTX25-IC-LORA-REF-VIDEO +// (#3020), and the argument for its absence is kept rather than deleted because +// it was correct and this row is exactly what made it false. +// +// It read: both appending VIDEO items pass a literal `attention_mask=None` +// (keyframe_cond.py:68-76, reference_video_cond.py:88-96), and +// `update_attention_mask` returns None when its argument is None and the state +// carries no mask (conditioning/mask_utils.py:110-143). The ONLY upstream route +// to a non-None mask is `ConditioningItemAttentionStrengthWrapper`, whose sole +// application site is the IC-LoRA path (ltx-pipelines/iclora_utils.py:169); // `combined_image_conditionings` (ltx-pipelines/utils/helpers.py:272-308), which -// is the route this engine mirrors, never wraps. So a field here would be one no -// ported item can populate — the unpassed-parameter shape .agents/reachability.md -// enumerates. An item that DID need one would have to grow this struct rather -// than silently dropping it, which is why the omission is stated here. +// is the route this engine mirrored, never wraps. So a field here would have been +// one no ported item could populate — the unpassed-parameter shape +// .agents/reachability.md enumerates — and the note ended: an item that DID need +// one would have to grow this struct rather than silently dropping it. +// +// #3020 is that item. `iclora_utils.py:162-169` is now REACHED: the reference +// item is built and, when a conditioning attention mask was supplied, wrapped. +// The field below is what the wrapper writes, and `ltx2_video.cpp` hands it to +// `Ltx2ModalityInput::attention_mask` — whose whole consumption chain existed +// and was assigned only from a test helper before this row. // `GeneratedKeyframeLayout` (types.py:220-247): where a state's generated // keyframe slot tokens live, and what they represent. // @@ -111,6 +119,35 @@ struct Ltx2LatentState { // about the render's SHAPE can see the difference. std::vector keyframes_mask; + // `LatentState.attention_mask` (types.py:251-287), a DENSE [tokens, tokens] + // STRENGTH mask in [0, 1]. EMPTY is upstream's `None`, which every state + // carries until an item wraps. + // + // DENSE and not the key-only broadcast form. `build_attention_mask` returns + // (B, N+M, N+M) unconditionally (mask_utils.py:220), and the block structure + // it fills is not expressible as one row: the noisy rows and the prior-ref + // rows get DIFFERENT values in the same column (`:236` against `:242`). + // + // GROWS WITH THE SEQUENCE, and that is the trap. An item that appends tokens + // without extending this leaves a mask the DiT reads as the key-only form — + // legal, differently shaped, and silently masking the wrong axis. That is why + // `Ltx2PadAttentionMaskForUnmaskedTokens` exists and why it is upstream's own + // answer at mask_utils.py:141-156 rather than a defensive addition here. + std::vector attention_mask; + + // `latent_tools.target_shape.token_count()`, which every appending item hands + // `update_attention_mask` as `num_noisy_tokens` (keyframe_cond.py:71, + // reference_video_cond.py:89). It is the count BEFORE any item ran, and it is + // NOT `tokens` once one has: rows between `noisy_tokens` and `tokens` are + // PRIOR reference tokens, and the block structure gives them a different value + // from the noisy rows in the same column (mask_utils.py:236 against `:242`). + // + // Carried on the state because this engine's phase loop rebuilds an + // `Ltx2LatentState` per item from a running `StreamState`, so `tokens` has + // already grown by the time the second item is applied and cannot stand in. + // Zero means "no mask can be built", which is what an audio state carries. + int64_t noisy_tokens = 0; + // `LatentState.generated_keyframe_layout` (types.py:246). Default-constructed // is upstream's `None`, and `Ltx2ConditionVideoByGeneratedKeyframeSlots` // refuses a second application on a state that already carries one diff --git a/include/vllm/model_executor/models/ltx2_iclora_reference.h b/include/vllm/model_executor/models/ltx2_iclora_reference.h new file mode 100644 index 000000000..91b088381 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_iclora_reference.h @@ -0,0 +1,144 @@ +// LTX-2.5 IC-LoRA REFERENCE CONDITIONING — the reference clip's own geometry, +// and the conditioning attention mask that rides it. +// +// Row LTX25-IC-LORA-REF-VIDEO, issue #3020, +// spec .agents/specs/ltx25-ic-lora-ref-video.md (gaps A15 and A16). +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f +// OURS <- UPSTREAM +// Ltx2IcLoraReferenceGeometry <- ltx-pipelines/iclora_utils.py:111-117 +// Ltx2TemporalSubsample <- ltx-pipelines/iclora_utils.py:87-90 +// Ltx2MaskVideoFromPixels <- ltx-pipelines/ic_lora.py:511-537 +// Ltx2DownsampleMaskVideoToLatent <- ltx-pipelines/iclora_utils.py:52-84 +// Ltx2ResolveCrossMask <- ltx-core/conditioning/mask_utils.py:13-73 +// Ltx2BuildAttentionMask <- ltx-core/conditioning/mask_utils.py:170-243 +// Ltx2UpdateAttentionMask <- ltx-core/conditioning/mask_utils.py:110-167 +// +// The PIXEL path is not re-implemented here. `video_preprocess` +// (media_io/decode.py:82-103) is per frame `resize_and_center_crop` then +// `normalize_images`, concatenated on the frame axis, and this tree already has +// that as `Ltx2ReadFrameDirectory` (ltx2_retake.h) — which row LTX25-RETAKE +// ported and gated. This header only says at WHAT GEOMETRY to call it. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY ────────────────────────────────────── +// * `temporal_subsample` KEEPS INDEX 1. It is `[0, *range(1, F, N)]` (`:89`), +// not `range(0, F, N)`. At N=2 over 5 frames the kept set is {0, 1, 3} and +// the plausible reading gives {0, 2, 4} — the same COUNT, so every shape +// check agrees and the reference simply describes different moments. +// * THE MASK'S FIRST LATENT FRAME IS PIXEL FRAME 0 ALONE (`:70`, `:80`), a +// causal carve-out that mirrors the VAE's own first-frame rule. Pooling all +// `f_pix` frames uniformly produces a correctly shaped mask of plausible +// values. +// * `build_attention_mask` PUTS THE CROSS BLOCK ON THE NOISY ROWS ONLY +// (`:236`, `:240`), and leaves the prev-ref-to-new-ref blocks at ZERO +// (`:242`). On a fixture with no PRIOR conditioning item that is elementwise +// equal to "cross on all existing rows", so the wrong reading is invisible +// until a second item exists. +// * AN ALL-ONES MASK IS THE IDENTITY. A downsample that lost its values, a +// strength that was never multiplied in, and a correct unmasked render are +// all the same bytes. Nothing about the render's shape can see it. +// +// ─── DTYPE ─────────────────────────────────────────────────────────────────── +// Upstream runs this whole path at bf16 (`self.dtype`, ic_lora.py:271-281, and +// `torch.bfloat16` at ic_lora.py:530). Every buffer here is f32, which is this +// tree's standing LTX host-path width and the A24 wave campaign's open +// divergence — NOT a decision this row made. Recorded in +// .agents/specs/ltx25-ic-lora-ref-video.md §4 R4 and in its `## Owed`, because a +// token gate and these goldens are both blind to a dtype that is too wide. +#pragma once + +#include +#include +#include + +namespace vllm { + +// `iclora_utils.py:111-117`. The resolution at which the reference clip is READ, +// which is the target's divided by the adapter's `reference_downscale_factor`. +// +// THE REFUSAL IS UPSTREAM'S AND IT IS GUARDED ON `scale != 1` (`:112`), not on +// divisibility alone: at scale 1 no dimension is ever refused, whatever it is. +// Refusing there would reject every render that supplies no adapter, since an +// absent `reference_downscale_factor` reads as 1 (`:35`). +struct Ltx2IcLoraReferenceGeometry { + int64_t height = 0; + int64_t width = 0; +}; +Ltx2IcLoraReferenceGeometry Ltx2ResolveIcLoraReferenceGeometry(int64_t height, int64_t width, + int64_t downscale_factor); + +// `temporal_subsample` (`iclora_utils.py:87-90`) over a channel-major +// `[channels, frames, height, width]` volume — the layout `Ltx2ReadFrameDirectory` +// returns and `Ltx2ConvVideoEncode` takes. +// +// Returns the KEPT FRAME INDICES, and the caller gathers. The indices are the +// whole content of the function and a vector of them is the only thing a gate +// can compare against upstream without also re-gating the copy. +std::vector Ltx2TemporalSubsampleIndices(int64_t frames, int64_t temporal_scale_factor); + +// Gather those indices out of a `[channels, frames, plane]` volume. +std::vector Ltx2TemporalSubsample(const std::vector& clip, int64_t channels, + int64_t frames, int64_t plane, + int64_t temporal_scale_factor); + +// `_load_mask_video`'s ARITHMETIC half (`ic_lora.py:530-536`): the three-channel +// pixel volume in [-1, 1] that `Ltx2ReadFrameDirectory` returns becomes a +// one-channel mask in [0, 1] — mean over channels, then `(x + 1) / 2`, then +// clamp. The READ half is `Ltx2ReadFrameDirectory` itself, at the stage's own +// height and width, which is what `ic_lora.py:460-461` spells `args.height // 2`. +std::vector Ltx2MaskVideoFromPixels(const std::vector& pixels, int64_t channels, + int64_t frames, int64_t plane); + +// `downsample_mask_video_to_latent` (`iclora_utils.py:52-84`). A pixel-space +// mask `[frames, height, width]` becomes flattened latent token weights +// `[f_lat * h_lat * w_lat]`, in the token order the video patchifier uses. +// +// AREA interpolation spatially (`:63-67`) — for the shapes this engine reaches +// it is exact box averaging — then the causal split: latent frame 0 is pixel +// frame 0 alone (`:70`, `:80`) and the remaining `f_pix - 1` frames mean-pool in +// groups of `t = (f_pix - 1) / (f_lat - 1)` (`:73`, `:78-79`). Refuses a +// non-divisible pair with upstream's own assertion (`:74-77`), and degenerates +// to the first frame alone when either axis has one frame (`:81-82`). +std::vector Ltx2DownsampleMaskVideoToLatent(const std::vector& mask, int64_t f_pix, + int64_t h_pix, int64_t w_pix, int64_t f_lat, + int64_t h_lat, int64_t w_lat); + +// `resolve_cross_mask` (`mask_utils.py:13-73`) at batch 1. `values` empty is the +// SCALAR arm (`:31-37`): `scalar` fills all `num_new_tokens`. Otherwise `values` +// is the 1-D `(M,)` form (`:49-54`) and its length must be `num_new_tokens`, +// refused by name exactly as `:50-53` refuses it. +std::vector Ltx2ResolveCrossMask(const std::vector& values, double scalar, + int64_t num_new_tokens); + +// `build_attention_mask` (`mask_utils.py:170-243`) at batch 1. Returns a dense +// `[(N+M) * (N+M)]` row-major mask in [0, 1]. +// +// noisy(Nn) prev_ref(N-Nn) new_ref(M) +// noisy existing existing cross +// prev_ref existing existing 0 +// new_ref cross 0 1 +// +// `existing` EMPTY is upstream's `existing_mask=None` and fills the top-left +// `N x N` block with ones (`:225-226`); otherwise it is preserved (`:224`), +// which is what keeps an earlier item's attenuation alive under a later one. +std::vector Ltx2BuildAttentionMask(const std::vector& existing, + int64_t num_noisy_tokens, int64_t num_new_tokens, + int64_t num_existing_tokens, + const std::vector& cross_mask); + +// `update_attention_mask` (`mask_utils.py:110-167`) at batch 1, for the +// `attention_mask is None` arm alone (`:141-156`). +// +// `existing` empty returns empty — upstream's "no mask requested and none +// present, return None" (`:142-143`). `existing` NON-empty pads the new tokens +// with FULL attention (`:147`) rather than returning the mask unchanged, which +// is what keeps the mask's dimensions equal to the growing sequence. A mask left +// short of the sequence is not a shape error anywhere downstream: it is read as +// a broadcastable key-only form and silently masks the wrong axis. +std::vector Ltx2PadAttentionMaskForUnmaskedTokens(const std::vector& existing, + int64_t num_noisy_tokens, + int64_t num_new_tokens, + int64_t num_existing_tokens); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 2d4ba645c..b12a5b884 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -988,6 +988,28 @@ struct Ltx2PipelineRecipe { // takes the keyframe item under BOTH of them. Ltx2ImageConditioningBuilder image_conditioning = Ltx2ImageConditioningBuilder::kCombined; + // WHETHER THIS PIPELINE CONDITIONS ON A REFERENCE CLIP. Row + // LTX25-IC-LORA-REF-VIDEO, issue #3020 (gaps A15 and A16). + // + // A FLAG ON THE RECIPE, for the reason `checkpoint_class` gives above, and + // because exactly one upstream pipeline class has this: `ICLoraPipeline` + // (ic_lora.py:60) calls `append_ic_lora_reference_video_conditionings` + // (`:381-402`) and no other pipeline in `ltx-pipelines` does. Serving the arm + // on `distilled_two_stage` would be inventing conditioning `distilled.py` has + // no call to. + // + // AND IT IS THE SAME PREDICATE THAT REFUSES. The phase loop reads this field + // once, binds it to a named local, and both the routing branch and the refusal + // beside it use that local. A refusal and its route predicate that are two + // expressions is how this campaign shipped a silently wrong answer. + // + // STAGE 1 ONLY is NOT this flag: it is the phase index, because upstream gives + // stage 1 `_create_conditionings` (which appends the reference item, + // ic_lora.py:269-281) and stage 2 plain `combined_image_conditionings` with no + // reference item at all (`:314-321`). A recipe-level "which phase" field would + // have exactly one value on the one recipe that sets this. + bool ic_lora_reference = false; + int64_t max_spatial_downscale() const; }; diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index c1c6ee39d..69af709a5 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -465,6 +465,48 @@ inline constexpr char kLtx2RetakeFrameRateExtra[] = "retake_frame_rate"; inline constexpr char kLtx2RegenerateVideoExtra[] = "regenerate_video"; inline constexpr char kLtx2RegenerateAudioExtra[] = "regenerate_audio"; +// ── IC-LoRA REFERENCE CONDITIONING. Row LTX25-IC-LORA-REF-VIDEO (#3020) ───── +// +// Read ONLY on an engine whose recipe declares `ic_lora_reference`, which is +// `pipeline_kind = ic_lora` alone. `ICLoraPipeline` is a pipeline CLASS upstream +// (ic_lora.py:60) and no other pipeline in `ltx-pipelines` calls +// `append_ic_lora_reference_video_conditionings`, so serving the arm elsewhere +// would be inventing conditioning the reference does not have. +// +// The reference CLIP itself is `VideoGenParams::ref_video_dir` — a directory of +// `frame_%06d.ppm`, this ABI's standing spelling for a video, because no demuxer +// is vendored here. Upstream reads a container (`decode_video_by_frame`, +// iclora_utils.py:141) and its folder arm is the adaptation row LTX25-RETAKE +// already recorded. + +// The STRENGTH half of `--video-conditioning PATH STRENGTH` +// (ic_lora.py:416-425), which becomes `VideoConditionByReferenceLatent.strength` +// (iclora_utils.py:166) and therefore the reference tokens' denoise mask +// `1 - strength`. Default 1.0. Upstream's flag is REQUIRED and takes both halves +// together; here the path is a request field that already exists, so only the +// strength needs a name. +inline constexpr char kLtx2RefVideoStrengthExtra[] = "ref_video_strength"; + +// The MASK_PATH half of `--conditioning-attention-mask MASK_PATH STRENGTH` +// (ic_lora.py:427-441), as a directory of `frame_%06d.ppm` for the reason above. +// Upstream loads it at the STAGE-1 resolution (`args.height // 2`, `:460-461`); +// this engine reads it at the stage's own grid, which is the same number derived +// rather than assumed. +inline constexpr char kLtx2CondAttentionMaskDirExtra[] = "conditioning_attention_mask_dir"; + +// Its STRENGTH half (`:454-455`), multiplied into the downsampled mask +// (iclora_utils.py:156). Default 1.0, and refused outside [0, 1] exactly as +// `ICLoraPipeline.__call__` refuses it. +// +// A STRENGTH BELOW 1 WITHOUT A MASK IS REFUSED BY NAME, and that is a mirror +// rather than a limitation. Upstream's scalar-only arm (`iclora_utils.py:157-158`) +// cannot be reached from the CLI: `conditioning_attention_strength` is assigned +// only inside `if args.conditioning_attention_mask is not None` +// (ic_lora.py:452-455) and is 1.0 otherwise, so a sub-1.0 strength always +// arrives with a mask. It is a Python-API-only branch, MEASURED as such by +// `kLtx2RefWrapScalarBelowOne`, and it is recorded owed rather than guessed at. +inline constexpr char kLtx2CondAttentionStrengthExtra[] = "conditioning_attention_strength"; + // ── TEXT-TO-AUDIO. Row LTX25-T2A-ONE-STAGE (#1005) ───────────────────────── // // These are read ONLY on a `pipeline_kind = t2a_one_stage` engine — that is a diff --git a/src/vllm/model_executor/models/ltx2_conditioning.cpp b/src/vllm/model_executor/models/ltx2_conditioning.cpp index dd733d639..4c3f82206 100644 --- a/src/vllm/model_executor/models/ltx2_conditioning.cpp +++ b/src/vllm/model_executor/models/ltx2_conditioning.cpp @@ -9,6 +9,8 @@ // (tools.py:158-176, keyframe_cond.py:57-65), so this one is not a widening. #include "vllm/model_executor/models/ltx2_conditioning.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + #include #include #include @@ -74,6 +76,27 @@ void AppendTokens(Ltx2LatentState* state, const std::vector& tokens, int6 // term to the wrong tokens. Ltx2ExtendKeyframesMask(state, token_count, /*marked=*/false); + // AND THE ATTENTION MASK, for the same reason and from the same pre-append + // state. Every appending item upstream calls `update_attention_mask` with a + // literal `attention_mask=None` (keyframe_cond.py:68-76, + // reference_video_cond.py:86-94), which returns None when the state carries no + // mask and otherwise pads the new tokens with FULL attention + // (mask_utils.py:141-156). Row LTX25-IC-LORA-REF-VIDEO (#3020) is what made + // the second branch reachable: before it no state could carry a mask at all. + // + // A stale mask is NOT a shape error downstream. `Ltx2ModalityInput` accepts + // both the dense [tokens, tokens] form and the key-only [1, tokens] broadcast, + // so a mask left at its pre-append size is read as a legal mask over a + // different axis, and the render comes out the right size with a trained term + // applied to the wrong tokens. + if (!state->attention_mask.empty()) { + VT_CHECK(state->noisy_tokens > 0, + "ltx2 conditioning: a state carrying an attention mask must know its own noisy-token " + "count, which is `latent_tools.target_shape.token_count()` upstream"); + state->attention_mask = Ltx2PadAttentionMaskForUnmaskedTokens( + state->attention_mask, state->noisy_tokens, token_count, before); + } + state->latent.resize(static_cast(after * state->width), 0.0f); state->clean.insert(state->clean.end(), tokens.begin(), tokens.end()); state->mask.insert(state->mask.end(), static_cast(token_count), @@ -126,6 +149,13 @@ Ltx2LatentState Ltx2CreateVideoLatentState(const Ltx2VideoLatentShape& shape, in if (out_keyframes_mask != nullptr) { *out_keyframes_mask = state.keyframes_mask; } + // `latent_tools.target_shape.token_count()` — what an appending item hands + // `update_attention_mask` as `num_noisy_tokens`. Recorded HERE, at the one + // moment it is knowable without searching: after the first append `tokens` is + // no longer it, and the rows between the two are prior REFERENCE tokens whose + // block in the mask differs from the noisy one (mask_utils.py:236 vs :242). + state.noisy_tokens = state.tokens; + return state; } @@ -487,6 +517,13 @@ Ltx2LatentState Ltx2CreateAudioLatentState(const Ltx2AudioLatentShape& shape, // The audio positions ARE the patch grid bounds, in seconds — there is no // get_pixel_coords and no fps division on this side (tools.py:271-279). state.positions = Ltx2AudioPatchTimings(shape, params); + // `latent_tools.target_shape.token_count()` — what an appending item hands + // `update_attention_mask` as `num_noisy_tokens`. Recorded HERE, at the one + // moment it is knowable without searching: after the first append `tokens` is + // no longer it, and the rows between the two are prior REFERENCE tokens whose + // block in the mask differs from the noisy one (mask_utils.py:236 vs :242). + state.noisy_tokens = state.tokens; + return state; } diff --git a/src/vllm/model_executor/models/ltx2_iclora_reference.cpp b/src/vllm/model_executor/models/ltx2_iclora_reference.cpp new file mode 100644 index 000000000..9fc12e763 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_iclora_reference.cpp @@ -0,0 +1,319 @@ +// LTX-2.5 IC-LoRA reference conditioning — see ltx2_iclora_reference.h for the +// port map and for the four failure modes this file exists to make impossible. +// +// Row LTX25-IC-LORA-REF-VIDEO (#3020). Upstream: Lightricks/LTX-2 @ fd4ded7f. +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + +#include +#include +#include +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { + throw std::runtime_error("ltx2 ic-lora reference: " + why); +} + +// `torch.nn.functional.interpolate(..., mode="area")` dispatches to +// `adaptive_avg_pool2d`, whose window for output index `i` over input extent `I` +// and output extent `O` is `[floor(i*I/O), ceil((i+1)*I/O))`. That is NOT the +// same as an integer-stride box filter whenever `O` does not divide `I`, and the +// difference is invisible on the divisible shapes a reduced fixture reaches — so +// the general form is written here rather than the special case. +int64_t PoolStart(int64_t index, int64_t in_extent, int64_t out_extent) { + return (index * in_extent) / out_extent; +} +int64_t PoolEnd(int64_t index, int64_t in_extent, int64_t out_extent) { + return ((index + 1) * in_extent + out_extent - 1) / out_extent; +} + +} // namespace + +Ltx2IcLoraReferenceGeometry Ltx2ResolveIcLoraReferenceGeometry(int64_t height, int64_t width, + int64_t downscale_factor) { + if (downscale_factor < 1) { + Fail("reference_downscale_factor must be at least 1, got " + + std::to_string(downscale_factor) + + ". Upstream's absent-metadata default is 1 (iclora_utils.py:35)"); + } + // `:112` — the guard is `scale != 1 AND (h % scale or w % scale)`. At scale 1 + // NOTHING is ever refused, which is what keeps every adapter-less render + // working, and a port that dropped the `scale != 1` half would refuse none of + // them differently (1 divides everything) while a port that dropped the + // divisibility half would refuse all of them. + if (downscale_factor != 1 && (height % downscale_factor != 0 || width % downscale_factor != 0)) { + Fail("Output dimensions (" + std::to_string(height) + "x" + std::to_string(width) + + ") must be divisible by reference_downscale_factor (" + + std::to_string(downscale_factor) + + "). This is upstream's own refusal (iclora_utils.py:112-115): the reference clip is read " + "at height // factor by width // factor and a truncating division would place the " + "reference tokens on a grid the target does not have"); + } + Ltx2IcLoraReferenceGeometry out; + out.height = height / downscale_factor; // :116 + out.width = width / downscale_factor; // :117 + return out; +} + +std::vector Ltx2TemporalSubsampleIndices(int64_t frames, int64_t temporal_scale_factor) { + if (frames < 1) Fail("a reference clip needs at least one frame"); + if (temporal_scale_factor < 1) { + Fail("reference_temporal_scale_factor must be at least 1, got " + + std::to_string(temporal_scale_factor)); + } + // `indices = [0, *list(range(1, video.shape[2], temporal_scale_factor))]` (:89). + // + // INDEX 1 IS ALWAYS KEPT when it exists, because the range starts AT 1 rather + // than stepping from 0. At factor 2 over 5 frames this is {0, 1, 3}; the + // plausible `range(0, F, N)` gives {0, 2, 4} — the same COUNT, so no shape + // check and no token count can tell them apart, and the reference simply + // describes different moments of the clip. Gated by `kLtx2TemporalSubsampleKept` + // against exactly that hypothesis. + std::vector indices; + indices.push_back(0); + for (int64_t i = 1; i < frames; i += temporal_scale_factor) indices.push_back(i); + return indices; +} + +std::vector Ltx2TemporalSubsample(const std::vector& clip, int64_t channels, + int64_t frames, int64_t plane, + int64_t temporal_scale_factor) { + const std::vector keep = Ltx2TemporalSubsampleIndices(frames, temporal_scale_factor); + const size_t expect = static_cast(channels) * static_cast(frames) * + static_cast(plane); + if (clip.size() != expect) { + Fail("the reference clip holds " + std::to_string(clip.size()) + " values but " + + std::to_string(channels) + " x " + std::to_string(frames) + " x " + + std::to_string(plane) + " is " + std::to_string(expect)); + } + const int64_t kept = static_cast(keep.size()); + std::vector out(static_cast(channels) * static_cast(kept) * + static_cast(plane)); + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < kept; ++t) { + const size_t src = (static_cast(c) * static_cast(frames) + + static_cast(keep[static_cast(t)])) * + static_cast(plane); + const size_t dst = (static_cast(c) * static_cast(kept) + + static_cast(t)) * + static_cast(plane); + std::copy(clip.begin() + static_cast(src), + clip.begin() + static_cast(src + static_cast(plane)), + out.begin() + static_cast(dst)); + } + } + return out; +} + +std::vector Ltx2MaskVideoFromPixels(const std::vector& pixels, int64_t channels, + int64_t frames, int64_t plane) { + if (channels < 1) Fail("a mask video needs at least one channel"); + const size_t expect = static_cast(channels) * static_cast(frames) * + static_cast(plane); + if (pixels.size() != expect) { + Fail("the mask video holds " + std::to_string(pixels.size()) + " values but " + + std::to_string(channels) + " x " + std::to_string(frames) + " x " + + std::to_string(plane) + " is " + std::to_string(expect)); + } + std::vector out(static_cast(frames) * static_cast(plane)); + for (int64_t t = 0; t < frames; ++t) { + for (int64_t p = 0; p < plane; ++p) { + // `mask_video.mean(dim=1, keepdim=True)` (:531). The pixels arrive + // CHANNEL-major, which is the layout the encoder takes, so the three + // samples of one pixel are a plane apart rather than adjacent. + double sum = 0.0; + for (int64_t c = 0; c < channels; ++c) { + const size_t at = (static_cast(c) * static_cast(frames) + + static_cast(t)) * + static_cast(plane) + + static_cast(p); + sum += static_cast(pixels[at]); + } + const double mean = sum / static_cast(channels); + // `(mask + 1.0) / 2.0` then `.clamp(0.0, 1.0)` (:534-536). The remap + // undoes `normalize_images`' `/127.5 - 1`, and the clamp is upstream's and + // not defensive: a mask frame that decoded above the range would otherwise + // become an attention weight above 1, which `_prepare_self_attention_mask` + // turns into a POSITIVE log-space bias — an amplifier, not an attenuator. + out[static_cast(t) * static_cast(plane) + static_cast(p)] = + static_cast(std::min(1.0, std::max(0.0, (mean + 1.0) / 2.0))); + } + } + return out; +} + +std::vector Ltx2DownsampleMaskVideoToLatent(const std::vector& mask, int64_t f_pix, + int64_t h_pix, int64_t w_pix, int64_t f_lat, + int64_t h_lat, int64_t w_lat) { + if (f_pix < 1 || h_pix < 1 || w_pix < 1 || f_lat < 1 || h_lat < 1 || w_lat < 1) { + Fail("every mask dimension must be positive"); + } + const size_t expect = + static_cast(f_pix) * static_cast(h_pix) * static_cast(w_pix); + if (mask.size() != expect) { + Fail("the mask holds " + std::to_string(mask.size()) + " values but " + + std::to_string(f_pix) + " x " + std::to_string(h_pix) + " x " + std::to_string(w_pix) + + " is " + std::to_string(expect)); + } + + // :63-67 — AREA interpolation, per pixel frame, to the LATENT spatial grid. + const size_t lat_plane = static_cast(h_lat) * static_cast(w_lat); + std::vector spatial(static_cast(f_pix) * lat_plane); + for (int64_t t = 0; t < f_pix; ++t) { + for (int64_t oh = 0; oh < h_lat; ++oh) { + const int64_t h0 = PoolStart(oh, h_pix, h_lat), h1 = PoolEnd(oh, h_pix, h_lat); + for (int64_t ow = 0; ow < w_lat; ++ow) { + const int64_t w0 = PoolStart(ow, w_pix, w_lat), w1 = PoolEnd(ow, w_pix, w_lat); + double sum = 0.0; + for (int64_t ih = h0; ih < h1; ++ih) { + for (int64_t iw = w0; iw < w1; ++iw) { + sum += static_cast( + mask[(static_cast(t) * static_cast(h_pix) + + static_cast(ih)) * + static_cast(w_pix) + + static_cast(iw)]); + } + } + spatial[static_cast(t) * lat_plane + static_cast(oh) * + static_cast(w_lat) + + static_cast(ow)] = sum / static_cast((h1 - h0) * (w1 - w0)); + } + } + } + + // :70 — the FIRST latent frame is pixel frame 0 ALONE. This is the causal + // carve-out, and it is the whole content of the temporal half: pooling all + // `f_pix` frames uniformly into `f_lat` groups produces a correctly shaped + // mask of entirely plausible values. Gated against exactly that hypothesis by + // `kLtx2MaskLatentUniformPool`. + std::vector out; + out.reserve(static_cast(f_lat) * lat_plane); + for (size_t i = 0; i < lat_plane; ++i) { + out.push_back(static_cast(spatial[i])); + } + // :72 — and the rest ONLY when both axes have more than one frame; otherwise + // the first frame alone is the whole answer (:81-82). + if (f_pix > 1 && f_lat > 1) { + const int64_t t = (f_pix - 1) / (f_lat - 1); // :73 + if ((f_pix - 1) % (f_lat - 1) != 0) { // :74-77, upstream's assertion + Fail("Pixel frames (" + std::to_string(f_pix) + ") not compatible with latent frames (" + + std::to_string(f_lat) + "): (f_pix - 1) must be divisible by (f_lat - 1). The mask " + "video and the reference clip must describe the same moments, and a truncating " + "group size would silently drop the tail of the mask"); + } + for (int64_t lf = 1; lf < f_lat; ++lf) { + for (size_t p = 0; p < lat_plane; ++p) { + double sum = 0.0; + for (int64_t k = 0; k < t; ++k) { + // :78-79 — the remaining frames reshape to [f_lat - 1, t] and mean + // over the group axis. Frame `1 + (lf - 1) * t + k`, because the + // reshape happens AFTER frame 0 has been taken out. + const int64_t src = 1 + (lf - 1) * t + k; + sum += spatial[static_cast(src) * lat_plane + p]; + } + out.push_back(static_cast(sum / static_cast(t))); + } + } + } + return out; +} + +std::vector Ltx2ResolveCrossMask(const std::vector& values, double scalar, + int64_t num_new_tokens) { + if (num_new_tokens < 1) Fail("a cross mask needs at least one new token"); + if (values.empty()) { + // :31-37 — the scalar arm fills every new token with the same weight. + return std::vector(static_cast(num_new_tokens), static_cast(scalar)); + } + // :49-53 — the 1-D `(M,)` form, whose length upstream refuses to broadcast. + if (static_cast(values.size()) != num_new_tokens) { + Fail("1-D attention_mask length must equal num_new_tokens (" + + std::to_string(num_new_tokens) + "), got " + std::to_string(values.size()) + + ". Upstream refuses rather than broadcasting (mask_utils.py:50-53), because a mask one " + "token short of the sequence would attenuate the wrong tokens and still render"); + } + return values; +} + +std::vector Ltx2BuildAttentionMask(const std::vector& existing, + int64_t num_noisy_tokens, int64_t num_new_tokens, + int64_t num_existing_tokens, + const std::vector& cross_mask) { + if (num_noisy_tokens < 1 || num_new_tokens < 1 || num_existing_tokens < num_noisy_tokens) { + Fail("the attention mask's block structure needs 1 <= num_noisy <= num_existing and at least " + "one new token; got noisy=" + + std::to_string(num_noisy_tokens) + " new=" + std::to_string(num_new_tokens) + + " existing=" + std::to_string(num_existing_tokens)); + } + if (static_cast(cross_mask.size()) != num_new_tokens) { + Fail("the cross mask must hold one weight per new token (" + std::to_string(num_new_tokens) + + "), got " + std::to_string(cross_mask.size())); + } + const int64_t total = num_existing_tokens + num_new_tokens; // :217 + if (!existing.empty() && + static_cast(existing.size()) != num_existing_tokens * num_existing_tokens) { + Fail("the existing mask must be [" + std::to_string(num_existing_tokens) + ", " + + std::to_string(num_existing_tokens) + "], got " + std::to_string(existing.size()) + + " values"); + } + + std::vector out(static_cast(total) * static_cast(total), 0.0F); // :220 + const auto At = [&out, total](int64_t r, int64_t c) -> float& { + return out[static_cast(r) * static_cast(total) + static_cast(c)]; + }; + + // :223-226 — the top-left N x N block is the existing mask, or ones when there + // was none. NOT ones-plus-existing and not zeros: an absent mask means every + // existing token attends to every other, INCLUDING prior reference tokens that + // carried no mask of their own (`:204-205`). + for (int64_t r = 0; r < num_existing_tokens; ++r) { + for (int64_t c = 0; c < num_existing_tokens; ++c) { + At(r, c) = existing.empty() + ? 1.0F + : existing[static_cast(r) * static_cast(num_existing_tokens) + + static_cast(c)]; + } + } + // :229 — the new reference tokens fully attend to THEMSELVES. + for (int64_t r = num_existing_tokens; r < total; ++r) { + for (int64_t c = num_existing_tokens; c < total; ++c) At(r, c) = 1.0F; + } + // :236 — noisy rows attend to the new reference tokens at the cross weight, + // one weight per COLUMN. `num_noisy_tokens` and NOT `num_existing_tokens`: + // rows between the two are PRIOR reference tokens and they stay at the zero + // `:242` leaves them at. On a fixture with no prior item the two readings are + // elementwise equal, which is why the golden's fixture carries one. + for (int64_t r = 0; r < num_noisy_tokens; ++r) { + for (int64_t j = 0; j < num_new_tokens; ++j) { + At(r, num_existing_tokens + j) = cross_mask[static_cast(j)]; + } + } + // :240 — and the transpose: new reference ROWS attend to the noisy tokens, one + // weight per ROW. + for (int64_t i = 0; i < num_new_tokens; ++i) { + for (int64_t c = 0; c < num_noisy_tokens; ++c) { + At(num_existing_tokens + i, c) = cross_mask[static_cast(i)]; + } + } + return out; +} + +std::vector Ltx2PadAttentionMaskForUnmaskedTokens(const std::vector& existing, + int64_t num_noisy_tokens, + int64_t num_new_tokens, + int64_t num_existing_tokens) { + // :142-143 — no mask requested and none present is upstream's `None`, and an + // empty vector is how this engine spells it. + if (existing.empty()) return {}; + // :147 — otherwise the new tokens get FULL attention, so the mask keeps + // describing the whole sequence. + const std::vector ones(static_cast(num_new_tokens), 1.0F); + return Ltx2BuildAttentionMask(existing, num_noisy_tokens, num_new_tokens, num_existing_tokens, + ones); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index d4d492740..cdf91f58f 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -2121,6 +2121,48 @@ Ltx2PipelineRecipe KeyframeInterpolationRecipe(const Ltx2PipelineParams& params, return recipe; } +// ICLoraPipeline (ltx-pipelines/ic_lora.py:60-402). Row LTX25-IC-LORA-REF-VIDEO, +// issue #3020. +// +// THE SCHEDULE IS THE DISTILLED TWO-STAGE ONE, EXACTLY, and that is a finding +// rather than a shortcut. `ICLoraPipeline.__call__` takes `stage_1_sigmas` +// defaulting to `DISTILLED_SIGMAS` and `stage_2_sigmas` to +// `STAGE_2_DISTILLED_SIGMAS` (the same two constants `distilled.py` imports), +// runs stage 1 at `width // 2, height // 2` (`:260-266`), and re-noises stage 2 +// to `stage_2_sigmas[0]` (`:333-343`). Every one of those is what +// `DistilledTwoStageRecipe` already carries. IC-LoRA differs from the distilled +// two-stage pipeline in its CONDITIONING and in where its adapter rides, not in +// its schedule — recorded here so a reader does not go looking for an IC-LoRA +// sigma set upstream does not have. +// +// TWO FIELDS SEPARATE IT, and they are the two `ICLoraPipeline` was written for. +Ltx2PipelineRecipe IcLoraRecipe(const std::string& version) { + Ltx2PipelineRecipe recipe = DistilledTwoStageRecipe(version); + if (recipe.phases.size() != 2) { + Refuse("The IC-LoRA recipe is the distilled two-stage recipe with IC-LoRA's adapter split, " + "and that recipe just returned " + + std::to_string(recipe.phases.size()) + + " phases. Stage 1 and stage 2 are addressed by INDEX below, so a changed phase count " + "would scope the adapter to the wrong one."); + } + recipe.phases[0].name = "ic_lora_lowres"; + recipe.phases[1].name = "ic_lora_refine"; + // `ic_lora.py:108` against `:119` — the MIRROR IMAGE of every other two-stage + // pipeline in this table. `A2VidTwoStageRecipe` and its siblings give stage 1 + // `kNoAdapters` and stage 2 the adapters; here the adapter rides stage 1 and + // stage 2 runs BARE, because the IC-LoRA is what teaches the model to read the + // reference and stage 2 only refines an already-conditioned latent. + // + // Getting this backwards renders a video. Stage 2 would run a 3-step + // refinement under an adapter it was never given, and the clip comes out the + // right length at the right resolution. + recipe.phases[1].loras = Ltx2PhaseLoraScope::kNoAdapters; + // `_create_conditionings` (`:269-281`, calling `:381-402`) against stage 2's + // plain `combined_image_conditionings` (`:314-321`). + recipe.ic_lora_reference = true; + return recipe; +} + } // namespace Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, @@ -2149,6 +2191,14 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // resolving DFR onto it would build a recipe whose first stage the engine // must then refuse at load. Refusing at the recipe table names the version. if (model_version == "2.5") return DfrRecipe(model_version); + } else if (pipeline_kind == "ic_lora") { + // 2.5 only, and 2.0 is refused rather than resolved. `ICLoraPipeline`'s + // reference item carries the adapter's `reference_downscale_factor` and + // `reference_temporal_scale_factor` (iclora_utils.py:30-49), which are + // LTX-2.5 IC-LoRA metadata keys; the 2.0 distilled row here exists for a + // generation that predates them, and resolving onto it would build a + // pipeline whose reference geometry is always the unscaled one. + if (model_version == "2.5") return IcLoraRecipe(model_version); } else if (pipeline_kind == "res2s_two_stage") { // 2.5 only — see `Res2sTwoStageRecipe`: `LTX_2_3_HQ_PARAMS` is a plain // constant with no `detect_params` lineage, so there is no second version to diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index a36beb85a..b4d0cbadf 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -46,6 +46,7 @@ #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" #include "vllm/model_executor/models/ltx2_samplers.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" #include "vllm/model_executor/models/ltx2_retake.h" #include "vllm/model_executor/models/ltx2_t2a.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" @@ -191,6 +192,16 @@ struct StreamState { // was supplied. Empty on the audio stream, whose args preprocessor upstream // builds with no keyframes_embedding_provider (model.py:333). std::vector keyframes_mask; + // `LatentState.attention_mask` (types.py:251-287), DENSE [tokens, tokens] in + // [0, 1], and the target token count every appending item measures the mask's + // block structure against. Row LTX25-IC-LORA-REF-VIDEO (#3020). + // + // EMPTY is upstream's `None` and is what every render carries unless an + // IC-LoRA conditioning attention mask was supplied. `noisy_tokens` is set once + // beside `tokens` on the target grid and never moves, which is exactly what + // `latent_tools.target_shape.token_count()` is upstream. + std::vector attention_mask; + int64_t noisy_tokens = 0; }; // ── StreamState <-> Ltx2LatentState (row LTX25-TOKEN-APPEND, issue #930) ──── @@ -223,6 +234,8 @@ Ltx2LatentState ToLatentState(const StreamState& s, int64_t pos_dims) { out.positions[i] = static_cast(s.positions[i]); } out.keyframes_mask = s.keyframes_mask; + out.attention_mask = s.attention_mask; + out.noisy_tokens = s.noisy_tokens; return out; } @@ -237,6 +250,8 @@ void FromLatentState(const Ltx2LatentState& in, StreamState* s) { s->positions[i] = static_cast(in.positions[i]); } s->keyframes_mask = in.keyframes_mask; + s->attention_mask = in.attention_mask; + s->noisy_tokens = in.noisy_tokens; } // `post_process_latent` (utils/helpers.py:462-464): @@ -482,7 +497,7 @@ constexpr char kLtx2AutoDurationExtra[] = "auto_duration"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 680 682 1324 1420 1516 1532 1667 1671 1829 1865 2017 2135 2177 2219 2221 +// 695 697 1339 1435 1531 1547 1682 1686 1844 1880 2032 2150 2192 2234 2236 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, @@ -2713,7 +2728,15 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2VideoSkipStepExtra || kv.first == kLtx2VideoStgBlocksExtra || kv.first == kLtx2A2vGuidanceScaleExtra || - kv.first == kLtx2V2aGuidanceScaleExtra; + kv.first == kLtx2V2aGuidanceScaleExtra || + // The IC-LoRA reference row (row LTX25-IC-LORA-REF-VIDEO, + // #3020). DEFINED here on every kind so that the refusal + // below can name the pipeline that serves them; a knob + // dropped from this list is refused as a TYPO, which + // tells a caller nothing about which pipeline to load. + kv.first == kLtx2RefVideoStrengthExtra || + kv.first == kLtx2CondAttentionMaskDirExtra || + kv.first == kLtx2CondAttentionStrengthExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -2727,7 +2750,9 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2AudioStgBlocksExtra + ", " + kLtx2VideoCfgScaleExtra + ", " + kLtx2VideoStgScaleExtra + ", " + kLtx2VideoRescaleScaleExtra + ", " + kLtx2VideoSkipStepExtra + ", " + kLtx2VideoStgBlocksExtra + ", " + - kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra); + kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra + ", " + + kLtx2RefVideoStrengthExtra + ", " + kLtx2CondAttentionMaskDirExtra + ", " + + kLtx2CondAttentionStrengthExtra); } } // ── the knobs that belong to ONE pipeline (#1005, corrected by #1092) ───── @@ -2767,7 +2792,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, kLtx2VideoRescaleScaleExtra, kLtx2VideoSkipStepExtra, kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, - kLtx2V2aGuidanceScaleExtra}; + kLtx2V2aGuidanceScaleExtra, + // A reference CLIP is a picture (#3020), + // and so is the mask that attenuates it. + kLtx2RefVideoStrengthExtra, + kLtx2CondAttentionMaskDirExtra, + kLtx2CondAttentionStrengthExtra}; for (const char* key : kNotOnT2a) { if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { Fail("the '" + std::string(key) + @@ -3217,111 +3247,119 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // extra tokens to a stage-1-only adapter (iclora_utils.py:112-117, :87-89, // :144-148). Serving the first says nothing about the second, so the second // stays refused and #975 stays open. - if (!wants_retake && (!gen.ref_image_paths.empty() || !gen.ref_video_dir.empty())) { - // TWO CAUSES REMAIN, AND NEITHER IS ONE THIS MESSAGE HAS EVER GIVEN. The - // message names both, and then names the three ruled-out reasons with what - // ruled each one out, because a reader who arrives here in a month should - // re-check the claim rather than re-derive the refutation for a third time. - // - // 1. THE REFERENCE CLIP HAS NO PIXEL PATH. Upstream resolves the reference - // at `height // scale` by `width // scale` (iclora_utils.py:116-117), - // refuses a target either axis of which the factor does not divide - // (:112-115), keeps frame 0 and then every Nth frame (`temporal_subsample`, - // :87-89, called at :144), and encodes the whole clip (:145-148). This - // engine's only pixel-to-latent route for a REFERENCE item is - // `Ltx2LoadImageAndPreprocess` followed by `Ltx2ConvVideoEncode` at - // `frame_count = 1` and the phase's OWN height and width, and it refuses - // an encode that returns more than one latent frame. - // - // THIS USED TO SAY "nothing anywhere reads `ref_video_dir`", and that was - // false about the tree even when it was written (#987): MiniMax-H3 has - // always consumed the directory in full — `ReadReferenceClipChw`, - // `minimax_h3_video.cpp:135`, called at `:650`. Since row LTX25-RETAKE - // (#924) the LTX-2.5 side reads it too, through - // `Ltx2ReadFrameDirectory`. So the missing piece is NOT a reader. It is - // the reference item's own geometry: the downscale-factor resize and the - // temporal subsample, neither of which retake performs and neither of - // which any reader supplies. - // - // THE SECOND REASON THIS MESSAGE GAVE IS NOW FALSE, and it is recorded - // here rather than deleted because it is the third reason in this block - // to come true and a reader needs to know which. It said: "the reference - // item belongs to stage 1 and stage 2 must run unfused — - // `ICLoraPipeline` gives stage 1 `loras=tuple(loras)` (ic_lora.py:108) - // and stage 2 `loras=()` (:119), and this engine holds ONE `Ltx2Dit`, - // fused at load, that every phase of the recipe runs. Serving the arm - // needs a second unfused DiT or a phase-scoped adapter." - // - // Row LTX25-PHASE-LORA (#1118) landed the phase-scoped adapter. - // `Ltx2PhaseRecipe::loras` (ltx2_pipeline.h) carries upstream's per-stage - // set and the phase loop in this file honours it through - // `Ltx2RebindDitLoras`, which re-materializes only the tensors an adapter - // targets — so a two-phase recipe CAN now give stage 1 the adapter and - // stage 2 none, which is exactly `ic_lora.py:108` against `:119`, and it - // does so without a second resident weight set. `A2VidTwoStageRecipe` is - // the executable proof it exists: it gives stage 1 `kNoAdapters` and the - // gate "the distilled adapter rides stage 2 ALONE" renders both states - // through this ABI and compares the pixels. - // - // What that leaves is reason 1 ALONE, and reason 1 is unrelated to - // weights: it is the reference clip's own geometry. The conditioning - // split is also still upstream's — stage 1 takes `_create_conditionings`, - // which appends the reference item (:269-278, :377-402), and stage 2 - // takes plain `combined_image_conditionings` with no reference item - // (:314-321) — but that is a conditioning question, not a fused-weight - // one, and serving the arm on one phase only is upstream's `skip_stage_2` - // (:302-308), a different request. - std::string factors = "no adapter was supplied, so none were read"; + // ── IC-LoRA REFERENCE CONDITIONING (row LTX25-IC-LORA-REF-VIDEO, #3020) ── + // + // ONE PREDICATE, READ ONCE, and it is what makes this block correct rather + // than merely present. `serves_reference` is bound here and used by BOTH the + // routing branch in the phase loop and every refusal below it. A refusal and + // its route predicate written as two expressions is how this campaign shipped + // a silently wrong answer, and the two are one local for exactly that reason. + // + // WHY IT IS A RECIPE FLAG. `ICLoraPipeline` (ic_lora.py:60) is the only + // pipeline in `ltx-pipelines` that calls + // `append_ic_lora_reference_video_conditionings`; `distilled.py`, `dfr`, + // `retake` and the four `*_two_stage` arms call none. Serving the reference on + // any of them would be inventing conditioning the reference does not have. + const bool serves_reference = im.recipe.ic_lora_reference; + const std::string cond_mask_dir = VideoExtra(gen.extras, kLtx2CondAttentionMaskDirExtra); + const double cond_attention_strength = + ExtraDouble(gen.extras, kLtx2CondAttentionStrengthExtra, 1.0); + const double ref_video_strength = ExtraDouble(gen.extras, kLtx2RefVideoStrengthExtra, 1.0); + + // `!wants_retake` IS LOAD-BEARING, and it narrows rather than weakens. Retake + // and IC-LoRA reference conditioning both arrive as `ref_video_dir`, and they + // consume it in completely different ways: retake encodes the clip at its OWN + // resolution and seeds the video stream's initial latent with it + // (retake.py:238-247, :273), while the reference item is downscaled by the + // adapter's factor, temporally subsampled, and APPENDED as extra tokens to a + // stage-1-only adapter (iclora_utils.py:112-117, :87-89, :144-148). + const bool wants_reference_video = !wants_retake && !gen.ref_video_dir.empty(); + + if (!wants_retake && !gen.ref_image_paths.empty()) { + // REFERENCE IMAGES ARE NOT UPSTREAM'S IC-LoRA SHAPE, and this stays refused + // whatever pipeline is loaded. `--video-conditioning` takes a video file or + // a directory of scene-linear EXR frames (ic_lora.py:416-425) and there is + // no `--reference-image` anywhere in `ltx-pipelines`. Serving a still here + // by encoding it at `frame_count = 1` would be a conditioning item upstream + // does not build, and it would silently answer a request for a different + // feature. Recorded owed in .agents/specs/ltx25-ic-lora-ref-video.md. + Fail( + "reference-IMAGE conditioning is not served, and it is not what an IC-LoRA reference is. " + "Upstream's IC-LoRA flag is `--video-conditioning PATH STRENGTH`, whose PATH is a video " + "file or a directory of scene-linear .exr frames (ic_lora.py:416-425); there is no " + "reference-image conditioning anywhere in `ltx-pipelines`, so serving a still here would " + "build a conditioning item the reference does not have. Use `ref_video_dir` with " + "'pipeline_kind' 'ic_lora' for a reference CLIP, first_frame_ppm / first_frame_path for " + "image-to-video, or last_frame_path for a closing keyframe. Recorded owed (#3020)."); + } + if (wants_reference_video && !serves_reference) { + // The SAME predicate the route uses, negated. A reference clip on a pipeline + // that has no reference conditioning is a request for a different pipeline, + // and saying so is more useful than saying the feature is missing — it is + // not missing, it is elsewhere. + std::string factors_note = "no adapter was supplied, so none were read"; if (im.dit.lora_fused_tensors > 0) { - factors = "the supplied adapter declares downscale=" + - std::to_string(im.dit.lora_reference.downscale) + - " temporal=" + std::to_string(im.dit.lora_reference.temporal) + - ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; + factors_note = "the supplied adapter declares downscale=" + + std::to_string(im.dit.lora_reference.downscale) + + " temporal=" + std::to_string(im.dit.lora_reference.temporal) + + ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; } - Fail( - "reference-image / reference-video conditioning is not served. TWO things are " - "missing. FIRST, the reference CLIP has no pixel path: upstream reads it at " - "`height // reference_downscale_factor` by `width // reference_downscale_factor` " - "(iclora_utils.py:116-117), refuses a target the factor does not divide (:112-115), " - "keeps frame 0 and then every Nth frame (`temporal_subsample`, :87-89, called at " - ":144) and encodes the whole clip (:145-148), while this engine's only " - "pixel-to-latent route for a REFERENCE item encodes exactly ONE frame at the phase's " - "own resolution. WHAT IS *NOT* THE REASON here: the READER. This message used to say " - "\"nothing reads `ref_video_dir` at all\", which was false about the tree when it was " - "written (#987) — MiniMax-H3 consumes the directory in full at " - "`minimax_h3_video.cpp:650` — and is doubly false now that row LTX25-RETAKE (#924) " - "reads it on this side through `Ltx2ReadFrameDirectory`. What is missing is the " - "reference item's own geometry, the downscale resize and the temporal subsample, " - "which no reader supplies. SECOND, the reference item is a STAGE-1 item and stage 2 " - "takes `combined_image_conditionings` with no reference item at all: `ICLoraPipeline` " - "gives stage 1 the reference conditioning (ic_lora.py:269-278) and stage 2 none " - "(:314-321), and this phase loop appends the same conditioning set to every phase. " - "That is a CONDITIONING gap and not a weights one. WHAT IS *NOT* THE REASON, because " - "this refusal has now given THREE reasons that later became false: (a) the IC-LoRA " - "METADATA. Row LTX25-IC-LORA (#923) " - "closed that; supply `lora_path` and the factors are read at load " - "(iclora_utils.py:30-49) — right now, " + factors + - ". (b) the TOKEN-APPEND machinery. This message blamed it on 2026-08-15 and row " - "LTX25-TOKEN-APPEND (#930) landed it in `c7cb59fbb` the next day: the phase loop " - "now binds a `target_tokens` local, grows `video.tokens` past it on an appending " - "item, carries the grown count through denoise, and trims back through " - "`Ltx2ClearConditioning` (ltx_core/tools.py:88-117) before unpatchify. The " - "last-frame keyframe arm is SERVED on exactly that machinery, which is the " - "executable proof it exists. (c) `Ltx2LatentState` carrying no attention-mask " - "field. On the DEFAULT arm upstream builds no mask: at " - "`conditioning_attention_strength >= 1.0` with no latent mask `attn_mask` is None " - "(iclora_utils.py:159-160) and `ConditioningItemAttentionStrengthWrapper` is " - "applied only `if attn_mask is not None` (:168-169). The sub-1.0 arm is owed by " - "#932, and it is not what blocks this one. (d) the FUSED-AT-LOAD adapter. This " - "message said until 2026-08-17 that stage 2 must run with no adapter while \"this " - "engine holds one DiT, fused at load, that every phase runs\", and row " - "LTX25-PHASE-LORA (#1118) closed it: `Ltx2PhaseRecipe::loras` carries upstream's " - "per-stage set and the phase loop rebinds the DiT through `Ltx2RebindDitLoras`, so " - "`loras=tuple(loras)` on stage 1 against `loras=()` on stage 2 (ic_lora.py:108, " - ":119) is now expressible with no second weight set. `a2vid_two_stage`'s stage 1 " - "runs `kNoAdapters` on exactly that machinery, which is the executable proof it " - "exists. Use first_frame_ppm / first_frame_path " - "for image-to-video, and last_frame_path for a closing keyframe."); + Fail("the '" + im.pipeline_kind + + "' pipeline does not condition on a reference clip, so `ref_video_dir` has no meaning on " + "it. `ICLoraPipeline` is the only pipeline in `ltx-pipelines` that calls " + "`append_ic_lora_reference_video_conditionings` (ic_lora.py:381-402); `distilled.py`, " + "`dfr_pipeline.py`, the four `*_two_stage` arms and `t2a_one_stage.py` call none of it. " + "Load with 'pipeline_kind' 'ic_lora' to serve a reference clip, or supply " + "'retake_start_time' and 'retake_end_time' to RETAKE a window of this directory instead " + "— which is a different operation: retake encodes the clip at its own resolution and " + "SEEDS the video latent with it (retake.py:238-247), while the reference item downscales " + "it, temporally subsamples it and APPENDS it as extra tokens (iclora_utils.py:112-117, " + ":87-89, :144-148). Right now, " + factors_note + "."); + } + if (!cond_mask_dir.empty() && !serves_reference) { + Fail("the '" + std::string(kLtx2CondAttentionMaskDirExtra) + "' extra attenuates the IC-LoRA " + "REFERENCE tokens' attention (iclora_utils.py:151-156, :168-169) and the '" + + im.pipeline_kind + + "' pipeline appends none, so there would be nothing for it to attenuate. Upstream applies " + "`ConditioningItemAttentionStrengthWrapper` at exactly one site and its argument is " + "always the `VideoConditionByReferenceLatent` built two lines above it; " + "`combined_image_conditionings` never wraps. Load with 'pipeline_kind' 'ic_lora'."); + } + if (serves_reference && !wants_reference_video) { + // `--video-conditioning` is `required=True` on this parser + // (ic_lora.py:416-425). An IC-LoRA render with no reference is upstream's + // distilled two-stage render, and defaulting to it silently would answer a + // request for a conditioned clip with an unconditioned one. + Fail("the 'ic_lora' pipeline conditions on a reference clip and this request supplied none. " + "`--video-conditioning PATH STRENGTH` is `required=True` upstream " + "(ic_lora.py:416-425), because `ICLoraPipeline` without it is exactly the distilled " + "two-stage pipeline. Set `ref_video_dir` to a directory of frame_%06d.ppm, or load with " + "'pipeline_kind' 'distilled_two_stage' for an unconditioned render."); + } + if (!cond_mask_dir.empty() && (cond_attention_strength < 0.0 || cond_attention_strength > 1.0)) { + Fail("'" + std::string(kLtx2CondAttentionStrengthExtra) + "' must be in [0.0, 1.0], got " + + std::to_string(cond_attention_strength) + + ". This is upstream's own refusal (ic_lora.py:230-233): the value multiplies a mask in " + "[0, 1] and the product becomes a log-space attention bias, so a value above 1 would " + "AMPLIFY attention rather than attenuate it."); + } + if (cond_mask_dir.empty() && cond_attention_strength < 1.0) { + // UPSTREAM HAS THIS BRANCH AND ITS CLI CANNOT REACH IT + // (`elif conditioning_attention_strength < 1.0`, iclora_utils.py:157-158). + // `conditioning_attention_strength` is assigned only inside + // `if args.conditioning_attention_mask is not None` (ic_lora.py:452-455) and + // is 1.0 otherwise, so a sub-1.0 strength always arrives WITH a mask. + // MEASURED rather than read: `kLtx2RefWrapScalarBelowOne` is what the pinned + // module returned for the no-mask, strength-0.5 call, and it is `true` — so + // the branch is real and this refusal is naming a Python-API-only arm rather + // than a branch that does not exist. Recorded owed (#3020). + Fail("'" + std::string(kLtx2CondAttentionStrengthExtra) + "' below 1.0 without '" + + std::string(kLtx2CondAttentionMaskDirExtra) + + "' is upstream's scalar-only attention arm (iclora_utils.py:157-158), which its own CLI " + "cannot reach: the strength is assigned only alongside a mask (ic_lora.py:452-455) and " + "is 1.0 otherwise. It is a Python-API-only branch, it is not ported, and it is recorded " + "owed (#3020) rather than guessed at. Supply a mask directory, or leave the strength at " + "1.0."); } if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { Fail( @@ -4169,6 +4207,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { StreamState video; video.width = vshape.channels; // patch_size 1 (VideoLatentPatchifier(1)) video.tokens = target_tokens; + // ...and remembers it, because every appending item measures the attention + // mask's block structure against the count BEFORE any item ran + // (mask_utils.py:236 gives the noisy rows the cross weight and `:242` leaves + // the prior-reference rows at zero). Bound to the SAME local the schedule + // reads, so a change to one moves both. + video.noisy_tokens = target_tokens; { std::vector volume(static_cast(vshape.channels) * static_cast(vshape.frames) * @@ -4753,6 +4797,175 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "size while omitting the trained embedding."); } + // ── THE IC-LoRA REFERENCE CLIP (row LTX25-IC-LORA-REF-VIDEO, #3020) ───── + // + // AFTER every image and keyframe item, because upstream appends it after + // `combined_image_conditionings` (ic_lora.py:377-402 runs at the END of + // `_create_conditionings`) and conditioning items are applied in list order + // (`state_with_conditionings`, helpers.py:448-458). Order is not cosmetic + // here: appended tokens land at the END of the sequence and + // `Ltx2ClearConditioning` trims from the end, so two appends in the wrong + // order swap their positions while every count still agrees. + // + // PHASE 0 ONLY, and that is upstream's split rather than an optimisation. + // `ICLoraPipeline` gives stage 1 `_create_conditionings`, which appends the + // reference item (ic_lora.py:269-281), and stage 2 plain + // `combined_image_conditionings` with NO reference item (`:314-321`). + // Appending it to stage 2 as well would put the reference tokens into a + // 3-step refinement that upstream runs unconditioned — and the clip would + // come out the right length at the right resolution. + // + // THE SAME `serves_reference` LOCAL the refusals above read. Two expressions + // for one predicate is how a refusal and its route drift apart. + if (serves_reference && phase_index == 0) { + // `scale = reference_downscale_factor` and the divisibility refusal + // (iclora_utils.py:111-115), then `height // scale` by `width // scale` + // (`:116-117`). Measured against the phase's OWN grid, which is stage 1's + // half resolution — the same number `ic_lora.py:460-461` spells + // `args.height // 2`, derived here rather than assumed. + const Ltx2IcLoraReferenceGeometry ref_geom = Ltx2ResolveIcLoraReferenceGeometry( + phase_h, phase_w, im.dit.lora_reference.downscale); + + // `decode_video_by_frame(path, frame_cap=num_frames)` then + // `video_preprocess(frames, ref_height, ref_width, ...)` + // (iclora_utils.py:141-142). `Ltx2ReadFrameDirectory` IS `video_preprocess` + // over the folder arm — row LTX25-RETAKE ported and gated it — so what is + // new here is only the GEOMETRY it is called at. + const Ltx2RetakeSourceGeometry ref_probe = Ltx2ProbeFrameDirectory(gen.ref_video_dir); + // `frame_cap` (`:141`). The reference may be longer than the render, and a + // clip read past the cap would append tokens for moments the target grid + // has no time for. + const int64_t ref_frames = std::min(ref_probe.frames, frames); + const std::vector ref_pixels = + Ltx2ReadFrameDirectory(gen.ref_video_dir, ref_geom.height, ref_geom.width); + const int64_t ref_channels = im.video_encoder_cfg.in_channels; + const int64_t ref_plane = ref_geom.height * ref_geom.width; + std::vector ref_clip; + if (ref_frames == ref_probe.frames) { + ref_clip = ref_pixels; + } else { + // The cap, applied on the FRAME axis of a channel-major volume. A plain + // prefix of the buffer would keep whole channels and drop others. + ref_clip.resize(static_cast(ref_channels * ref_frames * ref_plane)); + for (int64_t c = 0; c < ref_channels; ++c) { + const size_t src = static_cast(c * ref_probe.frames * ref_plane); + const size_t dst = static_cast(c * ref_frames * ref_plane); + std::copy(ref_pixels.begin() + static_cast(src), + ref_pixels.begin() + + static_cast(src + static_cast(ref_frames * ref_plane)), + ref_clip.begin() + static_cast(dst)); + } + } + + // `if reference_temporal_scale_factor > 1: video = temporal_subsample(...)` + // (`:143-144`). GUARDED, and the guard is measured: at factor 1 upstream + // hands the encoder all 5 frames of the probe fixture and at factor 2 it + // hands it 3, which `kLtx2RefEncodedFramesN1` and `...N2` pin. + int64_t ref_kept = ref_frames; + if (im.dit.lora_reference.temporal > 1) { + ref_clip = Ltx2TemporalSubsample(ref_clip, ref_channels, ref_frames, ref_plane, + im.dit.lora_reference.temporal); + ref_kept = static_cast( + Ltx2TemporalSubsampleIndices(ref_frames, im.dit.lora_reference.temporal).size()); + } + + // `encoded_video = video_encoder(video)` (`:148`) — the WHOLE clip, which + // is the same multi-frame encode the retake arm already drives. The + // `tiled_encode` arm (`:145-146`) is not reached: this engine tiles the + // target's decode and not a reference's encode, and it is recorded owed. + int64_t ref_cropped = 0; + const Ltx2LatentVolume ref_encoded = Ltx2ConvVideoEncode( + im.video_encoder_cfg, im.video_encoder_weights, ref_clip, ref_channels, ref_kept, + ref_geom.height, ref_geom.width, &ref_cropped); + im.trace.ic_lora_reference_digest = DigestF32(ref_encoded.data); + im.trace.ic_lora_reference_absmax = AbsMax(ref_encoded.data); + + // ── the MASK half (gap A16) ───────────────────────────────────────── + // + // Built BEFORE the item is applied, because it needs the reference + // LATENT's shape (`reference_video_shape`, `:149`) and because the + // wrapper measures its block structure against the PRE-append state + // (attention_strength_wrapper.py:49-64). + std::vector cross_mask; + if (!cond_mask_dir.empty()) { + // `_load_mask_video` (ic_lora.py:511-537): the same read, at the STAGE's + // resolution rather than the reference's — upstream passes + // `args.height // 2` and not `ref_height` (`:460-461`) — then mean over + // channels, `(x + 1) / 2`, clamp. + const Ltx2RetakeSourceGeometry mask_probe = Ltx2ProbeFrameDirectory(cond_mask_dir); + const int64_t mask_frames = std::min(mask_probe.frames, frames); + const std::vector mask_pixels = + Ltx2ReadFrameDirectory(cond_mask_dir, phase_h, phase_w); + const std::vector mask_video = Ltx2MaskVideoFromPixels( + mask_pixels, ref_channels, mask_probe.frames, phase_h * phase_w); + std::vector capped( + mask_video.begin(), + mask_video.begin() + static_cast(mask_frames * phase_h * phase_w)); + // `downsample_mask_video_to_latent(mask, target_latent_shape= + // reference_video_shape)` (`:151-155`) — the REFERENCE's latent shape and + // not the target's, because these weights attenuate the reference's own + // tokens. + std::vector latent_mask = Ltx2DownsampleMaskVideoToLatent( + capped, mask_frames, phase_h, phase_w, ref_encoded.frames, ref_encoded.height, + ref_encoded.width); + // `attn_mask = latent_mask * conditioning_attention_strength` (`:156`). + for (float& v : latent_mask) v *= static_cast(cond_attention_strength); + // `resolve_cross_mask`'s 1-D arm (mask_utils.py:49-54), which refuses a + // length that is not the new-token count rather than broadcasting. + cross_mask = Ltx2ResolveCrossMask(latent_mask, /*scalar=*/1.0, + static_cast(latent_mask.size())); + } + + // THE SEQUENCE LENGTH AND THE MASK BEFORE THIS ITEM. The wrapper snapshots + // the ORIGINAL state (attention_strength_wrapper.py:50) and builds against + // it; building against the post-append count would place the new block + // over the tokens it just added while every dimension still agreed. + const int64_t before_reference = video.tokens; + const std::vector mask_before = video.attention_mask; + + Ltx2LatentState state = ToLatentState(video, /*pos_dims=*/3); + Ltx2ConditionVideoByReference(&state, ref_encoded, /*patch_size=*/1, factors, fps, + im.dit.lora_reference.downscale, + im.dit.lora_reference.temporal, ref_video_strength, + /*causal_fix=*/true); + FromLatentState(state, &video); + + const int64_t ref_tokens = video.tokens - before_reference; + VT_CHECK(ref_tokens > 0, + "ltx2 video: the reference item must APPEND tokens " + "(reference_video_cond.py:96-108) and this one left the sequence unchanged"); + VT_CHECK(static_cast(video.latent.size()) == video.tokens * video.width && + static_cast(video.clean.size()) == video.tokens * video.width && + static_cast(video.mask.size()) == video.tokens && + static_cast(video.keyframes_mask.size()) == video.tokens && + static_cast(video.positions.size()) == 3 * video.tokens * 2, + "ltx2 video: after the reference append every per-token buffer must have one entry " + "per token. A buffer that did not grow with the others is invisible to the " + "render's SHAPE — the clip comes out the right size and describes the wrong " + "tokens."); + // MEASURED GROWTH, not a count recomputed from the latent's shape. A + // derived number agrees with itself on a build that computed the shape and + // appended nothing. + im.trace.ic_lora_reference_tokens = ref_tokens; + + // `ConditioningItemAttentionStrengthWrapper.apply_to` (`:55-71`): the mask + // is built from the ORIGINAL state and its `num_new_tokens` is the + // measured difference, then it REPLACES whatever the inner item's own + // `update_attention_mask(None, ...)` left (which is the pad-with-ones + // form, or nothing at all on a first item). + if (!cross_mask.empty()) { + VT_CHECK(static_cast(cross_mask.size()) == ref_tokens, + "ltx2 video: the conditioning attention mask resolved to " + + std::to_string(cross_mask.size()) + " weights but the reference item " + "appended " + std::to_string(ref_tokens) + + " tokens. The mask is downsampled to the REFERENCE latent's own shape " + "(iclora_utils.py:151-155), so these cannot differ unless one of the two " + "read a different grid."); + video.attention_mask = Ltx2BuildAttentionMask(mask_before, video.noisy_tokens, ref_tokens, + before_reference, cross_mask); + } + } + // The noiser draws VIDEO first, AUDIO second, from one generator // (blocks.py:554-563 builds the video state before the audio one; :576-580, // which this used to cite, is the TEARDOWN and proves nothing about order). @@ -5006,6 +5219,47 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "trained term and look exactly like a working render."); vin.keyframes_mask = video.keyframes_mask.data(); } + + // ── THE PRODUCTION ASSIGNMENT OF `attention_mask` (#3020) ──────────── + // + // THIS LINE IS WHAT THIS ROW EXISTS FOR. `Ltx2ModalityInput::attention_mask`, + // `Ltx2PrepareSelfAttentionMask`, `self_bias` on host and device, and the + // additive bias inside `vt::AttentionCross` were all built and all + // correct, and until this row NOTHING IN `src/` ASSIGNED EITHER FIELD: + // the only assignments in the tree were four lines of a `BuildModalities` + // helper in `test_ltx2.cpp` and `test_ltx2_device.cpp`. That is the shape + // .agents/reachability.md names — a unit test that constructs the type + // proves the class works, never that anything reaches it. Deleting this + // assignment is the mutation the reachability case must red on. + // + // THE DENSE FORM, `rows == tokens`. `build_attention_mask` returns + // (B, N+M, N+M) unconditionally (mask_utils.py:220) and its block + // structure is not expressible as one row. + // + // SIZE CHECKED BEFORE `data()` IS TAKEN, exactly as the keyframes mask + // above is and for the identical reason: an empty vector's `data()` is a + // null pointer, which the forward reads as upstream's legal "no mask". A + // mask built and then dropped renders a finite, correctly shaped, plausible + // clip with a trained term silently omitted. + if (!video.attention_mask.empty()) { + VT_CHECK(static_cast(video.attention_mask.size()) == + video.tokens * video.tokens, + "ltx2 video: a self-attention strength mask must be dense [tokens, tokens] " + "(mask_utils.py:220). One that is short of the sequence is NOT a shape error " + "downstream — it is read as the key-only broadcast form and masks the wrong " + "axis."); + vin.attention_mask = video.attention_mask.data(); + vin.attention_mask_rows = video.tokens; + // Observed off the buffer HANDED OVER, not off the one built. `min` and + // `max` both, because an all-ones mask is the identity: it renders + // correctly, it has the right shape, and it is exactly what a downsample + // that lost its values produces. + const auto range = + std::minmax_element(video.attention_mask.begin(), video.attention_mask.end()); + im.trace.ic_lora_attention_mask_rows = vin.attention_mask_rows; + im.trace.ic_lora_attention_mask_min = static_cast(*range.first); + im.trace.ic_lora_attention_mask_max = static_cast(*range.second); + } // AND THE HANDOVER IS CHECKED SEPARATELY FROM THE CONSTRUCTION, because the // check above cannot see the handover. It reads `video.keyframes_mask` — the // VECTOR — so it fires when the mask is built conditionally and stays silent diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 5e75f8210..2ef1e2443 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1774,96 +1774,57 @@ TEST_CASE("ltx2 video: keyframe and reference conditioning is refused BY WHAT IS const vllm::multimodal::VideoResult result = engine->Generate(gen); CHECK(result.frame_count == 9); } - SUBCASE("a reference video may not blame a seam THIS ENGINE demonstrably has") { - // WHAT THIS CASE USED TO DO, AND WHY THAT WAS THE DEFECT. It asserted five - // SUBSTRINGS of the refusal: `reference_video_cond.py`, `clear_conditioning`, - // `TOKEN-APPEND`, `NOT the IC-LoRA metadata`, and the absence of one retired - // phrase. Two of those five are UPSTREAM symbol names, which are present in - // the pinned checkout whatever this engine can do, and the other three are - // literals the message declares about itself. So not one of them could go - // red when the ENGINE changed — and the engine did change, twice, in two - // days: #923 made the metadata readable and #930 (`c7cb59fbb`) built the - // token-append seam. A reviewer replaced the local-cause sentence with a - // self-declared falsehood, kept all five substrings, and the whole suite - // stayed green. + SUBCASE("a reference video on a pipeline that does not condition on one names where it lives") { + // THIS SUBCASE USED TO ASSERT A REFUSAL OF THE FEATURE ITSELF, and row + // LTX25-IC-LORA-REF-VIDEO (#3020) SERVED it. What it required — that the + // refusal not blame the token-append seam this engine demonstrably has, and + // that every closed cause appear only under a `WHAT IS *NOT* THE REASON` + // marker — was the right constraint on a message whose job was to explain a + // MISSING capability. The capability is no longer missing. // - // SO THIS CASE MEASURES THE ENGINE FIRST and only then constrains the - // message. The measurement is the same instrument the token-append row - // gates itself with: `video_tokens` is written INSIDE the phase loop, so it - // can observe what the loop does, unlike every field filled before denoise. - const vllm::multimodal::VideoModelParams cond_params = ConditioningParams(ws.paths); - const std::string kf = ws.root + "/append_witness.ppm"; - WriteBytes(kf, ConditioningPpm(20, 28, 31)); - - auto tokens_of = [&](const std::string& tag, const std::string& keyframe) { - const std::unique_ptr own = - vllm::multimodal::LoadVideoEngine(cond_params); - auto* ltx2 = dynamic_cast(own.get()); - REQUIRE(ltx2 != nullptr); - vllm::multimodal::VideoGenParams g = FixtureGen(ws.root + "/" + tag); - if (!keyframe.empty()) { - g.last_frame_path = keyframe; - g.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; - } - const vllm::multimodal::VideoResult result = own->Generate(g); - // THE TRIM, observed from outside: the volume handed to unpatchify is the - // target grid, so the clip comes back at the requested length whether or - // not anything was appended. - CHECK(result.frame_count == 9); - return ltx2->last_conditioning().video_tokens; - }; - - const int64_t plain = tokens_of("ref_witness_plain", ""); - const int64_t grown = tokens_of("ref_witness_grown", kf); - // THE GROWTH. Both numbers are measured; pinning either to a literal would - // pass on a build that never grew anything. - REQUIRE_MESSAGE(grown > plain, - "this engine's phase loop did not grow its token sequence for an appending " - "conditioning item (" - << grown << " against a target of " << plain - << "), so the rest of this case cannot say what the refusal may claim"); - + // So the constraint is replaced by the stronger one the new shape allows: + // the message may not describe a missing feature AT ALL, because there is + // none. It has to name the pipeline that serves it. That is a property a + // reader can act on in one step, where the old message asked them to + // re-derive which of four causes still held. + // + // The token-append measurement the old subcase made is not dropped: it moved + // to `ltx2 ic-lora: a reference clip REACHES the render through the public + // ABI`, which measures the same growth by rendering through it. const std::string msg = refusal("a reference video", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { g.ref_video_dir = w.root; }); INFO(msg); - // BECAUSE THE TWO MEASUREMENTS ABOVE HOLD, the refusal may not CLAIM the - // phase loop. It may still MENTION it — the message's own convention is to - // record a ruled-out cause under `WHAT IS *NOT* THE REASON` so the next - // reader re-checks rather than re-derives — so the property asserted here is - // positional: every occurrence of a closed cause sits after that marker. - // - // That is what makes this case red for the mutation that motivated it. - // Restoring the pre-repair message leaves no marker at all AND puts - // `TOKEN-APPEND` in the first sentence, so both halves fire. - const size_t ruled_out = msg.find("WHAT IS *NOT* THE REASON"); - REQUIRE_MESSAGE(ruled_out != std::string::npos, - "the refusal carries no `WHAT IS *NOT* THE REASON` section, so a cause this " - "engine has already closed cannot be told apart from one it still has"); - for (const char* closed : {"TOKEN-APPEND", "fixed at the target grid's token count", - "nowhere to go", "nothing to trim"}) { - const size_t at = msg.find(closed); - const bool only_as_ruled_out = (at == std::string::npos) || (at > ruled_out); - CHECK_MESSAGE(only_as_ruled_out, + // IT NAMES WHERE THE FEATURE IS, and does not describe it as absent. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("ic_lora.py:381-402") != std::string::npos); + CHECK(msg.find("ref_video_dir") != std::string::npos); + // ...and it distinguishes the OTHER thing this directory can mean, which is + // the whole reason `!wants_retake` guards the route. + CHECK(msg.find("retake_start_time") != std::string::npos); + + // AND IT CLAIMS NO MISSING SEAM. Every phrase below named a cause this + // engine has since built, and each one appeared in this refusal at some + // point in the campaign; a message that reintroduces any of them is + // describing a build that no longer exists. There is no `WHAT IS *NOT* THE + // REASON` escape hatch here, because a refusal that only says "load a + // different pipeline" has no reason to mention a seam at all. + for (const char* closed : {"TOKEN-APPEND", "is not served", "has no pixel path", + "no reader", "second resident", "fused at load"}) { + CHECK_MESSAGE(msg.find(closed) == std::string::npos, "the refusal states '" << std::string(closed) - << "' as a cause rather than as a ruled-out one, and this case has just " - "MEASURED that the loop grows (" - << plain << " -> " << grown << ") and trims back to the target grid"); + << "', which describes a MISSING capability. The reference arm is served " + "on 'pipeline_kind' 'ic_lora' (#3020) and this message's only job is " + "to say so."); } - // The metadata half, same shape: the factors printed are READ from the - // adapter at load, so this asserts the read happened rather than asserting - // a sentence about it. `factors` says "no adapter was supplied" here. + // The adapter factors are still READ and reported, which is what says the + // metadata half happened rather than asserting a sentence about it. CHECK(msg.find("no adapter was supplied") != std::string::npos); - CHECK(msg.find("which this project does not read") == std::string::npos); - // And the two causes that DO remain are named, by the upstream anchors a - // reader can go and check. - CHECK(msg.find("iclora_utils.py:116-117") != std::string::npos); - CHECK(msg.find("ic_lora.py:108") != std::string::npos); - CHECK(msg.find("ref_video_dir") != std::string::npos); } + SUBCASE("reference audio names the AUDIO encoder, which this row did not build") { const std::string msg = refusal("reference audio", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { @@ -7379,9 +7340,13 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { // // IT IS STILL A REFUSAL AFTER ROW LTX25-IMAGE-COND (#644), which served the // first-frame arm and would have made a `first_frame_ppm` probe stop - // refusing. The reference arm stays refused for a reason this row did not - // touch (the IC-LoRA scale factors), so the probe was moved to it rather than - // to whatever happened to still throw. + // refusing, and after row LTX25-IC-LORA-REF-VIDEO (#3020), which served the + // reference CLIP. THE PROBE IS NOW A REFERENCE IMAGE and that is the durable + // choice rather than the convenient one: `--video-conditioning` takes a video + // file or an EXR directory (ic_lora.py:416-425) and there is no + // reference-IMAGE conditioning anywhere in `ltx-pipelines`, so this arm is + // refused because upstream has nothing to mirror — not because a seam here is + // missing. It is the one refusal in this file that no later row can serve. Workspace ws; const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); const std::unique_ptr engine = @@ -7398,7 +7363,7 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - CHECK(msg.find("reference-image / reference-video conditioning") != std::string::npos); + CHECK(msg.find("reference-IMAGE conditioning is not served") != std::string::npos); } const vllm::multimodal::Ltx2ConditioningTrace trace = ltx->last_conditioning(); @@ -8591,29 +8556,36 @@ TEST_CASE("ltx2 retake: the wrong recipe refuses, and the reference arm still do } } - SUBCASE("a reference clip WITHOUT the retake knobs is still refused, and #975 stays open") { - // THE LOCAL FACT this row's refusal edit turns on: the LTX side now reads - // `ref_video_dir`, so the sentence claiming nothing did is gone. A case that - // asserted only upstream symbol names could not see that going stale — which - // is exactly how the sentence survived (#987). + SUBCASE("a reference clip WITHOUT the retake knobs points at the pipeline that serves it") { + // THE LOCAL FACT this subcase turns on has CHANGED TWICE and the history is + // kept because that is the point of the case. It first asserted that the + // refusal must not claim nothing reads `ref_video_dir`, because the LTX side + // reads it (#987). Row LTX25-IC-LORA-REF-VIDEO (#3020) then SERVED the + // reference arm, on `pipeline_kind` `ic_lora`, so there is no missing seam + // left for a message to misdescribe. + // + // What survives is the shape: at least one assertion tied to a fact about + // THIS tree, so a message whose claim goes stale is visible. That fact is + // now that the engine HAS the feature and this pipeline is not it. vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refstill"); gen.ref_video_dir = clip; try { (void)engine->Generate(gen); - FAIL_CHECK("reference-video conditioning must still be refused"); + FAIL_CHECK("a reference clip on a distilled_two_stage engine must be refused"); } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - const size_t ruled_out = msg.find("WHAT IS *NOT* THE REASON"); - REQUIRE(ruled_out != std::string::npos); - const size_t stale = msg.find("nothing reads `ref_video_dir` at all"); - const bool only_as_ruled_out = (stale == std::string::npos) || (stale > ruled_out); - CHECK_MESSAGE(only_as_ruled_out, - "the refusal states as a CAUSE that nothing reads ref_video_dir, and this " - "suite has just rendered a retake that reads it"); - // And it names the reader that DOES exist, so the next reader goes looking - // for the reference item's geometry rather than for a file walker. - CHECK(msg.find("Ltx2ReadFrameDirectory") != std::string::npos); + // It names the pipeline that serves it, and the two operations this same + // directory can mean — which is what a caller can act on. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("retake_start_time") != std::string::npos); + CHECK(msg.find("SEEDS the video latent") != std::string::npos); + // AND IT CLAIMS NOTHING IS MISSING. This suite has just rendered a retake + // that reads this very directory, and the row above serves the reference + // arm from it, so a message that described either as absent would be + // describing a build that no longer exists. + CHECK(msg.find("nothing reads `ref_video_dir` at all") == std::string::npos); + CHECK(msg.find("is not served") == std::string::npos); } } } @@ -13867,18 +13839,18 @@ TEST_CASE("ltx2 video: auto_duration refuses MIN > MAX and refuses doubling an e } } + // ───────────────────────────────────────────────────────────────────────────── // IC-LoRA REFERENCE VIDEO and its CONDITIONING ATTENTION MASK // Row LTX25-IC-LORA-REF-VIDEO, issue #3020, // spec .agents/specs/ltx25-ic-lora-ref-video.md (gaps A15 and A16). // -// These are the REACHABILITY cases, and they are the only proof this row has. +// These are the REACHABILITY cases and they are the only proof this row has. // They enter through `LoadVideoEngine` and `Generate` — where -// `vllm_video_generate` arrives — and not through -// `Ltx2AppendIcLoraReferenceConditionings`. The value-level cases for the ported -// pieces live in `test_ltx2_iclora_reference` and per .agents/reachability.md -// they localize a failure rather than prove one: they stay green when the -// production call site is deleted. +// `vllm_video_generate` arrives — and not through the ported helpers. The +// value-level cases live in `test_ltx2_iclora_reference` and per +// .agents/reachability.md they localize a failure rather than prove one: they +// stay green when the production call site is deleted. // // THE SEAM THIS ROW RETIRES. Before it, `Ltx2ModalityInput::attention_mask` and // its whole consumption chain — `Ltx2PrepareSelfAttentionMask`, `self_bias` on @@ -13892,15 +13864,89 @@ namespace { // An `ic_lora` engine. The kind is a LOAD extra, because `ICLoraPipeline` is a // pipeline CLASS upstream (ic_lora.py:60) and not a per-request mode: stage 2 -// runs `loras=()` against stage 1's `loras=tuple(loras)` (`:108` against `:119`), -// which is a property of how the weights are held. -vllm::multimodal::VideoModelParams IcLoraParams(const ltx2_fixture::Paths& paths) { +// runs `loras=()` against stage 1's `loras=tuple(loras)` (`:108` against +// `:119`), which is a property of how the weights are held. +// +// `skip_stage_2` IS UPSTREAM'S OWN FLAG (`--skip-stage-2`, ic_lora.py:442-448), +// spelled here as the `max_phase` load extra this engine already has. It is not +// a test convenience, and the case below records exactly why one leg needs it. +vllm::multimodal::VideoModelParams IcLoraParams(const ltx2_fixture::Paths& paths, + const std::string& upsampler, + bool skip_stage_2 = false) { vllm::multimodal::VideoModelParams mp = FixtureParams(paths); mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "ic_lora"; mp.extras[vllm::multimodal::kLtx2CheckpointClassExtra] = FixtureCheckpointClass("ic_lora"); + // Stage 2 spatially upsamples, exactly as the distilled two-stage recipe this + // one derives from does (`ICLoraPipeline` calls `self.upsampler` at + // ic_lora.py:307). + mp.extras["upsampler_path"] = upsampler; + if (skip_stage_2) mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; return mp; } +// The spatial x2 upsampler stage 2 needs, written at the fixture's own reduced +// channel count so the phase's shapes follow the DiT contract rather than a +// literal. +std::string WriteIcLoraUpsampler(const std::string& path) { + vllm::Ltx2UpsamplerConfig cfg = + ltx2_fixture::ReducedUpsamplerConfig(ltx2_fixture::ReducedDitParams().in_channels); + cfg.dims = 2; + ltx2_fixture::WriteReducedUpsampler(cfg, path); + return path; +} + +// A clip of CONSTANT frames, and NOT `ConditioningPpm`'s seeded ramp. +// +// MEASURED, and it is the reduced-fixture trap this campaign has recorded twice. +// `ConditioningPpm` fills a frame with `(i * 37 + seed * 101) % 251` — a dense +// high-frequency ramp that a seed only PHASE-SHIFTS. The video VAE downsamples +// 32x spatially, so two seeds average to nearly the same latent: measured absmax +// 0.539 against 0.464, with the two renders identical to the byte. A fixture +// built on it cannot see whether the reference reached the DiT at all. Constant +// frames survive the downsample, and 20 against 235 encode to latents whose +// absmax is 0.434 against 0.637. +// +// The retake cases use the ramp and are right to: retake SEEDS the video latent +// with the clip, so a small latent difference propagates one-for-one. A +// reference's whole effect arrives through ATTENTION, which attenuates. +std::string WriteIcLoraClip(const std::string& dir, int frames, int height, int width, + unsigned char value) { + ::mkdir(dir.c_str(), 0755); + for (int i = 0; i < frames; ++i) { + std::string ppm = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + ppm.append(static_cast(height) * static_cast(width) * 3, + static_cast(value)); + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06d.ppm", i); + WriteBytes(dir + name, ppm); + } + return dir; +} + +// A grayscale mask clip: the LEFT half at `left` and the right half at `right`, +// so the mask is SPATIALLY structured rather than uniform. A uniform mask cannot +// separate "the mask reached the DiT" from "the strength scalar did", and an +// all-ones mask is the identity. +std::string WriteIcLoraMask(const std::string& dir, int frames, int height, int width, + unsigned char left, unsigned char right) { + ::mkdir(dir.c_str(), 0755); + for (int i = 0; i < frames; ++i) { + std::string ppm = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + for (int y = 0; y < height; ++y) { + for (int x = 0; x < width; ++x) { + const char v = static_cast(x < width / 2 ? left : right); + ppm.push_back(v); + ppm.push_back(v); + ppm.push_back(v); + } + } + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06d.ppm", i); + WriteBytes(dir + name, ppm); + } + return dir; +} + // 128x128 and NOT the 64x64 every other case uses, deliberately. The distilled // two-stage recipe runs stage 1 at half resolution, so the reference clip is // read at 64x64, and the fixture VAE's (8, 32, 32) factors then give a 2 x 2 x 2 @@ -13921,21 +13967,28 @@ vllm::multimodal::VideoGenParams IcLoraGen(const std::string& out_dir, const std return gen; } +std::string FirstFrame(const vllm::multimodal::VideoResult& result) { + return ReadAll(std::string(result.frame_dir) + "/frame_000000.ppm"); +} + } // namespace TEST_CASE("ltx2 ic-lora: a reference clip REACHES the render through the public ABI") { // THE RED-FIRST CASE. Before this row `pipeline_kind=ic_lora` resolved to no - // recipe and `ref_video_dir` was refused outright on every kind but `retake`, - // so nothing here could reach the new code at all. + // recipe at all and `ref_video_dir` was refused outright on every kind but + // `retake`, so nothing here could reach the new code. The literal red was + // ERROR: test case THREW exception: Unsupported LTX pipeline kind/version: + // 'ic_lora'/'2.5'. Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/ic_ups.safetensors"); const std::unique_ptr engine = - vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths)); + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups)); auto* ltx2 = dynamic_cast(engine.get()); REQUIRE(ltx2 != nullptr); // Stage 1 renders at 64x64 and the fixture's factors are (8, 32, 32), so the // reference clip encodes to 2 x 2 x 2 = 8 tokens. - const std::string clip = WriteRetakeClip(ws.root + "/ic_clip", 9, 64, 64); + const std::string clip = WriteIcLoraClip(ws.root + "/ic_clip", 9, 64, 64, /*value=*/20); const vllm::multimodal::VideoResult result = engine->Generate(IcLoraGen(ws.root + "/ic_out", clip)); @@ -13943,27 +13996,294 @@ TEST_CASE("ltx2 ic-lora: a reference clip REACHES the render through the public CHECK(trace.completed); CHECK_MESSAGE(trace.ic_lora_reference_tokens == 8, "the reference clip did not append one latent frame's worth of tokens per latent " - "frame; got " << trace.ic_lora_reference_tokens); - // The lower bound: a clip that encoded to zeros has the right shape, the right - // token count and renders. + "frame; got " + << trace.ic_lora_reference_tokens); + // The lower bound a token count cannot make: a clip that encoded to zeros has + // the right shape, the right count, and renders. CHECK(trace.ic_lora_reference_absmax > 0.0); - // The output geometry comes from the request, not from the reference. + CHECK(trace.ic_lora_reference_digest != 0); + // The output geometry comes from the REQUEST, not from the reference — which + // is where it differs from retake, whose geometry comes from the clip. CHECK(result.frame_count == 9); - - // THE RENDER DEPENDS ON THE REFERENCE CLIP'S PIXELS. `ic_lora_reference_absmax` - // observes the ENCODE and says nothing about whether the encoded latent was - // ever handed to the phase. A build that read the clip, encoded it, recorded - // the trace and then appended zeros satisfies every check above and renders a - // clip of the right length. Two DIFFERENT references at the same seed must - // therefore produce different pixels. - const std::string other = WriteRetakeClip(ws.root + "/ic_clip_b", 9, 64, 64, /*seed_base=*/113); + CHECK(result.width == 128); + CHECK(result.height == 128); + // NO MASK WAS SUPPLIED, so no mask was handed over. `attention_mask` stays a + // null pointer and the DiT takes upstream's `attention_mask is None` branch. + // Asserted, because a build that handed over an all-ones mask would render + // identically and would be measuring nothing at every case below. + CHECK(trace.ic_lora_attention_mask_rows == 0); + + // THE RENDER DEPENDS ON THE REFERENCE CLIP'S PIXELS, and this is the assertion + // the trace above cannot make. `ic_lora_reference_absmax` observes the ENCODE; + // it says nothing about whether the encoded latent was ever handed to the + // phase. A build that read the clip, encoded it, recorded the trace and then + // appended zeros satisfies every check above. + // + // ON THE SKIP-STAGE-2 ARM, AND THAT IS MEASURED RATHER THAN CONVENIENT. + // Upstream applies the reference to stage 1 ONLY (ic_lora.py:269-281 against + // `:314-321`) and stage 2 re-noises the upsampled latent to + // `STAGE_2_DISTILLED_SIGMAS[0]` = 0.909375, so 91% of stage 2's input is fresh + // noise. MEASURED on this reduced fixture: stage 1's latents for the two clips + // DO differ (clean digests 7109293506366579875 against 12136535927875385707) + // and stage 2's inputs still differ, but the surviving difference falls below + // 1/255 and the two-stage renders come out byte-identical. That is a property + // of a 2-block random-weight DiT, not of the port, and the honest answer is to + // make the pixel claim where the conditioning is applied. `--skip-stage-2` is + // upstream's own flag for exactly this configuration (ic_lora.py:442-448), so + // this leg is a supported request shape and not a test hook. + const std::unique_ptr stage1 = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + const std::string other = WriteIcLoraClip(ws.root + "/ic_clip_b", 9, 64, 64, /*value=*/235); REQUIRE(ReadAll(clip + "/frame_000000.ppm") != ReadAll(other + "/frame_000000.ppm")); - const vllm::multimodal::VideoResult from_other = - engine->Generate(IcLoraGen(ws.root + "/ic_out_b", other)); - const std::string frame_a = ReadAll(std::string(result.frame_dir) + "/frame_000000.ppm"); - const std::string frame_b = ReadAll(std::string(from_other.frame_dir) + "/frame_000000.ppm"); + const std::string frame_a = FirstFrame(stage1->Generate(IcLoraGen(ws.root + "/s1_a", clip))); + const std::string frame_b = FirstFrame(stage1->Generate(IcLoraGen(ws.root + "/s1_b", other))); REQUIRE(frame_a.size() == frame_b.size()); CHECK_MESSAGE(frame_a != frame_b, "two different reference clips rendered byte-identical frames at the same seed, " "so the encoded reference latent never reached the phase"); } + +TEST_CASE("ltx2 ic-lora: the conditioning attention MASK reaches the DiT and moves the render") { + // THE CASE THAT RETIRES THE DEAD SEAM. `Ltx2ModalityInput::attention_mask` had + // no assignment in `src/` before this row; the only four in the tree were in a + // `BuildModalities` test helper. This is the production assignment, reached + // from `Generate`. + Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/mask_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + const std::string clip = WriteIcLoraClip(ws.root + "/mask_clip", 9, 64, 64, /*value=*/20); + // The mask is read at the STAGE's resolution (64x64), which is what upstream + // spells `args.height // 2` (ic_lora.py:460-461), and downsampled to the + // REFERENCE latent's own 2 x 2 x 2 shape (iclora_utils.py:151-155). + const std::string mask = + WriteIcLoraMask(ws.root + "/mask_dir", 9, 64, 64, /*left=*/0, /*right=*/255); + + const std::string plain = FirstFrame(engine->Generate(IcLoraGen(ws.root + "/m_plain", clip))); + const vllm::multimodal::Ltx2ConditioningTrace no_mask = ltx2->last_conditioning(); + CHECK(no_mask.ic_lora_attention_mask_rows == 0); + + vllm::multimodal::VideoGenParams masked = IcLoraGen(ws.root + "/m_masked", clip); + masked.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = mask; + const std::string with_mask = FirstFrame(engine->Generate(masked)); + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + + // THE MASK WAS HANDED OVER, and the row count says it was the DENSE form. A + // build that constructed the mask and dropped the pointer leaves this at 0 + // while every other check in this case still passes. + // + // 8 target tokens + 8 reference tokens = 16. + CHECK_MESSAGE(trace.ic_lora_attention_mask_rows == 16, + "the self-attention strength mask did not reach the DiT as the dense " + "[tokens, tokens] form; rows = " + << trace.ic_lora_attention_mask_rows); + // AND IT IS NOT THE IDENTITY. An all-ones mask renders correctly, has the + // right shape, and is exactly what a downsample that lost its values produces. + // Both ends are asserted because one of them alone cannot see a constant mask. + CHECK_MESSAGE(trace.ic_lora_attention_mask_min < 1.0, + "every weight in the mask is 1.0, which is the identity: this case would be " + "measuring nothing"); + CHECK(trace.ic_lora_attention_mask_max == 1.0); + // The left half of the mask is 0 and the right half is 255, so after + // `(x + 1) / 2` the latent weights straddle 0 and 1 — a mask whose minimum is + // 0 is a token that is FULLY masked, which is `finfo.min` in the log-space + // bias and the one value a smooth attenuation cannot produce. + CHECK(trace.ic_lora_attention_mask_min == 0.0); + + REQUIRE(plain.size() == with_mask.size()); + CHECK_MESSAGE(plain != with_mask, + "a conditioning attention mask that fully masks half the reference's tokens " + "rendered byte-identical frames, so the mask never reached the attention"); + + // AND THE MASK'S VALUES MATTER, not merely its presence. Two DIFFERENT masks + // must render differently; a build that read the directory and then handed + // over a constant satisfies every check above. + const std::string other_mask = + WriteIcLoraMask(ws.root + "/mask_dir_b", 9, 64, 64, /*left=*/255, /*right=*/0); + vllm::multimodal::VideoGenParams flipped = IcLoraGen(ws.root + "/m_flipped", clip); + flipped.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = other_mask; + const std::string with_flipped = FirstFrame(engine->Generate(flipped)); + CHECK_MESSAGE(with_flipped != with_mask, + "a mask and its left-right mirror rendered identically, so the mask's VALUES " + "never reached the attention — only its presence did"); + + // AND THE STRENGTH SCALES IT (iclora_utils.py:156). Halving the strength + // halves every latent weight, so the maximum drops from 1.0 to 0.5 and the + // render moves again. + vllm::multimodal::VideoGenParams half = IcLoraGen(ws.root + "/m_half", clip); + half.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = mask; + half.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "0.5"; + const std::string with_half = FirstFrame(engine->Generate(half)); + const vllm::multimodal::Ltx2ConditioningTrace half_trace = ltx2->last_conditioning(); + CHECK_MESSAGE(half_trace.ic_lora_attention_mask_max == 1.0, + "the mask's own block still attends fully to itself (mask_utils.py:229); the " + "strength scales the CROSS weights, not the whole matrix"); + CHECK(with_half != with_mask); +} + +TEST_CASE("ltx2 ic-lora: the reference GEOMETRY follows the adapter, and refuses what it cannot divide") { + Workspace ws; + const std::string ups = WriteIcLoraUpsampler(ws.root + "/geom_ups.safetensors"); + const std::string clip = WriteIcLoraClip(ws.root + "/geom_clip", 9, 64, 64, /*value=*/20); + + SUBCASE("downscale 2 reads the clip at HALF the stage's grid, and appends fewer tokens") { + // The adapter's own metadata (iclora_utils.py:30-49), read at load by row + // LTX25-IC-LORA. Stage 1 runs at 64x64, so `reference_downscale_factor = 2` + // reads the clip at 32x32 and the (8, 32, 32) factors give a 2 x 1 x 1 + // reference latent — 2 tokens against the unscaled arm's 8. + vllm::multimodal::VideoModelParams mp = IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = WriteFixtureLora( + ws.root + "/ref_lora.safetensors", kFixtureLoraTarget, 1.0F, + {{"reference_downscale_factor", "2"}}); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + (void)engine->Generate(IcLoraGen(ws.root + "/geom_out", clip)); + CHECK_MESSAGE(ltx2->last_conditioning().ic_lora_reference_tokens == 2, + "the reference was not read at `height // reference_downscale_factor` " + "(iclora_utils.py:116-117); it appended " + << ltx2->last_conditioning().ic_lora_reference_tokens << " tokens"); + } + + SUBCASE("a target the factor does not divide is refused by upstream's own message") { + vllm::multimodal::VideoModelParams mp = IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = WriteFixtureLora( + ws.root + "/ref_lora3.safetensors", kFixtureLoraTarget, 1.0F, + {{"reference_downscale_factor", "3"}}); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + try { + // Stage 1 is 64x64 and 64 % 3 != 0. + (void)engine->Generate(IcLoraGen(ws.root + "/geom_bad", clip)); + FAIL("a target the reference_downscale_factor does not divide must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // Upstream's own wording, pinned by `kLtx2RefDivisibilityRefusal`. + CHECK(msg.find("must be divisible by reference_downscale_factor") != std::string::npos); + CHECK(msg.find("iclora_utils.py:112-115") != std::string::npos); + } + } +} + +TEST_CASE("ltx2 ic-lora: every refusal names what is wrong and where the feature lives") { + Workspace ws; + const std::string clip = WriteIcLoraClip(ws.root + "/ref_clip", 9, 64, 64, /*value=*/20); + + SUBCASE("a reference clip on a pipeline that has no reference conditioning") { + // THE SAME `serves_reference` PREDICATE the route uses, negated. A refusal + // and its route predicate written as two expressions is how this campaign + // shipped a silently wrong answer, and this case is what holds them together. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refuse_ref"); + gen.ref_video_dir = clip; + try { + (void)engine->Generate(gen); + FAIL("a reference clip on distilled_two_stage must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // It names the pipeline that DOES serve it, rather than saying the feature + // is missing — it is not missing, it is elsewhere. + CHECK(msg.find("'ic_lora'") != std::string::npos); + CHECK(msg.find("ic_lora.py:381-402") != std::string::npos); + // ...and it names retake as the other thing this directory can mean. + CHECK(msg.find("retake_start_time") != std::string::npos); + // The adapter factors are READ, so the message reports what was read + // rather than a sentence about it. + CHECK(msg.find("no adapter was supplied") != std::string::npos); + } + } + + SUBCASE("an ic_lora render with NO reference clip") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/none_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/no_ref", ""); + try { + (void)engine->Generate(gen); + FAIL("`--video-conditioning` is required=True upstream; an ic_lora render without one " + "must be refused rather than silently becoming a distilled render"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("required=True") != std::string::npos); + CHECK(msg.find("distilled_two_stage") != std::string::npos); + } + } + + SUBCASE("a mask on a pipeline that appends no reference tokens") { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refuse_mask"); + gen.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = ws.root; + try { + (void)engine->Generate(gen); + FAIL("a conditioning attention mask with no reference tokens to attenuate must refuse"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("combined_image_conditionings` never wraps") != std::string::npos); + CHECK(msg.find("'ic_lora'") != std::string::npos); + } + } + + SUBCASE("the scalar-only strength arm, which upstream's CLI cannot reach either") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/scalar_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/scalar", clip); + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "0.5"; + try { + (void)engine->Generate(gen); + FAIL("a sub-1.0 strength with no mask is upstream's Python-API-only arm and must be " + "refused by name rather than served"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("iclora_utils.py:157-158") != std::string::npos); + CHECK(msg.find("ic_lora.py:452-455") != std::string::npos); + CHECK(msg.find("owed") != std::string::npos); + } + } + + SUBCASE("a strength outside [0, 1]") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/range_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/range", clip); + gen.extras[vllm::multimodal::kLtx2CondAttentionMaskDirExtra] = ws.root; + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "1.5"; + try { + (void)engine->Generate(gen); + FAIL("a strength above 1 would AMPLIFY attention and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("[0.0, 1.0]") != std::string::npos); + CHECK(msg.find("AMPLIFY") != std::string::npos); + } + } + + SUBCASE("a reference IMAGE, which is not what an IC-LoRA reference is") { + const std::string ups = WriteIcLoraUpsampler(ws.root + "/img_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/refimg", clip); + gen.ref_image_paths.push_back(ws.paths.audio_embeds); + try { + (void)engine->Generate(gen); + FAIL("reference images are not an IC-LoRA shape and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("--video-conditioning") != std::string::npos); + CHECK(msg.find("first_frame_ppm") != std::string::npos); + } + } +} From c1e6979e320f1472efdd0c1b639ebb6286805f19 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 07:58:32 +0000 Subject: [PATCH 04/11] test(LTX25-IC-LORA-REF-VIDEO): value gates for the ported pieces, against the executed oracle Thirteen cases against `tests/vllm/models/ltx2_iclora_reference_goldens.inc`, which the generator produces by importing and executing the pinned module. These are NOT the proof that anything is reached and the file says so: every case here stays green when the production call site is deleted, which is why the reachability cases live in test_ltx2_video and render through Generate. What each case can actually see is measured rather than asserted in prose. The build_attention_mask fixture carries a PRIOR reference token because without one the true block structure and the plausible reading "cross on all existing rows" are elementwise equal; the downsample is checked against both bilinear spatial interpolation and uniform temporal pooling; temporal_subsample is checked against `range(0, F, N)` case by case rather than once over the set, because the two agree at factor 1 and on a single-frame clip and an aggregate check would pass on a build that got only the live cases wrong. 13 cases, 97 assertions, 0 failed. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- tests/CMakeLists.txt | 1 + .../models/test_ltx2_iclora_reference.cpp | 411 ++++++++++++++++++ 2 files changed, 412 insertions(+) create mode 100644 tests/vllm/models/test_ltx2_iclora_reference.cpp diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f14de2923..daf464fba 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -461,6 +461,7 @@ target_compile_definitions(test_ltx2_pipeline PRIVATE # CLI-stage validations. Its own target rather than cases inside # test_ltx2_pipeline, which several concurrent rows of this campaign edit. The # REACHABILITY proof for the same row lives in test_ltx2_video. +vllm_cpp_add_test(test_ltx2_iclora_reference vllm/models/test_ltx2_iclora_reference.cpp) vllm_cpp_add_test(test_ltx2_retake vllm/models/test_ltx2_retake.cpp) vllm_cpp_add_test(test_ltx2_lora vllm/models/test_ltx2_lora.cpp) vllm_cpp_add_test(test_ltx2_loader vllm/models/test_ltx2_loader.cpp) diff --git a/tests/vllm/models/test_ltx2_iclora_reference.cpp b/tests/vllm/models/test_ltx2_iclora_reference.cpp new file mode 100644 index 000000000..f66134072 --- /dev/null +++ b/tests/vllm/models/test_ltx2_iclora_reference.cpp @@ -0,0 +1,411 @@ +// LTX-2.5 IC-LoRA reference conditioning — VALUE gates against the EXECUTED +// pinned module. Row LTX25-IC-LORA-REF-VIDEO (#3020), +// spec .agents/specs/ltx25-ic-lora-ref-video.md. +// +// THESE ARE NOT THE PROOF THAT ANYTHING IS REACHED. Per .agents/reachability.md +// they localize a failure: every case here stays green when the production call +// site in `ltx2_video.cpp` is deleted. The reachability cases are +// `ltx2 ic-lora: ...` in `test_ltx2_video`, which render through `Generate`. +// +// Every golden is what upstream's own code RETURNED, and every case carries the +// hypothesis it rejects. `scripts/gen-ltx2-iclora-reference-goldens.py` refuses +// to emit a case whose separation is zero, and the separation table it writes at +// the bottom of the .inc is the record of what each case can actually see. +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_conditioning.h" +#include "vllm/model_executor/models/ltx2_iclora_reference.h" + +#include "ltx2_iclora_reference_goldens.inc" + +namespace { + +// The pin the goldens were generated against. Asserted rather than assumed: a +// regeneration against a different checkout must fail this gate instead of +// silently replacing the oracle. +constexpr const char* kExpectedUpstream = "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +std::string Message(void (*run)()) { + try { + run(); + } catch (const std::exception& e) { + return std::string(e.what()); + } + return std::string(); +} + +} // namespace + +TEST_CASE("ltx2 ic-lora goldens: the oracle pin is the one this suite claims") { + CHECK(std::string(vllm_test::kLtx2IcLoraRefUpstreamRevision) == kExpectedUpstream); +} + +TEST_CASE("ltx2 ic-lora: temporal_subsample keeps index 1, not every Nth from 0") { + // The whole content of `temporal_subsample` (iclora_utils.py:87-90) is WHICH + // indices it keeps, and the plausible wrong reading `range(0, F, N)` returns + // the SAME COUNT — so no shape check, no token count and no rendered frame + // count can tell them apart. The reference simply describes different moments. + size_t at = 0; + for (int c = 0; c < vllm_test::kLtx2TemporalSubsampleCases; ++c) { + const int64_t frames = vllm_test::kLtx2TemporalSubsampleFrames[c]; + const int64_t factor = vllm_test::kLtx2TemporalSubsampleFactor[c]; + const int64_t count = vllm_test::kLtx2TemporalSubsampleCount[c]; + INFO("frames = " << frames << ", factor = " << factor); + const std::vector got = vllm::Ltx2TemporalSubsampleIndices(frames, factor); + REQUIRE(static_cast(got.size()) == count); + for (int64_t i = 0; i < count; ++i) { + CHECK(got[static_cast(i)] == vllm_test::kLtx2TemporalSubsampleKept[at + i]); + } + // ...and it is NOT the rejected hypothesis, asserted case by case rather + // than once over the whole set: the two agree at factor 1 and on a + // single-frame clip, and a golden that only checked the aggregate would pass + // on a build that got the two live cases wrong and the degenerate ones right. + std::vector rejected; + for (int64_t i = 0; i < frames; i += factor) rejected.push_back(i); + if (frames > 1 && factor > 1) { + CHECK_MESSAGE(got != rejected, + "at frames = " << frames << " factor = " << factor + << " the port agrees with `range(0, F, N)`, which upstream is " + "not"); + } + at += static_cast(count); + } +} + +TEST_CASE("ltx2 ic-lora: the gather follows the indices on the FRAME axis") { + // `Ltx2TemporalSubsampleIndices` says which frames; this says the copy takes + // them out of a CHANNEL-MAJOR volume. A prefix of the buffer would keep whole + // channels and drop others, and the result is the right size either way. + const int64_t channels = 3, frames = 5, plane = 2; + std::vector clip(static_cast(channels * frames * plane)); + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < frames; ++t) { + for (int64_t p = 0; p < plane; ++p) { + clip[static_cast((c * frames + t) * plane + p)] = + static_cast(c * 100 + t * 10 + p); + } + } + } + const std::vector got = vllm::Ltx2TemporalSubsample(clip, channels, frames, plane, 2); + // Kept indices are {0, 1, 3}. + REQUIRE(got.size() == static_cast(channels * 3 * plane)); + const int64_t keep[] = {0, 1, 3}; + for (int64_t c = 0; c < channels; ++c) { + for (int64_t t = 0; t < 3; ++t) { + for (int64_t p = 0; p < plane; ++p) { + CHECK(got[static_cast((c * 3 + t) * plane + p)] == + doctest::Approx(static_cast(c * 100 + keep[t] * 10 + p))); + } + } + } +} + +TEST_CASE("ltx2 ic-lora: the reference geometry divides, and refuses only when scale != 1") { + // `iclora_utils.py:111-117`, executed. The refusal's guard is `scale != 1` + // AND indivisible; dropping the first half refuses nothing extra (1 divides + // everything) while dropping the second refuses every render. + const vllm::Ltx2IcLoraReferenceGeometry geom = + vllm::Ltx2ResolveIcLoraReferenceGeometry(448, 768, 4); + CHECK(geom.height == vllm_test::kLtx2RefGeomHeight); + CHECK(geom.width == vllm_test::kLtx2RefGeomWidth); + // ...and NOT the phase's own grid, which is the plausible wrong answer. + CHECK(geom.height != 448); + CHECK(geom.width != 768); + + // Scale 1 never refuses, whatever the dimensions. Measured on the oracle: + // 449 at scale 1 comes back as 449. + CHECK(vllm::Ltx2ResolveIcLoraReferenceGeometry(449, 771, 1).height == + vllm_test::kLtx2RefScaleOneHeight); + + const std::string refusal = + Message([] { (void)vllm::Ltx2ResolveIcLoraReferenceGeometry(448, 770, 4); }); + INFO(refusal); + // Upstream's own sentence, from the raised ValueError rather than from a + // reading of the f-string. + CHECK(refusal.find(vllm_test::kLtx2RefDivisibilityRefusal) != std::string::npos); +} + +TEST_CASE("ltx2 ic-lora: the mask downsample carves out the first latent frame") { + // `downsample_mask_video_to_latent` (iclora_utils.py:52-84). The INPUT comes + // out of the .inc so both sides read identical bytes rather than agreeing + // about a random stream. + const int64_t f_pix = vllm_test::kLtx2MaskPixFrames; + const int64_t h_pix = vllm_test::kLtx2MaskPixHeight; + const int64_t w_pix = vllm_test::kLtx2MaskPixWidth; + const int64_t f_lat = vllm_test::kLtx2MaskLatFrames; + const int64_t h_lat = vllm_test::kLtx2MaskLatHeight; + const int64_t w_lat = vllm_test::kLtx2MaskLatWidth; + const std::vector mask( + vllm_test::kLtx2MaskPixels, + vllm_test::kLtx2MaskPixels + static_cast(f_pix * h_pix * w_pix)); + + const std::vector got = + vllm::Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, f_lat, h_lat, w_lat); + REQUIRE(got.size() == static_cast(f_lat * h_lat * w_lat)); + + double max_diff = 0.0, bilinear_diff = 0.0, uniform_diff = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + max_diff = std::max(max_diff, std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentWeights[i]))); + bilinear_diff = std::max(bilinear_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentBilinear[i]))); + uniform_diff = std::max(uniform_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskLatentUniformPool[i]))); + } + // f32 area pooling accumulated in f64 against torch's own f32 reduction. + CHECK_MESSAGE(max_diff < 1e-6, "max |diff| against the executed oracle is " << max_diff); + // AND IT IS NOT EITHER REJECTED ANSWER. Both produce a correctly shaped mask + // of entirely plausible values, and nothing about a render's shape can see the + // difference — which is why they are asserted rather than described. + CHECK_MESSAGE(bilinear_diff > 1e-3, + "the port agrees with BILINEAR spatial interpolation; `mode=\"area\"` is what " + "upstream passes (iclora_utils.py:63-67)"); + CHECK_MESSAGE(uniform_diff > 1e-3, + "the port agrees with uniform temporal pooling, so the causal first-frame " + "carve-out (iclora_utils.py:70, :80) is not happening"); +} + +TEST_CASE("ltx2 ic-lora: the mask downsample refuses an incompatible frame pair") { + // Upstream's own assertion (`:74-77`). A truncating group size would silently + // drop the tail of the mask and still produce the right number of weights. + const std::string refusal = Message([] { + const std::vector mask(8 * 1 * 1, 0.5F); + (void)vllm::Ltx2DownsampleMaskVideoToLatent(mask, 8, 1, 1, 4, 1, 1); + }); + INFO(refusal); + CHECK(refusal.find("must be divisible by") != std::string::npos); + + // ...and the DEGENERATE arms return the first frame alone (`:81-82`) rather + // than refusing, which is what keeps a single-frame reference working. + const std::vector single(1 * 2 * 2, 0.25F); + CHECK(vllm::Ltx2DownsampleMaskVideoToLatent(single, 1, 2, 2, 1, 2, 2).size() == 4U); +} + +TEST_CASE("ltx2 ic-lora: the mask video's arithmetic remaps [-1, 1] to [0, 1]") { + // `_load_mask_video` (ic_lora.py:530-536) after the read: mean over channels, + // `(x + 1) / 2`, clamp. The pixels arrive CHANNEL-major, so the three samples + // of one pixel are a plane apart — reading them as adjacent would average + // three different pixels and still produce a plausible mask. + const int64_t channels = 3, frames = 1, plane = 2; + const std::vector pixels = { + -1.0F, 1.0F, // channel 0 + -1.0F, 1.0F, // channel 1 + -1.0F, 1.0F}; // channel 2 + const std::vector got = vllm::Ltx2MaskVideoFromPixels(pixels, channels, frames, plane); + REQUIRE(got.size() == 2U); + CHECK(got[0] == doctest::Approx(0.0)); + CHECK(got[1] == doctest::Approx(1.0)); + + // A pixel whose channels DISAGREE separates the channel-major read from the + // interleaved one: the mean of {-1, 0, 1} is 0 and maps to 0.5, while reading + // the first three values as one pixel would average {-1, 1, -1}. + const std::vector mixed = {-1.0F, 0.0F, 0.0F, 0.0F, 1.0F, 0.0F}; + const std::vector mixed_got = vllm::Ltx2MaskVideoFromPixels(mixed, channels, frames, plane); + CHECK(mixed_got[0] == doctest::Approx(0.5)); + + // The clamp is upstream's and it is not defensive: a value above 1 becomes a + // POSITIVE log-space bias, which amplifies attention rather than attenuating. + const std::vector hot(6, 4.0F); + CHECK(vllm::Ltx2MaskVideoFromPixels(hot, channels, frames, plane)[0] == doctest::Approx(1.0)); +} + +TEST_CASE("ltx2 ic-lora: build_attention_mask puts the cross block on the NOISY rows only") { + // THE FIXTURE CARRIES A PRIOR REFERENCE TOKEN, and that is the whole reason + // this case can fail. With `num_existing == num_noisy` the true block + // structure and the plausible reading `cross on ALL existing rows` are + // ELEMENTWISE EQUAL — measured 0 separating elements on the oracle — and the + // golden would be a mute switch. The generator refuses that fixture. + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t added = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(added)); + const std::vector got = + vllm::Ltx2BuildAttentionMask({}, noisy, added, existing, cross); + const int64_t total = existing + added; + REQUIRE(got.size() == static_cast(total * total)); + + size_t oracle_diff = 0, rejected_same = 0; + for (size_t i = 0; i < got.size(); ++i) { + if (got[i] != vllm_test::kLtx2AmMask[i]) ++oracle_diff; + if (got[i] == vllm_test::kLtx2AmMaskCrossOnAllRows[i]) ++rejected_same; + } + CHECK_MESSAGE(oracle_diff == 0, oracle_diff << " elements differ from the executed oracle"); + CHECK_MESSAGE(rejected_same < got.size(), + "the port is elementwise equal to `cross on ALL existing rows`, which is the " + "reading this fixture exists to separate"); +} + +TEST_CASE("ltx2 ic-lora: an existing mask is PRESERVED under a second item") { + // `:223-226`. Refilling the top-left block with ones would discard the first + // item's attenuation entirely, and the result is still a legal mask of the + // right shape whose values are all in [0, 1]. + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t first_new = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(first_new)); + const std::vector first = + vllm::Ltx2BuildAttentionMask({}, noisy, first_new, existing, cross); + const std::vector cross2(vllm_test::kLtx2AmCross2, vllm_test::kLtx2AmCross2 + 1); + const std::vector second = + vllm::Ltx2BuildAttentionMask(first, noisy, 1, existing + first_new, cross2); + + size_t diff = 0, matches_refilled = 0; + for (size_t i = 0; i < second.size(); ++i) { + if (second[i] != vllm_test::kLtx2AmMask2[i]) ++diff; + if (second[i] == vllm_test::kLtx2AmMask2Refilled[i]) ++matches_refilled; + } + CHECK_MESSAGE(diff == 0, diff << " elements differ from the executed oracle"); + CHECK_MESSAGE(matches_refilled < second.size(), + "the port discarded the existing mask and refilled the block with ones"); +} + +TEST_CASE("ltx2 ic-lora: resolve_cross_mask's scalar and 1-D arms, and its refusal") { + const std::vector scalar = vllm::Ltx2ResolveCrossMask({}, 0.5, 4); + REQUIRE(scalar.size() == 4U); + for (size_t i = 0; i < scalar.size(); ++i) CHECK(scalar[i] == vllm_test::kLtx2CrossScalar[i]); + + const std::vector values = {0.1F, 0.2F, 0.3F, 0.4F}; + const std::vector oned = vllm::Ltx2ResolveCrossMask(values, 1.0, 4); + REQUIRE(oned.size() == 4U); + for (size_t i = 0; i < oned.size(); ++i) CHECK(oned[i] == vllm_test::kLtx2CrossOneD[i]); + // ...and the 1-D arm is not the scalar fill, which is the failure a per-token + // mask that silently collapsed to a constant would produce. + CHECK(oned[0] != oned[3]); + + // `:50-53` refuses rather than broadcasting: a mask one token short of the + // sequence would attenuate the wrong tokens and still render. + const std::string refusal = Message([] { + const std::vector three = {0.1F, 0.2F, 0.3F}; + (void)vllm::Ltx2ResolveCrossMask(three, 1.0, 4); + }); + INFO(refusal); + CHECK(refusal.find("must equal num_new_tokens") != std::string::npos); +} + +TEST_CASE("ltx2 ic-lora: an unmasked append PADS the mask instead of leaving it short") { + // `update_attention_mask`'s `attention_mask is None` arm (`:141-156`). + const int64_t noisy = vllm_test::kLtx2AmNoisy; + const int64_t first_new = vllm_test::kLtx2AmNew; + const int64_t existing = vllm_test::kLtx2AmExisting; + const std::vector cross(vllm_test::kLtx2AmCross, + vllm_test::kLtx2AmCross + static_cast(first_new)); + const std::vector first = + vllm::Ltx2BuildAttentionMask({}, noisy, first_new, existing, cross); + + // No mask and none present is upstream's None. + CHECK(vllm::Ltx2PadAttentionMaskForUnmaskedTokens({}, noisy, 1, existing).empty()); + + const std::vector padded = + vllm::Ltx2PadAttentionMaskForUnmaskedTokens(first, noisy, 1, existing + first_new); + REQUIRE(padded.size() == static_cast((existing + first_new + 1) * + (existing + first_new + 1))); + size_t diff = 0; + for (size_t i = 0; i < padded.size(); ++i) { + if (padded[i] != vllm_test::kLtx2AmMaskPaddedOnes[i]) ++diff; + } + CHECK_MESSAGE(diff == 0, diff << " elements differ from the executed oracle"); + // GREW WITH THE SEQUENCE. A mask left at its pre-append size is not a shape + // error downstream — `Ltx2ModalityInput` accepts the key-only broadcast form + // too, so it is read as a legal mask over a different axis. + CHECK(padded.size() > first.size()); +} + +TEST_CASE("ltx2 ic-lora: an appending item extends the state's attention mask") { + // THE OBLIGATION LIVES IN `AppendTokens`, which is what upstream's docstring + // makes it (mask_utils.py:83-85 for the keyframes mask; every appending item + // calls `update_attention_mask` with a literal None at keyframe_cond.py:68-76 + // and reference_video_cond.py:86-94). Before this row no state could carry a + // mask at all, so this branch was unreachable. + vllm::Ltx2VideoLatentShape shape; + shape.batch = 1; + shape.channels = 2; + shape.frames = 1; + shape.height = 2; + shape.width = 2; + const vllm::Ltx2ScaleFactors factors; + vllm::Ltx2LatentState state = + vllm::Ltx2CreateVideoLatentState(shape, /*patch_size=*/1, factors, /*fps=*/24.0, + /*causal_fix=*/true, nullptr, nullptr); + REQUIRE(state.tokens == 4); + CHECK_MESSAGE(state.noisy_tokens == 4, + "the state must record `latent_tools.target_shape.token_count()` at construction; " + "after the first append `tokens` is no longer it"); + vllm::Ltx2LatentVolume ref; + ref.batch = 1; + ref.channels = 2; + ref.frames = 1; + ref.height = 2; + ref.width = 2; + ref.data.assign(8, 0.5F); + const auto apply = [&] { + vllm::Ltx2ConditionVideoByReference(&state, ref, /*patch_size=*/1, factors, /*fps=*/24.0, + /*downscale_factor=*/1, /*temporal_scale_factor=*/1, + /*strength=*/1.0, /*causal_fix=*/true); + }; + + // FIRST ITEM, no mask anywhere. Upstream's `update_attention_mask` returns + // None when its argument is None and the state carries none (`:142-143`), so + // an append on an unmasked state must leave the field empty rather than + // inventing an all-ones mask — which would be the identity and would make + // every later assertion vacuous. + apply(); + REQUIRE(state.tokens == 8); + CHECK_MESSAGE(state.attention_mask.empty(), + "an append on an unmasked state invented a mask; upstream returns None"); + + // Now the wrapper writes one, over the 8 tokens that exist. + state.attention_mask = vllm::Ltx2BuildAttentionMask({}, state.noisy_tokens, 4, 4, + {0.25F, 0.75F, 0.5F, 1.0F}); + REQUIRE(state.attention_mask.size() == 64U); + + // SECOND ITEM, and this is the branch this row made reachable. + const int64_t before = state.tokens; + apply(); + const int64_t added = state.tokens - before; + REQUIRE(added == 4); + // The mask covered 8 tokens and the state is now 12. A build that left it at + // 64 entries would hand the DiT a legal, differently shaped mask. + CHECK_MESSAGE(static_cast(state.attention_mask.size()) == 12 * 12, + "the appending item did not extend the attention mask; it holds " + << state.attention_mask.size() << " entries for " << state.tokens + << " tokens"); + // ...and it PADDED with full attention rather than refilling with ones: the + // first item's 0.25 is still there. + bool has_quarter = false; + for (const float v : state.attention_mask) { + if (v == 0.25F) has_quarter = true; + } + CHECK_MESSAGE(has_quarter, + "the earlier item's attenuation was discarded when the mask grew " + "(mask_utils.py:224 preserves the existing block)"); +} + +TEST_CASE("ltx2 ic-lora: the consumption seam converts [0,1] to an additive log-space bias") { + // `_prepare_self_attention_mask` (transformer_args.py:208-237), already ported + // as `Ltx2PrepareSelfAttentionMask`. Pinned HERE because nothing in `src/` + // assigned `Ltx2ModalityInput::attention_mask` before this row, so these + // numbers were previously gated only through a hand-built test struct. + const std::vector probe(vllm_test::kLtx2SelfMaskProbe, + vllm_test::kLtx2SelfMaskProbe + 4); + const std::vector bias = vllm::Ltx2PrepareSelfAttentionMask(probe.data(), 4); + REQUIRE(bias.size() == 4U); + for (size_t i = 0; i < bias.size(); ++i) { + INFO("i = " << i); + CHECK(bias[i] == vllm_test::kLtx2SelfMaskBias[i]); + } + // The two values a naive `log(x)` gets wrong, named rather than left to the + // element compare: a zero is `finfo.min` and NOT -inf, and a subnormal is + // clamped to `finfo.tiny` before the log rather than logged as-is. + CHECK(std::isfinite(bias[2])); + CHECK(bias[3] > -100.0F); +} From b018518a58c1189f99fb77416246cfd96a920f37 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:01:33 +0000 Subject: [PATCH 05/11] docs(LTX25-IC-LORA-REF-VIDEO): the ic_lora CLI, its knobs, and the two Owed entries this retires ltx2-gen grows the rest of upstream's two flags: --ref-video-strength is the second half of --video-conditioning PATH STRENGTH, and --conditioning-attention-mask with --conditioning-attention-strength is the whole of the other. The clip itself was already --ref-video. docs/models/ltx-2-5.md gets the invocation and, more usefully, the three things a reader cannot infer: the reference geometry comes from the ADAPTER's metadata and not from a flag, the adapter rides stage 1 and stage 2 runs bare, which is the mirror image of every other two-stage arm here, and a sub-1.0 strength with no mask is refused because upstream's own CLI cannot reach that branch either. ltx25-ic-lora.md's two Owed rows are struck through rather than deleted, with what discharged them and what did not: the reference-IMAGE arm is not discharged and never will be, because upstream has no such flag. Both rows cited #932 and #975, which 404 (#2899); they now point at #3020. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora.md | 14 +++++---- docs/FEATURES.md | 1 + docs/models/ltx-2-5.md | 53 ++++++++++++++++++++++++++++++++++ examples/ltx2_gen/main.cpp | 37 ++++++++++++++++++++++++ 4 files changed, 100 insertions(+), 5 deletions(-) diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md index 4e54dc39b..ecb02ad7d 100644 --- a/.agents/specs/ltx25-ic-lora.md +++ b/.agents/specs/ltx25-ic-lora.md @@ -353,8 +353,8 @@ Each is owed by this row and named in the commit and pull request bodies. | owed | issue | |---|---| -| serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter. §6 derives both. Token-append is NOT part of it any more — #930 landed in `c7cb59fbb` | [#975](https://github.com/mudler/vllm.cpp/issues/975) | -| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | [#932](https://github.com/mudler/vllm.cpp/issues/932) | +| ~~serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter~~ **DISCHARGED for the reference VIDEO arm by row `LTX25-IC-LORA-REF-VIDEO`** ([`ltx25-ic-lora-ref-video.md`](ltx25-ic-lora-ref-video.md), [#3020](https://github.com/mudler/vllm.cpp/issues/3020)), which found that the pixel path was already here — `Ltx2ReadFrameDirectory` IS `video_preprocess` over a frame directory and `Ltx2ConvVideoEncode` already runs multi-frame for retake — so what was missing was the reference item's own geometry and the stage split, both of which now land on a new `ic_lora` pipeline kind. The reference-IMAGE arm is NOT discharged and never will be: upstream has no such flag, and it is refused by name. #975 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| ~~the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`)~~ **DISCHARGED for the MASK arm by row `LTX25-IC-LORA-REF-VIDEO`** ([#3020](https://github.com/mudler/vllm.cpp/issues/3020)). `Ltx2LatentState` carries a mask, the block structure is ported and gated against the executed module, and the mask reaches the DiT — which also retired a seam that was built and dead: nothing in `src/` assigned `Ltx2ModalityInput::attention_mask` before that row. The `strength < 1.0` arm WITHOUT a mask stays owed and is refused by name, because upstream's own CLI cannot reach it either (`ic_lora.py:452-455`). #932 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | | ~~N-adapter fusion, which additionally needs upstream's SECOND rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`) that this row refuses rather than guesses~~ **DISCHARGED by row `LTX25-LORA-FUSION`** ([`ltx25-lora-fusion.md`](ltx25-lora-fusion.md)), which ported that second form and gated it as byte equality against the EXECUTED pinned module rather than against a transcription. The arity refusal is gone and `--lora` is repeatable. Lifting it made ONE new arm expressible — N adapters on a recipe with a `kNoAdapters` phase, whose stage 1 upstream gives a proper SUBSET — and that arm refuses by name and is owed in THAT row's `## Owed`, not here. **#932 itself stays OPEN**, because the row above is its other half and nothing has landed it | [#932](https://github.com/mudler/vllm.cpp/issues/932) | | GGUF k-quant LoRA fusion — **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | | a real-weights IC-LoRA fusion measurement | blocked on GPU authority; `dgx.casa` was under a long render for this row's duration and this row had no GPU authority | @@ -376,6 +376,10 @@ Each is owed by this row and named in the commit and pull request bodies. ## Now -`ACTIVE` — the adapter path is implemented and gated; the reference arm stays -refused on the two causes §6 derives, which are #975 and are neither of the two -this refusal has previously given. +`DONE` — the adapter path is implemented and gated. The reference arm that §6 +left refused was SERVED by row `LTX25-IC-LORA-REF-VIDEO` +([#3020](https://github.com/mudler/vllm.cpp/issues/3020)), together with the +conditioning attention mask; §6's derivation was correct about what was missing +and wrong about how much of it was, because the pixel path it named already +existed as `Ltx2ReadFrameDirectory`. What that row left owed is in its own +`## Owed`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 6659c5a3d..a34255962 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -228,6 +228,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` (#1009). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 Conv VAE decode DEVICE arm | LTX-2.5 video VAE | `test_ops_conv3d` 4/4 vs an independent scalar reference; `test_diffusion_device_seam` runs a non-CPU dispatch and requires byte-identical pixels | **f32**, which is what `vt::Conv3d` and the `kLtx2Vae` CUDA table serve, and what `Ltx2VideoEngine::Load` asks for whenever the render resolved a device queue ([#2853](https://github.com/mudler/vllm.cpp/issues/2853)). So this arm decodes 776.6 MiB wider than the CPU arm and off upstream's SDR bf16 grid; the bf16 DEVICE arm is owed and needs #1007. Routes through the new `vt::Conv3d` op on the queue the engine resolved at load (#1007). NO GPU has run the CUDA arm (#1452); other stages stay host-side (#1451) | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | +| LTX-2.5 IC-LoRA reference conditioning (`ICLoraPipeline`, A15 + A16) | LTX-2.5 DiT + video VAE encoder + an IC-LoRA adapter | `test_ltx2_iclora_reference` 13/13 (97 assertions) against the EXECUTED pinned module, every case carrying its rejected hypothesis; 4 `test_ltx2_video` cases entering through `Generate` | `--pipeline-kind ic_lora`. `--ref-video` is a `frame_%06d.ppm` DIRECTORY (no demuxer), read at `height // reference_downscale_factor` from the adapter's own metadata and temporally subsampled by its `reference_temporal_scale_factor`. The adapter rides STAGE 1 ONLY and stage 2 runs bare, which is the mirror image of every other two-stage arm here. `--conditioning-attention-mask` attenuates the reference per region and is the first production writer of `Ltx2ModalityInput::attention_mask`. REFUSED and owed: the EXR/HDR reference arm, `tiled_encode` for a reference, reference IMAGES (upstream has no such flag), and the scalar-only strength arm that upstream's own CLI cannot reach either | | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage` plus `--lora`, which is now REQUIRED (#1445, both stages, at ONE strength rather than upstream's 0.25/0.5; #1144). 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 (#1039) | diff --git a/docs/models/ltx-2-5.md b/docs/models/ltx-2-5.md index d4bc33936..074f381a9 100644 --- a/docs/models/ltx-2-5.md +++ b/docs/models/ltx-2-5.md @@ -117,6 +117,59 @@ ltx2-gen \ --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4 ``` +## Condition a render on a reference clip (IC-LoRA) + +```sh +ltx2-gen \ + --dit "$LTX_ROOT/diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors" \ + --model-version 2.5 --checkpoint-class distilled \ + --video-vae "$LTX_ROOT/vae/ltx-2.5-video-vae-conv-bf16.safetensors" \ + --audio-vae "$LTX_ROOT/vae/ltx-2.5-audio-vae-bf16.safetensors" \ + --upsampler "$LTX_ROOT/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors" \ + --pipeline-kind ic_lora \ + --lora "$LTX_ROOT/loras/ltx-2.5-ic-lora-depth.safetensors" \ + --ref-video /tmp/depth_frames --ref-video-strength 1.0 \ + --prompt-embeds "$LTX_VIDEO_EMBEDS" --audio-prompt-embeds "$LTX_AUDIO_EMBEDS" \ + --frames 25 --width 320 --height 192 --seed 20260812 \ + --workdir /tmp/ltx25ic --out /tmp/ltx25ic/video.mp4 +``` + +`--ref-video` is a directory of `frame_%06d.ppm`, not a container: upstream opens +one with PyAV and no demuxer is vendored here. The clip is read at +`height // reference_downscale_factor` by `width // reference_downscale_factor`, +and both factors come from the adapter's own metadata rather than from a flag. A +load with no adapter reads both as 1, which is upstream's default. A target +either axis of which the factor does not divide is refused by name, with +upstream's own sentence. + +The adapter rides **stage 1 only** and stage 2 runs bare. That is the mirror +image of `ti2vid_two_stage`, `a2vid_two_stage` and `keyframe_interpolation`, +where the adapter rides stage 2, and it is why IC-LoRA has its own pipeline kind +rather than being a mode of `distilled_two_stage`. A reference clip supplied to +any other kind is refused, and the message names this one. + +`--conditioning-attention-mask` takes a directory of grayscale +`frame_%06d.ppm` whose pixels attenuate the reference per region: black ignores +the conditioning there, white takes it in full. The frames are read at the +stage's own resolution and downsampled to the reference latent's grid, so a mask +must describe the same moments as the clip. `--conditioning-attention-strength` +scales the whole mask and must be in `[0, 1]`; above 1 it would amplify +attention rather than attenuate it, and is refused. + +A strength below 1 with **no** mask is refused rather than served. Upstream has +that branch and its own CLI cannot reach it either: the strength is assigned only +alongside a mask, so a sub-1.0 value always arrives with one. + +Through the C ABI and the server these are per-generation extras rather than +flags: `ref_video_strength`, `conditioning_attention_mask_dir` and +`conditioning_attention_strength`, beside the existing `ref_video` field. + +Not served, and refused by name: the EXR/HDR reference arm, which needs an +OpenEXR reader and a declared colour space; a tiled reference encode; and +reference IMAGES, which are not an IC-LoRA shape at all — upstream's flag takes a +video or an EXR directory and `ltx-pipelines` has no reference-image +conditioning. + `--lora` is repeatable, and each repetition takes an adapter path with an optional strength, exactly as upstream's own flag does: `--lora first.safetensors 0.8 --lora second.safetensors`. An omitted strength is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 2a2d0e34a..42f803046 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -188,6 +188,17 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "how much is read (default: the clip\'s own duration). Either without\n" "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n" "input, not a VAE round trip, which is upstream\'s deliberate choice.\n\n" + "IC-LoRA REFERENCE CONDITIONING conditions the whole render on a reference clip\n" + "(depth, pose, edges) and needs --pipeline-kind ic_lora, which is upstream's own\n" + "ICLoraPipeline: the adapter rides stage 1 and stage 2 runs bare. --ref-video is\n" + "the clip, as a DIRECTORY of frame_%%06d.ppm, and --ref-video-strength is the\n" + "second half of upstream's --video-conditioning PATH STRENGTH. Supply the\n" + "IC-LoRA itself with --lora: its metadata carries the downscale and temporal\n" + "factors the reference is read at, and without one both default to 1.\n" + "--conditioning-attention-mask names a grayscale frame directory whose pixels\n" + "attenuate the reference per region, and --conditioning-attention-strength\n" + "scales it. A strength below 1 with no mask is REFUSED rather than served: it is\n" + "upstream's Python-API-only branch, which its own CLI cannot reach either.\n\n" "RETAKE regenerates a time window of an existing clip and keeps the rest.\n" "--ref-video names a DIRECTORY of frame_%%06d.ppm (the layout minimax-h3-gen\n" "writes), not a container file: upstream opens one with PyAV and no demuxer is\n" @@ -289,6 +300,9 @@ int main(int argc, char** argv) { // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container. std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio; + // IC-LoRA reference conditioning (#3020): the strength half of + // `--video-conditioning`, and both halves of `--conditioning-attention-mask`. + std::string ref_video_strength, cond_mask_dir, cond_mask_strength; // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120). std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; @@ -422,6 +436,17 @@ int main(int argc, char** argv) { retake_end = Need(argc, argv, ++i, "--retake-end-time"); else if (f == "--retake-frame-rate") retake_fps = Need(argc, argv, ++i, "--retake-frame-rate"); + // IC-LoRA REFERENCE CONDITIONING (#3020). `--ref-video` above names the + // clip; these three are the rest of upstream's two flags — + // `--video-conditioning PATH STRENGTH` and + // `--conditioning-attention-mask MASK_PATH STRENGTH` (ic_lora.py:416-441). + // Per-generation, so they ride vp.extra_* like the retake knobs. + else if (f == "--ref-video-strength") + ref_video_strength = Need(argc, argv, ++i, "--ref-video-strength"); + else if (f == "--conditioning-attention-mask") + cond_mask_dir = Need(argc, argv, ++i, "--conditioning-attention-mask"); + else if (f == "--conditioning-attention-strength") + cond_mask_strength = Need(argc, argv, ++i, "--conditioning-attention-strength"); // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which // is a LOAD extra; these six are per-generation and are refused by name on // any other pipeline rather than accepted and ignored. @@ -517,6 +542,18 @@ int main(int argc, char** argv) { gen_keys.emplace_back("audio_path"); gen_values.push_back(audio_path); } + if (!ref_video_strength.empty()) { + gen_keys.emplace_back("ref_video_strength"); + gen_values.push_back(ref_video_strength); + } + if (!cond_mask_dir.empty()) { + gen_keys.emplace_back("conditioning_attention_mask_dir"); + gen_values.push_back(cond_mask_dir); + } + if (!cond_mask_strength.empty()) { + gen_keys.emplace_back("conditioning_attention_strength"); + gen_values.push_back(cond_mask_strength); + } if (!audio_start_time.empty()) { gen_keys.emplace_back("audio_start_time"); gen_values.push_back(audio_start_time); From 7af4b9fddf1877a5fedec68a77fe5cb8d660b080 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:09:24 +0000 Subject: [PATCH 06/11] test(LTX25-IC-LORA-REF-VIDEO): the mutation record, and a tripwire for the one residual it found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twelve mutations, each confirmed APPLIED and COMPILED before its result was recorded, each restored with an emptiness assertion on git status. Ten were DETECTED. Both survivors are identities, and the difference between an identity and a blind spot is the whole content of this commit. M6 drops `scale != 1` from the divisibility guard. For scale 1 the remaining conjunct is `h % 1 || w % 1`, which is 0 for every h and w, so the two predicates agree on every input and no fixture can separate them. Upstream writes the guard and this port mirrors it; it states intent, not behaviour. M11 replaces `serves_reference` in the refusal with a string compare on pipeline_kind. Those are the same predicate over the recipe table as it stands, because ic_lora_reference is set by IcLoraRecipe alone. They stop being the same the moment a second recipe sets the flag — and that is the day this campaign's "a refusal and its route predicate must be the SAME predicate" lesson would be paid for again. So test_ltx2_pipeline now walks all eleven kinds against all four versions and requires exactly ONE resolved pair to carry the flag, naming ltx2_video.cpp's refusals as what must be re-read. It does not make M11 detectable today, because today it changes nothing; it makes the identity guarded instead of assumed. test_ltx2_pipeline: 76 cases, 4712 assertions, 0 failed. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora-ref-video.md | 52 +++++++++++++++++++++--- tests/vllm/models/test_ltx2_pipeline.cpp | 47 +++++++++++++++++++++ 2 files changed, 94 insertions(+), 5 deletions(-) diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md index 06bd5ba3d..f6231e627 100644 --- a/.agents/specs/ltx25-ic-lora-ref-video.md +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -273,10 +273,50 @@ the production call site, not the helper. ## 6. Mutations -Every claimed guarantee is mutated in a scratch copy, each mutation confirmed -APPLIED and COMPILED before its result is recorded. Eight of eight A24 waves -shipped a guarantee that mutation showed nothing measured; this row assumes the -same until each one has fired. +Twelve, each applied by `scripts`-external `mutate.py`, which REFUSES when its +`old` text is not found and asserts the edit took; each then COMPILED, both +suites run, and the tree restored with `git checkout --` plus a `git status +--porcelain` emptiness assertion. `unit` is `test_ltx2_iclora_reference` +(97 assertions green), `reach` is `test_ltx2_video --test-case='ltx2 ic-lora*'` +(169 assertions green). + +| # | mutation | unit | reach | verdict | +|---|---|---|---|---| +| M1 | delete the production `vin.attention_mask` assignment | 97/97 pass | **4 failed** | DETECTED — the reachability property | +| M2 | never apply `Ltx2ConditionVideoByReference` | 97/97 pass | **threw** | DETECTED | +| M3 | `temporal_subsample` steps from 0 | **21 failed** | 169/169 pass | DETECTED | +| M4 | drop the causal first-frame carve-out | **1 failed** | 169/169 pass | DETECTED | +| M5 | cross block on ALL existing rows | **3 failed** | 169/169 pass | DETECTED | +| M6 | drop `scale != 1` from the divisibility guard | 97/97 pass | 169/169 pass | **SURVIVED — IDENTITY** | +| M7 | read the reference at the phase's own grid | **4 failed** | **1 failed** | DETECTED | +| M8 | apply the reference on every phase | 97/97 pass | **1 failed** | DETECTED | +| M9 | never multiply the mask by the strength | 97/97 pass | **1 failed** | DETECTED | +| M10 | an append no longer extends the mask | **1 failed** | 169/169 pass | DETECTED | +| M11 | the refusal takes a second predicate | 97/97 pass | 169/169 pass | **SURVIVED — IDENTITY** | +| M12 | broadcast a short 1-D mask instead of refusing | **1 failed** | 169/169 pass | DETECTED | + +**Ten of twelve detected. Neither survivor is a blind spot, and both are +identities provable rather than argued.** + +**M6.** The mutant drops `downscale_factor != 1` from +`downscale_factor != 1 && (h % downscale_factor || w % downscale_factor)`. For +`downscale_factor == 1` the right conjunct is `h % 1 || w % 1`, which is `0` +for every `h` and `w`. The two predicates therefore agree on every input, and no +fixture can separate them. Upstream writes the guard (`iclora_utils.py:112`) and +this port mirrors it; it is documentation of intent, not behaviour. + +**M11.** The mutant replaces `serves_reference` in the refusal with +`im.pipeline_kind != "ic_lora"`. `serves_reference` is +`im.recipe.ic_lora_reference`, and that field is set by `IcLoraRecipe` alone, +which `ResolveLtx2PipelineRecipe` reaches only for `pipeline_kind == "ic_lora"`. +So over the recipe table AS IT STANDS the two expressions are the same +predicate. **It stops being an identity the moment a second recipe sets the +flag**, and this row therefore adds the tripwire for that day: +`test_ltx2_pipeline`'s recipe-table case walks all eleven kinds against all four +versions and requires exactly ONE resolved pair to carry `ic_lora_reference`, +naming `ltx2_video.cpp`'s refusals as what must be re-read. That converts an +unmeasured residual into a guarded one; it does not make M11 detectable today, +because today it changes nothing. ## Owed @@ -305,4 +345,6 @@ same until each one has fired. ## Now -`ACTIVE` — implementation on `row/LTX25-IC-LORA-REF-VIDEO`. +`DONE` — landed on `row/LTX25-IC-LORA-REF-VIDEO`. A15 and A16 are served and +reached from `Generate`; the dead `Ltx2ModalityInput::attention_mask` seam has a +production writer. What stays owed is in `## Owed` and is refused by name. diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 3032e9b24..023bc9fff 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1173,6 +1173,53 @@ TEST_CASE("ltx2 the recipe table mirrors vLLM-Omni's, and refuses everything els CHECK(std::string(vllm_test::kLtx2OmniRecipeVersions[i]) != "2.5"); } + // `ic_lora_reference` IS TRUE FOR EXACTLY ONE KIND, and this walk is what + // keeps that true. Row LTX25-IC-LORA-REF-VIDEO (#3020). + // + // WHY IT EXISTS, stated as a MEASUREMENT rather than a worry. The phase loop + // routes on `im.recipe.ic_lora_reference` and its refusals read the SAME local, + // because a refusal and its route predicate written as two expressions is how + // this campaign shipped a silently wrong answer. Mutation M11 replaced the + // refusal's half with `im.pipeline_kind != "ic_lora"` and BOTH suites stayed + // green — 97/97 and 169/169 — because over the recipe table as it stands the + // two expressions are the same predicate. That is an IDENTITY, not a blind + // spot, and it stops being one the moment a second recipe sets this flag. + // + // So this case is the tripwire for that day: a second recipe carrying + // `ic_lora_reference` reds HERE, where the reader is told to reconcile the + // refusal, rather than in a render that silently refuses a supported request. + { + const char* const kAllKinds[] = {"one_stage", "distilled_two_stage", "dfr", + "res2s_two_stage", "dmd2", "retake", + "a2vid_two_stage", "ti2vid_two_stage", + "keyframe_interpolation", "t2a_one_stage", "ic_lora"}; + const char* const kAllVersions[] = {"2", "2.3", "2.4", "2.5"}; + int resolved_with_flag = 0; + for (const char* kind : kAllKinds) { + for (const char* version : kAllVersions) { + vllm::Ltx2PipelineRecipe recipe; + try { + recipe = vllm::ResolveLtx2PipelineRecipe(kind, version); + } catch (const std::exception&) { + continue; // an unkeyed pair; the table refuses it by name + } + INFO("kind = ", kind, " version = ", version); + if (recipe.ic_lora_reference) { + ++resolved_with_flag; + CHECK_MESSAGE(std::string(kind) == "ic_lora", + "a recipe other than 'ic_lora' declares ic_lora_reference. The phase " + "loop's refusals in ltx2_video.cpp must be re-read: they name 'ic_lora' " + "in their text, and that text is now wrong."); + } else { + CHECK(std::string(kind) != "ic_lora"); + } + } + } + CHECK_MESSAGE(resolved_with_flag == 1, + "exactly one (kind, version) pair carries ic_lora_reference; got " + << resolved_with_flag); + } + // The 2.4 and 2.5 rows this port adds, sourced from Lightricks. CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("one_stage", "2.4")); CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5")); From 975bc0906ac12be2dc07ebd971263cf3ed00ba6f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 08:17:39 +0000 Subject: [PATCH 07/11] docs(LTX25-IC-LORA-REF-VIDEO): record what the local gate covered and what it did not The declared gate named a whole-tree ctest and this run did not produce one, so the spec says which fifteen suites were built and run instead, with their numbers, and why the whole-tree run is PENDING on CI rather than passed. The reason is mechanical and worth writing down because it will happen again: a kill -9 during this row's scheduling corrupted build/.ninja_deps, ninja discarded its dependency graph and scheduled all 1338 edges, and under -j 2 on a box carrying four other compiling agents at load 20-33 that does not finish. It reached 76 of 1338. A pending result on an external resource is not a pass, and saying so is cheaper than a green nobody can reproduce. The fifteen suites are every target that links a translation unit this change touches, plus ltx2-gen, the shipped CLI that reaches the new knobs. Nothing was narrowed to reach a green. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora-ref-video.md | 32 ++++++++++++++++++------ 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md index f6231e627..039b7ac87 100644 --- a/.agents/specs/ltx25-ic-lora-ref-video.md +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -236,17 +236,35 @@ Every gate below is RUNNABLE from the repository root and all of them were run. ```sh cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=OFF -cmake --build build -j 2 --target test_ltx2_iclora_reference test_ltx2_video test_ltx2 test_ltx2_pipeline -./build/tests/test_ltx2_iclora_reference -./build/tests/test_ltx2_video -./build/tests/test_ltx2 -./build/tests/test_ltx2_pipeline -ctest --test-dir build -j 2 --output-on-failure -python3 scripts/check-pr-size.py --base origin/main +cmake --build build -j 2 --target \ + test_diffusion_device_seam test_ltx2 test_ltx2_device test_ltx2_dfr \ + test_ltx2_iclora_reference test_ltx2_image_cond test_ltx2_loader test_ltx2_lora \ + test_ltx2_pipeline test_ltx2_retake test_ltx2_text_encoder test_ltx2_tiling \ + test_ltx2_vae test_ltx2_video test_ltx2_video_device_forward ltx2-gen +ctest --test-dir build -j 2 --output-on-failure -R 'ltx2|diffusion' +python3 scripts/check-pr-size.py --base origin/main --head HEAD --branch row/LTX25-IC-LORA-REF-VIDEO python3 scripts/agent-issue-index.py --refresh && python3 scripts/check-agent-record.py scripts/agent-preflight.sh ``` +**WHAT THE LOCAL GATE COVERS, AND WHAT IT DOES NOT.** The fifteen suites above +are every test target that links a translation unit this change touches, plus +`ltx2-gen`, the shipped CLI that reaches the new knobs. Run on the rebased head: +`cmake --build` rc 0, `ctest` rc 0, **15/15 passed**, and +`test_ltx2_iclora_reference` 97/97, `test_ltx2_video` 5225/5225, `test_ltx2` +4581/4581, `test_ltx2_pipeline` 4712/4712, `test_ltx2_retake` 69/69 assertions. + +The WHOLE-TREE `ctest` was NOT run here, and the reason is mechanical rather +than a judgement about coverage: a `kill -9` during this row's scheduling +corrupted `build/.ninja_deps`, so ninja discarded its dependency graph and +scheduled all 1338 edges as a from-scratch rebuild. Under this row's `-j 2` +constraint, on a box carrying four other compiling agents at load 20-33, that +does not finish. It reached 76 of 1338 before it was stopped. **That is a +PENDING result on an external resource, not a pass**, and CI's own full build is +what discharges it on the pull request. Nothing about the local run was +weakened to reach a green: every suite that can observe this change was built +and run. + **The goldens are generated by executing the pinned module**, never by reading it: `tools/ltx2/gen_iclora_reference_goldens.py` imports `ltx_pipelines.iclora_utils` and `ltx_core.conditioning.mask_utils` from From 052288a92aaa61fe145320663b3053f107a5015d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 09:25:12 +0000 Subject: [PATCH 08/11] test(LTX25-IC-LORA-REF-VIDEO): gate the area window on a shape that can see it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mode="area"` pools output index i over `[floor(i*I/O), ceil((i+1)*I/O))`. The port writes that general form and its own comment names the hazard: an integer-stride box filter `[i*(I//O), (i+1)*(I//O))` is elementwise EQUAL to it whenever O divides I. Every fixture that reached the code was divisible. The only golden was 8 -> 2 and the e2e case reads a 64x64 mask and pools to 2x2, so replacing the general form with the stride form survived both suites entirely: 13/13 cases and 97/97 assertions in test_ltx2_iclora_reference, 131/131 and 5225/5225 in test_ltx2_video. Three mute fixtures, and the difference is not nothing — at 224 -> 3 it is ~2300x the port's error against torch. So this adds a second downsample fixture at 9 -> 2, where the two readings separate. The generator measures the separation the same way it measures every other case and refuses to emit a golden that cannot fail; this one separates on 0.08158010244369507, against a 1e-3 threshold. The same mutation now reds 2 assertions with max |diff| 0.0815801 against the oracle and 5.96e-08 against the stride reading, which is the mutant reproducing the rejected hypothesis exactly rather than merely differing from the truth. The fixture asserts its own indivisibility, because a later edit that made the shape divisible would restore the mute switch silently. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- scripts/gen-ltx2-iclora-reference-goldens.py | 48 +++++++++++++++++++ .../models/ltx2_iclora_reference_goldens.inc | 16 +++++++ .../models/test_ltx2_iclora_reference.cpp | 38 +++++++++++++++ 3 files changed, 102 insertions(+) diff --git a/scripts/gen-ltx2-iclora-reference-goldens.py b/scripts/gen-ltx2-iclora-reference-goldens.py index 094a32427..7bda56ac1 100755 --- a/scripts/gen-ltx2-iclora-reference-goldens.py +++ b/scripts/gen-ltx2-iclora-reference-goldens.py @@ -378,6 +378,54 @@ def preprocess(frames, h, w, dtype, device): E("// The two rejected answers, emitted so the suite asserts it is NOT them.") E(carray("kLtx2MaskLatentBilinear", bil_flat.flatten().tolist())) E(carray("kLtx2MaskLatentUniformPool", unif.flatten().tolist())) + + # A SECOND fixture whose spatial output does NOT divide its input. `area` + # pools output index i over `[floor(i*I/O), ceil((i+1)*I/O))`, which is only + # the integer-stride window `[i*(I//O), (i+1)*(I//O))` when O divides I. The + # 8 -> 2 case above is divisible, so it is elementwise equal to the stride + # reading and cannot see the difference; 9 -> 2 can. + nd_f_pix, nd_h_pix, nd_w_pix = 9, 9, 9 + nd_f_lat, nd_h_lat, nd_w_lat = 3, 2, 2 + nd_vals = deterministic(nd_f_pix * nd_h_pix * nd_w_pix, 0x9D19151B) + nd_mask = torch.tensor(nd_vals, dtype=torch.float32).reshape( + 1, 1, nd_f_pix, nd_h_pix, nd_w_pix) + nd_shape = VideoLatentShape.from_torch_shape((1, 1, nd_f_lat, nd_h_lat, nd_w_lat)) + nd_got = downsample(nd_mask, nd_shape) + + def stride_box_spatial(volume, h_out, w_out): + """The plausible wrong reading: an integer-stride box filter.""" + frames, h_in, w_in = volume.shape[2], volume.shape[3], volume.shape[4] + sh, sw = h_in // h_out, w_in // w_out + out_v = torch.empty(1, 1, frames, h_out, w_out, dtype=torch.float32) + for fi in range(frames): + for oh in range(h_out): + for ow in range(w_out): + out_v[0, 0, fi, oh, ow] = volume[ + 0, 0, fi, oh * sh:(oh + 1) * sh, ow * sw:(ow + 1) * sw].mean() + return out_v + + nd_t = (nd_f_pix - 1) // (nd_f_lat - 1) + box = stride_box_spatial(nd_mask, nd_h_lat, nd_w_lat) + box_rest = rearrange(box[:, :, 1:], "b 1 (f t) h w -> b 1 f t h w", t=nd_t).mean(3) + box_flat = rearrange(torch.cat([box[:, :, :1], box_rest], 2), "b 1 f h w -> b (f h w)") + require_separation("downsample_mask_video_to_latent at a NON-DIVIDING spatial pair (9 -> 2)", + "an integer-stride box filter, which `area` equals only when O divides I", + float((nd_got - box_flat).abs().max())) + E("// A NON-DIVIDING spatial pair, 9 -> 2. `area` pools output index i over") + E("// `[floor(i*I/O), ceil((i+1)*I/O))`; the integer-stride window") + E("// `[i*(I//O), (i+1)*(I//O))` agrees with it on every divisible shape and") + E("// only there, so the 8 -> 2 case above cannot see the difference and this") + E("// one can.") + E(f"inline constexpr int64_t kLtx2MaskNdPixFrames = {nd_f_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdPixHeight = {nd_h_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdPixWidth = {nd_w_pix};") + E(f"inline constexpr int64_t kLtx2MaskNdLatFrames = {nd_f_lat};") + E(f"inline constexpr int64_t kLtx2MaskNdLatHeight = {nd_h_lat};") + E(f"inline constexpr int64_t kLtx2MaskNdLatWidth = {nd_w_lat};") + E(carray("kLtx2MaskNdPixels", nd_vals)) + E(carray("kLtx2MaskNdLatentWeights", nd_got.flatten().tolist())) + E("// The rejected answer, emitted so the suite asserts it is NOT this one.") + E(carray("kLtx2MaskNdLatentStrideBox", box_flat.flatten().tolist())) E("") # ── 4. resolve_cross_mask + build_attention_mask (mask_utils.py) ────────── diff --git a/tests/vllm/models/ltx2_iclora_reference_goldens.inc b/tests/vllm/models/ltx2_iclora_reference_goldens.inc index 2379a0d1b..f014288ec 100644 --- a/tests/vllm/models/ltx2_iclora_reference_goldens.inc +++ b/tests/vllm/models/ltx2_iclora_reference_goldens.inc @@ -79,6 +79,21 @@ inline constexpr float kLtx2MaskLatentWeights[] = {0.534921169f, 0.474619985f, 0 // The two rejected answers, emitted so the suite asserts it is NOT them. inline constexpr float kLtx2MaskLatentBilinear[] = {0.496353656f, 0.681404233f, 0.361235648f, 0.389859647f, 0.488613337f, 0.519038856f, 0.347211421f, 0.502157032f, 0.469123542f, 0.545261562f, 0.613554657f, 0.502646863f}; inline constexpr float kLtx2MaskLatentUniformPool[] = {0.485171884f, 0.471803099f, 0.401835531f, 0.519039631f, 0.440979809f, 0.513357699f, 0.429122686f, 0.517239153f, 0.45536992f, 0.534099936f, 0.524896443f, 0.518256605f}; +// A NON-DIVIDING spatial pair, 9 -> 2. `area` pools output index i over +// `[floor(i*I/O), ceil((i+1)*I/O))`; the integer-stride window +// `[i*(I//O), (i+1)*(I//O))` agrees with it on every divisible shape and +// only there, so the 8 -> 2 case above cannot see the difference and this +// one can. +inline constexpr int64_t kLtx2MaskNdPixFrames = 9; +inline constexpr int64_t kLtx2MaskNdPixHeight = 9; +inline constexpr int64_t kLtx2MaskNdPixWidth = 9; +inline constexpr int64_t kLtx2MaskNdLatFrames = 3; +inline constexpr int64_t kLtx2MaskNdLatHeight = 2; +inline constexpr int64_t kLtx2MaskNdLatWidth = 2; +inline constexpr float kLtx2MaskNdPixels[] = {0.176431715f, 0.414751172f, 0.428075731f, 0.396973491f, 0.993891835f, 0.363071442f, 0.108931065f, 0.0180123448f, 0.528322339f, 0.528225362f, 0.898173034f, 0.225970566f, 0.284193099f, 0.122897267f, 0.871926129f, 0.905635357f, 0.975744545f, 0.339984655f, 0.347717524f, 0.107571602f, 0.216426015f, 0.891348183f, 0.687187612f, 0.361214876f, 0.122908354f, 0.398467839f, 0.859053314f, 0.552588582f, 0.0223608613f, 0.381933153f, 0.133411825f, 0.611968935f, 0.25096339f, 0.615133226f, 0.93135798f, 0.185379148f, 0.997653782f, 0.891151607f, 0.205703974f, 0.275035024f, 0.63247329f, 0.52415204f, 0.965521038f, 0.271311641f, 0.941007137f, 0.689656556f, 0.563211441f, 0.112410963f, 0.663916469f, 0.515940726f, 0.884365559f, 0.129287601f, 0.287406147f, 0.00508952141f, 0.829226613f, 0.913085282f, 0.0933603644f, 0.363948286f, 0.169794202f, 0.449740171f, 0.683066249f, 0.0454002619f, 0.887509406f, 0.352954507f, 0.0286763906f, 0.570546269f, 0.496683061f, 0.361666799f, 0.399156332f, 0.796284735f, 0.498372436f, 0.445922732f, 0.257750392f, 0.804583311f, 0.274433672f, 0.329919934f, 0.176333547f, 0.564143956f, 0.902864873f, 0.747000337f, 0.413248956f, 0.294984937f, 0.184212923f, 0.165899992f, 0.553884089f, 0.492762089f, 0.424729645f, 0.253381908f, 0.252146065f, 0.951010942f, 0.611447692f, 0.66240108f, 0.261108398f, 0.755251467f, 0.329187453f, 0.187325478f, 0.877708912f, 0.95155549f, 0.633138597f, 0.0419675708f, 0.232635796f, 0.66833657f, 0.273359537f, 0.412642837f, 0.525232077f, 0.810980916f, 0.261211991f, 0.947197556f, 0.9000718f, 0.368062198f, 0.0773826838f, 0.131558359f, 0.608510911f, 0.976616681f, 0.876119256f, 0.42066896f, 0.388753116f, 0.481222034f, 0.83724457f, 0.638977587f, 0.255988657f, 0.429601073f, 0.0447826385f, 0.397245526f, 0.654212952f, 0.982474685f, 0.0029489398f, 0.713418663f, 0.193713427f, 0.848715007f, 0.914697945f, 0.518172383f, 0.890321732f, 0.820030034f, 0.499699891f, 0.990263462f, 0.363884985f, 0.850643575f, 0.525403023f, 0.910626292f, 0.622525096f, 0.338403821f, 0.752064824f, 0.510951936f, 0.619227588f, 0.770158052f, 0.774864852f, 0.365872204f, 0.70408833f, 0.310323238f, 0.902955055f, 0.467824221f, 0.067301631f, 0.901615739f, 0.397632003f, 0.145964682f, 0.340275347f, 0.0559407473f, 0.55675292f, 0.983822882f, 0.787080526f, 0.174605727f, 0.65383178f, 0.442226827f, 0.453761578f, 0.097636044f, 0.671319366f, 0.145175278f, 0.901205122f, 0.937905014f, 0.620723784f, 0.23314029f, 0.0584473014f, 0.311875403f, 0.0517009497f, 0.0715442896f, 0.401783466f, 0.312211573f, 0.982579291f, 0.45889771f, 0.106410503f, 0.13699156f, 0.568376482f, 0.621710718f, 0.788453102f, 0.565417409f, 0.843536675f, 0.0152804852f, 0.161386311f, 0.499435306f, 0.133455098f, 0.287302732f, 0.649116874f, 0.747392416f, 0.0525169373f, 0.149493515f, 0.485217988f, 0.299057841f, 0.0585591197f, 0.800601184f, 0.705874681f, 0.760249972f, 0.255273879f, 0.450765669f, 0.854833901f, 0.269396067f, 0.204905152f, 0.123510599f, 0.994540036f, 0.251888454f, 0.313174903f, 0.722878397f, 0.125090957f, 0.300181627f, 0.319087744f, 0.470823646f, 0.0245870948f, 0.819541156f, 0.0128717422f, 0.167695045f, 0.656143904f, 0.432101429f, 0.402993858f, 0.739083767f, 0.00925147533f, 0.951627731f, 0.218733788f, 0.471463203f, 0.0320233107f, 0.900513947f, 0.335975945f, 0.325684547f, 0.724274635f, 0.813946486f, 0.00837951899f, 0.587917566f, 0.593036115f, 0.7367419f, 0.509701729f, 0.119342268f, 0.764099121f, 0.786913455f, 0.839243591f, 0.647685528f, 0.746626496f, 0.293163419f, 0.343819261f, 0.563849211f, 0.805668294f, 0.66591531f, 0.769356966f, 0.803707838f, 0.58803916f, 0.54597795f, 0.168208063f, 0.0741783381f, 0.229800701f, 0.494519413f, 0.362460494f, 0.637498617f, 0.161039829f, 0.489256442f, 0.092312634f, 0.823154628f, 0.832165182f, 0.247525454f, 0.294949114f, 0.748880327f, 0.451600671f, 0.40436697f, 0.51445049f, 0.640046299f, 0.356705248f, 0.432429552f, 0.825973094f, 0.623867393f, 0.922766864f, 0.0799279213f, 0.626308441f, 0.622054577f, 0.271120965f, 0.202672243f, 0.575166166f, 0.402850449f, 0.7335006f, 0.663229048f, 0.488371372f, 0.336939275f, 0.128312945f, 0.282105863f, 0.52510792f, 0.685502052f, 0.39796555f, 0.382492661f, 0.918198705f, 0.0393769741f, 0.746138871f, 0.476073146f, 0.446476877f, 0.99091053f, 0.178707838f, 0.565140069f, 0.348153234f, 0.187597871f, 0.109171689f, 0.623592556f, 0.778553843f, 0.00752681494f, 0.386353374f, 0.282530725f, 0.290333092f, 0.840117931f, 0.924435139f, 0.812919915f, 0.845062375f, 0.445622146f, 0.11066848f, 0.485649109f, 0.288371027f, 0.0302713513f, 0.903456211f, 0.318011343f, 0.108384132f, 0.861936212f, 0.503161728f, 0.52669543f, 0.857944191f, 0.945909142f, 0.792777896f, 0.672234178f, 0.133958578f, 0.690484643f, 0.557196558f, 0.177605987f, 0.550531685f, 0.856805444f, 0.118242562f, 0.704421759f, 0.229511619f, 0.0119316578f, 0.482146025f, 0.530855834f, 0.450839043f, 0.498370945f, 0.502073348f, 0.610073924f, 0.563346505f, 0.213598371f, 0.671187341f, 0.324515104f, 0.969206989f, 0.11610949f, 0.387870967f, 0.197868168f, 0.433820367f, 0.241572917f, 0.16652894f, 0.778164625f, 0.672078907f, 0.870177925f, 0.185845196f, 0.26016748f, 0.259678304f, 0.00730335712f, 0.817301929f, 0.308321118f, 0.142065346f, 0.602425873f, 0.0963254571f, 0.44181931f, 0.564264774f, 0.785137177f, 0.577842534f, 0.494561434f, 0.109093845f, 0.272757351f, 0.129696846f, 0.704673588f, 0.899068415f, 0.5506953f, 0.677172124f, 0.385102749f, 0.915118158f, 0.685579658f, 0.684499741f, 0.917760491f, 0.0294851065f, 0.160535574f, 0.138951302f, 0.747179031f, 0.453814983f, 0.993482947f, 0.493778586f, 0.368394077f, 0.759645939f, 0.270795047f, 0.281280041f, 0.440939605f, 0.377233982f, 0.627582848f, 0.0953028202f, 0.770352066f, 0.311038017f, 0.652339518f, 0.953617692f, 0.938524067f, 0.248702705f, 0.849620342f, 0.473485649f, 0.178462207f, 0.481127799f, 0.359116018f, 0.996653497f, 0.460334182f, 0.766413271f, 0.947203696f, 0.38217622f, 0.472835422f, 0.548464417f, 0.994021535f, 0.505486369f, 0.112265825f, 0.649771154f, 0.484351277f, 0.433071434f, 0.178781092f, 0.34736073f, 0.926333845f, 0.447269499f, 0.0620956421f, 0.322753251f, 0.104726017f, 0.402469754f, 0.111355186f, 0.858210564f, 0.102687776f, 0.0320823789f, 0.36260885f, 0.613993347f, 0.553087652f, 0.764120519f, 0.876651525f, 0.805870414f, 0.784868956f, 0.377298594f, 0.250272334f, 0.339644015f, 0.379174411f, 0.547709107f, 0.859425783f, 0.365451097f, 0.28319329f, 0.437577188f, 0.417913139f, 0.763182938f, 0.794263363f, 0.214469671f, 0.845055819f, 0.878148019f, 0.968808532f, 0.645396233f, 0.433998883f, 0.924810052f, 0.197890162f, 0.551473022f, 0.153582096f, 0.579355478f, 0.0588880777f, 0.398751438f, 0.397648275f, 0.343063891f, 0.220753551f, 0.836437583f, 0.78373307f, 0.876475275f, 0.251742601f, 0.131393731f, 0.00924360752f, 0.0378209352f, 0.920103848f, 0.239557207f, 0.785963535f, 0.458949089f, 0.917492151f, 0.82664746f, 0.824898958f, 0.748798013f, 0.0437062383f, 0.275460839f, 0.802760303f, 0.562115431f, 0.253444493f, 0.721078157f, 0.270546675f, 0.289948046f, 0.264674067f, 0.988374114f, 0.921821356f, 0.639254689f, 0.357419848f, 0.0164500475f, 0.987020433f, 0.134222806f, 0.470599413f, 0.214927495f, 0.804601073f, 0.00276684761f, 0.108661234f, 0.1645329f, 0.930413604f, 0.156331539f, 0.826165974f, 0.992276132f, 0.788320959f, 0.839853644f, 0.86417073f, 0.00915694237f, 0.359093964f, 0.153696716f, 0.950254858f, 0.110191941f, 0.0497874618f, 0.756811023f, 0.407120287f, 0.833793044f, 0.768642783f, 0.129186273f, 0.603067219f, 0.391770661f, 0.459852576f, 0.544464171f, 0.412319064f, 0.261648715f, 0.0876111388f, 0.913850784f, 0.621434808f, 0.907304585f, 0.0707091093f, 0.96202302f, 0.148205638f, 0.329700232f, 0.164689302f, 0.999581754f, 0.549011528f, 0.836014986f, 0.1472615f, 0.821966588f, 0.080082655f, 0.472028852f, 0.149907827f, 0.470775306f, 0.178379297f, 0.252197027f, 0.486711025f, 0.138778448f, 0.355213642f, 0.0720499754f, 0.524939477f, 0.06805861f, 0.678370833f, 0.811598957f, 0.386808276f, 0.546884239f, 0.939647734f, 0.124761701f, 0.983676314f, 0.53366524f, 0.407992423f, 0.748240411f, 0.186389923f, 0.456374049f, 0.264949262f, 0.101999223f, 0.840950489f, 0.935143232f, 0.737413049f, 0.572279096f, 0.453462601f, 0.424460053f, 0.0912488699f, 0.661141574f, 0.352765918f, 0.14255327f, 0.0666944981f, 0.512379944f, 0.040235579f, 0.0467900038f, 0.563910425f, 0.0949526429f, 0.824949086f, 0.786893725f, 0.480244696f, 0.512091756f, 0.253281355f, 0.371683776f, 0.750189245f, 0.236807227f, 0.250991404f, 0.148420811f, 0.760418773f, 0.706954002f, 0.483227968f, 0.860947609f, 0.958144605f, 0.204591811f, 0.346984088f, 0.330379963f, 0.260562778f, 0.454497993f, 0.175507128f, 0.874245822f, 0.655669034f, 0.133049369f, 0.681265354f, 0.0751605034f, 0.381592751f, 0.65002501f, 0.332112014f, 0.664795637f, 0.0514030457f, 0.520705462f, 0.428389311f, 0.695400119f, 0.903649747f, 0.591303825f, 0.103616059f, 0.83450526f, 0.955841482f, 0.135760486f, 0.620482624f, 0.523535073f, 0.502091646f, 0.189656317f, 0.936189651f, 0.590592086f, 0.583584547f, 0.610268474f, 0.443874002f, 0.130639553f, 0.61188519f, 0.877523184f, 0.420557976f, 0.567815006f, 0.255108118f, 0.364960551f, 0.185195923f, 0.476062536f, 0.600627542f, 0.558772326f, 0.957621634f, 0.3928231f, 0.455191553f, 0.00377166271f, 0.62539798f, 0.225390792f, 0.177161932f, 0.169970393f, 0.661524951f, 0.412112474f, 0.132826149f, 0.0998225212f, 0.661162794f, 0.0239869952f, 0.944482744f, 0.544507325f, 0.9350034f, 0.575270712f, 0.519179463f, 0.451092482f, 0.25096643f, 0.54721272f, 0.374691486f, 0.774538517f, 0.722382426f, 0.564935863f, 0.707167804f, 0.114523411f, 0.201274157f, 0.161640704f, 0.597701728f, 0.378583431f, 0.567182422f, 0.617218018f, 0.420417905f, 0.0773469806f, 0.00652354956f, 0.650327325f, 0.171233237f, 0.856792927f, 0.207099199f, 0.348393142f, 0.928106427f, 0.729782224f, 0.825324774f, 0.0101621747f, 0.731117189f, 0.597788632f, 0.566959321f, 0.672528088f, 0.413965285f, 0.682504356f, 0.509997785f, 0.150990427f, 0.0506671667f, 0.739574969f, 0.57869482f, 0.853643f, 0.209993958f, 0.966246784f, 0.978174984f, 0.852503657f, 0.941322565f, 0.538044572f, 0.509447157f, 0.230799377f, 0.555043757f, 0.62337774f, 0.479802608f, 0.826905191f, 0.681678236f, 0.559722781f, 0.143384337f, 0.509182394f, 0.288674951f, 0.293766856f, 0.00271993876f, 0.570420086f, 0.619341314f, 0.831060112f}; +inline constexpr float kLtx2MaskNdLatentWeights[] = {0.456964582f, 0.543460608f, 0.462804705f, 0.507882357f, 0.439289629f, 0.513226748f, 0.477039456f, 0.520850718f, 0.453535885f, 0.485347092f, 0.484792441f, 0.527994156f}; +// The rejected answer, emitted so the suite asserts it is NOT this one. +inline constexpr float kLtx2MaskNdLatentStrideBox[] = {0.37538448f, 0.521206975f, 0.502951264f, 0.475871205f, 0.418908179f, 0.512575507f, 0.466769427f, 0.535788894f, 0.447849631f, 0.473493338f, 0.482950687f, 0.520915866f}; // --- 4. build_attention_mask (mask_utils.py:170-243) --------------------- // @@ -130,6 +145,7 @@ inline constexpr float kLtx2SelfMaskBias[] = {0.0f, -0.693147182f, -3.40282347e+ // the divisibility refusal vs refusing whenever height % scale != 0, scale==1 included -> 1.0 // downsample_mask_video_to_latent vs bilinear spatial interpolation -> 0.20678424835205078 // downsample_mask_video_to_latent vs uniform temporal pooling with no causal first-frame carve-out -> 0.049749284982681274 +// downsample_mask_video_to_latent at a NON-DIVIDING spatial pair (9 -> 2) vs an integer-stride box filter, which `area` equals only when O divides I -> 0.08158010244369507 // build_attention_mask vs cross on ALL existing rows, not only the noisy ones -> 4.0 // build_attention_mask with an existing mask vs discarding the existing mask and refilling the block with ones -> 1.0 // resolve_cross_mask's 1-D arm vs the scalar fill -> 0.4000000059604645 diff --git a/tests/vllm/models/test_ltx2_iclora_reference.cpp b/tests/vllm/models/test_ltx2_iclora_reference.cpp index f66134072..770b2449e 100644 --- a/tests/vllm/models/test_ltx2_iclora_reference.cpp +++ b/tests/vllm/models/test_ltx2_iclora_reference.cpp @@ -173,6 +173,44 @@ TEST_CASE("ltx2 ic-lora: the mask downsample carves out the first latent frame") "carve-out (iclora_utils.py:70, :80) is not happening"); } +TEST_CASE("ltx2 ic-lora: the area window is the general form, not an integer stride") { + // `mode="area"` dispatches to `adaptive_avg_pool2d`, whose window for output + // index `i` is `[floor(i*I/O), ceil((i+1)*I/O))`. An integer-stride box filter + // `[i*(I//O), (i+1)*(I//O))` is elementwise EQUAL to it on every divisible + // shape, so the 8 -> 2 fixture above cannot see the difference and neither can + // the e2e case, which reads a 64x64 mask and pools to 2x2. This fixture is + // 9 -> 2, where the two readings separate. + const int64_t f_pix = vllm_test::kLtx2MaskNdPixFrames; + const int64_t h_pix = vllm_test::kLtx2MaskNdPixHeight; + const int64_t w_pix = vllm_test::kLtx2MaskNdPixWidth; + const int64_t f_lat = vllm_test::kLtx2MaskNdLatFrames; + const int64_t h_lat = vllm_test::kLtx2MaskNdLatHeight; + const int64_t w_lat = vllm_test::kLtx2MaskNdLatWidth; + REQUIRE(h_pix % h_lat != 0); // the fixture is only a gate while it stays indivisible + REQUIRE(w_pix % w_lat != 0); + const std::vector mask( + vllm_test::kLtx2MaskNdPixels, + vllm_test::kLtx2MaskNdPixels + static_cast(f_pix * h_pix * w_pix)); + + const std::vector got = + vllm::Ltx2DownsampleMaskVideoToLatent(mask, f_pix, h_pix, w_pix, f_lat, h_lat, w_lat); + REQUIRE(got.size() == static_cast(f_lat * h_lat * w_lat)); + + double max_diff = 0.0, stride_diff = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + max_diff = std::max(max_diff, std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskNdLatentWeights[i]))); + stride_diff = std::max(stride_diff, + std::fabs(static_cast( + got[i] - vllm_test::kLtx2MaskNdLatentStrideBox[i]))); + } + CHECK_MESSAGE(max_diff < 1e-6, "max |diff| against the executed oracle is " << max_diff); + CHECK_MESSAGE(stride_diff > 1e-3, + "the port agrees with an integer-stride box filter, so the `area` window is " + "the divisible special case rather than the general form " + "(iclora_utils.py:63-67, ltx2_iclora_reference.cpp PoolStart/PoolEnd)"); +} + TEST_CASE("ltx2 ic-lora: the mask downsample refuses an incompatible frame pair") { // Upstream's own assertion (`:74-77`). A truncating group size would silently // drop the tail of the mask and still produce the right number of weights. From 0edf627af47ccaafef1286d0ecc1fd6536a56ffe Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 09:35:29 +0000 Subject: [PATCH 09/11] fix(LTX25-IC-LORA-REF-VIDEO): refuse an out-of-range strength the way upstream does, with no mask `ic_lora.py:230-233` is UNCONDITIONAL. It sits in the method body, after `assert_resolution` and outside every `if args.conditioning_attention_mask is not None`. This port guarded its copy on a non-empty mask directory, and the two refusals below it did not close the hole: 1.5 is not below 1.0, so the scalar-arm refusal does not fire either. The result was a knob that accepted 1.5, refused nothing, and was then never read, because its only reader is inside the mask branch. Accepted, ignored, silent, on every pipeline. The new subcase supplies no mask directory at all and asserts the refusal by name. Without the fix it does not merely mis-assert: the render is SERVED, and the case fails on its own FAIL rather than on a CHECK, which is the shape of the defect rather than a restatement of it. The refusal now reads: ltx-2.5 video: 'conditioning_attention_strength' must be in [0.0, 1.0], got 1.500000. This is upstream's own refusal (ic_lora.py:230-233): the value multiplies a mask in [0, 1] and the product becomes a log-space attention bias, so a value above 1 would AMPLIFY attention rather than attenuate it. Ordering is unchanged where it mattered: 0.5 with no mask still passes the range check and still lands on the scalar-arm refusal. Two record repairs ride along, because both are prose this row falsified. The paragraph above `wants_image` said the reference arm "stays refused and #975 stays open". This row serves it, #975 is one of the five 404s #2899 tracks, and the paragraph was orphaned besides: it sat between an unrelated local and the new section header, and the correct version of the same text already stands beside `wants_reference_video`. Deleted rather than corrected, because the surviving copy is the one in the right place. `docs/models/ltx-2-5.md` named `ltx-2.5-ic-lora-depth.safetensors`, which the pinned checkout does not contain anywhere. The doc's convention is real published filenames, so an invented one reads as a pin. It is now an unmistakable placeholder, with the adapters upstream actually publishes named beside it and the reason none of them fits: upstream's only 2.5 IC-LoRA is DFRPipeline's stage-2 detailing adapter, and every published reference-control adapter is 2.3-era or 19b. Nothing was downloaded; this was verified against the pinned checkout at fd4ded7f alone. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/models/ltx-2-5.md | 14 +++++++++++- src/vllm/multimodal/ltx2_video.cpp | 17 ++++++--------- tests/vllm/multimodal/test_ltx2_video.cpp | 26 +++++++++++++++++++++++ 3 files changed, 46 insertions(+), 11 deletions(-) diff --git a/docs/models/ltx-2-5.md b/docs/models/ltx-2-5.md index 074f381a9..a6f577340 100644 --- a/docs/models/ltx-2-5.md +++ b/docs/models/ltx-2-5.md @@ -127,13 +127,25 @@ ltx2-gen \ --audio-vae "$LTX_ROOT/vae/ltx-2.5-audio-vae-bf16.safetensors" \ --upsampler "$LTX_ROOT/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors" \ --pipeline-kind ic_lora \ - --lora "$LTX_ROOT/loras/ltx-2.5-ic-lora-depth.safetensors" \ + --lora "$LTX_ROOT/loras/.safetensors" \ --ref-video /tmp/depth_frames --ref-video-strength 1.0 \ --prompt-embeds "$LTX_VIDEO_EMBEDS" --audio-prompt-embeds "$LTX_AUDIO_EMBEDS" \ --frames 25 --width 320 --height 192 --seed 20260812 \ --workdir /tmp/ltx25ic --out /tmp/ltx25ic/video.mp4 ``` +`--lora` is a PLACEHOLDER above, and deliberately so. The pinned upstream +checkout names no reference-conditioning IC-LoRA for 2.5: its only 2.5 IC-LoRA is +[`ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler), +which is `DFRPipeline`'s detailing adapter and rides stage 2 (`README.md:95-96`). +The published control adapters that DO condition on a reference clip are 2.3-era +and 19b — `LTX-2.3-22b-IC-LoRA-Union-Control`, +`LTX-2.3-22b-IC-LoRA-Motion-Track-Control` and +`LTX-2-19b-IC-LoRA-Pose-Control` (`MODELS-LTX-2.3.md:32-35`) — so none of them +pairs with the 2.5 22b transformer this recipe loads. No adapter is named here +because naming one would claim a pairing nothing has run. The mechanism below is +what this row gated; the weights are not. + `--ref-video` is a directory of `frame_%06d.ppm`, not a container: upstream opens one with PyAV and no demuxer is vendored here. The clip is read at `height // reference_downscale_factor` by `width // reference_downscale_factor`, diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index b4d0cbadf..8981580a8 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3238,15 +3238,6 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { const bool wants_first_frame = !gen.first_frame_path.empty() || !gen.first_frame_ppm.empty(); const bool wants_last_frame = !gen.last_frame_path.empty(); const bool wants_image = wants_first_frame || wants_last_frame; - // `!wants_retake` IS LOAD-BEARING, and it narrows this refusal rather than - // weakening it. Retake and IC-LoRA reference conditioning both arrive as - // `ref_video_dir`, and they consume it in completely different ways: retake - // encodes the clip at its OWN resolution and seeds the video stream's initial - // latent with it (retake.py:238-247, :273), while the reference item is - // downscaled by the adapter's factor, temporally subsampled, and APPENDED as - // extra tokens to a stage-1-only adapter (iclora_utils.py:112-117, :87-89, - // :144-148). Serving the first says nothing about the second, so the second - // stays refused and #975 stays open. // ── IC-LoRA REFERENCE CONDITIONING (row LTX25-IC-LORA-REF-VIDEO, #3020) ── // // ONE PREDICATE, READ ONCE, and it is what makes this block correct rather @@ -3336,7 +3327,13 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "two-stage pipeline. Set `ref_video_dir` to a directory of frame_%06d.ppm, or load with " "'pipeline_kind' 'distilled_two_stage' for an unconditioned render."); } - if (!cond_mask_dir.empty() && (cond_attention_strength < 0.0 || cond_attention_strength > 1.0)) { + // UNCONDITIONAL, exactly as upstream's is. `ic_lora.py:230-233` sits in the + // method body, before anything about the mask and outside every `if + // args.conditioning_attention_mask is not None`. Guarding it on a mask made + // this engine ACCEPT an out-of-range strength whenever none was supplied and + // then never read it, because its only reader is inside the mask branch: a + // knob that takes a value, refuses nothing, and does nothing. + if (cond_attention_strength < 0.0 || cond_attention_strength > 1.0) { Fail("'" + std::string(kLtx2CondAttentionStrengthExtra) + "' must be in [0.0, 1.0], got " + std::to_string(cond_attention_strength) + ". This is upstream's own refusal (ic_lora.py:230-233): the value multiplies a mask in " diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 2ef1e2443..e64473441 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -14270,6 +14270,32 @@ TEST_CASE("ltx2 ic-lora: every refusal names what is wrong and where the feature } } + SUBCASE("a strength outside [0, 1] with NO mask, which upstream refuses just the same") { + // `ic_lora.py:230-233` is UNCONDITIONAL: it is in the method body, not + // inside `if args.conditioning_attention_mask is not None`. Guarding this + // engine's copy on a mask directory made an out-of-range strength with no + // mask pass the range check AND pass the sub-1.0 scalar-arm refusal below + // it, leaving a value that is accepted, never read (its only reader is + // inside the mask branch) and silently ignored on every pipeline. + const std::string ups = WriteIcLoraUpsampler(ws.root + "/nomask_range_ups.safetensors"); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(IcLoraParams(ws.paths, ups, /*skip_stage_2=*/true)); + vllm::multimodal::VideoGenParams gen = IcLoraGen(ws.root + "/nomask_range", clip); + // NO kLtx2CondAttentionMaskDirExtra. That is the whole point of the case. + gen.extras[vllm::multimodal::kLtx2CondAttentionStrengthExtra] = "1.5"; + try { + (void)engine->Generate(gen); + FAIL("an out-of-range strength must be refused whether or not a mask was supplied; " + "upstream's check does not consult the mask"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("[0.0, 1.0]") != std::string::npos); + CHECK(msg.find("AMPLIFY") != std::string::npos); + CHECK(msg.find("ic_lora.py:230-233") != std::string::npos); + } + } + SUBCASE("a reference IMAGE, which is not what an IC-LoRA reference is") { const std::string ups = WriteIcLoraUpsampler(ws.root + "/img_ups.safetensors"); const std::unique_ptr engine = From b7be218a09ec3367df8743fad1f4e747030add85 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 09:43:13 +0000 Subject: [PATCH 10/11] docs(LTX25-IC-LORA-REF-VIDEO): repair the record, and derive the kind list the tripwire walks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five record defects, each of which read as a measurement and was not one. The separation table recorded 0.1890125870704651 and 0.16791561245918274 for the mask downsample. The committed `.inc` says 0.20678424835205078 and 0.049749284982681274, and a byte-identical regeneration against the pinned checkout says the same. The wrong pair had already propagated through two briefings, so these are re-derived from the generated file rather than copied forward again. The new non-dividing case joins them. The generator was cited as `tools/ltx2/gen_iclora_reference_goldens.py`. That directory does not exist; the file is `scripts/gen-ltx2-iclora-reference-goldens.py`. `Ltx2ReadMaskFrameDirectory(dir, height, width)` was named in the anchor table and again in §4.2. `grep -rn` finds it in those two lines and nowhere else: the tree composes the mask read at the call site out of `Ltx2ReadFrameDirectory` and `Ltx2MaskVideoFromPixels`. A function named in a spec and absent from the tree sends the next reader looking for a seam that was never built. The mutation table said M1 reds four assertions. Re-measured, it reds three, and WHICH fourth it is matters more than the count: `trace.ic_lora_attention_mask_rows == 16` still passes under the mutant, because the trace records that the mask was built and the deleted line is the one that hands it to the model. Only the pixel compares see the drop. That is now written down, because a trace assertion is cheaper than a render and the next reader will be tempted. The table's other counts are left as measured and dated instead, since restating a mutation result against a tree it was not run on is the drift these tables exist to prevent. `ref_video_strength` with no `ref_video_dir` is the same accept-and-ignore shape as the strength defect this round fixed. It is recorded owed rather than fixed, because upstream cannot express it — `--video-conditioning` is `nargs=2` — so any refusal here would be a product decision and not a port. The M11 tripwire now DERIVES its kind list. It walked a hand-copied eleven-entry array, which is complete only until the next kind lands: a twelfth would have gone unwalked while the case went on reporting "exactly one". The kinds now come out of every `pipeline_kind == "..."` branch in `ltx2_pipeline.cpp`, the file a new kind must be written into to exist, and the count is asserted before it is used so a derivation that found nothing cannot pass silently. Measured: adding a twelfth branch to the resolver reds the case with `declares 12 kinds, not the 11 this tripwire was written against`, and the assertion count rises by two, which is the new kind actually being walked. The hand-copied version stayed green. Issue #3020's records section named `docs/USAGE.md`, and this change does not write it. Struck there rather than written, because that file no longer carries per-model commands: its own "Find a model recipe" section routes commands, weights, settings and limits to `docs/models/.md`, which is where these knobs are, and it names no `ltx2-gen` flag at all. Writing them in both places would put one fact in two documents. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora-ref-video.md | 51 +++++++++++++++++++----- docs/FEATURES.md | 2 +- tests/vllm/models/test_ltx2_pipeline.cpp | 43 +++++++++++++++++--- 3 files changed, 80 insertions(+), 16 deletions(-) diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md index 039b7ac87..16b216cad 100644 --- a/.agents/specs/ltx25-ic-lora-ref-video.md +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -87,7 +87,7 @@ Root `Lightricks/LTX-2 @ fd4ded7f`. | `Ltx2BuildAttentionMask` | `ltx-core/conditioning/mask_utils.py:170-243` | | `Ltx2UpdateAttentionMask` | `ltx-core/conditioning/mask_utils.py:110-167` | | the wrapper's apply order | `ltx-core/conditioning/types/attention_strength_wrapper.py:43-71` | -| `Ltx2ReadMaskFrameDirectory` | `ltx-pipelines/ic_lora.py:511-537` (`_load_mask_video`) | +| the mask read (`Ltx2ReadFrameDirectory` + `Ltx2MaskVideoFromPixels`, inlined at the call site) | `ltx-pipelines/ic_lora.py:511-537` (`_load_mask_video`) | | the stage split | `ltx-pipelines/ic_lora.py:269-281` (stage 1) vs `:314-321` (stage 2) | | the CLI shape | `ltx-pipelines/ic_lora.py:415-441`, `:452-463`, `:481-498` | | the consumption (already ported) | `ltx-core/model/transformer/transformer_args.py:208-237`, `:289` | @@ -133,9 +133,11 @@ expression, bound once in a named local, so the two cannot drift. ### 4.2 A16 — the attention mask -`Ltx2ReadMaskFrameDirectory(dir, height, width)` mirrors `_load_mask_video`: -read the frames through the SAME `Ltx2ReadFrameDirectory`, mean over the three -channels, remap `(x + 1) / 2`, clamp to `[0, 1]`. It is read at the STAGE-1 +The mask read mirrors `_load_mask_video`, and it is NOT a function of its own: +`ltx2_video.cpp` composes it at the call site out of the SAME +`Ltx2ReadFrameDirectory` the reference clip uses plus `Ltx2MaskVideoFromPixels`, +which means over the three channels, remaps `(x + 1) / 2` and clamps to +`[0, 1]`. It is read at the STAGE-1 resolution, which upstream spells `args.height // 2` at `ic_lora.py:460-461` and this engine reads from the phase's own grid — the same number, derived from the phase rather than assumed from the CLI. @@ -266,7 +268,7 @@ weakened to reach a green: every suite that can observe this change was built and run. **The goldens are generated by executing the pinned module**, never by reading -it: `tools/ltx2/gen_iclora_reference_goldens.py` imports +it: `scripts/gen-ltx2-iclora-reference-goldens.py` imports `ltx_pipelines.iclora_utils` and `ltx_core.conditioning.mask_utils` from `/home/mudler/_git/LTX-2` at `fd4ded7f` and writes `tests/vllm/models/ltx2_iclora_reference_goldens.inc`. It emits, for every case, @@ -279,8 +281,9 @@ Measured separations, which are what makes each case able to fail: |---|---|---| | `build_attention_mask` with a prior ref | cross on ALL existing rows | 4 elements | | the same with `num_existing == num_noisy` | the same | **0 — REFUSED by the generator** | -| `downsample_mask_video_to_latent` | bilinear spatial | 0.1890125870704651 | -| the same | uniform temporal pooling, no causal carve-out | 0.16791561245918274 | +| `downsample_mask_video_to_latent` | bilinear spatial | 0.20678424835205078 | +| the same | uniform temporal pooling, no causal carve-out | 0.049749284982681274 | +| the same at 9 -> 2 | an integer-stride box filter | 0.08158010244369507 | | `_prepare_self_attention_mask` on `[1, 0.5, 0, 1e-30]` | — | `[0, -0.6931471824645996, -3.4028234663852886e+38, -69.07755279541016]` | **Reachability.** `test_ltx2_video` renders through the public `Generate` on the @@ -298,9 +301,17 @@ suites run, and the tree restored with `git checkout --` plus a `git status (97 assertions green), `reach` is `test_ltx2_video --test-case='ltx2 ic-lora*'` (169 assertions green). +**The counts in this table are AS MEASURED, before fresh review.** The review +round that followed added the non-dividing pooling golden and the no-mask range +refusal, so `unit` is now 102 assertions over 14 cases and the full +`test_ltx2_video` is 5237 over 131. The table is not rewritten to those numbers, +because a mutation result is a measurement and restating it against a tree it +was not run on is exactly the drift these tables exist to prevent. Only M1's +count is corrected below, and it is corrected because it was RE-MEASURED. + | # | mutation | unit | reach | verdict | |---|---|---|---|---| -| M1 | delete the production `vin.attention_mask` assignment | 97/97 pass | **4 failed** | DETECTED — the reachability property | +| M1 | delete the production `vin.attention_mask` assignment | 97/97 pass | **3 failed** | DETECTED — the reachability property | | M2 | never apply `Ltx2ConditionVideoByReference` | 97/97 pass | **threw** | DETECTED | | M3 | `temporal_subsample` steps from 0 | **21 failed** | 169/169 pass | DETECTED | | M4 | drop the causal first-frame carve-out | **1 failed** | 169/169 pass | DETECTED | @@ -316,6 +327,17 @@ suites run, and the tree restored with `git checkout --` plus a `git status **Ten of twelve detected. Neither survivor is a blind spot, and both are identities provable rather than argued.** +**M1 reds THREE assertions, not four.** This spec recorded four; fresh review +re-ran it and measured three, at `test_ltx2_video.cpp:14097`, `:14109` and +`:14124`. The fourth is worth naming because of WHICH one it is: +`trace.ic_lora_attention_mask_rows == 16` still PASSED under the mutant. The +trace records that the mask was built and how many rows it has, and the deleted +line is the one that hands it to the model — so the counter is still correct +about a mask nothing consumes. **Only the pixel compares can see the drop.** A +future reader tempted to gate this seam on the trace alone, because a trace +assertion is cheaper than a render, would be gating the construction and not the +consumption. + **M6.** The mutant drops `downscale_factor != 1` from `downscale_factor != 1 && (h % downscale_factor || w % downscale_factor)`. For `downscale_factor == 1` the right conjunct is `h % 1 || w % 1`, which is `0` @@ -330,9 +352,17 @@ which `ResolveLtx2PipelineRecipe` reaches only for `pipeline_kind == "ic_lora"`. So over the recipe table AS IT STANDS the two expressions are the same predicate. **It stops being an identity the moment a second recipe sets the flag**, and this row therefore adds the tripwire for that day: -`test_ltx2_pipeline`'s recipe-table case walks all eleven kinds against all four +`test_ltx2_pipeline`'s recipe-table case walks every kind against all four versions and requires exactly ONE resolved pair to carry `ic_lora_reference`, -naming `ltx2_video.cpp`'s refusals as what must be re-read. That converts an +naming `ltx2_video.cpp`'s refusals as what must be re-read. **The kind list is +DERIVED**, from every `pipeline_kind == "..."` branch in +`ltx2_pipeline.cpp` — the file a new kind has to be written into to exist. It was +a hand-copied eleven-entry array until fresh review, which is a list that is +complete only until the next kind lands: a twelfth kind would have gone unwalked +while the case went on reporting "exactly one". Measured rather than argued — +adding a twelfth branch to the resolver now reds the case with +`declares 12 kinds, not the 11 this tripwire was written against`, and the +assertion count rises by two, which is the new kind actually being walked. That converts an unmeasured residual into a guarded one; it does not make M11 detectable today, because today it changes nothing. @@ -346,6 +376,7 @@ because today it changes nothing. | the reference-AUDIO arm, unchanged by this row and still refused by name | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | | the bf16 storage width of every buffer this row adds (R4) | the A24 dtype campaign | | a real-weights IC-LoRA reference render; this row has no GPU authority and no checkpoint | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | +| `ref_video_strength` with no `ref_video_dir` is ACCEPTED and then never read. It is the same shape as the `conditioning_attention_strength` defect this round fixed, and it is left open rather than fixed because upstream cannot express it: `--video-conditioning` is `nargs=2` (`ic_lora.py:416-425`), so upstream has no reading of a strength without a clip to be mirrored, and inventing a refusal would be a product decision rather than a port | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) | ## 7. Stop conditions diff --git a/docs/FEATURES.md b/docs/FEATURES.md index a34255962..18a2c3ef4 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -228,7 +228,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` (#1009). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 Conv VAE decode DEVICE arm | LTX-2.5 video VAE | `test_ops_conv3d` 4/4 vs an independent scalar reference; `test_diffusion_device_seam` runs a non-CPU dispatch and requires byte-identical pixels | **f32**, which is what `vt::Conv3d` and the `kLtx2Vae` CUDA table serve, and what `Ltx2VideoEngine::Load` asks for whenever the render resolved a device queue ([#2853](https://github.com/mudler/vllm.cpp/issues/2853)). So this arm decodes 776.6 MiB wider than the CPU arm and off upstream's SDR bf16 grid; the bf16 DEVICE arm is owed and needs #1007. Routes through the new `vt::Conv3d` op on the queue the engine resolved at load (#1007). NO GPU has run the CUDA arm (#1452); other stages stay host-side (#1451) | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 IC-LoRA reference conditioning (`ICLoraPipeline`, A15 + A16) | LTX-2.5 DiT + video VAE encoder + an IC-LoRA adapter | `test_ltx2_iclora_reference` 13/13 (97 assertions) against the EXECUTED pinned module, every case carrying its rejected hypothesis; 4 `test_ltx2_video` cases entering through `Generate` | `--pipeline-kind ic_lora`. `--ref-video` is a `frame_%06d.ppm` DIRECTORY (no demuxer), read at `height // reference_downscale_factor` from the adapter's own metadata and temporally subsampled by its `reference_temporal_scale_factor`. The adapter rides STAGE 1 ONLY and stage 2 runs bare, which is the mirror image of every other two-stage arm here. `--conditioning-attention-mask` attenuates the reference per region and is the first production writer of `Ltx2ModalityInput::attention_mask`. REFUSED and owed: the EXR/HDR reference arm, `tiled_encode` for a reference, reference IMAGES (upstream has no such flag), and the scalar-only strength arm that upstream's own CLI cannot reach either | +| LTX-2.5 IC-LoRA reference conditioning (`ICLoraPipeline`, A15 + A16) | LTX-2.5 DiT + video VAE encoder + an IC-LoRA adapter | `test_ltx2_iclora_reference` 14/14 (102 assertions) against the EXECUTED pinned module, every case carrying its rejected hypothesis, including a NON-DIVIDING 9 -> 2 pooling shape that separates `area` from an integer-stride box filter; 4 `test_ltx2_video` cases entering through `Generate` | `--pipeline-kind ic_lora`. `--ref-video` is a `frame_%06d.ppm` DIRECTORY (no demuxer), read at `height // reference_downscale_factor` from the adapter's own metadata and temporally subsampled by its `reference_temporal_scale_factor`. The adapter rides STAGE 1 ONLY and stage 2 runs bare, which is the mirror image of every other two-stage arm here. `--conditioning-attention-mask` attenuates the reference per region and is the first production writer of `Ltx2ModalityInput::attention_mask`. REFUSED and owed: the EXR/HDR reference arm, `tiled_encode` for a reference, reference IMAGES (upstream has no such flag), and the scalar-only strength arm that upstream's own CLI cannot reach either | | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage` plus `--lora`, which is now REQUIRED (#1445, both stages, at ONE strength rather than upstream's 0.25/0.5; #1144). 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 (#1039) | diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 023bc9fff..ad8022643 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1189,13 +1189,46 @@ TEST_CASE("ltx2 the recipe table mirrors vLLM-Omni's, and refuses everything els // `ic_lora_reference` reds HERE, where the reader is told to reconcile the // refusal, rather than in a render that silently refuses a supported request. { - const char* const kAllKinds[] = {"one_stage", "distilled_two_stage", "dfr", - "res2s_two_stage", "dmd2", "retake", - "a2vid_two_stage", "ti2vid_two_stage", - "keyframe_interpolation", "t2a_one_stage", "ic_lora"}; + // THE KIND LIST IS DERIVED, not hand-copied. A hand-copied list is complete + // only until the next kind is added, and this tripwire's whole job is to + // fire on that day — a list that silently omits the twelfth kind would go on + // reporting "exactly one" while the new recipe went unwalked. So the kinds + // come out of `ResolveLtx2PipelineRecipe`'s own source: every + // `pipeline_kind == "..."` branch in ltx2_pipeline.cpp, which is where a new + // kind has to be written down to exist at all. + const std::filesystem::path resolver_source = + std::filesystem::path(VLLM_CPP_SOURCE_ROOT) / + "src/vllm/model_executor/models/ltx2_pipeline.cpp"; + std::ifstream resolver_in(resolver_source); + REQUIRE_MESSAGE(resolver_in.good(), + "cannot read " << resolver_source.string() + << "; VLLM_CPP_SOURCE_ROOT is wrong"); + const std::string resolver((std::istreambuf_iterator(resolver_in)), + std::istreambuf_iterator()); + std::vector kAllKinds; + const std::string needle = "pipeline_kind == \""; + for (size_t at = resolver.find(needle); at != std::string::npos; + at = resolver.find(needle, at + 1)) { + const size_t start = at + needle.size(); + const size_t end = resolver.find('"', start); + REQUIRE(end != std::string::npos); + kAllKinds.push_back(resolver.substr(start, end - start)); + } + // A derivation that found nothing would pass every assertion below without + // walking anything, so the count is asserted before it is used. Eleven is + // what the table carries today; a twelfth kind reds HERE, which is the point. + CHECK_MESSAGE(kAllKinds.size() == 11, + "ResolveLtx2PipelineRecipe declares " << kAllKinds.size() + << " kinds, not the 11 this tripwire was " + "written against. Re-read the " + "ic_lora_reference refusals in " + "ltx2_video.cpp, then update this " + "count."); + REQUIRE(kAllKinds.size() >= 11); const char* const kAllVersions[] = {"2", "2.3", "2.4", "2.5"}; int resolved_with_flag = 0; - for (const char* kind : kAllKinds) { + for (const std::string& kind_s : kAllKinds) { + const char* const kind = kind_s.c_str(); for (const char* version : kAllVersions) { vllm::Ltx2PipelineRecipe recipe; try { From 0719d23c496a0a179ebb6aa75f192ad5551df271 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 6 Sep 2026 10:26:43 +0000 Subject: [PATCH 11/11] docs(LTX25-IC-LORA-REF-VIDEO): discharge the whole-tree gate, which is no longer pending This spec recorded the whole-tree `ctest` as PENDING on CI, and gave the reason: a corrupted `.ninja_deps` turned it into a from-scratch 1338-edge rebuild, which under `-j 2` alongside four other compiling agents at load 20-33 reached 76 edges and stopped. That was an honest PENDING and it is now discharged, twice, on an idle box: fresh review ran it on `975bc0906` and this round ran it again on the merged head, both 100% tests passed, 0 failed out of 714, with the same 7 skips, every one GPU- or checkpoint-gated and none in this row's reach. The per-suite numbers move with it, because this round's repairs added assertions: 102 over 14 cases on the value suite and 5237 over 131 on test_ltx2_video. Recording it here rather than leaving the pull request to say it, because the spec is what a reader consults after the branch is gone. FOLLOWING_AGENTS_PROTOCOL Refs: #3020 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora-ref-video.md | 44 +++++++++++++++--------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/.agents/specs/ltx25-ic-lora-ref-video.md b/.agents/specs/ltx25-ic-lora-ref-video.md index 16b216cad..aa3a9d5c1 100644 --- a/.agents/specs/ltx25-ic-lora-ref-video.md +++ b/.agents/specs/ltx25-ic-lora-ref-video.md @@ -249,23 +249,33 @@ python3 scripts/agent-issue-index.py --refresh && python3 scripts/check-agent-re scripts/agent-preflight.sh ``` -**WHAT THE LOCAL GATE COVERS, AND WHAT IT DOES NOT.** The fifteen suites above -are every test target that links a translation unit this change touches, plus -`ltx2-gen`, the shipped CLI that reaches the new knobs. Run on the rebased head: -`cmake --build` rc 0, `ctest` rc 0, **15/15 passed**, and -`test_ltx2_iclora_reference` 97/97, `test_ltx2_video` 5225/5225, `test_ltx2` -4581/4581, `test_ltx2_pipeline` 4712/4712, `test_ltx2_retake` 69/69 assertions. - -The WHOLE-TREE `ctest` was NOT run here, and the reason is mechanical rather -than a judgement about coverage: a `kill -9` during this row's scheduling -corrupted `build/.ninja_deps`, so ninja discarded its dependency graph and -scheduled all 1338 edges as a from-scratch rebuild. Under this row's `-j 2` -constraint, on a box carrying four other compiling agents at load 20-33, that -does not finish. It reached 76 of 1338 before it was stopped. **That is a -PENDING result on an external resource, not a pass**, and CI's own full build is -what discharges it on the pull request. Nothing about the local run was -weakened to reach a green: every suite that can observe this change was built -and run. +**THE WHOLE-TREE GATE IS NO LONGER PENDING.** This spec recorded it as a +PENDING result on an external resource, because a `kill -9` during the row's +scheduling had corrupted `build/.ninja_deps`, ninja discarded its dependency +graph and scheduled all 1338 edges, and under `-j 2` on a box carrying four +other compiling agents at load 20-33 that did not finish — it reached 76 of +1338. **It has now been run twice on an idle box** and it is a pass, so the +PENDING is discharged locally and not only by CI: + +* Fresh review, on head `975bc0906`: full build rc 0, then + `ctest -j 2` — **100% tests passed, 0 tests failed out of 714**, 7 skipped. +* This repair round, on the merged head: full build rc 0 over all 1337 edges, + then `ctest --test-dir build -j 2 --output-on-failure` — **100% tests passed, + 0 tests failed out of 714**, rc 0, the same 7 skipped. Every skip is + GPU-gated or checkpoint-gated and none is in this row's reach. + +Per-suite on the merged head: `test_ltx2_iclora_reference` 102/102 over 14 +cases, `test_ltx2_video` 5237/5237 over 131, `test_ltx2` 4581/4581, +`test_ltx2_pipeline` 4726/4726, `test_ltx2_retake` 69/69. +`scripts/agent-preflight.sh` rc 0 with 155 `ok` and 5 `SKIP`, every skip being a +checker preflight cannot supply arguments for; `check-pr-size.py` was therefore +run explicitly, rc 0. `check-agent-record.py` rc 0 with the snapshot PRESENT, +refreshed by `agent-issue-index.py --refresh` (316 open issues). + +The fifteen suites the recipe above names are still the right focused set — they +are every target that links a translation unit this change touches, plus +`ltx2-gen`, the shipped CLI that reaches the new knobs — but they are no longer +the whole of what was run. **The goldens are generated by executing the pinned module**, never by reading it: `scripts/gen-ltx2-iclora-reference-goldens.py` imports