feat(LTX25-IC-LORA-REF-VIDEO): serve the IC-LoRA reference clip and its conditioning attention mask

.agents/specs/ltx25-ic-lora.md

@@ -353,8 +353,8 @@ Each is owed by this row and named in the commit and pull request bodies.
 
 | owed | issue |
 |---|---|
-| serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter. §6 derives both. Token-append is NOT part of it any more — #930 landed in `c7cb59fbb` | [#975](https://github.com/mudler/vllm.cpp/issues/975) |
-| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | [#932](https://github.com/mudler/vllm.cpp/issues/932) |
+| ~~serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter~~ **DISCHARGED for the reference VIDEO arm by row `LTX25-IC-LORA-REF-VIDEO`** ([`ltx25-ic-lora-ref-video.md`](ltx25-ic-lora-ref-video.md), [#3020](https://github.com/mudler/vllm.cpp/issues/3020)), which found that the pixel path was already here — `Ltx2ReadFrameDirectory` IS `video_preprocess` over a frame directory and `Ltx2ConvVideoEncode` already runs multi-frame for retake — so what was missing was the reference item's own geometry and the stage split, both of which now land on a new `ic_lora` pipeline kind. The reference-IMAGE arm is NOT discharged and never will be: upstream has no such flag, and it is refused by name. #975 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) |
+| ~~the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`)~~ **DISCHARGED for the MASK arm by row `LTX25-IC-LORA-REF-VIDEO`** ([#3020](https://github.com/mudler/vllm.cpp/issues/3020)). `Ltx2LatentState` carries a mask, the block structure is ported and gated against the executed module, and the mask reaches the DiT — which also retired a seam that was built and dead: nothing in `src/` assigned `Ltx2ModalityInput::attention_mask` before that row. The `strength < 1.0` arm WITHOUT a mask stays owed and is refused by name, because upstream's own CLI cannot reach it either (`ic_lora.py:452-455`). #932 is one of the three issue numbers that 404 (tracked by #2899); the live record is #3020 | [#3020](https://github.com/mudler/vllm.cpp/issues/3020) |
 | ~~N-adapter fusion, which additionally needs upstream's SECOND rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`) that this row refuses rather than guesses~~ **DISCHARGED by row `LTX25-LORA-FUSION`** ([`ltx25-lora-fusion.md`](ltx25-lora-fusion.md)), which ported that second form and gated it as byte equality against the EXECUTED pinned module rather than against a transcription. The arity refusal is gone and `--lora` is repeatable. Lifting it made ONE new arm expressible — N adapters on a recipe with a `kNoAdapters` phase, whose stage 1 upstream gives a proper SUBSET — and that arm refuses by name and is owed in THAT row's `## Owed`, not here. **#932 itself stays OPEN**, because the row above is its other half and nothing has landed it | [#932](https://github.com/mudler/vllm.cpp/issues/932) |
 | GGUF k-quant LoRA fusion — **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a |
 | a real-weights IC-LoRA fusion measurement | blocked on GPU authority; `dgx.casa` was under a long render for this row's duration and this row had no GPU authority |
@@ -376,6 +376,10 @@ Each is owed by this row and named in the commit and pull request bodies.
 
 ## Now
 
-`ACTIVE` — the adapter path is implemented and gated; the reference arm stays
-refused on the two causes §6 derives, which are #975 and are neither of the two
-this refusal has previously given.
+`DONE` — the adapter path is implemented and gated. The reference arm that §6
+left refused was SERVED by row `LTX25-IC-LORA-REF-VIDEO`
+([#3020](https://github.com/mudler/vllm.cpp/issues/3020)), together with the
+conditioning attention mask; §6's derivation was correct about what was missing
+and wrong about how much of it was, because the pixel path it named already
+existed as `Ltx2ReadFrameDirectory`. What that row left owed is in its own
+`## Owed`.

CMakeLists.txt

@@ -972,6 +972,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp
   src/vllm/model_executor/models/ltx2_audio_vae_encoder_load.cpp
   src/vllm/model_executor/models/ltx2_audio_input.cpp
+  src/vllm/model_executor/models/ltx2_iclora_reference.cpp
   src/vllm/model_executor/models/ltx2_image_preprocess.cpp
   # LTX-2.5 RETAKE (row LTX25-RETAKE, #924): the temporal region mask, the
   # truncate-or-pad conform, and the frame-directory source. Its own TU for the

docs/FEATURES.md

@@ -228,6 +228,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` (#1009). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count |
 | LTX-2.5 Conv VAE decode DEVICE arm | LTX-2.5 video VAE | `test_ops_conv3d` 4/4 vs an independent scalar reference; `test_diffusion_device_seam` runs a non-CPU dispatch and requires byte-identical pixels | **f32**, which is what `vt::Conv3d` and the `kLtx2Vae` CUDA table serve, and what `Ltx2VideoEngine::Load` asks for whenever the render resolved a device queue ([#2853](https://github.com/mudler/vllm.cpp/issues/2853)). So this arm decodes 776.6 MiB wider than the CPU arm and off upstream's SDR bf16 grid; the bf16 DEVICE arm is owed and needs #1007. Routes through the new `vt::Conv3d` op on the queue the engine resolved at load (#1007). NO GPU has run the CUDA arm (#1452); other stages stay host-side (#1451) |
 | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated |
+| LTX-2.5 IC-LoRA reference conditioning (`ICLoraPipeline`, A15 + A16) | LTX-2.5 DiT + video VAE encoder + an IC-LoRA adapter | `test_ltx2_iclora_reference` 14/14 (102 assertions) against the EXECUTED pinned module, every case carrying its rejected hypothesis, including a NON-DIVIDING 9 -> 2 pooling shape that separates `area` from an integer-stride box filter; 4 `test_ltx2_video` cases entering through `Generate` | `--pipeline-kind ic_lora`. `--ref-video` is a `frame_%06d.ppm` DIRECTORY (no demuxer), read at `height // reference_downscale_factor` from the adapter's own metadata and temporally subsampled by its `reference_temporal_scale_factor`. The adapter rides STAGE 1 ONLY and stage 2 runs bare, which is the mirror image of every other two-stage arm here. `--conditioning-attention-mask` attenuates the reference per region and is the first production writer of `Ltx2ModalityInput::attention_mask`. REFUSED and owed: the EXR/HDR reference arm, `tiled_encode` for a reference, reference IMAGES (upstream has no such flag), and the scalar-only strength arm that upstream's own CLI cannot reach either |
 | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name |
 | LTX-2.5 HQ preset (`TI2VidTwoStagesHQPipeline`, `res_2s` sampler) | LTX-2.5 DiT | 6 `test_ltx2_pipeline` cases + 2 `test_ltx2_video` cases through `Generate`, vs UPSTREAM'S OWN loop run at `fd4ded7f`: video latents BIT-EXACT on 3 of 5 fixtures, 1 ulp on 2. 20 mutations, 18 DETECTED | `--pipeline-kind res2s_two_stage` plus `--lora`, which is now REQUIRED (#1445, both stages, at ONE strength rather than upstream's 0.25/0.5; #1144). 2.5 only. TWO denoiser calls per step plus a terminal one, and stage 1 is GUIDED at cfg 3.0 / modality 3.0, so 15 + 3 steps is 38 calls and 100 forwards. The preset IS the sampler |
 | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 (#1039) |

docs/models/ltx-2-5.md

@@ -117,6 +117,71 @@ ltx2-gen \
   --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4
 ```
 
+## Condition a render on a reference clip (IC-LoRA)
+
+```sh
+ltx2-gen \
+  --dit "$LTX_ROOT/diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors" \
+  --model-version 2.5 --checkpoint-class distilled \
+  --video-vae "$LTX_ROOT/vae/ltx-2.5-video-vae-conv-bf16.safetensors" \
+  --audio-vae "$LTX_ROOT/vae/ltx-2.5-audio-vae-bf16.safetensors" \
+  --upsampler "$LTX_ROOT/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors" \
+  --pipeline-kind ic_lora \
+  --lora "$LTX_ROOT/loras/<YOUR-IC-LORA>.safetensors" \
+  --ref-video /tmp/depth_frames --ref-video-strength 1.0 \
+  --prompt-embeds "$LTX_VIDEO_EMBEDS" --audio-prompt-embeds "$LTX_AUDIO_EMBEDS" \
+  --frames 25 --width 320 --height 192 --seed 20260812 \
+  --workdir /tmp/ltx25ic --out /tmp/ltx25ic/video.mp4
+```
+
+`--lora` is a PLACEHOLDER above, and deliberately so. The pinned upstream
+checkout names no reference-conditioning IC-LoRA for 2.5: its only 2.5 IC-LoRA is
+[`ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler),
+which is `DFRPipeline`'s detailing adapter and rides stage 2 (`README.md:95-96`).
+The published control adapters that DO condition on a reference clip are 2.3-era
+and 19b — `LTX-2.3-22b-IC-LoRA-Union-Control`,
+`LTX-2.3-22b-IC-LoRA-Motion-Track-Control` and
+`LTX-2-19b-IC-LoRA-Pose-Control` (`MODELS-LTX-2.3.md:32-35`) — so none of them
+pairs with the 2.5 22b transformer this recipe loads. No adapter is named here
+because naming one would claim a pairing nothing has run. The mechanism below is
+what this row gated; the weights are not.
+
+`--ref-video` is a directory of `frame_%06d.ppm`, not a container: upstream opens
+one with PyAV and no demuxer is vendored here. The clip is read at
+`height // reference_downscale_factor` by `width // reference_downscale_factor`,
+and both factors come from the adapter's own metadata rather than from a flag. A
+load with no adapter reads both as 1, which is upstream's default. A target
+either axis of which the factor does not divide is refused by name, with
+upstream's own sentence.
+
+The adapter rides **stage 1 only** and stage 2 runs bare. That is the mirror
+image of `ti2vid_two_stage`, `a2vid_two_stage` and `keyframe_interpolation`,
+where the adapter rides stage 2, and it is why IC-LoRA has its own pipeline kind
+rather than being a mode of `distilled_two_stage`. A reference clip supplied to
+any other kind is refused, and the message names this one.
+
+`--conditioning-attention-mask` takes a directory of grayscale
+`frame_%06d.ppm` whose pixels attenuate the reference per region: black ignores
+the conditioning there, white takes it in full. The frames are read at the
+stage's own resolution and downsampled to the reference latent's grid, so a mask
+must describe the same moments as the clip. `--conditioning-attention-strength`
+scales the whole mask and must be in `[0, 1]`; above 1 it would amplify
+attention rather than attenuate it, and is refused.
+
+A strength below 1 with **no** mask is refused rather than served. Upstream has
+that branch and its own CLI cannot reach it either: the strength is assigned only
+alongside a mask, so a sub-1.0 value always arrives with one.
+
+Through the C ABI and the server these are per-generation extras rather than
+flags: `ref_video_strength`, `conditioning_attention_mask_dir` and
+`conditioning_attention_strength`, beside the existing `ref_video` field.
+
+Not served, and refused by name: the EXR/HDR reference arm, which needs an
+OpenEXR reader and a declared colour space; a tiled reference encode; and
+reference IMAGES, which are not an IC-LoRA shape at all — upstream's flag takes a
+video or an EXR directory and `ltx-pipelines` has no reference-image
+conditioning.
+
 `--lora` is repeatable, and each repetition takes an adapter path with an
 optional strength, exactly as upstream's own flag does: `--lora
 first.safetensors 0.8 --lora second.safetensors`. An omitted strength is

examples/ltx2_gen/main.cpp

@@ -188,6 +188,17 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "how much is read (default: the clip\'s own duration). Either without\n"
       "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n"
       "input, not a VAE round trip, which is upstream\'s deliberate choice.\n\n"
+      "IC-LoRA REFERENCE CONDITIONING conditions the whole render on a reference clip\n"
+      "(depth, pose, edges) and needs --pipeline-kind ic_lora, which is upstream's own\n"
+      "ICLoraPipeline: the adapter rides stage 1 and stage 2 runs bare. --ref-video is\n"
+      "the clip, as a DIRECTORY of frame_%%06d.ppm, and --ref-video-strength is the\n"
+      "second half of upstream's --video-conditioning PATH STRENGTH. Supply the\n"
+      "IC-LoRA itself with --lora: its metadata carries the downscale and temporal\n"
+      "factors the reference is read at, and without one both default to 1.\n"
+      "--conditioning-attention-mask names a grayscale frame directory whose pixels\n"
+      "attenuate the reference per region, and --conditioning-attention-strength\n"
+      "scales it. A strength below 1 with no mask is REFUSED rather than served: it is\n"
+      "upstream's Python-API-only branch, which its own CLI cannot reach either.\n\n"
       "RETAKE regenerates a time window of an existing clip and keeps the rest.\n"
       "--ref-video names a DIRECTORY of frame_%%06d.ppm (the layout minimax-h3-gen\n"
       "writes), not a container file: upstream opens one with PyAV and no demuxer is\n"
@@ -289,6 +300,9 @@ int main(int argc, char** argv) {
   // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to
   // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container.
   std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio;
+  // IC-LoRA reference conditioning (#3020): the strength half of
+  // `--video-conditioning`, and both halves of `--conditioning-attention-mask`.
+  std::string ref_video_strength, cond_mask_dir, cond_mask_strength;
   // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of
   // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120).
   std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale;
@@ -422,6 +436,17 @@ int main(int argc, char** argv) {
       retake_end = Need(argc, argv, ++i, "--retake-end-time");
     else if (f == "--retake-frame-rate")
       retake_fps = Need(argc, argv, ++i, "--retake-frame-rate");
+    // IC-LoRA REFERENCE CONDITIONING (#3020). `--ref-video` above names the
+    // clip; these three are the rest of upstream's two flags —
+    // `--video-conditioning PATH STRENGTH` and
+    // `--conditioning-attention-mask MASK_PATH STRENGTH` (ic_lora.py:416-441).
+    // Per-generation, so they ride vp.extra_* like the retake knobs.
+    else if (f == "--ref-video-strength")
+      ref_video_strength = Need(argc, argv, ++i, "--ref-video-strength");
+    else if (f == "--conditioning-attention-mask")
+      cond_mask_dir = Need(argc, argv, ++i, "--conditioning-attention-mask");
+    else if (f == "--conditioning-attention-strength")
+      cond_mask_strength = Need(argc, argv, ++i, "--conditioning-attention-strength");
     // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which
     // is a LOAD extra; these six are per-generation and are refused by name on
     // any other pipeline rather than accepted and ignored.
@@ -517,6 +542,18 @@ int main(int argc, char** argv) {
     gen_keys.emplace_back("audio_path");
     gen_values.push_back(audio_path);
   }
+  if (!ref_video_strength.empty()) {
+    gen_keys.emplace_back("ref_video_strength");
+    gen_values.push_back(ref_video_strength);
+  }
+  if (!cond_mask_dir.empty()) {
+    gen_keys.emplace_back("conditioning_attention_mask_dir");
+    gen_values.push_back(cond_mask_dir);
+  }
+  if (!cond_mask_strength.empty()) {
+    gen_keys.emplace_back("conditioning_attention_strength");
+    gen_values.push_back(cond_mask_strength);
+  }
   if (!audio_start_time.empty()) {
     gen_keys.emplace_back("audio_start_time");
     gen_values.push_back(audio_start_time);