FrameCrafter: Novel View Synthesis as Video Completion

Qi Wu, Khiem Vuong, Minsik Jeon, Srinivasa Narasimhan, Deva Ramanan

Carnegie Mellon University

News

• [2026-06] FrameCrafter is accepted to ECCV 2026! 🎉
• [2026-05] Training code released. See Training for the LoRA recipe used to produce the released checkpoint.
• [2026-04] Inference code released.

Overview

We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models: given K (≈ 5) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be invariant to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks.

Environment Setup

conda create -n framecrafter python=3.11 -y
conda activate framecrafter
pip install -e .

Backbone Weights

The model requires pre-trained Wan2.1 backbone weights (DiT 14B, T5 text encoder, VAE, CLIP image encoder). By default these are downloaded automatically on first run into ./models/ — no setup needed.

Choosing the download source. Defaults to ModelScope. To download from HuggingFace instead (often faster outside China):

export DIFFSYNTH_DOWNLOAD_SOURCE=huggingface

Custom download location. Override the default ./models/ directory:

export DIFFSYNTH_MODEL_BASE_PATH=/path/to/backbone/weights

Or pass it explicitly when loading the model:

model = FrameCrafter(
    "ckpt/framecrafter.safetensors",
    base_model_dir="/path/to/backbone/weights",
)

Model Weights

Download the FrameCrafter checkpoint into ckpt/:

Option A -- Hugging Face (recommended):

huggingface-cli download szqwu/FrameCrafter framecrafter.safetensors --local-dir ckpt

Option B -- Google Drive:

bash prepare/download_ckpt.sh

Both place the checkpoint at ckpt/framecrafter.safetensors (the default path used by the scripts).

Quick Start

Demo

Run on bundled example scenes. Backbone weights will be downloaded automatically to ./models/ on first run. The three examples (example1, example2, example3) showcase different M-to-N configurations: 6-to-1, 6-to-2, and 3-to-1.

# Run first example scene
python demo.py

# Run a specific scene
python demo.py --scene example2

# Run all 3 example scenes
python demo.py --all

Generated frames are saved to demo_output/<scene_name>/. Use --output_dir <path> to change the location.

Python API

from model import FrameCrafter
import numpy as np
from PIL import Image

# Load model (backbone weights auto-download to ./models/ on first run;
# pass base_model_dir=... to use a pre-downloaded location)
model = FrameCrafter("ckpt/framecrafter.safetensors")

# Prepare inputs -- M context images, M+N poses (M context + N target)
M = 6  # number of context views (flexible)
images = [Image.open(f"ctx_{i}.png") for i in range(M)]
data = np.load("scene.npz")
w2c_poses = data["w2c_poses"]    # (M+N, 4, 4) -- OpenCV w2c convention
intrinsics = data["intrinsics"]  # (M+N, 3, 3) -- at original image resolution

# Generate with automatic resize + center-crop
video = model.generate(
    images=images,
    w2c_poses=w2c_poses,
    intrinsics=intrinsics,
    height=480, width=832,
    resize_mode="crop",
)

# video[:M] are context frames, video[M:] are generated novel views
for i, novel_view in enumerate(video[M:]):
    novel_view.save(f"novel_view_{i}.png")

CLI

python infer.py \
    --images ctx_0.png ctx_1.png ctx_2.png ctx_3.png ctx_4.png ctx_5.png \
    --poses_npz scene.npz \
    --height 480 --width 832 \
    --resize_mode crop \
    --output_dir results/

Add --base_model_dir /path/to/backbone/weights if you have the backbone weights pre-downloaded somewhere other than ./models/.

Input Format

Poses (.npz)

The .npz file should contain:

Key	Shape	Description
w2c_poses	(M+N, 4, 4)	World-to-camera extrinsics in OpenCV convention
intrinsics	(M+N, 3, 3)	Camera intrinsic matrices at original image resolution

The first M entries correspond to the M context images (matching the order of --images), and the remaining N entries are target camera positions for novel view generation. Both M and N are flexible.

Camera Conventions

All camera poses are w2c (world-to-camera) in OpenCV convention:

• x-right, y-down, z-forward

Image Preprocessing

The model operates at a fixed resolution. Three modes are available:

1. resize_mode="crop" (recommended): Resize preserving aspect ratio, then center-crop to model resolution. Intrinsics are adjusted automatically.

2. resize_mode="stretch": Plain resize ignoring aspect ratio. Intrinsics are adjusted automatically.

3. resize_mode=None (default): No preprocessing. Provide images and intrinsics already at model resolution.

Intrinsics are automatically scaled based on the input image dimensions.

Tips

• VRAM: Set --vram_limit (in GB) to fit different GPUs. The model offloads weights to CPU when VRAM is limited. For example, use --vram_limit 22 for a 24 GB card.
• Speed: Reduce --num_inference_steps to speed up generation (default 50; 10–20 still gives reasonable quality).

Training

FrameCrafter is trained as a LoRA adapter (rank 32, q,k,v,o,ffn.0,ffn.2) on top of the Wan2.1-I2V-14B backbone. Training is done on DL3DV-10K (960P, 1K-scene subset).

# 1. Train at 192x336 (fixed 6-input -> 1-target views)
bash model_training/train_192_336_6to1.sh

# 2. Train at 480x832 (same 6-to-1 split, full inference resolution).
#    Uncomment the --resume_checkpoint line in the script to start from previous ckpt.
bash model_training/train_480_832_6to1.sh

LoRA weights are written to ./models/train/framecrafter-<run_name>/ as .safetensors, drop-in compatible with the FrameCrafter(...) loader used for inference.

Optional -- mixed M-to-N training. For better generalization to variable input/output view counts at inference time, optionally continue fine-tuning with a randomized input/target split (M ∈ [3, 9], N = 10 − M):

bash model_training/train_480_832_mixed.sh

Dataset layout

Each scene must be a directory in DL3DV-10K-960P format:

<dataset_base_path>/
├── <scene_a>/
│   ├── images_4/             # RGB frames, sorted alphabetically
│   └── transforms.json       # nerfstudio-style intrinsics + per-frame
│                             # c2w transform_matrix (OpenGL convention)
└── <scene_b>/
    └── ...

By default the scripts point at ../DL3DV-10K_960P/1K -- edit --dataset_base_path / --dataset_metadata_path in each script to match your layout.

Hardware & multi-GPU

All scripts launch via accelerate over 8 processes (bf16). The 192×336 run fits on 8× 48 GB GPUs (e.g. A6000s) thanks to DeepSpeed ZeRO-2, configured via the bundled model_training/my_config.yaml. The 480×832 runs require 8× 80 GB GPUs (e.g. H100s) and use vanilla accelerate launch -- no ZeRO sharding needed. Adjust num_processes in my_config.yaml (or via accelerate config) to match your node, and tune --gradient_accumulation_steps to keep the effective batch size constant on smaller setups. Wan2.1-I2V-14B backbone weights download automatically to ./models/ on first run (see Backbone Weights to relocate or switch source). Logs stream to wandb under project framecrafter; export WANDB_MODE=offline to disable network logging.

File Structure

FrameCrafter/
├── model.py              # FrameCrafter class: model loading + generation
├── infer.py              # CLI for inference
├── demo.py               # Demo script for bundled examples
├── camera_utils.py       # Plucker ray computation & pose normalisation
├── diffsynth/            # Core diffusion library (incl. training utilities)
├── model_training/       # LoRA training entry point + three-stage scripts
├── examples/             # Bundled demo scenes (inputs + poses)
├── prepare/              # Scripts for downloading checkpoints
├── pyproject.toml        # Package config (pip install -e .)
└── LICENSE               # Apache 2.0

License

This project is licensed under the Apache 2.0 License. See LICENSE for details.

Acknowledgement

We thank the authors of DiffSynth-Studio and Wan2.1 for releasing their code and pretrained models, which this project builds upon.

Citation

If you find this work useful, please consider citing:

@article{Wu2026framecrafter,
  title={Novel View Synthesis as Video Completion},
  author={Qi Wu and Khiem Vuong and Minsik Jeon and Srinivasa Narasimhan and Deva Ramanan},
  year={2026},
  journal={arXiv preprint arXiv:2604.08500},
}