ControlFix3D

ControlFix3D: Improving Sparse-View 3D Gaussian Splatting with Pose-Aware Diffusion Models

Overview

Sparse-view 3D Gaussian Splatting often produces incomplete geometry and artifacts at unseen viewpoints. ControlFix3D uses a frozen diffusion model, ControlFix, to repair novel-view renderings and distill the corrected views back into the 3D representation.

ControlFix is conditioned on two observed reference images and their relative camera poses. The full pipeline supports reconstruction from raw photographs, 3DGS refinement, and novel-view rendering.

Results

Video

View the high-quality MP4

ControlFix image repair

Setup

The project was developed on Linux with an NVIDIA GPU and CUDA 12.8.

git clone --recursive https://github.com/Bochidaru/ControlFix3D.git
cd ControlFix3D

conda env create -f environment.yml
conda activate controlfix3d

If you cloned the repository without submodules:

git submodule update --init --recursive

Place the ControlFix checkpoint at:

ControlFix/checkpoints/control_sd15_3dgs.ckpt

You can also provide a custom checkpoint with --cnfix_ckpt.

Quickstart

Reconstruct a scene from photographs

Put your JPG or PNG images in a folder and run:

python infer.py \
    --images path/to/images \
    --out outputs/my_scene

The pipeline will:

1. Estimate cameras and initial geometry with VGGT.
2. Train the sparse-view 3D Gaussian representation.
3. Refine it using ControlFix.
4. Export the point clouds and novel-view videos.

Main outputs:

outputs/my_scene/
├── 3dgs_15000.ply
├── controlfix3d_24600.ply
├── orbit_3dgs.mp4
├── orbit_controlfix3d.mp4
├── model/
└── scene/

Use a specific number of input views with:

python infer.py -i path/to/images -o outputs/my_scene --n_support 3

To reuse an existing reconstruction:

python infer.py -i path/to/images -o outputs/my_scene --skip_vggt --skip_train

Train on a prepared COLMAP scene

python train.py \
    -s path/to/scene \
    -m outputs/my_scene \
    --n_support 9

Render the final model:

python render.py \
    -s path/to/scene \
    -m outputs/my_scene \
    --iteration -1 \
    --fixer cnfix \
    --n_support 9

Evaluate the reconstruction:

python metrics.py \
    -s path/to/scene \
    -m outputs/my_scene \
    --n_support 9

Web demo

A Streamlit interface is included for uploading photographs and viewing the reconstruction results.

pip install streamlit==1.62.0
streamlit run app.py --server.port 8531

Then open http://localhost:8531.

Acknowledgements

This project builds on:

• 3D Gaussian Splatting
• ControlNet
• VGGT
• DiFix3D

License

See LICENSE.md for details. Third-party components and model weights are subject to their respective licenses.