A-TDOM: Active TDOM via On-the-Fly 3DGS

Yiwei Xu, Xiang Wang, Yifei Yu, Wentian Gan, Luca Morelli, Giulio Perda, Xiongwu Xiao, Xin Wang*, Zongqian Zhan, and Fabio Remondino

Important Notice (Please Read before Start!!!)

The Python environment configuration differs significantly between Windows and Linux in this project, and the supported functionalities also vary across the two platforms.

If you intend to run the project on Windows and execute it concurrently with On-the-Fly SfM, thereby enabling true simultaneous camera pose acquisition, 3DGS training, and TDOM generation, please refer to Windows Env Setting and Windows Run.

If you intend to run the project on Linux and only perform Simulation Experiments, please refer to Linux Env Setting and Linux Run.

We also provide a video demonstrating how to use On-the-Fly SfM and A-TDOM, you can download it from here.

Cloning the Repository

# SSH
git clone git@github.com:xywjohn/A-TDOM.git --recursive

or

# HTTPS
git clone https://github.com/xywjohn/A-TDOM --recursive

After that, please manually decompress the diff-gaussian-rasterization.zip, fused-ssim.zip and simple-knn.zip archive located under the submodules directory.

Windows Env Setting

If you are configuring the Python environment for A-TDOM on Windows, please do not use environment.yml. Instead, set up the environment using the following commands:

conda create -n ATDOM python=3.9.0
conda activate ATDOM
conda install cudatoolkit=11.6 -c conda-forge
conda install pytorch=1.12.1 torchvision=0.13.1 torchaudio=0.12.1 -c pytorch

pip install plyfile
pip install tqdm
pip install opencv-python
pip install lpips
pip install scikit-image

pip install submodules/diff-gaussian-rasterization
pip install submodules/simple-knn
pip install submodules/fused-ssim

Since PyTorch3D does not provide precompiled binaries for Windows, you need to manually download and build it from source. Please refer to the official repository: PyTorch3D GitHub. When compiling PyTorch3D, please pay close attention to the compatibility between CUB, PyTorch3D, and the version of Visual Studio. In this project, we use PyTorch3D v0.7.1 together with CUB v1.15.0, and Visual Studio 2019 or 2022 is recommended.

Please ensure that all prerequisites required to build PyTorch3D are properly installed, including compatible versions of CUB and Visual Studio, and that you have downloaded the correct version of the PyTorch3D source code.

Then, open either x64 Native Tools Command Prompt for VS 2019 or x64 Native Tools Command Prompt for VS 2022, and run the following commands:

conda activate ATDOM
cd ...\pytorch3d-0.7.1
python setup.py install

Windows Run

When running on Windows, we provide visualization support for TDOM generation, allowing the results of orthographic splatting from the current 3DGS field to be displayed during 3DGS progressive training. In addition, A-TDOM can be executed concurrently with On-the-Fly SfM to achieve true simultaneous camera pose acquisition, 3DGS training, and TDOM generation. Alternatively, A-TDOM can also be run after On-the-Fly SfM has finished exporting all intermediate results, enabling simulation experiments.

In this project, we provide executable versions of On-the-Fly SfM, which can be download here from Google Drive or BaiDuCloud, Extract Code=w7m3 If you would like to learn how to export the intermediate results, please refer to Data Preparation.

Before Run A-TDOM Concurrently With On-the-Fly SfM

If you would like to run A-TDOM concurrently with On-the-Fly SfM, you need to start On-the-Fly SfM first (please refer to Data Preparation for usage details), and then launch A-TDOM. If all image data are captured and stored in Image_Folder_Path and On-the-Fly SfM saves all SfM results in Source_Path_Dir, your directory structure should be organized as follows:

.../Image_Folder_Path
=> NULL (Waiting for new images)

.../Source_Path_Dir
=> GaussianSplatting
==> NULL (Waiting for SfM results from On-the Fly SfM) ...

Before Simulation Experiments On Windows

Simulation Experiments require running On-the-Fly SfM in advance to generate and export the intermediate results. If the intermediate results generated by On-the-Fly SfM are stored in Source_Path_Dir, and the image data are stored in Image_Folder_Path, your directory structure should be organized as follows:

.../Image_Folder_Path
=> 0001.jpg, 0002.jpg, 0003.jpg, ...

.../Source_Path_Dir
=> GaussianSplatting
==> 3
===> bin, ...
==> 4
===> bin, ...
...

3DGS Training and TDOM Generation On Windows

If you would like to output the results of 3DGS training to Model_Path_Dir, please run the following command to perform 3DGS training:

python Window-On-the-Fly-Train.py 
  --Source_Path_Dir .../your_path 
  --Model_Path_Dir .../your_path 
  --Image_Folder_Path .../your_path 
  --OriginImageHeight [int] 
  --OriginImageWidth [int] 
  --points_per_triangle [int] 
  --LogMaskThreshold [float] 
  --Use_Tri_Mask

If you would like to generate TDOM simultaneously during training, please run the following command:

python Window-On-the-Fly-Train.py 
  --Source_Path_Dir .../your_path 
  --Model_Path_Dir .../your_path 
  --Image_Folder_Path .../your_path 
  --OriginImageHeight [int] 
  --OriginImageWidth [int] 
  --points_per_triangle [int] 
  --LogMaskThreshold [float] 
  --show_TDOM
  --Use_Tri_Mask
  --render_TDOM 
  --GSD_x [float] 
  --GSD_y [float]

Necessary command Line Arguments for ContinuosProgressiveTrain4.py

--Source_Path_Dir {str}

Path to the source directory containing all camera pose information and sparse point cloud.

--Model_Path_Dir {str}

Path where the trained model should be stored.

--Image_Folder_Path {str}

Path to directory containing all image data.

--IterationFirstScene {int}

Training iterations for initial training phase.

--FinalOptimizationIterations {int}

Training iterations for final refinement phase.

--StartFromImageNo {int}

It indicates from which image the progressive training begins. This image and all the images before it are used for scene initialization.

--OriginImageHeight {int}

The original image height input into On-Fly SfM.

--OriginImageWidth {int}

The original image width input into On-Fly SfM.

--points_per_triangle {int}

The number of points collected in each triangle when conducting point sampling based on the Delunay triangulation.

--LogMaskThreshHold {float}

Gradient difference threshold used to determine whether new Gaussians should be added. A larger value results in fewer Gaussians being added, while a smaller value leads to more.

--show_TDOM

When enabled, the generated TDOM is continuously visualized during progressive 3DGS training.

--Use_Tri_Mask

When enabled, use the mask generated from Delaunay Triangulation for 3DGS training.

--NoDebug

When enabled, rendering quality evaluation is disabled during progressive 3DGS training.

--render_RGB

When enabled, the newly added image is rendered after completing the training stage for each new image.

--render_TDOM

When enabled, orthogonal splatting is performed on the current 3DGS field after each training stage, and a TDOM is generated.

--GetDemo

When enabled, a demo is generated based on the rendered results during training. This option must be used together with --render_RGB or --render_TDOM.

--GSD_x {float}

GSD of the generated TDOM along the Y-axis. If set to 0, the GSD will be automatically determined.

--GSD_y {float}

GSD of the generated TDOM along the Y-axis. If set to 0, the GSD will be automatically determined.

Example

We use the phantom3-ieu dataset (you can download it here) as an example. The following command demonstrates how to perform 3DGS training with simultaneous TDOM generation:

python Window-On-the-Fly-Train.py 
  --Source_Path_Dir .../phantom3-ieu 
  --Model_Path_Dir .../phantom3-ieu 
  --Image_Folder_Path .../images 
  --OriginImageHeight 1049 
  --OriginImageWidth 1870 
  --points_per_triangle 30 
  --LogMaskThreshold 0.15 
  --show_TDOM
  --Use_Tri_Mask 
  --render_TDOM 
  --GSD_x 0.00
  --GSD_y 0.00

Due to the absence of GCPs, On-the-Fly SfM performs camera pose estimation and sparse point cloud reconstruction in a relative coordinate system. As a result, the scale of the coordinate system may vary across different runs of 3DGS training and TDOM generation.

Therefore, we temporarily set --GSD_x and --GSD_y to 0, allowing the system to automatically determine the appropriate GSD for each rendering based on the current scene scale.

Linux Env Setting

If you are configuring the Python environment for A-TDOM on Linux, please use the following command directly:

# Env
cd .../A-TDOM
conda env create --file environment.yml
conda activate ATDOM

pip install pytorch3d
pip install opencv-python
pip install lpips
pip install scikit-image

Linux Run

Since On-the-Fly SfM is not currently available on Linux, running A-TDOM on Linux requires first executing On-the-Fly SfM on Windows and saving the intermediate results. These results should then be transferred to the Linux environment to proceed with subsequent 3DGS training and TDOM generation.

In this project, we provide executable versions of On-the-Fly SfM, which can be download here from Google Drive or BaiDuCloud, Extract Code=w7m3 If you would like to learn how to export the intermediate results, please refer to Data Preparation.

Before Run On Linux

If the intermediate results generated by On-the-Fly SfM are stored in Source_Path_Dir, and the image data are stored in Image_Folder_Path, your directory structure should be organized as follows:

.../Image_Folder_Path
=> 0001.jpg, 0002.jpg, 0003.jpg, ...

.../Source_Path_Dir
=> GaussianSplatting
==> 3
===> bin, ...
==> 4
===> bin, ...
...

3DGS Training and TDOM Generation On Linux

If you would like to output the results of 3DGS training to Model_Path_Dir, please run the following command to perform 3DGS training:

python ContinuosProgressiveTrain4.py 
  --Source_Path_Dir .../your_path 
  --Model_Path_Dir .../your_path 
  --Image_Folder_Path .../your_path 
  --OriginImageHeight [int] 
  --OriginImageWidth [int] 
  --points_per_triangle [int] 
  --LogMaskThreshold [float] 
  --Use_Tri_Mask

If you would like to generate TDOM simultaneously during training, please run the following command:

python ContinuosProgressiveTrain4.py 
  --Source_Path_Dir .../your_path 
  --Model_Path_Dir .../your_path 
  --Image_Folder_Path .../your_path 
  --OriginImageHeight [int] 
  --OriginImageWidth [int] 
  --points_per_triangle [int] 
  --LogMaskThreshold [float] 
  --Use_Tri_Mask
  --render_TDOM 
  --GSD_x [float] 
  --GSD_y [float]

Necessary command Line Arguments for ContinuosProgressiveTrain4.py

--Source_Path_Dir {str}

Path to the source directory containing all camera pose information and sparse point cloud.

--Model_Path_Dir {str}

Path where the trained model should be stored.

--Image_Folder_Path {str}

Path to directory containing all image data.

--IterationFirstScene {int}

Training iterations for initial training phase.

--FinalOptimizationIterations {int}

Training iterations for final refinement phase.

--StartFromImageNo {int}

It indicates from which image the progressive training begins. This image and all the images before it are used for scene initialization.

--OriginImageHeight {int}

The original image height input into On-Fly SfM.

--OriginImageWidth {int}

The original image width input into On-Fly SfM.

--points_per_triangle {int}

The number of points collected in each triangle when conducting point sampling based on the Delunay triangulation.

--LogMaskThreshHold {float}

Gradient difference threshold used to determine whether new Gaussians should be added. A larger value results in fewer Gaussians being added, while a smaller value leads to more.

--NoDebug

When enabled, rendering quality evaluation is disabled during progressive 3DGS training.

--Use_Tri_Mask

When enabled, use the mask generated from Delaunay Triangulation for 3DGS training.

--render_RGB

When enabled, the newly added image is rendered after completing the training stage for each new image.

--render_TDOM

When enabled, orthogonal splatting is performed on the current 3DGS field after each training stage, and a TDOM is generated.

--GetDemo

When enabled, a demo is generated based on the rendered results during training. This option must be used together with --render_RGB or --render_TDOM.

--GSD_x {float}

GSD of the generated TDOM along the Y-axis. If set to 0, the GSD will be automatically determined.

--GSD_y {float}

GSD of the generated TDOM along the Y-axis. If set to 0, the GSD will be automatically determined.

Example

We use the phantom3-ieu dataset (you can download it here) as an example. The following command demonstrates how to perform 3DGS training with simultaneous TDOM generation:

python ContinuosProgressiveTrain4.py 
  --Source_Path_Dir .../phantom3-ieu 
  --Model_Path_Dir .../phantom3-ieu 
  --Image_Folder_Path .../images 
  --OriginImageHeight 1049 
  --OriginImageWidth 1870 
  --points_per_triangle 30 
  --LogMaskThreshold 0.15 
  --Use_Tri_Mask 
  --render_TDOM 
  --GSD_x 0.00
  --GSD_y 0.00

Due to the absence of GCPs, On-the-Fly SfM performs camera pose estimation and sparse point cloud reconstruction in a relative coordinate system. As a result, the scale of the coordinate system may vary across different runs of 3DGS training and TDOM generation.

Therefore, we temporarily set --GSD_x and --GSD_y to 0, allowing the system to automatically determine the appropriate GSD for each rendering based on the current scene scale.

Data Preparation

On-the-Fly SfM is a semi real-time Structure-from-Motion (SfM) system that currently supports only Windows. It can process data while images are being captured, as long as the acquired images can be transmitted to the computer in time. You can use the mobile image transmission app we provide for this purpose: https://sygant.github.io/onthefly/SfMClienten.html

At the same time, On-the-Fly SfM also supports processing previously collected datasets. Both modes use exactly the same workflow within On-the-Fly SfM. You can download the On-the-Fly SfM here from Google Drive or BaiDuCloud, Extract Code=w7m3

After extracting the downloaded archive, before run, you have to open the config.ini and modify the PythonHome, PythonWorkspace, and GlobalFeatureModel paths (absolute path) to the corresponding locations on your computer.

All of these paths should point to other folders located in the same directory as config.ini, as shown below:

[Python]
# Python 虚拟环境目录
# Python virtual environment directory
PythonHome = ...\On-the-FlyGS\Python

# Python 脚本工作目录
# Working directory for Python scripts
PythonWorkspace = ...\On-the-FlyGS\pyscripts

# 全局特征模型路径
# Path to the global feature model
GlobalFeatureModel = ...\On-the-FlyGS\pyscripts

Then, locate and run On-the-FlySfM.exe in the bin folder. After the interface opens, please wait until the following message appears in the Log panel at the bottom before proceeding with the subsequent operations:

Initializing PyTorch environment...
Python environment initialize success!
Python version: 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]
The global extractor has been successfully loaded!
The local extractor has been successfully loaded!
The similar matcher has been successfully loaded!

Next, click Project -> New Project in the top-left corner. For Images, directly select the folder that contains the input images. For Database, choose a directory where you want to store the SfM output results.

Note that the paths specified here must be consistent with the --Image_Folder_Path and --Source_Path_Dir parameters set in Windows Run or Linux Run. As shown below:

Database = ...\output\scene1                   # Same as --Source_Path_Dir
Images = ...\dataset\scene1\images             # Same as --Image_Folder_Path

--Source_Path_Dir ...\output\scene1            # Same as Database
--Image_Folder_Path ...\dataset\scene1\images  # Same as Images

Now, please start On-the-Fly SfM. If you want to run A-TDOM Concurrently With On-the-Fly SfM, please proceed to Before Run A-TDOM Concurrently With On-the-Fly SfM + 3DGS Training and TDOM Generation On Windows to continue with the next steps.

If you intend to conduct simulation experiments, please wait until On-the-Fly SfM has completed full point cloud generation and camera pose estimation, and proceed to Before Simulation Experiments On Windows + 3DGS Training and TDOM Generation On Windows or Before Run On Linux + 3DGS Training and TDOM Generation On Linux to continue with the next steps.

When you need to close On-the-FlySfM.exe, it is recommended to use Taskmgr.exe to completely close it to prevent some threads from continuing to run in the background.

BibTex

@article{ATDOM,
  title={A-TDOM: Active TDOM via On-the-Fly 3DGS},
  author={Xu, Yiwei and Wang, Xiang and Yu, Yifei and Gan, Wentian and Morelli, Luca and Perda, Giulio and Xiao, Xiongwu and Zhan, Zongqian and Wang, Xin and Remondino, Fabio},
  journal={arXiv preprint arXiv:2509.12759},
  year={2025}
}

Name		Name	Last commit message	Last commit date
Latest commit History 77 Commits
On-The-Fly		On-The-Fly
arguments		arguments
assets		assets
demo_data		demo_data
gaussian_renderer		gaussian_renderer
lpipsPyTorch		lpipsPyTorch
scene		scene
submodules		submodules
utils		utils
0.png		0.png
ContinuosProgressiveTrain4.py		ContinuosProgressiveTrain4.py
DatasetPrepare.py		DatasetPrepare.py
DownSample.py		DownSample.py
GetDemo.py		GetDemo.py
README.md		README.md
Test.py		Test.py
Window-On-the-Fly-Train.py		Window-On-the-Fly-Train.py
WindowsTrain.py		WindowsTrain.py
convert.py		convert.py
environment.yml		environment.yml
full_eval.py		full_eval.py
metrics.py		metrics.py
render.py		render.py

Folders and files

Latest commit

History

Repository files navigation

A-TDOM: Active TDOM via On-the-Fly 3DGS

Important Notice (Please Read before Start!!!)

Cloning the Repository

Windows Env Setting

Windows Run

Before Run A-TDOM Concurrently With On-the-Fly SfM

Before Simulation Experiments On Windows

3DGS Training and TDOM Generation On Windows

--Source_Path_Dir {str}

--Model_Path_Dir {str}

--Image_Folder_Path {str}

--IterationFirstScene {int}

--FinalOptimizationIterations {int}

--StartFromImageNo {int}

--OriginImageHeight {int}

--OriginImageWidth {int}

--points_per_triangle {int}

--LogMaskThreshHold {float}

--show_TDOM

--Use_Tri_Mask

--NoDebug

--render_RGB

--render_TDOM

--GetDemo

--GSD_x {float}

--GSD_y {float}

Example

Linux Env Setting

Linux Run

Before Run On Linux

3DGS Training and TDOM Generation On Linux

--Source_Path_Dir {str}

--Model_Path_Dir {str}

--Image_Folder_Path {str}

--IterationFirstScene {int}

--FinalOptimizationIterations {int}

--StartFromImageNo {int}

--OriginImageHeight {int}

--OriginImageWidth {int}

--points_per_triangle {int}

--LogMaskThreshHold {float}

--NoDebug

--Use_Tri_Mask

--render_RGB

--render_TDOM

--GetDemo

--GSD_x {float}

--GSD_y {float}

Example

Data Preparation

BibTex

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 1

Languages

Packages