Discussion about the VGGT-SLAM result

[PR5GRAMM2R]

I run VGGT-SLAM with the dataset TUM.

And I got the following result.

Run,Dataset,RMSE
1,rgbd_dataset_freiburg1_360,0.051752
1,rgbd_dataset_freiburg1_desk,0.026684
1,rgbd_dataset_freiburg1_desk2,0.056097
1,rgbd_dataset_freiburg1_floor,0.144438
1,rgbd_dataset_freiburg1_plant,0.022208
1,rgbd_dataset_freiburg1_room,0.123293
1,rgbd_dataset_freiburg1_rpy,0.038118
1,rgbd_dataset_freiburg1_teddy,0.043309
1,rgbd_dataset_freiburg1_xyz,0.013879
1,Average,.05775311111111111111
2,rgbd_dataset_freiburg1_360,0.058878
2,rgbd_dataset_freiburg1_desk,0.027564
2,rgbd_dataset_freiburg1_desk2,0.032104
2,rgbd_dataset_freiburg1_floor,0.359750
2,rgbd_dataset_freiburg1_plant,0.024654
2,rgbd_dataset_freiburg1_room,0.135679
2,rgbd_dataset_freiburg1_rpy,0.030353
2,rgbd_dataset_freiburg1_teddy,0.038054
2,rgbd_dataset_freiburg1_xyz,0.018845
2,Average,.08065344444444444444
3,rgbd_dataset_freiburg1_360,0.090233
3,rgbd_dataset_freiburg1_desk,0.024864
3,rgbd_dataset_freiburg1_desk2,0.069978
3,rgbd_dataset_freiburg1_floor,0.146341
3,rgbd_dataset_freiburg1_plant,0.020198
3,rgbd_dataset_freiburg1_room,0.168063
3,rgbd_dataset_freiburg1_rpy,0.030523
3,rgbd_dataset_freiburg1_teddy,0.040171
3,rgbd_dataset_freiburg1_xyz,0.014934
3,Average,.06725611111111111111
4,rgbd_dataset_freiburg1_360,0.050741
4,rgbd_dataset_freiburg1_desk,0.033998
4,rgbd_dataset_freiburg1_desk2,0.051289
4,rgbd_dataset_freiburg1_floor,0.171805
4,rgbd_dataset_freiburg1_plant,0.022991
4,rgbd_dataset_freiburg1_room,0.142180
4,rgbd_dataset_freiburg1_rpy,0.037055
4,rgbd_dataset_freiburg1_teddy,0.043363
4,rgbd_dataset_freiburg1_xyz,0.013270
4,Average,.06296577777777777777
5,rgbd_dataset_freiburg1_360,0.070748
5,rgbd_dataset_freiburg1_desk,0.034087
5,rgbd_dataset_freiburg1_desk2,0.039767
5,rgbd_dataset_freiburg1_floor,0.099062
5,rgbd_dataset_freiburg1_plant,0.022626
5,rgbd_dataset_freiburg1_room,0.105492
5,rgbd_dataset_freiburg1_rpy,0.036673
5,rgbd_dataset_freiburg1_teddy,0.037590
5,rgbd_dataset_freiburg1_xyz,0.013504
5,Average,.05106100000000000000

From this results, I found that eval_tum evalustion file run VGGT-SLAM 5 times.

I want to know why it runs more than 1 time.

And, I checked VGGT-SLAM paper. I found your TUM results in 8th page, but I wonder how that results came from.

Thanks for reading my issue.

[Dominic101]

From the paper: "Due to potential randomness in our approach caused by RANSAC, we report the average performance over five runs, which have a low spread (small standard deviation) as shown in Sec. 5.5."

The README provides information as well on how to change this to run a different number of trials here: https://github.com/MIT-SPARK/VGGT-SLAM?tab=readme-ov-file#running-evaluations

[PR5GRAMM2R]

Thanks for reply. I understood.

[PR5GRAMM2R]

Hello, I am here to ask you why the paper's result is different compare with the result that I got after running [ TUM Dataset , Sim(3) , w = 32 ] . I executed the code as posted on GitHub without changing the parameters.

Below is what I executed using eval_tum.sh .

python main.py --image_folder "$dataset_name" --use_sim3 --max_loops 1 --min_disparity 50 --conf_threshold 25 --submap_size "$submap_size" --log_results --skip_dense_log --log_path "$(pwd)/logs/logs_tum/${dataset}_run${run}_w${submap_size}.txt"

I got these result.

Run,Dataset,RMSE
1,rgbd_dataset_freiburg1_360,0.047326
1,rgbd_dataset_freiburg1_desk,0.023354
1,rgbd_dataset_freiburg1_desk2,0.031112
1,rgbd_dataset_freiburg1_floor,0.100596
1,rgbd_dataset_freiburg1_plant,0.023657
1,rgbd_dataset_freiburg1_room,0.062659
1,rgbd_dataset_freiburg1_rpy,0.026040
1,rgbd_dataset_freiburg1_teddy,0.033972
1,rgbd_dataset_freiburg1_xyz,0.014127
1,Average,.04031588888888888888
2,rgbd_dataset_freiburg1_360,0.047326
2,rgbd_dataset_freiburg1_desk,0.023354
2,rgbd_dataset_freiburg1_desk2,0.031112
2,rgbd_dataset_freiburg1_floor,0.100596
2,rgbd_dataset_freiburg1_plant,0.023657
2,rgbd_dataset_freiburg1_room,0.062659
2,rgbd_dataset_freiburg1_rpy,0.026040
2,rgbd_dataset_freiburg1_teddy,0.033972
2,rgbd_dataset_freiburg1_xyz,0.014127
2,Average,.04031588888888888888
3,rgbd_dataset_freiburg1_360,0.047326
3,rgbd_dataset_freiburg1_desk,0.023354
3,rgbd_dataset_freiburg1_desk2,0.031112
3,rgbd_dataset_freiburg1_floor,0.100596
3,rgbd_dataset_freiburg1_plant,0.023657
3,rgbd_dataset_freiburg1_room,0.062659
3,rgbd_dataset_freiburg1_rpy,0.026040
3,rgbd_dataset_freiburg1_teddy,0.033972
3,rgbd_dataset_freiburg1_xyz,0.014127
3,Average,.04031588888888888888
4,rgbd_dataset_freiburg1_360,0.047326
4,rgbd_dataset_freiburg1_desk,0.023354
4,rgbd_dataset_freiburg1_desk2,0.031112
4,rgbd_dataset_freiburg1_floor,0.100596
4,rgbd_dataset_freiburg1_plant,0.023657
4,rgbd_dataset_freiburg1_room,0.062659
4,rgbd_dataset_freiburg1_rpy,0.026040
4,rgbd_dataset_freiburg1_teddy,0.033972
4,rgbd_dataset_freiburg1_xyz,0.014127
4,Average,.04031588888888888888
5,rgbd_dataset_freiburg1_360,0.047326
5,rgbd_dataset_freiburg1_desk,0.023354
5,rgbd_dataset_freiburg1_desk2,0.031112
5,rgbd_dataset_freiburg1_floor,0.100596
5,rgbd_dataset_freiburg1_plant,0.023657
5,rgbd_dataset_freiburg1_room,0.062659
5,rgbd_dataset_freiburg1_rpy,0.026040
5,rgbd_dataset_freiburg1_teddy,0.033972
5,rgbd_dataset_freiburg1_xyz,0.014127
5,Average,.04031588888888888888

Of course, Optimization using Sim(3) doesn't use RANSAC, so the result is identical at each runs. But, in the paper, [ TUM Dataset , Sim(3) , w = 32 ] result is 0.123 0.040 0.055 0.254 0.022 0.088 0.041 0.032 0.016 and the average is 0.074 .

I wonder why this difference happens. Thank you for your concern.