Some details are missing in the paper

[sangaram]

Dear authors, thank you for your great work and congratulations for the publication at CVPR 2025 ! After reading your paper, I feel like some details are still missing.

First, about your data curation pipeline, you explain that the optical flow score is used to filter videos (mostly static ones I guess) without giving details on how this score is computed. The given reference [1] don't explicitly mention it either. Can you give more details on that and also explain the threshold of 3.

Secondly, you explain that you use camera motion detection and motion segmentation [2] for filtering out videos with significant camera motion. Can you give more details on how do you do this ? For instance how you get the center points of each object, and also elaborate a little bit more on the given thresholds ?

Finally, In order to evaluate the trajectory controllability, you explain that you use the average L1 distance between predefined and generated trajectories. How do you extract the trajectories of the generated videos ? Formally, how to express the TrajError metric ?

Having more details on these points would be much appreciated, and again congratulations for your great work ! I might be wrong on some remarks, my apologies if that's the case.

References

[1] Xu et. al, Unifying flow, stereo and depth estimation, IEEE TPAMI 2023
[2] Zhao et. al, ParticleSfM: Exploiting dense point trajectories for localizing moving cameras in the wild, ECCV 2022

[antragoudaras]

Great observations @sangaram, I was also wondering on the filtering pipeline which seems to not be released in the codebase repository. Maybe @zhenghao977 or @Leojc can elaborate on it and if there are plans for releasing the pre-processing related scripts. If that is not the case it would nice to have access to the filtered data used for fine-tuning. However if I am not mistaken since part of the dataset is propitiatory, I think I read in #4 that they are no plans to be released. Yet a version containing the filtered data used for fine-tuning would be of great convenience

[zhenghao977]

@sangaram @antragoudaras
1)Most of our processing can refer to this repo for calculating optical flow scores. The score you use for filtering depends on your dataset and the FPS you’re using. You can pick a good value by checking the stats from your training.
2) We used motion segmentation to define video backgrounds and only used background pixels for camera motion detection. But now with VLMs improving, there are open-source tools like SkyCaptioner-V1 that use human-labeled data to detect camera motion more effectively, you can try it in your work.
3)The trajectories of the generated videos are also predicted using RAFT.

[sangaram]

Thank you for these answers @zhenghao977, I would like to have some further clarifications.

In your paper, it is said:

We train Tora for 2 epochs with dense optical flow and fine-tune for 1 epoch with sparse trajectories. The maximum number of sampling trajectories N is set to 16.

From what I understood, you use the method of [2] to get the trajectories of foreground and background pixels, those of the background are used to estimate camera poses. These poses are then used to filter out videos with significant background motion as explained in your paper.

The detected camera movement angles, calculated based on the background from the motion segmentation results, are valid as follows: [0° , 20°], [160° , 200°], [340°, 360°].

Are these the valid x, y, z camera rotation angles for your videos ? Moreover I guess you use the trajectories of your foreground pixels as the control trajectories. Then do you randomly sample between 1 and 16 among those in the second training stage ? How do you make sure those trajectories belong to each object ?

[antragoudaras]

Thank you for these answers @zhenghao977, I would like to have some further clarifications.

In your paper, it is said:

We train Tora for 2 epochs with dense optical flow and fine-tune for 1 epoch with sparse trajectories. The maximum number of sampling trajectories N is set to 16.

From what I understood, you use the method of [2] to get the trajectories of foreground and background pixels, those of the background are used to estimate camera poses. These poses are then used to filter out videos with significant background motion as explained in your paper.

The detected camera movement angles, calculated based on the background from the motion segmentation results, are valid as follows: [0° , 20°], [160° , 200°], [340°, 360°].

Are these the valid x, y, z camera rotation angles for your videos ? Moreover I guess you use the trajectories of your foreground pixels as the control trajectories. Then do you randomly sample between 1 and 16 among those in the second training stage ? How do you make sure those trajectories belong to each object ?

@sangaram Looking at the implementation I believe that for the second stage fine-tuning on sparse trajectories, the flow/trajectories are still predicted with the RAFT model. Look at this script https://github.com/alibaba/Tora/blob/main/sat/train_video.py#L124-L126. The sampling happening inside the get_keypoint method takes as an argument the first flow (first_flow)

[sangaram]

Thank you @antragoudaras for this clarification, so they are randomly initializing a set of tracking points and warping them with the RAFT-computed optical flow in order to get the sparse flow. Let's wait for the answer of @zhenghao977 on the question on camera motion filtering.