Potential MegaDepth split leakage in training? Request for results under the standard train split 关于 MegaDepth 训练划分可能存在泄露的问题，以及希望补充标准训练划分下的结果

[supremepig]

Hi, thanks for open-sourcing this great work. I really enjoyed reading the paper and code.

I noticed a possible issue regarding the MegaDepth training split. It seems that the training code directly loads all .npz files under data/megadepth/megadepth_indices/scene_info_0.1_0.7, instead of using the standard train_list.txt split as in prior works such as LoFTR.

If I understand this correctly, this may cause overlap between the training data and the MegaDepth-1500 evaluation scenes, which would introduce data leakage.

I may be mistaken, so I would appreciate your clarification:

Does RDD training use all files under scene_info_0.1_0.7, or only the standard training subset?
If all files are used, are MegaDepth-1500 scenes included during training?
Could you please report the results of v1 / v2 / v3 under the standard non-leaking MegaDepth split?

Thanks again for the excellent work.

你好，感谢开源这项很棒的工作，我很喜欢这篇论文和代码实现。

我注意到一个可能和 MegaDepth 训练划分相关的问题：当前训练代码似乎直接读取了
data/megadepth/megadepth_indices/scene_info_0.1_0.7
下的全部 .npz 文件，而不是像 LoFTR 等以往方法那样使用标准的 train_list.txt 训练划分。

如果我的理解没错，这可能会导致训练数据和 MegaDepth-1500 测试场景发生重叠，从而引入数据泄露。

当然也可能是我理解有误，因此想请教作者澄清一下：

RDD 训练时到底是使用了 scene_info_0.1_0.7 下的全部文件，还是只使用标准训练子集？
如果用了全部文件，MegaDepth-1500 的场景是否已经在训练中出现过？
是否可以补充给出在标准、无泄露划分下 v1 / v2 / v3 的结果？

再次感谢你们的出色工作。

[xtcpete]

Yes, I think you are right that there is a leakage. Thanks for pointing it out. I had incorrectly assumed that those files were excluded from scene_info_0.1_0.7. I will update the code and try to get some updated metrics.

[xtcpete]

A quick update, here is the new metrics for v3 after fixing the leakage. I will update the code and paper once I have run through all experiments.

	MegaDepth-1500			MegaDepth-View			Air-to-Ground
	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°
Before	53.6	69.3	81.3	56.9	71.9	82.3	56.3	71.2	81.7
After	52.0	67.8	80.0	52.3	67.1	78.1	56.0	71.0	81.4

[supremepig]

Thanks for the update. I noticed another issue that may also affect the reported results.

In the v3 branch benchmark scripts, failed pose estimation cases seem to set e_t, e_R = 0, 0 rather than inf. If so, failed cases would be counted as zero error, which would artificially increase the AUC. So I think the v3 results may need to be recomputed after fixing this evaluation issue.

I also wanted to ask for clarification on training: were v2 and v3 trained with any Air-to-Ground data? I am asking because Air-to-Ground is usually considered a harder benchmark, yet the reported AUC there is higher than on MegaDepth.

Could you please clarify this, and if needed, re-run the evaluation with the corrected benchmark scripts?

Thanks again for your work.
感谢更新。我又注意到一个可能同样会影响结果的问题。

在 v3 分支的 benchmark 脚本里，位姿估计失败的样本似乎被设成了 e_t, e_R = 0, 0，而不是 inf。如果是这样，那么失败样本就会被当成零误差，从而人为抬高 AUC。因此我觉得 v3 的结果可能也需要在修正这个评测问题之后重新计算一次。

另外我还想确认一下训练设定：v2 和 v3 是否使用了任何 Air-to-Ground 数据进行训练？我之所以这样问，是因为 Air-to-Ground 通常被认为是更难的 benchmark，但目前报告的 AUC 却高于 MegaDepth，这一点让我有些疑惑。

能否请你澄清一下这一点，并在必要时用修正后的 benchmark 脚本重新测试一次？

再次感谢你们的工作。

[xtcpete]

V3 is trained on a combination of the MegaDepth and Air-to-Ground training datasets. After rerunning the benchmarks:

	MegaDepth-1500			MegaDepth-View			Air-to-Ground
	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°
With Air-to-Ground	51.6	67.4	79.6	52.0	66.5	77.8	56.2	70.9	81.0
MegaDepth Only	51.9	68.2	80.6	48.5	63.0	74.3	44.7	59.1	70.5

I hope this clarify things up.

[supremepig]

Thanks for the clarification and for re-running the benchmarks.

This helps clear things up a lot. I am also very interested in the MegaDepth-only setting, and I look forward to seeing the v2 and v3 MegaDepth-only checkpoints released soon. I think they would be very helpful for fair comparison and reproducibility.

Thanks again for your work.
感谢澄清，也感谢重新测试 benchmark。

这些说明让我清楚了很多。我也很关心 MegaDepth-only 的设定，期待能够尽快看到 只用 MegaDepth 训练的 v2 和 v3 权重 发布出来。我觉得这对于公平比较和复现都会很有帮助。

再次感谢你们的工作。

[stz-source]

Hi , thank you for sharing the V3 MegaDepth-only numbers.

I have one additional clarification request: would it be possible to also report the same ablation for the V1 model (CVPR paper verison )?

Specifically, for V1, could you share AUC @5°/@10°/@20° on:

MegaDepth-1500
MegaDepth-View
Air-to-Ground
for both settings:

with Air-to-Ground training data
MegaDepth-only training
I am asking because this would help separate architecture/version gains from domain-data gains more clearly.

Thanks again for your time and for the very helpful responses.

[xtcpete]

Yes, working on it. This could take some time

[xtcpete]

I have retrained the v1 with the updated training code. After rerunning the benchmarks, the results I got has better performance compared to previous number. I think the usage of lightening stabilizes the training and recollected air-to-ground dataset has better quality. I should be able to update the code and the weights this week.

	MegaDepth-1500			MegaDepth-View			Air-to-Ground
	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°	AUC 5°	AUC 10°	AUC 20°
With Air-to-Ground	48.6	64.7	77.2	38.3	51.3	62.3	49.8	64.7	75.8
MegaDepth Only	48.5	64.6	77.2	37.1	50.2	61.1	40.5	54.6	65.9

[xtcpete]

I have updated the code and released new weights. Please let me know if there are any other problems.