Is it possible to project 3d points back onto the 2d images?

[jpgallegoar]

Let's say I place 3d markers on the 3d splat, how can I project it back onto the original images accurately?

[brandonsmartdev]

Hello @jpgallegoar,

Because we do not directly predict or regress camera poses, we currently don't provide a fool-proof method for doing this. The images are in the coordinate system of the first image, so if you wish to project onto the first image and know the first image's camera intrinsics, you should be able to reliably achieve this. Otherwise, you will likely need to regress the camera poses from point cloud predictions. We base our model on MASt3R, and the official MASt3R repo contains examples of how to regress camera poses and intrinsics from the predicted pointsmaps. For example, you can look at the demo, which I believe uses this regression code from DUSt3R, but you may need to verify this.

Like MASt3R/DUSt3R, we expose MASt3R's point cloud predictions in pred1['pts3d'] and pred2['pts3d_in_other_view'], so you should be able to directly use their existing code to predict the camera poses using our predictions. Once you have the estimated camera poses/intrinsics, you should be able to project 3D markers onto the original images using regular 3D->2D projection.