How to perform depth integration with the predicted deformation

[BaldrLector]

Hi, @pablorpalafox @AljazBozic ,
thanks for your kindly sharing, I learn so much from this source code, and I love it!

I have some questions:

1. However, it seems the depth integration is not implemented at this repository.
   Could you shed some light on how to perform depth integration, as the gif presented?
   Is there any code for reference?

2. I plan to perform non-rigid reconstruction form RGBD images with this method. Now i can get rgb image and depth image, how should I prepare other needed data (the graph nodes, edges, graph_edges_weights, graph_clusters, pixel_anchors and pixel_weights ) ?

Thank you in advance, your reply will vary appreciated!

[pablopalafox]

Hi @BaldrLector, thanks for your interest in the project!

Regarding 1., unfortunately we didn't have the time to properly clean up the part of the code for depth integration. But it's basically an implementation of DynamicFusion.

As to how to get the other needed data, you can check out the Data section in the readme. You should first download DeepDeform by following the instructions in that repo. Then use the link we provide in the readme to get the additional data. You should then merge both datasets, such that within a given sequence folder (see Data Organization of DeepDeform) you have all the data from DeepDeform plus the graph nodes, edges, etc.

Hopefully this clarifies it a bit more. Thanks for pointing it out.

[BaldrLector]

Hi @BaldrLector, thanks for your interest in the project!

Regarding 1., unfortunately we didn't have the time to properly clean up the part of the code for depth integration. But it's basically an implementation of DynamicFusion.

As to how to get the other needed data, you can check out the Data section in the readme. You should first download DeepDeform by following the instructions in that repo. Then use the link we provide in the readme to get the additional data. You should then merge both datasets, such that within a given sequence folder (see Data Organization of DeepDeform) you have all the data from DeepDeform plus the graph nodes, edges, etc.

Hopefully this clarifies it a bit more. Thanks for pointing it out.

1. Got it, thanks, your reply is very kind, and I appreciate it. I will try to implement the dynamic fusion with the predicted deformation. By the way, will you plan to release this part code in the future?

2. Sorry for the confusing description, I mean if I want to utilize new data ( capture by my self with Kinect azure, there are only RGB and depth images), how should I do to get graph nodes, edges, graph_edges_weights, graph_clusters, pixel_anchors and pixel_weights?

[pablopalafox]

Oh, got you. We are planning to clean up the code for generating this part of the dataset. It wasn't a priority, since we also released the corresponding graph data for DeepDeform, but we'll keep it mind now that you mentioned it. Essentially you need to sample nodes in the source depth map. Then compute your graph edges using geodesic connectivity and find your anchor nodes for each pixel, as well as their weights. We'll try to release the scripts for this relatively soon. Thanks for letting us know.

[BaldrLector]

Oh, got you. We are planning to clean up the code for generating this part of the dataset. It wasn't a priority, since we also released the corresponding graph data for DeepDeform, but we'll keep it mind now that you mentioned it. Essentially you need to sample nodes in the source depth map. Then compute your graph edges using geodesic connectivity and find your anchor nodes for each pixel, as well as their weights. We'll try to release the scripts for this relatively soon. Thanks for letting us know.

Got it, thanks for your responsive reply!
I will close this issue now.
And if there is any update, I will update it under this issue.

[BaldrLector]

@pablorpalafox
Hi, Here to me again. After reading the referenced paper, I am writing code to reconstruct meshes with predicted deformation using Projective TSDF. If I am right, the core I need to do is:

1. build canonical TSDF volume(weight, tsdf value, and color), record each voxel center coordinate
2. Init TSDF with first RGBD frame( in this resp, it is denoted as source) with Camera parameters(fx, fy, cx, cy).
3. for next all RGBD frames, we warp each voxel center to the live frame with predicted deformation and translation. compute Projective tsdf value, and update the canonical volume.

Now I finished the 1) and 2) steps, I face a problem at step 3)

How to rotate and translate each voxel center with predicted deformation?

Although the resp provide the "warp_deform_3d" function, which utilizes the pixel_anchors pixel_weights, node_positions, node_rotations and node_translations. Now we have node_positions, node_rotations, and node_translations. How I compute the pixel_anchors pixel_weights for each voxel?

[Algomorph]

@pablorpalafox, I'm also interested in the DynamicFusion portion of the code, whenever you can get to it. I'll add a +1 here so others can simply upvote in the future.

[pablopalafox]

Hi @BaldrLector and @Algomorph!

We're now working on a coming deadline, but we'll definitely let you guys know if we eventually upload the reimplementation of DynamicFusion. @BaldrLector, your pipeline makes sense, the problem is that the warp_deform_3d function you mention is not exactly the one used for the final reconstruction, i.e., the dynamicfusion part is a separate repo. Again, sorry for the inconvenience of not releasing the whole thing, but we wanted to have a clean repo with the tracking part only, since in any case it was the main contribution of the paper. Anyways, we'll let you know :)

[BaldrLector]

@pablorpalafox thanks for your reply, I will keep work on the reconstruction part, and I am still looking for your new info eventually. By the way, have a nice year :)

[shubhMaheshwari]

Hi, @pablorpalafox @AljazBozic,
This is a groundbreaking work with so many applicants. thanks for this source code. I am also trying to implement 3D Non-Rigid Reconstruction. Your approach to use multiple keyframes for reconstruction is very promising for future work.

I am confused about the integration of the various keyframes(every 50 frames in the sequence). Previous methods take an initial frame (generally it is the frame at time step-1). Then we keep on updating the canonical pose using only the RGBD image at the next timestep.
Specifically:

1. For a particular timestep, is the warp field estimated between the estimated canonical pose(extracting RGBD using rasterization) and every keyframe, and then somehow these all warp fields are averaged out.
2. Do you update the deformable graph similar to DynamicFusion at each timestep.
3. Or is it more like you update canonical pose only for 50 frames. eg. 50-51, 50-52,.. 50-100 and then find merge all key frames poses to get the final canonical pose.

@BaldrLector if you can provide some insight that would be very helpful.

Thank you everyone.

[Algomorph]

@shubhMaheshwari and @BaldrLector , it seems like all three of us are working on the same thing: trying to integrate the neural tracking code from the current repository into a complete DynamicFusion pipeline.

Would either of you like to pool our efforts together?

What I have, at time of this post:

1. Open3D-based TSDF (dynamic VRam allocation, voxel hashing) -- this is really just the stock Open3D implementation, but it's very good. I'm able to build it from source and have made contributions to Open3D, so I think I would be able to modify it for the fusion step.
2. Isosurface extraction using Marching Cubes from above (also not my implementation)
3. A way to deform the extracted mesh to frame t-1 using dual-quaternion blending (through a small C++ library with port to python, I believe it's single-threaded)
4. A way to render the mesh to a depth image (generating input for NeuralTracking), by using PyTorch3D

What is still missing:

1. Integrating all of this together
2. A way to fuse frame-t data into the frame-0 TSDF
3. Topological updates to the motion graph, like in DynamicFusion

If you'd like to collaborate, please, reply so here. I'll email you at your respective email in your GitHub profile, and we can continue the discussion.

[shubhMaheshwari]

@Algomorph definitely that's a great idea.