Depth registration#14
Merged
Merged
Conversation
This is essentially what Spot does onboard for its *_depth_in_visual_frame sources. This allows us to request the *_depth targets instead, which are ~1/3 the size (with no additional useful depth info). Also Added a README file.
🧹 Ruff — ❌ 1 files need formatting
Files needing format: Important
Tip VSCode: Set up auto-format and code actions on saveSettings UI: Search "format on save" and "code actions on save" settings.json: "editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll.ruff": "explicit",
"source.organizeImports.ruff": "explicit"
}🟢 Fixed Issues (2)📦 RUF — Ruff
📦 I — isort
Summary
Bot commands for this PRType
Reactions: 👀 = processing, 🚀 = all passing, 😕 = issues remain (details in reply) |
There was a problem hiding this comment.
Pull request overview
This PR adds client-side GPU depth registration so the system can request Spot’s smaller raw *_depth sources and reproject them into the RGB camera frame (matching *_depth_in_visual_frame), while also introducing a runtime toggle for EMA depth averaging in the vision pipeline and improving build/docs.
Changes:
- Switch depth streaming from robot-registered
*_depth_in_visual_framesources to raw*_depthsources and register depth→RGB on the client GPU (CUDA), with matching Python implementation + tests. - Add/adjust vision-pipeline depth caching/EMA behavior and expose a per-stream C API (
SOb_SetDepthAveraging) to toggle it at runtime. - Add repository README and improve Unity Plugin API header discovery in CMake.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/integ-test/integ-test.cpp | Updates integration test logging setup and includes the HAND camera in the requested mask. |
| src/vision-pipeline.cpp | Initializes depth cache and adds an EMA toggle path in postprocess. |
| src/unity-cuda-interop.cpp | Moves Windows/D3D12 and CUDA includes into the .cpp for cleaner header usage. |
| src/spot-observer.cpp | Replaces the old depth-averaging toggle export with SOb_SetDepthAveraging(...). |
| src/spot-connection.cpp | Requests raw depth sources, builds registration params, stages raw depth, and registers into RGB-sized depth buffers. |
| src/include/vision-pipeline.h | Adds an atomic runtime toggle for depth averaging. |
| src/include/unity-cuda-interop.h | Shrinks header dependencies to Unity’s base interface types. |
| src/include/spot-connection.h | Plumbs depth-registration params and raw-depth scratch storage into the ring buffer. |
| src/include/cuda_kernels.cuh | Adds DepthRegistrationParams + register_depth_to_rgb and expands postprocess API for fused EMA update. |
| src/cuda_kernels.cu | Implements depth→RGB registration kernels and a fused postprocess+cache update path. |
| README.md | Adds top-level documentation for building, layout, and usage context. |
| PySpotObserver/tests/test_depth_registration.py | Adds unit tests for the Python depth registration implementation. |
| PySpotObserver/tests/test_depth_registration_robot.py | Adds optional robot-in-the-loop validation against *_depth_in_visual_frame. |
| PySpotObserver/tests/test_config.py | Updates expected depth source names to the raw *_depth sources. |
| PySpotObserver/pyspotobserver/depth_registration.py | Adds Python depth registration implementation mirroring the CUDA approach. |
| PySpotObserver/pyspotobserver/config.py | Switches Python depth source naming to raw depth sources and documents the behavior. |
| PySpotObserver/pyspotobserver/camera_stream.py | Registers raw depth into the RGB frame on the client side and caches per-camera registration params. |
| PySpotObserver/examples/config_example.yaml | Updates the sample config content (notably credentials fields). |
| include/spot-observer.h | Exposes SOb_SetDepthAveraging(robot_id, cam_stream_id, enable) in the public C API. |
| CMakeLists.txt | Auto-detects Unity Plugin API headers and provides a clear override path. |
Comments suppressed due to low confidence (1)
src/spot-observer.cpp:666
- This log message inside
SOb_ToggleDepthAveragingWithOpticalFlowstill says "SOb_ToggleDepthAveraging", which is misleading when debugging (and is now especially confusing since the non-optical-flow function was renamed toSOb_SetDepthAveraging). Update the message to match the function name.
}
UNITY_INTERFACE_EXPORT
bool UNITY_INTERFACE_API SOb_ToggleDepthAveragingWithOpticalFlow(bool enable) {
SOb::LogMessage("SOb_ToggleDepthAveraging called with enable: {}", enable);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+6
to
+8
| robot_ip: "128.148.138.22" | ||
| username: "" | ||
| password: "" | ||
| username: "user" | ||
| password: "bigbubbabigbubba" |
Comment on lines
+102
to
+106
| if (!rgb_source.has_pinhole() || !depth_source.has_pinhole()) { | ||
| LogMessage("build_depth_registration_params: Missing pinhole intrinsics " | ||
| "(rgb '{}': {}, depth '{}': {})", | ||
| rgb_source.name(), rgb_source.has_pinhole(), | ||
| depth_source.name(), depth_source.has_pinhole()); |
Comment on lines
1359
to
1363
| float generated_val = generated_depth[idx]; | ||
| float new_val = sparse_depth[idx]; | ||
| //float new_val = sparse_depth[idx]; | ||
| float old_val = cached_depth[idx]; | ||
|
|
||
| // Check if new depth value is valid |
Avoid converting u16 depth data to floats on the CPU. This is now done on the GPU during the course of registering the depth to the RGB frame. Also addressed review comments.
| } | ||
| } | ||
|
|
||
| } // namespace STb |
| // | ||
| // LogMessage("ReaderWriterCBuf::push: latency: {:.4f} ms", latency_ms); | ||
| // start_time = end_time; // Reset start time for next push | ||
| static time_point<high_resolution_clock> start_time = high_resolution_clock::now(); |
Comment on lines
+154
to
+156
| # Paint far-to-near so the nearest surface wins where samples overlap. | ||
| order = np.argsort(zs)[::-1] | ||
| out[ys[order], xs[order]] = zs[order] |
Comment on lines
1361
to
1365
| int idx = y * width + x; | ||
| float generated_val = generated_depth[idx]; | ||
| float new_val = sparse_depth[idx]; | ||
| float old_val = cached_depth[idx]; | ||
|
|
||
| // Check if new depth value is valid | ||
| bool new_valid = (new_val >= min_valid_depth && new_val <= max_valid_depth); | ||
|
|
||
| bool old_valid = (old_val >= min_valid_depth && old_val <= max_valid_depth); |
Run nvcc only for sm86 and sm89. Previously Torch was mucking with CMAKE_CUDA_ARCHITECTURES so it was building for more archs. Fix the DLL copying nightmare in cmake. Fix corner case where depth cameras transmit different-sized depth. Rework the python depth registration to avoid reallocating buffers every iteration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is essentially what Spot does onboard for its *_depth_in_visual_frame sources. This allows us to request the *_depth targets instead, which are ~1/3 the size (with no additional useful depth info).
The depth registration is now also responsible for converting u16 depth values to floats, cutting down CPU->GPU bandwidth requirement by half.
Also added EMA depth averaging (togglable).
And a README file.