[video_ingestion] OSRB compliance: dual-license LICENSE, NOTICE, SPDX…#91
Closed
nv-liuw wants to merge 238 commits into
Closed
[video_ingestion] OSRB compliance: dual-license LICENSE, NOTICE, SPDX…#91nv-liuw wants to merge 238 commits into
nv-liuw wants to merge 238 commits into
Conversation
…t and formats it with the ManoSharpaData schema and saves to file. Then second script loads the data file and runs retargeting filling the robot state related fields. These changes also add code that loads the TACO dataset. The new data loading flow works for both Arctic and Taco.
fix num_envs = 1 fix left hand init and reset add box no collision
- Improve the landing page of the github page - Update ci/cd
# v2d hand alignment + ego e2e pipeline ## Summary Adds an end-to-end egocentric pipeline for reconstructing a hand and held object from a single RGB video, along with the `v2d_hand_alignment` module that depth-aligns DynHaMR hand poses to monocular depth and computes a coherent world-frame scene. --- ## New modules ### `v2d_hand_alignment` Full Docker module for aligning egocentric hand reconstructions against monocular depth estimates. - **`hand_object_alignment.py`** — `HandObjectAlignment` class: given DynHaMR world results, per-frame depth images, and per-frame object masks, computes the camera-space translation offset that minimises the hand–depth discrepancy per frame. `compute_offset_reprojected` corrects for intrinsics mismatch between DynHaMR and the depth source (MoGe/ViPE) by reprojecting the shifted centroid into depth pixel space before applying the offset. - **`align_world_results.py`** — reads `world_results.npz`, runs per-frame `compute_offset_reprojected`, stores `trans_aligned` (depth-corrected translation) and `intrins_aligned` ([fx,fy,cx,cy] of the depth used for alignment) alongside all original fields. Optionally ingests FoundationPose object poses and stores them in both camera and world frame. - **`render_dynhamr_video.py`** — renders a 2×2 grid overlay video (RGB + hand mesh + object mesh + combined). Intrinsics are picked up automatically from `intrins_aligned` in the `.npz` when present; an explicit `--intrinsics_path` override is also supported. - **`render_multiview_video.py`**, **`render_mano_params_video.py`**, **`reproject_hand_mesh.py`**, **`smooth_hand_mesh.py`**, **`recover_mano_params.py`** — supporting render and post-processing steps. ### `v2d_depth_anything` New Docker module wrapping Depth Anything V2 for monocular depth estimation (image and video). Follows the same host/container split as other `v2d_*` modules. ### `v2d_viz` Lightweight 3D viewer package: a Python HTTP server (`run_3d_viewer.py`) serving a Three.js web UI (`static/index.html`) for inspecting point clouds and meshes interactively. --- ## New pipelines (`v2d_pipelines`) ### `run_v2d_ego_e2e.py` Single-script end-to-end pipeline for ego hand + object reconstruction: | Step | What runs | |------|-----------| | 1 | Ego hand reconstruction (ViPE + DynHaMR) | | 2 | DynHaMR EXR depth → uint16 PNG + intrinsics | | 3 | Frame extraction | | 4 | MoGe monocular depth + intrinsics | | 5 | Grounding DINO object detection (reference frame) | | 6 | SAM2 object mask tracking | | 7 | SAM3D textured mesh generation | | 8 | FoundationPose scale estimation | | 9 | FoundationPose 6-DOF tracking (with optional IoU-based re-registration) | | 10 | EKF pose smoothing | | 11 | Hand/object depth alignment | | 12–13 | Aligned + unaligned overlay video renders | `--depth_source moge|vipe` selects which depth map feeds FoundationPose, EKF, and hand alignment; both sources are always computed so results can be compared side by side. ### `run_v2d_ego_video.py` Lighter pipeline variant: runs only the hand reconstruction and depth estimation steps (steps 1–4), without object tracking. --- ## Module enhancements ### `v2d_moge` - `video_to_depth` now optionally exports a **pointmap** folder (`--points_folder`): per-frame `(H, W, 3)` float32 `.npy` files with metric 3D points, useful for SAM3D and depth alignment. - Intrinsics stabilisation: `stabilize_intrinsics` smooths per-frame focal lengths into a single stable estimate across the video. ### `v2d_sam3d` - Accepts a pointmap input as an alternative depth source for mesh generation. - `run_image_to_mesh` exposes the pointmap path via both the lib and docker wrappers. ### `v2d_foundation_pose` - `run_video_to_poses` gains `--reregister_iou_thresh`: when set, re-registers the object from scratch whenever the projected mask IoU drops below the threshold (uses the new `track_one_with_recovery` path in `FoundationPoseTracker`). ### `v2d_mesh` - `Mesh.load` / `Mesh.save` extended; `mesh_transform.py` and `mesh_render_image.py` updated to support the render pipeline. ### `v2d_depth` - `align_depth_sequence.py` and `stabilize_intrinsics.py` added as shared utilities. --- ## Documentation - **`docs/ego_e2e_setup.md`** — minimal setup guide: prerequisites, package install, Docker build, weight download, and how to run the pipeline. --- ## Output schema (world_results_aligned.npz) All original DynHaMR fields, plus: | Field | Shape | Description | |-------|-------|-------------| | `trans_aligned` | `(B, T, 3)` | Depth-aligned translation (world units) | | `intrins_aligned` | `(4,)` | `[fx, fy, cx, cy]` of the depth source used for alignment | | `object_pose_cam` | `(T, 4, 4)` | Object-to-camera SE(3) (if `--object_poses_dir`) | | `object_pose_world` | `(T, 4, 4)` | Object-to-world SE(3) (if `--object_poses_dir`) | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- **Adaptive auto-curriculum** (`curriculum.py`): CV-gated first VOC decay, WSR-retention gate for subsequent decays, custom VOC schedule (1.0→0.75→…→0.0), force-timeout fallback; new `Sharpa-V2P-AutoCurr-v0` env config and pool/priority support in OSMO configs - **Eval pipeline overhaul** (`eval_callback.py`): two-pass eval per checkpoint — Pass A (from-start, tc=0) measures full-trajectory completion; Pass B (random reset) measures trajectory coverage; logs `eval/completion_ratio_mean`, `eval/completion_ratio_std`, `eval/full_completion_pct`, `eval/completion_ratio_mean_random` - **Train/eval distribution gap fix**: `reset_to_first_frame_prob` causes a fraction of training resets to go to tc=0, closing the gap between training distribution and from-start eval - **VOC-gated eval**: eval runs before each VOC decay and on a 1000-iter fallback, providing signal timed to curriculum transitions - **Bug fixes**: `custom_schedule` VOC not decaying per-env, eval video now records from frame 0, post-warmup completion ratio denominator, spurious training reward peaks after eval, inference mode not set at step 0, VOC not restored correctly on resume --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a SOMA-X (NVlabs SOMA-MHR) -> G1 whole-body retargeter that mirrors
the NVHuman pipeline's motion_v1 output schema. The new path is
config-driven from day one: every robot-specific transform, IK weight,
posture-task gain, and foot-framing parameter is loaded from per-robot
JSON in `configs/<robot>/{frame_alignment,retargeter}.json` rather than
embedded in `params.py`, so adding a second humanoid is a JSON drop-in.
New modules
- `retarget/read_soma.py`: SOMA-X loader. Reads `soma_params.npz`,
applies SOMA's joint-orient + level-order FK, anchors the body to
frame-0 pelvis at the origin (`normalize=True`), and exposes per-frame
joints, world-rotation quats, and posed mesh vertices.
- `retarget/robot_config.py`: `RobotRetargetConfig` dataclass +
`load_robot_config(robot_name)`. Reads `frame_alignment.json` (world
axis swap + per-bone q_offset / t_offset) and `retargeter.json` (URDF,
IK map, foot framing, posture-task), composes pre-multiplied matrices,
validates the bundle.
- `retarget/whole_body_kinematics.py`: add
`ConfigDrivenWholeBodyKinematics` plus a posture-task hook
(`_extra_tasks`) so q0 + previous-q tracking can run alongside the frame
tasks.
- `retarget/ground_alignment.py`: generalize `ReferencePlane` to a
`(normal, offset)` pair; add `vertical_offset_to_plane` and a
reconstructed-plane loader so retargeting can honour the
reconstruction's `ground_plane.json` when present (falls back to
`ReferencePlane.horizontal(z=0.0)`).
New scripts
- `scripts/retarget/soma_to_g1.py` (1226 lines): SOMA -> G1 entrypoint.
Loads SOMA, applies CV->SOMA + first-frame anchor to the object
trajectory, runs the IK loop with optional posture-task regularization,
anchors per-frame ground plane, and writes a motion_v1 parquet under
`whole_body/soma/...`. Supports `--start-frame/--end-frame` slicing and
IK diagnostics.
- `scripts/retarget/soma_loader_probe.py`: read-only verification probe
for the SOMA loader; viser playback of body + object in source frame.
- `scripts/retarget/process_soma_sequence.sh`: interactive driver that
walks one sequence through the loader probe, retarget, support
reconstruction, and Isaac Lab replay stages. `--auto` runs the
non-visual stages hands-off.
- `scripts/setup_soma_assets.py`: one-shot bootstrap that fetches the
SOMA-MHR body model assets into `assets/body_models/soma/`.
Configs + docs
- `configs/g1/frame_alignment.json`: source-to-robot frame change for G1
(world axis swap + per-bone q_offsets / t_offsets). Per-bone q_offsets
seeded from NVIDIA/soma-retargeter and refined empirically.
- `configs/g1/retargeter.json`: G1 IK map, foot framing, posture-task
weights tuned for the snack-box / book sequences.
- `configs/README.md`: schema for both files, conventions (frames,
units, rotation composition), "add a new robot" walkthrough, three
legitimate sources for q_offsets (upstream copy / rest-pose derivation /
hand-tune in viewer), and a verification recipe.
- `retarget/scene_utils/scene_viewer_env_cfg.py`: seed
`env_cfg.scene.robot.init_state` from the saved frame-0 robot pose so
the viewer spawns the URDF where the retargeted motion expects it.
- `pyproject.toml`, `workflow/Dockerfile`: add the SOMA dep group +
body-model fetcher.
Tests
- `tests/test_reference_plane.py` (10 cases): horizontal equivalence,
inclined signed distance, vertical-offset conversion, normalization,
normal-z reorientation, near-vertical / zero-norm rejection, batch-shape
support, trailing-dim guard. All pass.
Bundled data
- `assets/human_motion_data/whole_body/soma/...`: one retargeted fixture
sequence + reconstructed support-surface USDA, for end-to-end regression
from this branch.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Touches modules that I don't own: - (1506ab3) FrameSource/FrameWriter abstraction to support reading/writing frames as images, video, or HDF5. In most places, this is simply replacing code lines doing image read/write. For SAM2, I modified the LazyFrameLoader. - (3477889) Reorder contents of several Dockerfiles to use build cache better (v2d_sam2, v2d_foundation_pose, and some mv-specific modules) - (12b1498) Small fix in v2d_mesh to load GLB meshes with orientation Other changes: - Added OSMO workflow configuration and management system for dataset sequences - Add MHR->SOMA conversion in v2d_sam3d_body module, simplify output format - Add ground plane estimation - Add symmetric object handling in MV FP tracking - Add data exporting - Improve detectron2 with ByteTracker
… to hand alignment (#62) # Add v2d_anycalib + fisheye-aware hand alignment ## Summary - New `v2d_anycalib` module: single-view camera calibration (intrinsics + distortion) and undistortion via AnyCalib, for image / image folder / video inputs. Lets the e2e pipeline run on undistorted frames so ViPE/MoGe/FoundationPose/DynHaMR don't fight the fisheye. - Fixes for `v2d_hand_alignment` aligned/unaligned renders on fisheye footage: intrinsics misuse, left-hand mirror, and missing per-vertex size correction under depth-source intrinsics. ## Changes **`v2d_anycalib`** — `lib/` calibration + undistortion, `docker/` host-side wrappers (image, image folder, video, shell, weights download), Dockerfile, pyproject, `v2d_common.datatypes` fields for distortion + undistorted intrinsics. **`v2d_hand_alignment`** - `run_v2d_ego_e2e.render_unaligned` now passes `intrinsics_vipe` (raw `trans/cam_R/cam_t` were optimized against ViPE's pinhole; depth-source intrinsics break it). - `align_world_results` mirrors `delta_world.x` for left hands — the renderer flips x *after* adding `trans`, so the world-space delta must match. - `HandObjectAlignment.compute_alignment(side, t)` does one render pass and returns `{offset, offset_reprojected, scale, n_pixels}`. Old methods are thin shims. - `align_world_results` aggregates per-frame scales into a constant `hand_scale` (`n_pixels`-weighted median, rejecting <256-pixel or out-of-`[0.2, 5.0]` frames) and saves it on `world_results_aligned_*.npz`. MANO has no native scale knob, so a post-MANO scalar is the right schema. - `render_dynhamr_video` applies `hand_scale` around the per-frame centroid only when `use_trans_aligned=True`; falls back to 1.0 when absent (backward-compat). **Misc** — `v2d_nlf` Docker build fix. ## Why `hand_scale = (fx_vipe / fx_moge) × (z_moge / z_vipe)` resolves both the ViPE-vs-MoGe focal mismatch (wrong silhouette size) and the MoGe-vs-DynHaMR depth disagreement (centroid at wrong depth). Approximately constant across time, so a single global value works. ## Test plan - [ ] `run_v2d_ego_e2e` on raw fisheye and AnyCalib-undistorted inputs. - [ ] `render_unaligned_moge.mp4` matches DynHaMR `*_src_cam.mp4` modulo OneEuro smoothing (known). - [ ] `render_aligned_moge.mp4` hand silhouette overlays image hand at correct size. - [ ] `world_results_aligned_*.npz` contains `hand_scale`; old npzs without it still render. - [ ] `mano.ipynb`: scaled hand sits on depth point cloud, not just along its ray. - [ ] AnyCalib end-to-end: per-frame `dz` smaller on undistorted stream than on raw fisheye.
…uction (#60) - Add reconstruction interface to integrate video agent with reconstruction module - Integrate the e2e hand reconstuction pipeline with webapp - Add tests Test plan: - [x] Unit tests - [x] Integrated tests with webapp: https://drive.google.com/file/d/1h_8kcBLoSJEFKnjE_4ckBpCMdAvXkK4e/view?usp=drive_link Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Adds MANO→Dex3 retargeting for ARCTIC + TACO mirroring the sharpa
scripts on main: new `arctic_to_dex3.py` and `taco_to_dex3.py` use the
same loader output (`<dataset>_loaded/`), CLI surface, and per-frame IK
loop.
- Implements MANO source in `Dex3HandKinematics` (replaces the existing
`NotImplementedError` stub) and adds `setup_dex3_kinematics`,
`DEX3_TO_MANO_MAPPING` (palm + thumb/index/middle tips), `ManoDex3Data`
schema (BASE + MANO + DEX3 + OBJECT, 7 finger joints, 23 frames, 4
tasks).
- `dataset_registry.py`: refactors `retarget_script: str` →
`retarget_scripts: dict[str, str]` keyed by robot name. arctic and taco
gain `"dex3"` entries; sharpa entries unchanged.
- `run_retarget.py`: new `--robot {sharpa_wave,dex3}` flag dispatches to
the right per-robot retargeter.
- `vis_retargeted.py`: same `--robot` flag generalises the viewer.
Bundles the changes needed to launch whole-body ReconBody training on the new soma TPV sequences (snack_box pick-and-place, dark_blue_book pick-handover-place, blue_trash_can drag) and bring across the VOC and metric infrastructure they depend on. Feature work: - MotionData.trim(start, end) at the schema layer; consumed by the whole-body TrackingCommand loader and by load_replay_trajectory so train, eval, and replay all subset motions consistently. - TrackingCommandCfg gains motion_start_frame / motion_end_frame (int sentinel for end so Hydra int overrides aren't rejected). - scripts/replay_motion.py exposes --start_frame / --end_frame and threads them through Trajectory and ContactOverlay. - WholeBodyFixedTimestepVOCCurriculum + G1SonicReconBodyVOCCurriculumCfg port the VOC scale schedule onto v2p_whole_body. - TrackingCommand allocates and populates per-env tracking metrics including virtual_object_controller_scale_factor. - DualHandsObjectTrackingCommand logs the per-env VOC scale. - motion_joint_pos_error_exp converted to ManagerTermBase so configs can pass joint_names; G1SonicReconBodyRewardsCfg now scopes the joint-pos reward to G1_SONIC_JOINT_NAMES. - Strengthens resampling_time_range docstring to spell out how the (1e6, 1e6) sentinel disables IsaacLab's mid-episode resample, since the trim feature relies on that path staying off. Experiments + assets: - New experiment configs recon_body_snack_box_pick_and_place_01 and recon_body_dark_blue_book_pick_handover_place_02, registered in experiments/registry.yaml. - snack_box config drives motion_start_frame=250 / motion_end_frame=500 and reset_shoulder_spread=0.2 as a working example of the trim + shoulder-spread reset combo. - Adds soma sequence assets for snack_box_pick_and_place_01 and blue_trash_can_drag_007 (mesh + URDF + parquet) plus the reconstructed_stage USDA referenced by the snack_box scene. Tests: - v2p_whole_body command tests cover update_metrics shape contract and the per-env VOC scale propagation. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Tooling (the bulk of this change): - scripts/train_tpv_wholebody_workflow.sh: 729-line host-side driver for the TPV ReconBody training pipeline. Provides an interactive `run` mode that walks init -> replay -> set-frames -> submit with stage-skip detection, and per-stage subcommands (`init`, `replay`, `set-frames`, `submit`) for scripted reruns. Orchestrates Docker (`./workflow/run.sh exec`) and OSMO submission from the host; aborts if invoked inside the container. - scripts/retarget/create_soma_task.py: 441-line scaffolder that writes `experiments/<exp_id>/config.yaml` from the existing recon_body_* template and idempotently registers the id in `experiments/registry.yaml` under the whole-body section, preserving comments/blank lines. Used by stage 6 of the workflow and runnable standalone. - experiments/run_experiment.py (+169), experiments/utils.py (+52), scripts/rsl_rl/eval.py (+43): wiring needed by the workflow driver. ## Task config: - v2p_whole_body/config/sonic/g1/g1_sonic_env_cfg.py: switch the timeout DoneTerm from `il_mdp.time_out` to `timestep_termination` keyed on the motion command, so episode length follows the per-motion frame range instead of a fixed wall-clock budget. ## New experiments + assets: - experiments/recon_body_corn_can_right_left_handover_01 and experiments/recon_body_sit_skinny_wood_chair_01: config.yaml + registry entries, retargeted G1 motion_v1 parquet, and textured object meshes (.obj/.urdf/.mtl/.png) under the SOMA whole_body asset tree. ## Existing-config tweaks: - recon_body_blue_trash_can_drag_007: motion frame range 0..250 -> 50..400. - recon_body_snack_box_pick_and_place_01: motion frame range 250..500 -> 300..-1, episode_length_s 5.0. ## Tests - [x] Run the full data processing pipeline with `process_soma_sequnces.sh` - [x] Run the task initialization and training pipeline with `train_tpv_wholebody_workflow.sh` - [x] launched all five jobs in OSMO, [example 1](https://wandb.ai/nvidia-isaac/v2d-whole-body-tpv/runs/mxno4i5z?nw=nwuserhuihuaz), [example 2](https://wandb.ai/nvidia-isaac/v2d-whole-body-tpv/runs/qtgv5it3?nw=nwuserhuihuaz) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add SECURITY.md file to repo for public-facing releases. Thanks!
…hot updates (#74) Lower bound stays on 0.12.x: 0.13+ introduces a ~6x per-video slowdown on Qwen3-VL multimodal prefill at TP=8 (verified by A/B on identical HoT3D Aria 1080p workload, groot-h100-01). Upper bound already dodged the 0.20+ Qwen3-VL deepstack-tokens regression (`num_tokens > deepstack_input_embeds_num_tokens`); kept. Update the screenshots of the v2d webapp Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Motionbricks planner migration into video_to_data + planning-time
validation
Replaces the legacy MFM-based planner with the new motionbricks backend
(stacked on the task-40 dex3-retargeting MR) and adds planning-time
validation so the output parquet is trainable end-to-end with no
post-hoc
patches.
## Summary
- Self-contained `torch.package` motionbricks backend bundled at
`planner/motionbricks/assets/models/planner.pkg` (Git LFS). No gr00t
runtime dependency.
- `g1_planner.py` rewired to the new backend; output is the unified
`motion_v1` parquet schema, Hive-partitioned at
`<output>/planner_processed/sequence_id=…/robot_name=…/*.parquet`.
- Support surface reconstruction split into `planner/support_recon.py`
with a phantom-tool filter (a tool resting on the target no longer
emits a support disk that intersects the target body in sim).
- New `planner/utils/` subpackage for shared helpers (coordinate
transforms, motion loading, validation). Legacy MFM backend kept under
`planner/mfm/` for reference and explicitly noqa-ed; everything else
enforces ruff/mypy.
## Pipeline
V2P retargeted parquet
→ Step 1: nominal FK
→ Step 2: load + interpolate + (optional contact-driven trim)
→ Step 3–4: transform_reference (local frame fix → yaw → position align
→ workspace offset)
→ Step 5: build warmup trajectory (hold nominal → interp → hold start →
reference)
→ Step 6: motionbricks chunked autoregressive inference
→ Step 7: assemble full qpos (planner body + reference fingers +
optional fixed legs/root)
→ Step 8: save motion_v1 parquet + post-write invariant assertions
→ Step 8b: reconstruct support USDA with phantom-tool filter
→ Step 9: optional MuJoCo viewer
## What's new vs the legacy planner
### Planning-time validation (`planner/utils/validation.py`)
Two entry points called from `g1_planner.py`:
- **`warn_reference_issues(motion, ref_data, robot_type)`** — pre-plan.
Warns when the input motion is missing required fields, has an
off-FPS source, references asset paths that don't resolve locally, or
has URDF `<mesh filename=>` deps not present in the workspace. These
are upstream (retargeting / asset pipeline) responsibilities; the
planner never auto-fixes them.
- **`assert_motion_parquet_invariants(parquet_path, robot_type)`** —
post-plan, hard-fail. Verifies:
- `object_root_position` length matches `object_body_position` and
mirrors body 0 to within 1 mm.
- `ee_link_names` matches the actual EE body in the data
(`*_hand_palm_link` for dex3, `*_wrist_yaw_link` for sharpa).
- `hand_frames_w[palm]` agrees with `ee_pose_w` per side (catches a
skipped V2P→planner rigid transform on hand keypoints).
- `robot_joint_names` contains the tracked finger joints (catches the
case where the env's tracking_command would raise on
`cfg.joint_names` resolution).
- Every contact-field shape is `(2, T, N, 3)` with T matching the
body trajectory.
- Every `object_urdf_paths` / `object_mesh_paths` entry exists; every
URDF mesh dependency exists.
### Output parquet contract enforced in `save_planner_parquet`
- `robot_joint_names` / `robot_joint_positions` cover every actuated
joint (body + fingers) in MuJoCo joint order. The per-side
`hand_finger_joints` / `hand_finger_joint_names` lists stay
populated for callers that want the side-segregated view.
- `object_root_position` / `object_root_axis_angle` derived from body 0
of the planner-frame object pose so the env's articulated scene init
always lands where the trajectory starts.
- `ee_link_names` is robot-conditional (palm for dex3, wrist_yaw_link
for default).
- `hand_frames_w` and the four contact-field groups are lifted into the
planner frame via the same per-frame rigid transform that
`transform_reference` applies to `ee_pose_w` / `object_body_position`,
so every field of the output parquet lives in a single coherent
frame.
### Support surface reconstruction
`planner/support_recon.py` runs after the parquet write. The new
phantom-tool filter drops a body's support disk if any other body sits
in its xy circle for ≥ 90 % of frames within Δz < 10 cm — catches the
case where a tool is "still" only because it's resting on the target.
Detection uses per-body trajectories from the parquet, no name-prefix
convention needed.
## Module layout
planner/
├── g1_planner.py CLI orchestration; sole consumer of utils/
├── support_recon.py support reconstruction with phantom-tool filter
├── trajectory.py warmup trajectory builder
├── visualization.py MuJoCo viewer
├── motionbricks/ motionbricks model backend (torch.package)
├── mfm/ legacy MFM backend (kept; noqa-ed)
└── utils/ pure helpers, no planner state
├── transforms.py quat math + rigid primitives + transform_reference
pipeline
├── loader.py V2P motion resampling (linear + SLERP + masked contact
interp)
└── validation.py pre-plan warn + post-plan assert
`planner.utils.*` is consumed only by `g1_planner.py`; the orchestration
script is the only entry point in the package.
## Lint / typing
- File-level `# ruff: noqa: ANN001, ANN201, ANN202, ANN204, D102, D103,
D107, D417` blocks removed from every active file. Annotations and
docstrings added on the public surface.
- Suppression retained on `planner/mfm/*` only — the legacy backend
will be removed; not worth churning.
- Local mypy / ruff pre-commit hooks pass; CI mypy fixes (`_dc._is_type`
attr-defined ignores + a `previous_range` None-narrowing assertion)
are in the final commit.
## Verification
Run end-to-end under the `v2p` conda env on two known-good references:
| Sequence | Robot | T frames | FPS | Wrist EE | Notes |
|---|---|---|---|---|---|
| `taco_skim_off__spoon__pan_20230926_011` | dex3 | 1980 | 150 | 78.3 mm
| phantom support on `tool` correctly filtered (tool sits on target 100
% of frames) |
| `dataset_s01_mixer_use_02` | dex3 | 2660 | 100 | 73.7 mm | no phantom
support |
Both runs:
- Pre-plan: emit only `REFERENCE WARNING` lines for unresolvable
container paths in the upstream parquet (planner's path remapper
then re-roots them locally).
- Post-plan: `planner_validation: <hash>.parquet passed all
invariants.` — zero diagnostics.
- Output parquets load cleanly via
`motion_schema.reader.load_motion_data_parquet` and through
`SceneConfig.from_motion_file(...)` for env spawn.
Run command (taco_skim example):
```bash
PYTHONPATH=robotic_grounding/source/robotic_grounding \
MUJOCO_GL=egl python -m robotic_grounding.planner.g1_planner \
--robot dex3 \
--v2p_parquet <…>/assets/human_motion_data/taco/taco_dex3_processed \
--v2p_robot_name dex3 \
--v2p_sequence taco_skim_off__spoon__pan_20230926_011 \
--target_fps 150 \
--output <out_dir> \
--no_viewer
Out of scope / known caveats
- A few task-48-era taco assets
(assets/urdfs/taco/{040,067}_rigid.urdf +
assets/meshes/taco/obj_{040,067}_visual.stl) are still untracked in
this worktree. They're not required for the sequences exercised
above; including them in this MR is left as a follow-up.
- planner/mfm/ is retained verbatim. It can be removed once the
motionbricks backend has soaked in training for a release cycle.
- Env-side G1SonicReconHandEnvCfg.episode_length_s and similar
consumer-side fixes live downstream of this MR; this PR's
invariants run on the planner output only.
Files changed (high level)
- Added: planner/motionbricks/, planner/utils/{__init__,transforms,loader,validation}.py,
planner/support_recon.py, planner/motionbricks/assets/models/planner.pkg (LFS).
- Modified: planner/g1_planner.py (orchestration + validation
wiring), planner/visualization.py (inertial on injected vis
bodies, mesh scale, warmup-hold), planner/README.md,
planner/trajectory.py (lint).
- Removed: planner/transforms.py, planner/frame_transforms.py,
planner/v2p_loader.py, planner/planner_validation.py (moved /
merged into planner/utils/).
) Root-level dotfiles had accumulated rules that only applied to one subproject (mainly robotic_grounding). Move them into each package's own copies and add a CI workflow that prevents the drift from coming back. Moves and content shifts: - .pre-commit-config.yaml and .envrc -> robotic_grounding/. The robotic_grounding CI was already keying its cache on robotic_grounding/.pre-commit-config.yaml; this lines that up. - Trim root .gitignore and .gitattributes to universal rules only; push package-anchored rules (robotic_grounding/checkpoints/, reconstruction/soma_data/, taco LFS paths, *.pkg, ...) into the matching subproject .gitignore / .gitattributes. - Drop CLAUDE.md ignore rules from root and robotic_grounding gitignores so per-package agent guidance is committable; add video_ingestion_agent/CLAUDE.md which had been silently ignored. New CI: .github/workflows/monorepo_hygiene.yml runs two jobs on changes to root config: - check_root_files: fails if root .gitignore / .gitattributes contain patterns anchored to a subproject path, or if .pre-commit-config.yaml / .envrc exist at root. - agent_context_check: fails if the root .gitignore ignores CLAUDE.md or AGENTS.md (per-package gitignores are out of scope by design). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#76) atomic-clip segmentation + auto-discovered penetration filter
## Summary
New hand-reconstruction stack, end-to-end pipeline rework, and Apache
2.0 LICENSEs for the authored modules.
**New Docker modules**
- `v2d_mediapipe` — single-image hand bbox + handedness (CPU).
- `v2d_hamer` — per-frame MANO from SAM2 hand masks + overlay renderer +
shared `align_hands`.
- `v2d_wilor` — hand bbox + MANO from image / image-list / video / SAM2
masks.
- `v2d_droid_slam` — monocular SLAM → per-frame camera poses (+ optional
keyframe depth / PLY). Accepts single or per-frame intrinsics; optional
metric-depth folder for scale alignment.
- `v2d_gsplat_refinement` — 2DGS / hand-Gaussian refinement, with
face-tied hand mode, bg init, and a visualizer.
**Pipeline**
- `run_v2d_ego_e2e.py` rewritten to use the hamer-style per-frame
cam-frame alignment (`dynhamr_to_hamer_tracks` → `align_hands`). Robust
to bad ViPE intrinsics / camera poses because the alignment never
touches inter-frame extrinsics.
- Same renderer drives both the aligned and unaligned comparison videos
via a new `use_pre_dz_cam_t` flag.
- SAM3D mesh generation is seedable (`--seed`) for reproducibility.
- MoGe accepts an `--input_intrinsics` focal-length prior.
**Bug fix**
- SAM2 multi-frame prompts now run under `torch.autocast(bfloat16)` —
without it, the cross-attention dtype mismatch broke any prompt at
`frame_idx > 0`.
**Cleanup**
- Removed legacy `v2d_hand_alignment` modules superseded by the
hamer-style path: `align_world_results.py`,
`run_align_world_results.py`, `render_dynhamr_video.py`,
`run_render_dynhamr_video.py`.
**Licensing (commit `d323b3c0`)**
Adds 31 Apache 2.0 LICENSE files: umbrella `reconstruction/LICENSE` +
one per module for every module whose code is entirely authored (Docker
wrappers, lib wrappers, shared packages).
Intentionally omitted pending legal review of in-tree upstream code:
- `v2d_foundation_pose` — `lib/FoundationPose/` is in-tree (NVIDIA Sec
3.3).
- `v2d_nlf` — `lib/lib_smpl/smplpytorch/` is in-tree (GPL-3).
- `v2d_sam3d_body` — `lib/sam_3d_body/` is in-tree (Meta SAM License).
For these three, the upstream LICENSE files at the in-tree subdirectory
level already cover the vendored code; the wrapper-level
derivative-vs-independent call is deferred. `v2d_raft` is also
unlicensed — empty placeholder.
## Test plan
- [ ] `./scripts/build_containers.sh` succeeds for all listed modules
- [ ] `python modules/v2d_pipelines/run_v2d_ego_e2e.py …` produces both
`render_aligned_{ds}.mp4` and `render_unaligned_{ds}.mp4` from a single
`align_hands` pass
- [ ] `--seed 0` yields a deterministic SAM3D mesh; `--seed -1` produces
fresh meshes each run
- [ ] SAM2 mask tracking succeeds with a multi-frame prompt JSON
(regression test for the bfloat16 fix)
- [ ] HaMeR / WiLoR module CLIs still run as standalone wrappers
- [ ] DROID-SLAM `run_video_to_slam` produces poses + optional depth on
a sample clip
- [ ] Confirm with legal: deferred LICENSE on `v2d_foundation_pose`,
`v2d_nlf`, `v2d_sam3d_body` is the right call before release
…build scripts (#84) ## Summary Unblocks the EVT-02 ego e2e walkthrough by fixing two QA-reported setup bugs and tightening the install/build story. - **bug 6197910** — `scripts/install_packages.sh` had drifted out of sync with the modules tree; missing entries (`v2d_depth`, `v2d_hand_alignment/docker`, etc.) caused `ModuleNotFoundError` mid-pipeline. Script now installs every host-side package under `modules/`. `v2d_pipelines/pyproject.toml` also now declares its direct host-side imports (`trimesh`, `OpenEXR`, `numpy`) so no ad-hoc `pip install` is needed. - **bug 6203170** — MANO assets now live at `data/weights/hand/models/MANO_RIGHT.pkl` (manotorch's required convention). `v2d_ego_hand_reconstruction/docker/run_reconstruction.py` reads from the nested path and copies into the vendored container's flat location. Docs updated. - **New focused scripts** — `scripts/install_ego_e2e_packages.sh` and `scripts/build_ego_e2e_containers.sh` install/build only what `run_v2d_ego_e2e.py` needs. `docs/ego_e2e_setup.md` links to both the focused and the full scripts. ## Test plan - [ ] Fresh `.venv` + `./scripts/install_ego_e2e_packages.sh` resolves cleanly - [ ] Fresh `.venv` + `./scripts/install_packages.sh` resolves cleanly - [ ] `python -c "import v2d.pipelines.run_v2d_ego_e2e"` succeeds without ad-hoc `pip install` steps - [ ] `./scripts/build_ego_e2e_containers.sh` builds all required images - [ ] End-to-end `run_v2d_ego_e2e.py` runs against `assets/airplane.mp4` with `data/weights/hand/models/MANO_RIGHT.pkl` (no manotorch assertion) - [ ] QA re-runs EVT-02 and confirms the bug 6197910 / 6203170 workarounds are no longer needed
#85) ## Summary - `app.py:87-91` gates the cross-tab `reconstruct_btn.click()` registration on `"reconstruction" in services`. That key is only set when `AppConfig.reconstruction` is a populated dict AND `ReconstructionService(...)` constructs without raising. - The default config path — `AppConfig.from_project_configs()`, taken when `scripts/run_webapp.py` is launched without `--config` — reads only `retrieval.yaml` and `ingestion.yaml`. Neither has a `reconstruction:` block. Result: the Reconstruct button appears on the page, gets enabled after a successful query, but has **no click handler** — clicking is a silent no-op (QA-reported behavior). - This patch adds an `elif` branch that wires a fallback handler firing a `gr.Warning(...)` toast explaining what to launch with. Happy path is unchanged. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#87) ## Summary - `database_tab.load_database()` called `resolve_graph_db_path(db_dir)` **outside** its `try/except`. If that call raised — `Path(...)` on adversarial input, `OSError` / `PermissionError` on an unreadable parent, etc. — the function exited without producing the 10-tuple the Gradio `.click()` binding expects, and the frontend sat in a loading spinner forever with no error surfaced. This violates the F.1 Error paths spec contract ("no silent hangs in any case") and is the symptom QA reported when typing a bad path into the Database directory input. - Fix: hoist the `try/except` to wrap the resolve call, and add `gr.Warning(...)` toasts on both failure paths so the user gets a visible signal regardless of which output widget is animating. - Also switched the exception logger to `logger.exception(...)` so the stack trace lands in stderr for future audits. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#86) - `_read_worker_progress()` returns **deltas** since its previous call (it tracks consumed JSONL lines per worker via `progress_lines_read`). Both call sites — the 2 s poll loop and the final read — were *assigning* the tuple back into the outer totals, so once the JSONL was fully drained, the next call returned (0, 0, 0) and clobbered the prior counts. - Result before this patch: a successful 2/2-video, 6-clip batch run shows `Videos processed: 0/0, Failed: 0, Total clips: 0` in the webapp, while `summary_worker_*.json` on disk records the real successful counts. ("Workers" and "Total time" were correct because they weren't routed through the buggy helper — exactly what QA flagged.) - Fix: accumulate (`+=`) the delta tuple at both call sites; no change to `_read_worker_progress()` itself. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#82) Three independent bug fixes that surfaced while running the HoT3D benchmark suite at scale; isolated here for fast review and merge. 1. Dockerfile: `uv sync` was missing `--extra local`. The ingestion `__init__` chain eagerly imports `transformers` via `visual_extractor`, which needs `accelerate`. Without `--extra local`, entity_extraction crashes with `ImportError: accelerate` on EVERY clip in the image, producing 0 entities written. This affected the `s3_ingest_v10b` batch ingest (100% entity-extraction failure rate) before being diagnosed. The local extra is light (transformers, accelerate, timm, bitsandbytes) and does not conflict with the vLLM-backed code path. 2. entity_graph_nodes.py: frame_id values written into vector.db were NOT video-unique. `video_processor.py` emits `frame_<idx>` and we passed it through unchanged into add_frames_batch. The vector.db schema's UNIQUE(frame_id) constraint combined with INSERT OR REPLACE caused cross-video overwrites: in a multi-video ingest, only the last writer for each `frame_<idx>` slot survived, leaving the table with ~150 rows total instead of 30k+. This silently destroyed ~99.5% of frame embeddings on the v10c batch ingest with no error surfaced. Fix: namespace with `f"{video_id_str}/{frame_id}"` on write. After fix, v10d ingest landed 21,539 frame embeddings vs the prior 154. 3. scripts/run_benchmark.py: parallel workers all defaulted to cuda:0 for SigLIP-2 embedding regardless of backend. The old guard skipped CUDA_VISIBLE_DEVICES pinning for the vllm backend on the reasoning that vLLM owns the GPU, but the per-worker SigLIP-2 embedding model still runs in-process on the worker, so all N workers OOM'd on a shared cuda:0 (v10 ingest: 11/136 videos completed). Fix mirrors run_batch_ingestion.py: unconditional pinning, honors upstream CUDA_VISIBLE_DEVICES (so OSMO container device sets are respected), modulos worker_id across visible devices. Also threads a `--worker-id` arg so per-worker progress files don't collide and surfaces non-zero worker exits for the caller. Each fix was validated: * Dockerfile: local docker smoke ingestion completed end-to-end with entities + relationships populated. * frame_id: v10d OSMO ingest emitted 21,539 namespaced frame IDs matching `<video_id>/frame_<idx>` pattern. * run_benchmark.py: v10d ingest hit 198/198 videos with no per-worker OOM in the logs. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… headers, DCO sign-off
NVIDIA OSRB approval for the dual CC-BY-4.0 + Apache-2.0 release
flagged four BEFORE-WE-CAN-CLOSE items. This commit addresses all of
them on the video_ingestion_agent surface.
1. License file (`video_ingestion_agent/LICENSE`)
Replace the previous 15-line Apache-2.0 short notice with the full
CC-BY-4.0-Apache2 Dual License text (~700 lines): NVIDIA preamble +
verbatim Creative Commons Attribution 4.0 International + verbatim
Apache License 2.0.
2. Third-party notices (`video_ingestion_agent/NOTICE`)
New file. Enumerates the 41 direct dependencies declared in
pyproject.toml (top-level + every optional-dependencies extra). Each
entry follows the NVIDIA modstar template: owner + SPDX license +
license URL + package name/version + project URL. Versions sourced
from uv.lock; licenses hand-curated against each project's upstream
LICENSE file.
3. SPDX headers normalized across the package (~125 files)
Every code/config file under video_ingestion_agent/ now carries the
2-line dual header:
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0
Previously the headers used the 3-line Apache-2.0-only template (107
files) or a few variant formats (Dockerfile, docs/conf.py, scripts/
run_osmo.py docstring variant — 9 files), and 9 files (osmo_workflows
yamls and reconstruction_interface __init__.py files) had no header
at all. Year normalized to 2026 throughout.
4. Contribution policy (`CONTRIBUTING.md`, `.github/pull_request_template.md`,
`README.md`)
Add a Sign-off section to CONTRIBUTING.md referencing the Developer
Certificate of Origin (https://developercertificate.org/) and
requiring `git commit -s` on every commit. Add a top-level
pull_request_template.md with a Checklist including the DCO
reminder. Expand the README License section to mention the dual
license + the NOTICE file, and add a brief Contributing section
pointing at CONTRIBUTING.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Wei Liu <liuw@nvidia.com>
… Your Work" wording Replace the prior compact Sign-off section with the canonical "Signing Your Work" template — bullet-list rationale plus the full verbatim Developer Certificate of Origin v1.1 text — so the policy is self-contained in the repo rather than relying on the external link. Update the anchor references in `.github/pull_request_template.md` and `video_ingestion_agent/README.md` from the old `#sign-off-developer-certificate-of-origin` anchor to the new `#signing-your-work` anchor. No functional change vs. the prior compact section — same DCO, same git commit -s requirement. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Wei Liu <liuw@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NVIDIA OSRB approval for the dual CC-BY-4.0 + Apache-2.0 release flagged four BEFORE-WE-CAN-CLOSE items. This commit addresses all of them on the video_ingestion_agent surface.
License file (
video_ingestion_agent/LICENSE) Replace the previous 15-line Apache-2.0 short notice with the full CC-BY-4.0-Apache2 Dual License text (~700 lines): NVIDIA preamble + verbatim Creative Commons Attribution 4.0 International + verbatim Apache License 2.0.Third-party notices (
video_ingestion_agent/NOTICE) New file. Enumerates the 41 direct dependencies declared in pyproject.toml (top-level + every optional-dependencies extra). Each entry follows the NVIDIA modstar template: owner + SPDX license + license URL + package name/version + project URL. Versions sourced from uv.lock; licenses hand-curated against each project's upstream LICENSE file.SPDX headers normalized across the package (~125 files) Every code/config file under video_ingestion_agent/ now carries the 2-line dual header: # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0 Previously the headers used the 3-line Apache-2.0-only template (107 files) or a few variant formats (Dockerfile, docs/conf.py, scripts/ run_osmo.py docstring variant — 9 files), and 9 files (osmo_workflows yamls and reconstruction_interface init.py files) had no header at all. Year normalized to 2026 throughout.
Contribution policy (
CONTRIBUTING.md,.github/pull_request_template.md,README.md) Add a Sign-off section to CONTRIBUTING.md referencing the Developer Certificate of Origin (https://developercertificate.org/) and requiringgit commit -son every commit. Add a top-level pull_request_template.md with a Checklist including the DCO reminder. Expand the README License section to mention the dual license + the NOTICE file, and add a brief Contributing section pointing at CONTRIBUTING.md.