Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
timeout-minutes: 20
# sphinx-multiversion runs `git worktree add` for each version, which
# materializes the full repo at that SHA and triggers Git LFS smudging
# on sibling-package assets (parquet / pkl / npz under robotic_grounding/).
# on large sibling-package LFS assets (parquet / pkl / npz).
# We never need those file contents to build docs -- skipping smudging
# keeps them as LFS pointer files in the worktree.
env:
Expand Down
111 changes: 0 additions & 111 deletions .github/workflows/robotic_grounding_ci.yml

This file was deleted.

31 changes: 6 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,31 +8,30 @@ Monorepo for **Video to Data (V2D)** — an end-to-end pipeline that converts hu
┌───────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ ┌────────────────────────────┐
│ Human demo │ → │ 1. Video Ingestion │ → │ 2. Reconstruction │ → │ 3. Robotic Grounding │
│ video / rosbag│ │ Agent │ │ depth · masks · │ │ retargeting → Isaac Lab │
│ │ │ action segments · │ │ meshes · 6D pose · │ │ RL training (RSL-RL PPO)
│ │ │ entity graph · │ │ SMPL body │ │
│ │ │ action segments · │ │ meshes · 6D pose · │ │ RL training
│ │ │ entity graph · │ │ body pose │ │ Coming soon
│ │ │ visual embeddings │ │ │ │ │
└───────────────┘ └──────────────────────┘ └──────────────────────┘ └────────────────────────────┘
video_ingestion_agent/ reconstruction/ robotic_grounding/
```

1. **Video Ingestion Agent** — a LangGraph-driven agentic workflow that segments demonstration videos into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The result is a queryable action database (`graph.db` + `vector.db`) that lets downstream stages select which clips to process via natural-language retrieval, instead of brute-forcing the full video.
2. **Reconstruction** — containerized vision modules turn the selected RGB (or stereo) clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. Multi-view pipelines (`run_mv_hoi_reconstruction`, `run_mv_calibration`) orchestrate the full reconstruction from a rosbag.
3. **Robotic Grounding** — human motion (e.g. Arctic) is retargeted onto the target robot embodiment (Sharpa), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies.
3. **Robotic Grounding** — human motion is retargeted onto the target robot embodiment (Sharpa or G1), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RL to produce deployable policies.

## Packages

| Package | Role | Runtime |
|---|---|---|
| [`video_ingestion_agent/`](video_ingestion_agent/) | Video → action segments + entity scene graph + frame embeddings. LangGraph pipeline (segment → verify/refine → entity graph → embeddings) plus an EGAgent-style natural-language retrieval agent and an optional Gradio UI. | Python venv + vLLM server |
| [`reconstruction/`](reconstruction/) | Video → depth, masks, meshes, 6D poses, human body. 18 containerized modules + multi-view pipelines. | Docker (per-module images) |
| [`robotic_grounding/`](robotic_grounding/) | RL training on NVIDIA Isaac Lab 2.3.1 with RSL-RL (PPO); motion retargeting utilities. | Docker (`nvcr.io/nvstaging/isaac-amr`) |
| [`robotic_grounding/`](robotic_grounding/) | RL training on NVIDIA Isaac Lab with PPO; motion retargeting utilities. **Code will be published in a later release.** | Coming soon |

## Prerequisites

- Docker with GPU support ([install](https://docs.docker.com/engine/install/ubuntu/))
- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)
- Python 3.10+
- NVIDIA driver 580.126.09 / CUDA 13.0 recommended (for `robotic_grounding`)

## Quickstart

Expand Down Expand Up @@ -95,27 +94,9 @@ See [reconstruction/README.md](reconstruction/README.md) for the complete module

### Robotic Grounding (data → RL policy)

```bash
cd robotic_grounding

# One-time host setup (git-lfs, pre-commit)
bash workflow/setup_deps.sh

# Build + enter the Isaac Lab container
./workflow/run.sh build [version]
./workflow/run.sh start [version] [gpu_id]

# Inside the container — train a policy
python scripts/rsl_rl/train.py --task Sharpa-V2P-v0
```

See [robotic_grounding/README.md](robotic_grounding/README.md) for retargeting, debug environments, and task definitions.

### Visualizer (retargeting gallery)

Browse retargeted sequences as 3D animations at **http://10.111.83.14:8080/**
The Robotic Grounding stage (motion retargeting + Isaac Lab RL training) will be publically available in a later release. See [robotic_grounding/README.md](robotic_grounding/README.md) for an overview.

See [robotic_grounding/README.md#visualizer](robotic_grounding/README.md#visualizer) for setup instructions.
The technical report and project website are available on the project page: [Webpage](https://nvidia-isaac.github.io/video_to_data/chord/).

## Design Philosophy

Expand Down
3 changes: 0 additions & 3 deletions reconstruction/modules/v2d_task_library_loader/.gitignore

This file was deleted.

36 changes: 0 additions & 36 deletions reconstruction/modules/v2d_task_library_loader/docker/Dockerfile

This file was deleted.

This file was deleted.

This file was deleted.

57 changes: 0 additions & 57 deletions reconstruction/modules/v2d_task_library_loader/docker/build.py

This file was deleted.

This file was deleted.

Loading
Loading