diff --git a/README.md b/README.md index 59b8cd72..8e2e5a81 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,42 @@ -# Video to Data +# Video to Data (V2D) -Monorepo for **Video to Data (V2D)** — an end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data. +> An end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data. -Documentation is available at [nvidia-isaac.github.io/video_to_data](https://nvidia-isaac.github.io/video_to_data/). +**[Documentation](https://nvidia-isaac.github.io/video_to_data/)** · **[Robotic Grounding Project Page](https://nvidia-isaac.github.io/video_to_data/chord/)** · **[Robotic Grounding Tech Report](https://nvidia-isaac.github.io/video_to_data/chord/chord.pdf)** -## End-to-End Workflow +![Video to Data pipeline — from human demonstration video through ingestion, reconstruction, and robotic grounding in Isaac Lab to a physics-grounded policy, dataset, and real-robot deployment](docs/figures/v2d_overview.png) -``` - ┌───────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ ┌────────────────────────────┐ - │ Human demo │ → │ 1. Video Ingestion │ → │ 2. Reconstruction │ → │ 3. Robotic Grounding │ - │ video / rosbag│ │ Agent │ │ depth · masks · │ │ retargeting → Isaac Lab │ - │ │ │ action segments · │ │ meshes · 6D pose · │ │ RL training │ - │ │ │ entity graph · │ │ body pose │ │ Coming soon │ - │ │ │ visual embeddings │ │ │ │ │ - └───────────────┘ └──────────────────────┘ └──────────────────────┘ └────────────────────────────┘ - video_ingestion_agent/ reconstruction/ robotic_grounding/ -``` +--- + +## Contents + +- [Overview](#overview) +- [Demos](#demos) +- [Packages](#packages) +- [Prerequisites](#prerequisites) +- [Quickstart](#quickstart) + - [Video Ingestion Agent](#video-ingestion-agent-video--queryable-action-database) + - [Reconstruction](#reconstruction-video--reconstructed-trajectory-and-simulation-assets) + - [Robotic Grounding](#robotic-grounding-reconstructed-trajectory--rl-policy-and-dataset) +- [Design philosophy](#design-philosophy) +- [Contributing](#contributing) + +--- + +## Overview + +Video to Data (V2D) turns raw human demonstrations into robot-ready training data through three composable stages. Each stage runs independently and writes its artifacts to disk, so you can stop, inspect, cache, and recompose the pipeline at any boundary. 1. **Video Ingestion Agent** — a LangGraph-driven agentic workflow that segments demonstration videos into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The result is a queryable action database (`graph.db` + `vector.db`) that lets downstream stages select which clips to process via natural-language retrieval, instead of brute-forcing the full video. 2. **Reconstruction** — containerized vision modules turn the selected RGB (or stereo) clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. Multi-view pipelines (`run_mv_hoi_reconstruction`, `run_mv_calibration`) orchestrate the full reconstruction from a rosbag. 3. **Robotic Grounding** — human motion is retargeted onto the target robot embodiment (Sharpa or G1), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RL to produce deployable policies. +## Demos + +The pipeline in action — from a raw human demonstration, to grounded policies trained in Isaac Lab, to deployment on a physical robot. + +Raw human demonstration Grounded robot policies in Isaac Lab Deploy to real robot + ## Packages | Package | Role | Runtime | @@ -62,23 +78,17 @@ python scripts/run_webapp.py See [video_ingestion_agent/README.md](video_ingestion_agent/README.md) for hardware requirements, the full extras list, the verify/refine loop, and batch-ingestion across multiple GPUs. Pre-publication TODOs are tracked in [video_ingestion_agent/docs/release_readiness.md](video_ingestion_agent/docs/release_readiness.md). -### Reconstruction - -We include a variety of algorithms and pipelines that are useful for human-object reconstruction. These packages are contained in the [reconstruction](reconstruction) subfolder. - -For the initial release, we provide an example pipeline for ego-centric hand-object reconstruction. - -To get started, please follow the following the instructions [here](reconstruction/docs/ego_e2e_setup.md). +### Reconstruction (video → reconstructed trajectory and simulation assets) -> Please note: The reconstruction subfolder contains a wide variety of packages, many of which are partially tested or in development. You may find these packages useful, but please note they are subject to change. The above ego-centric pipeline has been tested, and is officially included as part of the initial video to data release. If there is a package you would like to see supported, or you have any feedback, please let us know by opening an issue on GitHub! +The [reconstruction](reconstruction) subfolder contains a variety of algorithms and pipelines for human-object reconstruction. For the initial release, we provide an example pipeline for ego-centric hand-object reconstruction — follow the setup instructions [here](reconstruction/docs/ego_e2e_setup.md). -### Robotic Grounding (data → RL policy) +> **Note:** The reconstruction subfolder contains a wide variety of packages, many of which are partially tested or in development. You may find these packages useful, but please note they are subject to change. The ego-centric pipeline above has been tested and is officially included as part of the initial Video to Data release. If there is a package you would like to see supported, or you have any feedback, please open an issue on GitHub. -The Robotic Grounding stage (motion retargeting + Isaac Lab RL training) will be publically available in a later release. See [robotic_grounding/README.md](robotic_grounding/README.md) for an overview. +### Robotic Grounding (reconstructed trajectory → RL policy and dataset) -The technical report and project website are available on the project page: [Webpage](https://nvidia-isaac.github.io/video_to_data/chord/). +The Robotic Grounding stage (motion retargeting + Isaac Lab RL training) will be publicly available in a later release. See [robotic_grounding/README.md](robotic_grounding/README.md) for an overview. The [tech report](https://nvidia-isaac.github.io/video_to_data/chord/chord.pdf) and [project page](https://nvidia-isaac.github.io/video_to_data/chord/) are available. -## Design Philosophy +## Design philosophy - **Host orchestration, containerized inference.** The host runs thin Python wrappers that `docker run` each module; all ML dependencies live inside their respective images. No CUDA or PyTorch is ever installed on the host. - **Typed contracts between packages.** Modules communicate through strongly-typed dataclasses in [`v2d_common`](reconstruction/modules/v2d_common/) (`DepthImage`, `CameraIntrinsics`, `Transform3d`, `BoundingBox`, `Mask`) — never raw arrays across package boundaries. diff --git a/docs/figures/bg-cover.jpg b/docs/figures/bg-cover.jpg new file mode 100644 index 00000000..e65193ae Binary files /dev/null and b/docs/figures/bg-cover.jpg differ diff --git a/docs/figures/human.gif b/docs/figures/human.gif new file mode 100644 index 00000000..e655b665 Binary files /dev/null and b/docs/figures/human.gif differ diff --git a/docs/figures/logo.png b/docs/figures/logo.png new file mode 100644 index 00000000..a647a759 Binary files /dev/null and b/docs/figures/logo.png differ diff --git a/docs/figures/real.gif b/docs/figures/real.gif new file mode 100644 index 00000000..347e88a8 Binary files /dev/null and b/docs/figures/real.gif differ diff --git a/docs/figures/sim.gif b/docs/figures/sim.gif new file mode 100644 index 00000000..f9d9b813 Binary files /dev/null and b/docs/figures/sim.gif differ diff --git a/docs/figures/v2d_overview.png b/docs/figures/v2d_overview.png new file mode 100644 index 00000000..d1579391 Binary files /dev/null and b/docs/figures/v2d_overview.png differ diff --git a/docs/index.html b/docs/index.html index 73ca9ebe..61109afb 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1,526 +1,227 @@ - + - - - Video to Data — NVIDIA Isaac - - - - - + + +Video to Data — NVIDIA Isaac + + + + +
-
- - - - Video to Data / Isaac + + Video to Data / Isaac - - - - - - - + + +
-
+
+ + +
+
+
+ + NVIDIA Isaac +
+

Video → Data

+

An end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data.

+ +
+
+
-
-

NVIDIA · Isaac

-

Video → Data

-

- An end-to-end pipeline that converts human demonstration videos into - simulation-ready assets and physics-grounded robot training data. -

-
+
-

End-to-end workflow

-
    -
  1. - 1 -

    Video Ingestion Agent

    -

    LangGraph workflow that segments demos into action clips, extracts an entity-relation scene graph, and stores SigLIP-2 frame embeddings.

    - video_ingestion_agent/ -
    2. - -
  2. - 2 -

    Reconstruction

    -

    Containerized vision modules turn selected clips into per-frame depth, masks, textured meshes, 6-DoF object poses, and parametric human hand and body models.

    - reconstruction/ -
    3. - -
  3. - 3 -

    Robotic Grounding

    -

    Retarget human motion onto the target embodiment, then drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies.

    - robotic_grounding/ -
    4. -
+
+ + Pipeline +
+

From human video to robot data

+

Raw human video becomes video segments, reconstructed trajectories, simulation assets, grounded policies, and robot data.

+ + Video to Data pipeline overview — from human demonstration video through ingestion, reconstruction, human-object trajectory and simulation environments, robotic grounding, and data augmentation in Isaac Lab to a physics-grounded dataset, foundation models, and real-robot deployment + +
+
+ 1 +

Video Ingestion Agent

+

LangGraph workflow that segments demos into action clips, extracts an entity-relation scene graph, and stores SigLIP-2 frame embeddings.

+
video_ingestion_agent/
+
+
+ 2 +

Reconstruction

+

Containerized vision modules turn selected clips into per-frame depth, masks, textured meshes, 6-DoF object poses, and parametric human hand and body models.

+
reconstruction/
+
+
+ 3 +

Robotic Grounding

+

Retarget human motion onto the target embodiment, then drive Isaac Lab environments trained with RL to produce deployable policies.

+
robotic_grounding/
+
+
-
-

Packages

- +
+
+ + Demos +
+

See it run, stage by stage

+
+
+ Raw human demonstration clip +
Raw human demonstration
+
+
+ Grounded robot policies in Isaac Lab +
Grounded robot policies in Isaac Lab
+
+
+ Deploy to real robot +
Deploy to real robot
+
+
-
-

Get started

-
- - Read the quickstart - +
+
+ + Packages +
+

Explore the toolkit

+ +
+ +
+
+
+
+

Get started

+

Clone the repo, follow the quickstart, and turn your first demonstration video into training data.

+
+ +
-