ObAct

Minimal, hardware-agnostic repository of the offline observer stage from ObAct.

Given three recorded RGB-D observations, camera intrinsics, object masks, and camera-to-world poses, this package samples candidate next views, reconstructs with an external InstantSplat checkout, scores rendered candidates against a reference image, and returns the selected next camera pose in the original world frame.

InstantSplat receives only the three RGB images. Depth, masks, intrinsics, and poses are used by ObAct for candidate sampling, scoring, and frame alignment.

Scope

Included:

• Offline observer pipeline: three sparse RGB-D views to a next-best-view pose.
• A tiny recorded mug example with segmentation masks.
• Pure Python dataset loading, pose alignment, candidate sampling, and result packaging.
• A thin bridge to the original external InstantSplat repo.
• Optional local pose refinement through the original InstantSplat Gaussian renderer.

Not included:

• ROS, MoveIt, URDFs, robot drivers, live camera topics, or policy execution.
• InstantSplat source, checkpoints, or model weights.
• Segmentation models. Masks are precomputed inputs in this release.

Repository Layout

• obact/: reusable package.
• examples/mug_demo/inputs/: three-view example input bundle.
• examples/mug_demo/config.yaml: public example config.
• examples/mug_demo/outputs/mug_demo/: curated example outputs from a successful run.
• scripts/run_example.py: one-command entrypoint.
• scripts/visualize_reconstruction.py: optional point-cloud visualization.
• tests/: lightweight package tests.

Installation

Install the package:

cd obact_public
python3 -m pip install -e .

For the real InstantSplat-backed example, first set up the original InstantSplat environment, then install ObAct's optional matcher and refinement dependencies into that environment:

python -m pip install -e .[matching,refinement]

The matching extra requests romatch, torch, and torchvision; keep the Torch build compatible with the InstantSplat CUDA extensions in your environment. ObAct uses RoMa dense matching for view ranking, and RoMa downloads model weights on first use. If romatch is unavailable from PyPI in your environment, install it directly:

python -m pip install git+https://github.com/Parskatt/RoMa.git

Keep InstantSplat outside this repo. The example config expects a local checkout path and calls ObAct's bridge into that checkout.

Run The Example

Edit examples/mug_demo/config.yaml and set:

"instantsplat_root": "/absolute/path/to/InstantSplat"

Then run:

python scripts/run_example.py --config examples/mug_demo/config.yaml

For machine-specific paths, copy the config to examples/mug_demo/config.local.json. That filename is gitignored so private paths do not leak into the public repo.

Input Contract

Each sequence contains exactly three observations:

• rgb_001.png, rgb_002.png, rgb_003.png
• depth_001.npy, depth_002.npy, depth_003.npy
• mask_001.png, mask_002.png, mask_003.png
• pose_001.npy, pose_002.npy, pose_003.npy
• intrinsics.npy

Reference files live next to the sequence:

• reference_rgb.png
• reference_mask.png

All poses are 4x4 camera-to-world transforms. For custom datasets, override rgb_files, depth_files, mask_files, pose_files, or reconstruction_rgb_files in the config.

Outputs

The configured output directory contains:

• best_view_pose.npy: selected next view in the input world frame.
• render_preview.png: side-by-side selection preview.

When backend.refinement.enabled is true, the pipeline also writes:

• refined_view_pose.npy
• refinement_loss.png
• refinement.mp4

The default refinement starts from the best coarse candidate only (top_k: 1) and locally optimizes the camera pose with a bounded SE(3) update through the original InstantSplat Gaussian renderer. It is a local refinement step, not a global optimality guarantee.

Backend Notes

ObAct does not vendor InstantSplat. Instead it:

1. Creates a temporary workspace with the three reconstruction RGB images.
2. Runs external reconstruction through the configured backend command.
3. Aligns InstantSplat poses back to the input world frame.
4. Renders and scores candidate views with RoMa dense matching inside the reference mask.
5. Optionally refines the selected pose with differentiable rendering.

The public pipeline is intentionally hardware-agnostic at the interface level: any recorded RGB-D source can be used as long as it provides the files above.

Visualization

After running the example, you can visualize the fused RGB-D observations and selected camera frustums:

python -m pip install -e .[viz]
python scripts/visualize_reconstruction.py

This writes fused_reconstruction.ply to the example output directory.

Citation

If you find this repository useful for your project, please consider citing us!

@article{wang2025observer,
  title={Observer Actor: Active Vision Imitation Learning with Sparse View Gaussian Splatting},
  author={Wang, Yilong and Qian, Cheng and Fan, Ruomeng and Johns, Edward},
  journal={arXiv preprint arXiv:2511.18140},
  year={2025}
}