feat: add MujocoCamera module for sim-based perception

[ruthwikdasyam]

Problem

Sim-based manipulation requires a camera for perception
This blocks testing and developing the full perception, planning, execution pipeline in simulation

Closes DIM-733

Solution

• Added MujocoCamera a drop-in replacement for RealSenseCamera that renders RGB + depth from a named camera in the MuJoCo sim loop.
• Shared engine registry: get_or_create_engine() keyed by resolved MJCF path lets the sim adapter and camera independently resolve the same engine without explicit wiring
• Gripper fix: Corrected inverted linear interpolation in sim_manip_interface.py
• PointCloud2 pickle fix: Strip bounding_box_dimensions cached property before pickling (was missing, could cause stale cache after unpickling)
• Sequence[str] → list[str] in RobotModelConfig.tf_extra_links for Pydantic v2 compatibility

Breaking Changes

None

Important to Note: - SimMujocoAdapter now uses the engine registry - two adapters pointing to the same MJCF share state (previously each got an independent engine). This is intentional for the camera sharing use case.

How to Test

# Unit tests
pytest dimos/simulation/manipulators/test_mujoco_camera.py -v -m mujoco

# Full sim-agent blueprint
dimos run xarm-perception-sim-agent
humancli
# Verify: RGB/depth frames render in rerun, able to command through humancli, observe and manipulate

Contributor License Agreement

• I have read and approved the CLA.

Note:

Added rerun in sim blueprint for a reason to view rgb and depth camera views
Can remove later, if not a need

Object detection is not great, should be improved later (if any optimzes needed for sim-cam)

[greptile-apps]

Greptile Summary

This PR adds MujocoCamera, a drop-in replacement for RealSenseCamera that renders RGB + depth frames from a named MuJoCo camera inside the existing sim loop. It introduces a shared engine registry (get_or_create_engine) so the SimMujocoAdapter and MujocoCamera can independently resolve the same MujocoEngine instance without explicit wiring. Additional fixes include corrected gripper interpolation in sim_manip_interface.py, a PointCloud2 pickle fix for bounding_box_dimensions, and a Pydantic v2 compatibility fix in RobotModelConfig.

Key findings:

• P1 — set_engine() bypass silently breaks frame delivery: When set_engine(engine) is called before start(), self._engine is already populated, so the if self._engine is None guard in start() skips the get_or_create_engine() call. This means the CameraConfig is never appended to engine._camera_configs, no renderer is ever created in the sim thread, and read_camera() returns None forever. The publish loop spins without ever emitting a frame.
• P2 — _publish_loop doesn't exit on engine disconnect: After the initial connected-gate, the loop has no check for engine.connected. If the shared engine is stopped (e.g., adapter disconnects), the camera thread hangs until stop() is called explicitly.
• P2 — Lazy from pathlib import Path import inside start(): Path is not imported at module level in camera.py but is imported inside the start() method. It should be a top-level import.
• P2 — Test helper imported from another test file: _patch_mujoco_engine is pulled in from test_sim_adapter; shared test utilities should live in conftest.py or a dedicated helpers module.
• The thread-safety concerns from the previous round (race on _camera_configs, headless mismatch warning) have been addressed.

Confidence Score: 4/5

Safe to merge once the set_engine() / camera-registration bug is fixed; all other findings are P2 style or robustness improvements.

One confirmed P1 bug: using set_engine() leaves the camera never registered with the engine, so the advertised alternative initialization path produces no frames. The primary registry-based path (via address) works correctly. Fixing the P1 before merge is recommended.

dimos/simulation/manipulators/camera.py — set_engine() + start() interaction needs the camera-config registration fix.

Important Files Changed

Filename	Overview
dimos/simulation/manipulators/camera.py	New MujocoCamera module — drop-in replacement for RealSenseCamera; P1 bug: set_engine() bypasses camera config registration so frames never arrive; P2: lazy Path import and missing engine-disconnect detection in publish loop.
dimos/simulation/engines/mujoco_engine.py	Added shared engine registry and camera rendering infrastructure; thread-safety (camera_lock) correctly applied; headless mismatch warning added; previously reported race condition and redundant import are resolved.
dimos/hardware/manipulators/sim/adapter.py	Updated to use shared get_or_create_engine() registry; optional engine injection parameter added; logic is clean and correct.
dimos/manipulation/blueprints.py	New xarm_perception_sim blueprint wiring MujocoCamera + SimMujocoAdapter via shared engine registry; pre_grasp_offset and home_joints made configurable.
dimos/simulation/manipulators/test_mujoco_camera.py	Good test coverage for engine registry, camera intrinsics, lifecycle, and TF publishing; test helper imported from another test module (minor coupling concern).

Sequence Diagram

sequenceDiagram
    participant BP as Blueprint
    participant Adapter as SimMujocoAdapter
    participant Camera as MujocoCamera
    participant Registry as engine registry
    participant Engine as MujocoEngine (sim thread)

    BP->>Adapter: connect()
    Adapter->>Registry: get_or_create_engine(path, headless)
    Registry-->>Adapter: engine (new)
    Adapter->>Engine: engine.connect() → starts sim thread

    BP->>Camera: start()
    Camera->>Registry: get_or_create_engine(path, cameras=[CameraConfig])
    Registry->>Engine: merge CameraConfig into _camera_configs (via _camera_lock)
    Registry-->>Camera: engine (existing, shared)

    loop sim tick (sim thread)
        Engine->>Engine: _init_new_cameras() → create Renderer
        Engine->>Engine: _render_cameras() → write CameraFrame
    end

    Camera->>Engine: read_camera(name) → CameraFrame
    Camera->>Camera: publish color_image, depth_image, TF

Loading

_{Reviews (2): Last reviewed commit: "xarm7 updated mujoco env" | Re-trigger Greptile}