Decouple camera observable refresh from video recording

[RuiyangSi]

Fixes #25.

Problem

--record-video changed task outcomes, not just whether media was saved.

_do_robosuite_step gated robosuite's observable refresh on _record_frames:

need_render = (self._record_frames and self._sim_step_count % self._subsample_rate == 0) \
              or hasattr(self, "viser_server")
if need_render:
    self.robosuite_env.step(sliced)
else:
    self.robosuite_env.step(sliced, skip_render_images=True)

skip_render_images=True makes robosuite skip every _image / _depth /
_segmentation_instance observable. The affected get_observation() methods
then call _get_observations() without force_update=True, so they return the
cached _current_observed_value instead of invoking the sensor.

With recording off — the default, and what scripts/run_capbench_sweep.py uses —
perception therefore reused the RGB-D captured at reset() for the entire
episode. Programs that re-perceive after moving an object grasped stale poses
and failed deterministically.

Physics was never affected: both branches reach sim.step(). The divergence is
closed-loop (stale perception -> wrong object pose -> wrong IK target).

Measured on cube_restack, after the program relocates the green cube and
re-queries its pose:

	pose changes after motion?	mean |Δx|
video off	0 / 40 paired trials	0 m
video on	40 / 40	0.115 m (max 0.153 m)

Change

• _do_robosuite_step no longer looks at _record_frames; only the viser debug
  view still forces a full step.
• The four affected simulators (cube_lift, cubes, cubes_restack,
  spill_wipe) refresh cameras on demand via _get_observations(force_update=True).
• The same recording-gated branches in robosuite_handover.py are removed for
  consistency. That env already forced updates, so its behaviour is unchanged.

This keeps the original optimization — cameras render only when an observation is
actually requested — while guaranteeing perception sees current state.
_record_frame() is untouched; it uses the read-only sim.render().

Not affected, and deliberately left alone: robosuite_nut_assembly.py and
robosuite_handover.py already pass force_update=True;
robosuite_two_arm_lift.py steps independently of recording; privileged (S1)
APIs read state observables, which skip_render_images never skips.

Verification

cube_restack, 20 trials, avg reward / task completed:

	before	after
--record-video False	0.002 / 0	0.502 / 10
--record-video True	0.454 / 9	0.452 / 9

The two modes now agree, which is the actual correctness criterion here.

Unchanged, as expected (their oracles only perceive before motion):
cube_lifting 0.977 / 19, spill_wipe 1.000 / 20.

tests/test_environments.py gives identical results before and after this
change on my setup (the pre-existing failures there are unrelated —
LIBERO/Isaac deps).

Impact on published numbers

Benchmark cells run with recording off are affected: 4 tasks x the 8
non-privileged tiers. cube_restack is definitively invalidated (its near-zero
vision scores were this bug, not a perception-grounding ceiling); cube_stack
is likely materially affected. cube_lifting and spill_wipe single-turn
oracles are not, though generated multi-turn programs that re-perceive are.

[Copilot]

Pull request overview

Decouples Robosuite camera observation refresh from video recording so perception uses current simulation state.

Changes:

• Refreshes camera observations on demand in four simulators.
• Removes recording-dependent stepping branches.
• Preserves read-only video rendering and Viser debugging.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Summary
capx/envs/simulators/robosuite_spill_wipe.py	Forces fresh camera observations.
capx/envs/simulators/robosuite_handover.py	Removes recording-gated render branches.
capx/envs/simulators/robosuite_cubes.py	Forces fresh camera observations.
capx/envs/simulators/robosuite_cubes_restack.py	Forces fresh camera observations.
capx/envs/simulators/robosuite_cube_lift.py	Forces fresh camera observations.
capx/envs/simulators/robosuite_base.py	Decouples simulation stepping from recording.

Suppressed comments (1)

capx/envs/simulators/robosuite_base.py:145

• When viser_server is present, this normal step() already refreshes the camera observables, and _update_viser_server() then immediately calls get_observation(), which now force-updates them again. Each debug update therefore renders the RGB-D/segmentation sensors twice and discards the first result; keep the step's image skip (as the handover paths do) or pass the step observation through to the viewer.

        if hasattr(self, "viser_server"):
            self.robosuite_env.step(sliced)
        else:
            self.robosuite_env.step(sliced, skip_render_images=True)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.