Virtual leader teleop + episode recording for the SO-101

[MJohnson459]

Teleoperation with no leader arm, and the recordings that make it worth doing.
Branch arm/virtual-leader-teleop (committed, not pushed). Design and workflow:
mote_arm/TELEOP.md.

What was chosen and why. Of the three candidate shapes, this is the second: a
virtual leader publishing joint targets. LeRobot's own keyboard teleop was the
cheapest option on paper but means running LeRobot's robot class and bus driver
on the Pi, which is exactly what task 163's stack decision exists to avoid.
LeRobot is where the dataset goes, not where the arm is driven from — that
split is the design. IK jog was out for v1 and stays out.

Teleop. virtual_leader (keyboard, pixi run arm-teleop) holds a leader
pose and publishes leader/joint_states; arm_mirror turns it into arm/goal.
The frontend is deliberately replaceable — the mirror's whole contract is
leader/joint_states plus a latched teleop/estop, so a slider GUI or a gamepad
is a drop-in with no code change. Every safety rule is decided in teleop.py and
nowhere else, so all of it is unit-tested without a bus: soft-limit clamping, a
0.5 rad/s rate limit (a leader that jumps becomes a ramp, never a lunge), the
deadman, the panic latch, and re-seeding from the measured pose on resume so a
pause cannot bank up motion. The deadman is the leader's liveness — a released
key, a closed window and a dropped SSH session all arrive as "no fresh pose" —
and on that transition the mirror issues one goal at the arm's present position,
stopping it there rather than letting it coast to the setpoint it was still
travelling towards.

Episodes. episode_record samples joint_states (observation), arm/goal
(action — the mirror's output, because a policy replaces whatever produces goals)
and /image_raw/compressed at 20 Hz into a capture under $MOTE_HOME/episodes:
JSON lines plus the compressed frames byte-for-byte, standard library only, since
the Pi carries no parquet or ffmpeg. tools/lerobot_export.py converts a capture
to a real LeRobotDataset off-board in a new linux-64 lerobot pixi env, through
LeRobot's own API rather than emitting files — the format already moved once
(v2.1 -> v3.0) and a hand-rolled writer would be wrong the next time. It resamples
onto the exact 1/fps grid first (LeRobot derives timestamps from the frame index,
so a slipped capture would export as if its timing had been perfect) and loads
the result back to verify. episode_replay reads the capture, not the dataset,
so replay needs nothing off-board; it approaches the first pose, replays at a
quarter speed, and stops on sustained lag.

Verified (all headless, against mock_arm — the driver's exact interface with
no bus, plus a synthetic camera encoded with zlib and struct):

• pixi run arm-teleop-test runs leader -> mirror -> follower -> record -> replay
  -> export plan end to end. 220 frames over 11.0 s, 0 dropped ticks; replay
  finished within 0.0000 rad of the last action at 0.010 rad steady lag.
• Export produced a v3.0 dataset (aggregated parquet + MP4 shards), which loads
  back through LeRobotDataset with the right shapes and reads in LeRobot's own
  lerobot-dataset-viz (wrote a 780 KB .rrd headless).
• The state-only path forced by the camera/arm clash (GitHub The camera doesn't fit when the SO-101 arm is attached #2) exports and
  verifies identically.
• 21 new tests over the safety rules and the capture format; 447 tests pass
  overall, lint clean.

Not verified: anything on the arm itself. The three safety observations (the
arm stopping at a limit, halting on a released key, going limp on panic) and a
real arm retracing an episode need a human at the bench. That is BENCH.md step 8
and pixi run arm-bench-teleop, which prompts for those observations and writes a
report — nothing is recorded as passing that the operator did not say they saw.

Two bugs found by running it, both fixed: two publishers on arm/goal fight
(caught by the stall guard before the loop script learned to stop the leader
first), and destroying a node while rclpy.spin() still holds it aborts the
process — cli.shutdown now joins the spin thread first. cli.parse also cuts
the --ros-args block out and parses strictly, because parse_known_args
silently discards a mistyped --max-travel.

Incidental: lag supervision moved to motion.py, shared with arm-pose go
rather than reimplemented; poses.py dropped its duplicate of mote_home.

Follow-up filed as #259 (move jog/arm_pose onto the same shutdown/parsing,
which wants a bench re-run of BENCH.md steps 5-7).

[claude]

CLAUDE.md compliance: main() hand-rolls its own rclpy.init()/spin()/destroy_node()/shutdown() teardown instead of going through cli.py, even though it already uses cli.parse(parser) above.

Per CLAUDE.md's mote_arm section:

"Every arm CLI exits and parses through cli.py, because both properties fail silently when hand-rolled. cli.shutdown(node, spinner) shuts the context down, joins the spin thread, and only then destroys the node... This is not a rare race, so a new arm CLI must not hand-roll the teardown."

Every other new CLI in this PR (virtual_leader.py, mirror.py, episode_record.py, episode_replay.py) uses cli.spin_background(node) + cli.shutdown(node, spinner). This file diverges — and the # noqa: BLE001 - the context is gone, see cli.spin_background comment on line 217 even references the function this file doesn't call, suggesting the exception handling was copied from a spin_background-based file without adopting the pattern itself.

Suggest switching to spinner = cli.spin_background(node) / cli.shutdown(node, spinner) for consistency with the rest of the PR and the documented rule.

[MJohnson459]

Changed in 0fa5c1b — mock_arm now spins on a worker thread and exits through cli.shutdown. The stale # see cli.spin_background comment pointing at a function the file never called is gone with it; that was the real defect here, and it is exactly the reading you drew from it.

One correction to the severity, because it changes what the rule is buying. This was not a latent abort. I measured both forms, SIGINT after 4 s, three runs each:

	exit	terminate called
before	0, 0, 0	0
after	0, 0, 0	0

The reason is that mock_arm spun in the main thread, so spin() has already returned by the time the finally block destroys the node — there is no executor still holding it, which is the thing cli.shutdown exists to prevent. Task 259 measured the opposite for jog and arm-pose list (exit 134, 3/3), and those two spin on a background thread. So the documented "not a rare race" is about the background-spin pattern specifically.

Worth having the change anyway: CLAUDE.md's rule is stated for every arm CLI without qualification, one teardown is better than two, and the main thread now has somewhere for SIGINT to land rather than relying on where spin() happened to be running.

[claude]

Orphaned process leak: cleanup()/stop() (line 53) both do a plain kill "$pid" on the PID of a backgrounded ros2 run ... invocation (see the call sites at lines 62-63/67).

Per CLAUDE.md's "Stray ROS processes" section:

"ros2 run is a wrapper that Popens the real executable and handles no SIGTERM (only KeyboardInterrupt, assuming a terminal Ctrl-C reached the whole group), so proc.terminate() on it kills the wrapper and hands the node to init — once per run, on the success path... Every ros2 run spawn goes through it [spawn_reapable/reap_group]."

This script has no setsid, no process-group kill (kill -- -$pid), and no equivalent of spawn_reapable/reap_group — just a bare kill "$pid" on the wrapper's PID. Since this is pixi run arm-teleop-test, meant to be run repeatedly as a pre-bench gate, each run risks leaking a mock_arm, arm_mirror, and virtual_leader process — the exact class of leak pixi run sweep exists to catch.

[MJohnson459]

Fixed in 0fa5c1b, and confirmed by measurement first: arm_mirror and mock_arm both survived a run that exited 0.

Every job is now setsid-ed into a session of its own and torn down by process group and then by session id — the sim smoke test's scoping (run_sim_smoke.sh), and the shell equivalent of spawn_reapable/reap_group. The session id is read back with ps -o sid= and dropped if it matches the script's own, since a setsid that did not detach would otherwise have the teardown kill the script and its caller.

Same run afterwards: 0 surviving nodes, loop still passes.

[claude]

Bug: when a frame declares an image filename that's missing on disk, this crashes with an unhandled FileNotFoundError instead of reporting the diagnostic it just computed.

missing (lines 57-59) correctly detects frames with an absent image file and records a problem for it, but sizes (line 64) filters only on f.image being truthy — not on the file actually existing — so it re-.stat()s the very same missing file and raises, skipping the problems report entirely (and any problems for later episodes).

Suggested change

	if spec.camera is not None:
	missing = [
	f for f in frames if not f.image or not (path / f.image).exists()
	]
	if missing:
	problems.append(
	f"{name}: {len(missing)}/{len(frames)} frames have no image"
	)
	sizes = {(path / f.image).stat().st_size for f in frames if f.image}
	sizes = {
	(path / f.image).stat().st_size
	for f in frames
	if f.image and (path / f.image).exists()
	}

[MJohnson459]

Fixed in 0fa5c1b. Reproduced first with a capture whose last frame names a file that never landed: FileNotFoundError before, PROBLEM episode_000: 1/30 frames have no image after.

Taken slightly differently from the suggestion — the fix is one reading of "this frame has an image", used by both checks, rather than the same predicate written twice:

present = [f for f in frames if f.image and (path / f.image).exists()]
if len(present) < len(frames):
    problems.append(f"{name}: {len(frames) - len(present)}/{len(frames)} frames have no image")
sizes = {(path / f.image).stat().st_size for f in present}
if present and len(sizes) < 2:
    problems.append(f"{name}: every camera frame is byte-identical")

Two copies of the predicate is how the two drifted apart in the first place. The present and guard is the other half of it: with no images at all, sizes is empty and the byte-identical check would otherwise fire as a second, confusing complaint about frames that are already reported missing.