fix(policies/lerobot_async): rename_map renames the camera it declares - #3496
Conversation
The client declared observation.images.<robot cam> and forwarded rename_map to the server, whose prepare_raw_observation resizes each declared image by the checkpoint's image features before the rename step, so a renamed camera was a KeyError and every observation returned no actions. The client now applies an image rename itself: the handshake and the raw observation carry the model's camera name.
Fragment for PR 3496.
cagataycali
left a comment
There was a problem hiding this comment.
The fix is sound and I verified the mechanism against lerobot 0.6.1 rather than the description: PolicyServer sizes every declared image by policy_image_features (policy_server.py:345) ahead of the rename step, and RenameObservationsProcessorStep.observation passes an absent key through (rename_processor.py:45-48), so the image entry that is still forwarded to the server is a harmless no-op once the client has renamed. Lazy from lerobot... inside the method is this module's existing shape (lines 348, 373, 437), so no concern there.
Two things, both one commit and a plain push:
- The
rename_mapcontract is now described in two places and they disagree - see the thread on_camera_items. - The new test module needs the
importorskipits sibling carries - see the thread on the test.
The fork runs were held at action_required; released this cycle, so CI is running on this head.
…ient-side image rename; test gates on importorskip (addresses thread policy.py:411 + test:17)
cagataycali
left a comment
There was a problem hiding this comment.
R1 landed in f21c572 and answers both threads exactly: the Args: entry and the docs table row now say which side applies which kind of rename_map entry and why (the server resizes by policy_image_features before its rename step), and the test module gates on pytest.importorskip("lerobot"). Both threads resolved. Superseding the request for changes.
The fork runs on this head were held at action_required and are released. Note for the merge: this head was pushed under the reviewing account, so require_last_push_approval needs a second approver; this approval clears the review decision but cannot satisfy that rule on its own.
cagataycali
left a comment
There was a problem hiding this comment.
Verified the mechanism against lerobot 0.6.2 rather than taking the description on trust, and it holds as stated:
async_inference/policy_server.py:340-351runsraw_observation_to_observation(step 1) beforeself.preprocessor(observation)(step 2), and the rename step lives in the preprocessor (policy_server.py:162). Resize precedes rename.async_inference/helpers.py:166subscriptspolicy_image_features[key]directly on the declared key, so a camera declared under the robot's name is aKeyErrorthere, not a rename.RenameObservationsProcessorStep.observationiterates the observation's own keys, so the camera entry still forwarded to the server is a harmless no-op once the client has applied it.
Both cells fail on the merge base 0759ececd with the pre-fix keys (observation.images.front declared, front on the wire); tests/policies/lerobot_async/ is 28 passed on this head. ruff check, ruff format --check and mypy strands_robots tests tests_integ clean (Success, 1958 files).
The Args: entry and the docs row now say which side applies which kind of entry and why, and the module-level importorskip("lerobot") matches the sibling roundtrip test. Both threads addressed.
|
Green and ready: rollup SUCCESS on |
yinsong1986
left a comment
There was a problem hiding this comment.
Summary
Fixes rename_map camera entries in the lerobot_async provider by applying the image rename client-side: _camera_items now returns the model's feature name as the wire key (via rename_map lookup keyed on observation.images.<key>), so both the SendPolicyInstructions handshake and the raw observation carry the checkpoint's own camera name. This matches the server's actual pipeline, where prepare_raw_observation resizes every declared image by policy_image_features before the RenameObservationsProcessorStep runs, turning a robot-named camera into a server-side KeyError. State entries are still forwarded to the server's rename_map as before. The docstring, docs table, and changelog were updated in R1 to describe the client-side/server-side split, and the pin test gates on pytest.importorskip("lerobot").
What's good
- Pin tests assert on outputs (
_build_lerobot_featureskeys,_to_raw_observationkeys) and fail on pre-fix code, per AGENTS.md testing conventions. - Scope discipline: the diff touches exactly the rename path plus its docs and changelog; the forwarded server-side map entry was verified a harmless no-op rather than speculatively removed.
- Docstrings and the docs table were kept in sync with the new semantics in the same PR.
What
lerobot_asyncapplies arename_mapcamera entry on the client, so the handshake and the raw observation carry the checkpoint's own camera name.Why
lerobot's
PolicyServerresizes every declaredobservation.images.<key>bypolicy_image_features[key]inprepare_raw_observation, before theRenameObservationsProcessorStepruns (0.6.1 and main). A camera declared under the robot's name is aKeyErrorthere, and the client raised "server returned no actions". On main:Against a real
policy_serveron CPU (smolvla_base,front -> camera1), the same call now returns a 10-action chunk.Tests
tests/policies/lerobot_async/test_rename_map_renames_the_camera_the_server_resizes.py; ruff, mypy clean on touched files.Review rounds
Args:entry + docs table still said "forwarded to the server" for image keys (now client-side); test missingimportorskip("lerobot")f21c572