Context capture for physical AI. A pure-Python hardware abstraction layer (HAL) that connects physical devices -- including Reachy Mini -- to the OpenClaw agent runtime. Robots, wearables, and edge compute in one open source SDK.
Most embodied AI frameworks are agent-to-device: the agent issues commands and the hardware executes them. The OpenClaw Embodiment SDK flips the direction.
Device-to-agent: Hardware events capture context and trigger AI responses. The device notices; the agent thinks; the device acts. Actuation is a response path, not the primary control flow.
Compare:
| Framework | Direction | Paradigm |
|---|---|---|
| OM1 (2,668 stars) | Agent → Device | Command-control |
| OpenClaw Embodiment SDK | Device → Agent | Context capture |
No ROS dependency. No proprietary lock-in. Six device profiles ship out of the box. One HAL interface covers robots, AR glasses, edge cameras, and single-board computers.
┌─────────────────────────────────────────────────────────┐
│ Physical Hardware │
│ IMU · Camera · Microphone · Display · Actuator · Power │
└────────────────────────┬────────────────────────────────┘
│ HAL ABCs (openclaw_embodiment.hal)
┌────────────────────────▼────────────────────────────────┐
│ TriggerDetector (visual) + AudioTriggerDetector │
│ IDLE → SACCADE → FIXATION → CAPTURE │
│ TriggerArbiter │
│ (FIRST_WINS | AUDIO_PRIORITY | HIGHEST_CONFIDENCE) │
└────────────────────────┬────────────────────────────────┘
│ TriggerEvent + HeartbeatWake
┌────────────────────────▼────────────────────────────────┐
│ TransportHal: BLE · HTTP · LocalMLX · Attachment │
│ (latency-aware: BLE 50ms | HTTP 10ms | MLX 5ms) │
└────────────────────────┬────────────────────────────────┘
│
┌────────────────────────▼────────────────────────────────┐
│ OpenClaw Agent Runtime │
│ Memory · Skills · Multi-agent · Cron │
└────────────────────────┬────────────────────────────────┘
│ AgentResponse (onAgentEvent)
┌────────────────────────▼────────────────────────────────┐
│ AgentResponseListener (Gate 4 -- bidirectional) │
│ AudioOutputHal.speak() · DisplayHal.show_card() │
│ ActuatorHal.execute() · StatusIndicatorHal.pulse() │
└─────────────────────────────────────────────────────────┘
Gate 4 complete: The full device → agent → device loop is operational. Reachy hears, sees, and responds.
| Device | Type | Transport | Status | Profile Name |
|---|---|---|---|---|
| Reachy Mini Lite (Pollen Robotics) | Robot | HTTP | Tested on CM5 | reachy-mini |
| Reachy Mini Wireless (Pollen Robotics) | Robot (built-in IMU + RPi 4) | HTTP | In Progress | reachy-mini-wireless |
| Reachy 2 (Pollen Robotics) | Full humanoid (7-DOF arms × 2, stereo cameras) | HTTP | In Progress | reachy2 |
| Rabbit r1 ✦ | AI companion device | WiFi + creations-sdk | Reference implementation | rabbit-r1 |
| Raspberry Pi 5 + PiCamera Module 3 | Edge compute | BLE | Implemented (spec-based, no hw validation) | pi5-picam |
| Raspberry Pi Zero 2W + PiCamera | Edge compute (constrained) | BLE | Implemented (spec-based, no hw validation) | pi-zero2w |
| Luxonis OAK-D | AI camera | USB + HTTP | Implemented (spec-based, no hw validation) | luxonis-oakd |
| Brilliant Labs Frame | AR glasses | BLE | Implemented (hw validation needed) | frame-glasses |
| Even Realities G2 | Smart glasses | BLE dual-arm | Protocol-verified (BLE UUIDs confirmed, hw validation in progress) | even-g2 |
✦ Rabbit r1 note: The r1 runs OpenClaw as a voice terminal. The Embodiment SDK extends this with a full hardware HAL -- IMU, dual-mic array, rotating 8MP camera, and display output -- giving agents bidirectional hardware access beyond voice. It serves as the reference implementation for the gap between "connected device" and full embodiment.
Eight abstract base classes define the contract between hardware and agents. Each device profile implements the subset that applies.
| HAL | Purpose | Key Methods |
|---|---|---|
IMUHal |
Motion and orientation capture | get_orientation(), get_acceleration() |
CameraHal |
Frame capture | capture_frame(), get_raw_frame() |
MicrophoneHal |
Audio capture + direction of arrival | capture_audio(), get_doa() |
AudioOutputHal |
Speaker / TTS output | speak(), play_audio() |
DisplayHal |
Screen or LED expression | render(), set_expression() |
TransportHal |
Bidirectional data transport | send(), receive(), connect() |
ActuatorHal |
Physical actuation | execute(command), stop_all(), get_joint_states() |
PowerHal |
Battery state and charging | get_battery_level(), on_low_battery(cb) |
CameraHal.get_raw_frame() is the escape hatch for device-native frame formats (e.g., OAK-D's depthai.ImgFrame).
MicrophoneHal.get_doa() returns Direction of Arrival as a stub -- wired to the AudioTriggerDetector in v1.1.
pip install openclaw-embodiment
openclaw-embodiment doctorThe doctor command checks Python version, BLE adapter availability, USB devices (OAK-D), and OpenClaw runtime connectivity.
Load a device profile:
from openclaw_embodiment.profiles import load_profile
config = load_profile("reachy-mini")
# Returns a fully configured HAL bundle for Reachy Mini LiteRun the end-to-end pipeline:
from openclaw_embodiment.profiles import load_profile
from openclaw_embodiment.core.pipeline import EmbodimentPipeline
config = load_profile("reachy-mini")
pipeline = EmbodimentPipeline(config)
pipeline.run() # Blocks; handles trigger→capture→transport→actuate loopThe e2e_loop.py example runs a full simulated embodiment loop -- no hardware required.
# Clone the repo
git clone https://github.com/mmartoccia/openclaw-embodiment
cd openclaw-embodiment
# Install dev dependencies
pip install -e ".[dev]"
# Run the simulation demo (no hardware needed)
openclaw-embodiment demo
# Or run the example directly
python examples/reachy_openclaw/e2e_loop.py --profile simulatorThe demo uses SimulatorHal implementations that emit synthetic IMU, camera frames, and audio events through the full pipeline -- same code path as real hardware.
Each device has a tuned TriggerProfile that controls when a context capture fires. The state machine runs IDLE → SACCADE → FIXATION → CAPTURE.
| Profile Constant | Device | Saccade Threshold | Fixation Duration | Capture Cooldown |
|---|---|---|---|---|
REACHY_MINI_TRIGGER_PROFILE |
Reachy Mini | Low (head motor latency) | 400 ms | 2.0 s |
GLASSES_TRIGGER_PROFILE |
Frame AR glasses | Medium (eye/head) | 300 ms | 1.5 s |
G2_TRIGGER_PROFILE |
Even G2 | Medium (IMU-based) | 350 ms | 1.5 s |
OAKD_TRIGGER_PROFILE |
Luxonis OAK-D | High (static mount) | 500 ms | 3.0 s |
Profiles are importable and overridable:
from openclaw_embodiment.core.trigger import REACHY_MINI_TRIGGER_PROFILE, TriggerDetector
detector = TriggerDetector(profile=REACHY_MINI_TRIGGER_PROFILE)| Command | Description |
|---|---|
openclaw-embodiment init |
Initialize SDK config in current directory |
openclaw-embodiment check |
Check system readiness (connectivity, permissions) |
openclaw-embodiment demo |
Run simulated end-to-end demo |
openclaw-embodiment doctor |
Full dependency and permission checker |
| Gate | Description | Status |
|---|---|---|
| Gate 1 | HAL ABC design + package scaffolding | Complete |
| Gate 2 | Reachy Mini HAL implementation + TriggerDetector | Complete |
| Gate 3 | CM5 hardware validation + 6 device profiles + 22 tests | Complete |
| Gate 4 | AudioTriggerDetector + TriggerArbiter (v1.1) | In progress |
- AudioTriggerDetector -- sound energy + DoA → orient → capture; parallel trigger path alongside visual saccade detection
- TriggerArbiter -- multi-modal fusion: visual and audio triggers with configurable priority policy
- Latency-aware actuation --
get_expected_latency_ms()onTransportHal; pipeline adjusts actuation timing - StatusIndicatorHal -- LED state HAL:
set_color(),blink(),pulse()for device status feedback openclaw-embodiment demoauto-detection -- profile detection from connected hardware at startup- Meta Ray-Ban profile -- pending Meta Ray-Ban SDK public release; priority-1 wearable target
- Apple Vision Pro profile -- via VisionProTeleop bridge
- HalOrchestrator -- explicit
trigger → capture → transport → actuateloop as a first-class object; replaces manual pipeline wiring - SystemHealthHal -- unified device health: thermals, connectivity, storage, sensor availability
- Transport abstraction unification -- BLE and HTTP transports share a single interface with latency metadata
- Profile auto-discovery -- detect connected hardware and load matching profile automatically
Read CONTRIBUTING.md. Hardware validation PRs are the highest priority -- Frame glasses and Even G2 need real-device smoke tests before they leave spec-based status.
Apache 2.0. LICENSE
OpenClaw Embodiment SDK -- the device notices, the agent thinks, the device acts.