Summary
Capture a much-later product experiment, not committed roadmap work: explore whether Motion's on-device laptop-camera tracking could become an optional, low-bandwidth control layer for supervising coding agents.
This should not try to replace the keyboard or general mouse input. The plausible use case is controlling agents while standing back from the desk, thinking aloud, managing several agents, or needing an alternative input method.
Motion already has a browser webcam path using MediaPipe hand landmarks. A future spike could translate those landmarks into semantic commands and route them through a desktop/app adapter.
Candidate gesture vocabulary
- Pinch and hold: push-to-talk; releasing turns the microphone off.
- Thumbs up: approve a reversible agent action.
- Thumbs down: reject an agent action.
- Hold up 1–4 fingers: directly select Low / Medium / High / Max reasoning complexity.
- Swipe right or left: move to the next or previous agent/tab based on the active context.
Prefer direct complexity selection over repeated increments so the requested state is deterministic.
Interaction and safety constraints
- Run recognition on-device; never transmit or store camera frames.
- Require explicit camera and macOS Accessibility permissions.
- Use a calibrated command zone, gesture dwell, visible command preview, and cooldown to prevent accidental activation.
- Show the active target before navigation or approval.
- Keep destructive, irreversible, credential, deployment, migration, and production actions behind physical confirmation.
- Separate semantic gesture events from application-specific adapters so recognition does not depend on one editor or agent product.
Product-risk gate
The default assumption is that this may remain a compelling demo rather than a durable product. A keyboard is faster and more reliable whenever the user's hands are already at the desk.
Before building beyond a minimal prototype:
- Test only pinch-to-talk, approve/reject, and navigation.
- Put it in front of people who actually supervise multiple agents or need alternative input.
- Observe whether they voluntarily keep using gestures after the novelty period.
- Stop or leave it experimental if users consistently return to keyboard shortcuts when both options are available.
Revisit only when
- Motion's core game/control-feel work is validated on physical devices.
- There is concrete user evidence for away-from-keyboard agent supervision or an accessibility need.
- A narrow target application and semantic command adapter can be named.
Non-goals
- No implementation now.
- No general-purpose gesture replacement for keyboard and mouse.
- No production dependency or OS-wide control work until the product-risk gate is passed.
Summary
Capture a much-later product experiment, not committed roadmap work: explore whether Motion's on-device laptop-camera tracking could become an optional, low-bandwidth control layer for supervising coding agents.
This should not try to replace the keyboard or general mouse input. The plausible use case is controlling agents while standing back from the desk, thinking aloud, managing several agents, or needing an alternative input method.
Motion already has a browser webcam path using MediaPipe hand landmarks. A future spike could translate those landmarks into semantic commands and route them through a desktop/app adapter.
Candidate gesture vocabulary
Prefer direct complexity selection over repeated increments so the requested state is deterministic.
Interaction and safety constraints
Product-risk gate
The default assumption is that this may remain a compelling demo rather than a durable product. A keyboard is faster and more reliable whenever the user's hands are already at the desk.
Before building beyond a minimal prototype:
Revisit only when
Non-goals