[Feature] Proposal: Battery-aware sensor management and power modes

[eclemm]

Problem

Proposal: Battery-Aware Sensor Management and Power Modes

Problem

OpenVision’s live voice and vision capabilities make it possible to use Ray-Ban Meta glasses as a general-purpose AI assistant. However, frequent camera use, continuous video streaming, active microphone sessions, persistent network connections, and local model inference can increase battery consumption on both the glasses and the connected iPhone.

For users who want to wear the glasses throughout the day, continuous sensing is not always necessary. Many requests can be handled through voice, conversation context, tools, or a single photograph without maintaining an active camera stream.

The objective is to make OpenVision continuously available while avoiding unnecessary sensor, network, and compute activity.

Proposed Solution

Introduce battery-aware sensor management based on the following principle:

«OpenVision should use the lowest-power combination of sensing, computation, and networking capable of completing the user’s request.»

Camera-Off Default

The glasses camera should remain inactive while OpenVision is idle and during requests that do not require visual context.

Examples include:

• General conversation
• Web searches
• Reminders and tool calls
• Calculations
• Questions answered from conversation history
• Voice-only OpenClaw tasks

Camera resources should close after a visual request completes, is cancelled, times out, or fails.

Snapshot-First Vision

Visual requests should begin with a single image whenever possible.

Suggested escalation order:

1. No visual input
2. One snapshot
3. An additional snapshot when necessary
4. A short live-video session
5. Explicit continuous video mode

Continuous vision should remain available, but it should be an intentional high-power mode rather than the default path for visual requests.

Automatic Resource Cleanup

Camera, microphone, speech-recognition, network, timer, frame-buffer, and model resources should be released during every completion path:

• Successful completion
• User cancellation
• Inactivity timeout
• Error
• App backgrounding
• Glasses disconnection
• Repeated network failure

This could also include an audit for camera sessions, WebSockets, timers, retry loops, frame buffers, and inference tasks that remain active longer than required.

Configurable Power Profiles

OpenVision could expose several operating profiles.

All-Day Mode

• Camera off unless required
• Snapshot-first visual requests
• Short inactivity timeouts
• Aggressive resource cleanup
• Minimal network keep-alive
• No continuous video without confirmation
• Reduced model preloading

Balanced Mode

• Snapshot-first behavior
• Short live-video sessions when useful
• Moderate inactivity timeouts
• Brief retention of active sessions for follow-up questions

Performance Mode

• Faster model availability
• Longer warm sessions
• Higher image quality or frame rate where useful
• Clear indication that the mode consumes more battery

Live Vision Mode

• Explicit continuous-video operation
• Configurable frame rate and resolution
• Configurable inactivity and maximum-duration limits
• Audible or visible indication while active
• Automatic downgrade when battery or temperature reaches a threshold

Reserve Mode

• Voice-first operation
• Camera disabled unless explicitly requested
• No continuous video
• Minimal background synchronization
• Automatic activation below a configurable battery level

Battery-Aware Behavior

The application could adjust behavior based on battery state.

Example behavior:

• Normal operation at high battery levels
• Discourage unnecessary continuous video at moderate battery levels
• Prefer snapshots and shorter sessions at low battery levels
• Enter reserve mode at critically low levels

Thresholds should be configurable rather than hard-coded.

Adaptive Live-Video Processing

Live-video sessions could reduce unnecessary work by:

• Allowing configurable frame rate and resolution
• Reducing frame rate when the scene is static
• Skipping near-duplicate frames
• Increasing quality only when motion or fine detail requires it
• Using a high-resolution still image instead of sustained high-quality video where appropriate

Optional Battery Diagnostics

An optional diagnostic view could help measure improvements and identify regressions.

Potential metrics:

• Session duration
• Phone and glasses battery before and after a session
• Camera-active duration
• Microphone-active duration
• Frames captured, transmitted, and analyzed
• Network-active duration
• Backend and model used
• Thermal warnings
• Cleanup failures

Diagnostics should collect operational measurements only and should not retain personal audio, images, or transcripts.

Alternative Approaches Considered

Continuous Vision by Default

This provides the richest environmental context but is likely unsuitable for frequent all-day use because the camera, wireless connection, and model pipeline remain active.

Continuous vision should remain an explicit mode for tasks that genuinely require it.

User-Managed Camera Control Only

Users could manually start and stop every visual session. This is simple to implement but makes resource leaks and forgotten active sessions more likely.

Automatic cleanup and inactivity timeouts would provide a safer default.

One Fixed Low-Power Configuration

A single conservative configuration would reduce battery use but would limit users who prefer higher performance for short periods.

Selectable profiles provide clearer control over the battery-versus-performance tradeoff.

Cloud Processing Instead of Local Processing

Moving inference to the cloud may reduce phone compute usage, but it still requires wireless communication and introduces latency, connectivity, privacy, and API-cost considerations.

The most efficient route may vary by task, so local and cloud options should remain available.

Suggested Incremental Implementation

This proposal could be implemented in stages:

1. Audit and guarantee resource cleanup across success, cancellation, timeout, error, backgrounding, and disconnect paths.
2. Enforce camera-off-by-default behavior for non-visual requests.
3. Prefer single snapshots over live video.
4. Add configurable inactivity and maximum-session timeouts.
5. Add power profiles and battery thresholds.
6. Add adaptive frame processing.
7. Add optional diagnostics and repeatable battery benchmarks.

Success Criteria

• Lower idle battery consumption
• Reduced camera-active time for static visual requests
• No lingering camera, microphone, network, or inference sessions
• Reduced transmission of duplicate or unnecessary frames
• Predictable low-battery behavior
• Clear control over power and performance tradeoffs
• Longer practical daily use of OpenVision with connected glasses

[rayl15]

Thanks for such a thoughtful, well-structured proposal, @eclemm — and welcome! This is the repo's first community feature request, and it's a genuinely good one. 🙌

You're right that all-day wearable use makes power the central constraint, and your north star is exactly the right one: "use the lowest-power combination of sensing, computation, and networking capable of completing the request."

A few notes so we build on what already ships rather than redoing it:

Already in place today

• Camera-off by default — the camera only starts for a request that actually needs vision; it's idle otherwise.
• Snapshot-first ("click-and-go") — a visual request grabs a single frame and stops the stream before running inference, so the camera/LED aren't held during the multi-second model call.
• Continuous video is an explicit mode — sustained streaming only happens in an intentional "Live video" mode, never as the default path.
• Partial cleanup — session teardown, conversation-inactivity timeouts, and background/GPU guards already release resources on several paths.

Where your proposal adds real, missing value (the good stuff 👇)

1. Resource-cleanup audit — guaranteeing every exit path releases camera, mic, sockets, timers, frame buffers, and inference tasks. We do some of this, but not as one verified sweep.
2. Battery-/thermal-aware auto-downgrade — there is currently no reading of battery level or thermal state anywhere. Auto-preferring snapshots / a reserve mode at low battery or high temperature is entirely new and high-impact.
3. Adaptive live-video — skipping near-duplicate frames and lowering FPS on static scenes.
4. Optional battery diagnostics — measurable before/after benchmarking.

Scoping — the full proposal is essentially a roadmap (~7 features), so rather than one giant PR I've labeled this as the tracking issue and broken out the highest-value first slices:

• Resource-cleanup audit: release sensors/network/compute on every completion path #38 — Resource-cleanup audit across all completion paths (good first issue)
• Battery- and thermal-aware auto-downgrade (snapshot-first / reserve mode) #39 — Battery-/thermal-aware auto-downgrade (help wanted)
• Adaptive live-video: skip near-duplicate frames, lower FPS on static scenes #40 — Adaptive live-video: skip duplicate frames / lower FPS on static scenes (help wanted)

If you'd like to take one on, I'd start with #38 — happy to point you at the relevant files (VoiceAgentView.swift, GlassesManager.swift, GemmaLocalService.swift) and review a PR. No pressure at all if you'd rather just have proposed the idea — either way it's a valuable contribution.

Thanks again, and glad to have you here. 🚀