Skip to content

feat(app): add HandsFree computer use extension#1908

Open
benjaminshafii wants to merge 3 commits into
daytona-server-sandboxfrom
handsfree-computer-use-extension
Open

feat(app): add HandsFree computer use extension#1908
benjaminshafii wants to merge 3 commits into
daytona-server-sandboxfrom
handsfree-computer-use-extension

Conversation

@benjaminshafii
Copy link
Copy Markdown
Member

Summary

  • add @openwork/handsfree as a non-UI package with the native macOS computer-use runtime
  • expose semantic AX refs, strict background mode, postToPid input, background activation, screenshots, and CUA compatibility tools
  • register HandsFree Computer Use as an MCP-backed OpenWork extension and resolve its local dev command from desktop IPC

Integration Direction

  • Recommended path: ship HandsFree as a local MCP-backed extension in the existing Extensions surface. This keeps OpenWork's extension model consistent while letting the native runtime remain reusable.
  • Alternative considered: embed it directly into the desktop process. Rejected for now because macOS permissions, Swift build output, and long-running computer-use sessions are cleaner behind a separate process.
  • Future follow-up: add a richer config slot for permission diagnostics and strict-mode toggles once the extension is connected.

Testing

  • pnpm --filter @openwork/handsfree check
  • pnpm --filter @openwork/app typecheck
  • MCP wrapper smoke test: initialize + tools/list via node packages/handsfree/bin/openwork-handsfree-computer-use.mjs mcp

Notes

  • No video attached: the full flow requires macOS Accessibility and Screen Recording permissions plus a target app/window. The smoke test verifies server startup and tool registration; native build verifies the Swift runtime.
  • Swift emits expected deprecation warnings for CGWindowListCreateImage; this PR intentionally uses that API for the target-window capture path.

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented May 22, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment May 23, 2026 1:20pm
openwork-den Ready Ready Preview, Comment May 23, 2026 1:20pm
openwork-den-worker-proxy Ready Ready Preview, Comment May 23, 2026 1:20pm
openwork-landing Ready Ready Preview, Comment, Open in v0 May 23, 2026 1:20pm
openwork-share Ready Ready Preview, Comment May 23, 2026 1:20pm

* fix: isolate dev browser cdp port

* fix: pin openwork browser prompt port
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant