Skip to content

feat(computer-use): add native desktop control over MCP - #124

Closed
GrothKeiran wants to merge 1 commit into
uvwt:mainfrom
GrothKeiran:feat/native-computer-use
Closed

GrothKeiran wants to merge 1 commit into
uvwt:mainfrom
GrothKeiran:feat/native-computer-use

Conversation

@GrothKeiran

Copy link
Copy Markdown

Summary

  • add computer_apps, computer_snapshot, and computer_act as opt-in built-in MCP tools
  • provide native Windows capture/input, macOS capture/automation, and Linux X11 adapters with an XWD screenshot fallback
  • return screenshots as both standard MCP image content and authenticated short-lived Artifacts
  • enforce latest-snapshot binding, serialized action batches, bounds checks, conservative system-key policy, and input cleanup after partial failures
  • expose configuration in the Windows/macOS control panels, runtime status, CLI/environment settings, and English/Chinese documentation

Validation

  • go test ./...
  • go test -race ./internal/tool/computer
  • go vet ./internal/tool/computer ./internal/app ./internal/mcp ./internal/config ./internal/desktopruntime ./internal/httpx
  • git diff --check
  • cross-compiled internal/app for Windows amd64/arm64 and macOS amd64/arm64
  • cross-compiled internal/desktopruntime for Windows amd64/arm64
  • ran a real Linux MCP end-to-end smoke through initialize, tools/list, computer_apps, computer_snapshot, and a non-mutating computer_act wait action
  • Linux smoke returned 11 visible windows, a 1280x800 PNG MCP image, a snapshot id, and an authenticated Artifact

Notes

Windows WPF could not be compiled in this environment because dotnet is unavailable. macOS AppKit and native Windows/macOS permission/input smoke tests still need to run on those operating systems. Real mouse and keyboard mutation was intentionally not exercised on the active Linux desktop; native action dispatch is covered with injected-driver tests, and the MCP action path was exercised using a wait action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant