SENTINEL — Safety-First macOS Desktop Agent

SENTINEL is a local desktop operator that plans once, executes deterministic steps on macOS, remembers successful workflows, and keeps destructive actions behind explicit safety gates. It is designed to make useful automation possible within free-tier model limits.

Design question: how much capable desktop automation can run locally when model calls are scarce, expensive, and untrusted?

The approach

task intake → one model-generated plan → local deterministic executor
                                         ├── Accessibility actions
                                         ├── guarded shell
                                         ├── browser/app control
                                         └── auditable memory

Planning uses a model; execution reuses local primitives. Successful procedures can be recalled later, so repeated tasks can run with zero additional planning calls.

Highlights

• Queue-based task intake and transparent plan view
• Native macOS Accessibility-tree targeting instead of pixel-only clicks
• Free-tier-aware model routing, quotas, fallbacks, and request reserves
• Guarded shell execution with confirmation and hard refusal tiers
• Hybrid local memory with keyword and embedding retrieval
• Reusable playbooks reinforced by successful outcomes
• Voice, image, and PDF task intake
• Kill switch, sensitive-app blocklist, redaction, and audit trail
• Fully functional mock mode with no API key

Quick start

Requires macOS and Node.js 24+.

cd sentinel
npm run setup
npm run dev

Open http://localhost:4777. Mock mode demonstrates the complete queue, planning, execution, memory, and audit flow without spending model quota.

For live use, add GEMINI_API_KEY to .env and grant Accessibility and Screen Recording permissions to the terminal running SENTINEL.

npm test
npm run typecheck

See the complete technical guide for architecture, configuration, safety boundaries, permissions, tests, and roadmap.

Repository map

Path	Purpose
sentinel/src/	Planner, runner, native actions, memory, safety, and server
sentinel/dashboard/	Local delegation and audit interface
sentinel/test/	Budget, safety, execution, memory, and reuse tests
sentinel/README.md	Full operator and engineering documentation

Safety boundary

SENTINEL is a research-grade local operator, not an unattended production agent. Cloud models may receive unblocked screenshots or attachments; textual redaction cannot sanitize binary media. Review the safety notes before live use.

Contributors

Contributor	Role
Priyadeep Jaiswal	Owner; product design, architecture, implementation, and testing

Priyadeep set the architecture and safety constraints, integrated and tested the system, and owns the resulting work.