Desktop Action Agent

A safe, sandboxed local computer automation agent reference architecture with window detection, application allowlists, and SHA-256 audit logging.
🧠 Intelligence • 🛠️ Tools • 🛡️ Verification

---

What Problem It Solves

Computer-use agents that interact directly with the operating system risk executing destructive shell commands, launching unapproved binaries, or performing runaway mouse/keyboard loops. Desktop Action Agent provides a deterministic safety sandbox featuring:

1. Strict Application Allowlists: Blocks any binary not explicitly declared in the sandbox policy.
2. Command Pattern Blocklists: Unconditionally intercepts hazardous patterns (format, del, rmdir, shutdown, powershell -enc).
3. Bounded Input Coordinates: Validates screen coordinates to prevent invalid clicks.
4. Cryptographic Audit Trail: Computes an immutable SHA-256 hash for every attempted, allowed, or blocked action.

---

Architecture

  Goal ──► Permission Policy ──► Window Detection ──► Bounded Action ──► SHA-256 Audit ──► Verification

---

Key Subsystems

1. Window Manager (core/window_manager.py)

Discovers active desktop windows, filters by title/process name, and safely controls window focus state (includes in-memory mock provider for headless CI and tests).

2. Application Launcher (core/launcher.py)

Spawns processes strictly verified against the sandbox allowlist. Disallowed binaries are blocked and logged.

3. Action Executor (core/actions.py)

Dispatches bounded mouse clicks, typing keystrokes, and privacy-preserving screenshot metadata capturers.

4. Sandbox Policy (core/sandbox.py)

Manages security tiers: READ_ONLY, ALLOW_WHITELISTED, and PROMPT_ALWAYS.

5. Audit Logger (core/audit_logger.py)

Appends immutable action records to a JSONL log file with cryptographic SHA-256 signatures.

---

Quick Start

Installation

# Clone repository
git clone https://github.com/dax0056/desktop-action-agent.git
cd desktop-action-agent

# Install with development dependencies
pip install -e .[dev]

Basic Usage

from desktop_agent import DesktopActionAgent, SandboxPolicy, PermissionLevel

# Initialize agent with whitelisted policy
policy = SandboxPolicy(
    permission_level=PermissionLevel.ALLOW_WHITELISTED,
    allowed_applications={"notepad", "calculator", "code"}
)
agent = DesktopActionAgent(policy=policy)

# Find and focus target window
agent.find_and_focus_window("Editor")

# Launch verified app
launch_res = agent.launch_application("notepad", ["workspace.txt"])
print(f"Launch status: {launch_res.success} (PID: {launch_res.process_id})")

# Execute bounded click & typing
agent.click_at(400, 300)
agent.type_keys("System verification complete.")

---

Live Demo & Execution Walkthrough

📹 Download Full 1080p HD Demo Video with Studio Audio (MP4)

Here is the actual execution trace and audit record generated by Desktop Action Agent:

[DesktopAgent] Initialized with policy: ALLOW_WHITELISTED (4 apps permitted)
[WindowManager] Querying window with title matching 'Editor' -> Window 101 found.
[WindowManager] Window 101 focused.
[Launcher] Launching application 'notepad' with args: ['example.txt']
[Sandbox] Application 'notepad' verified on allowlist. Process spawned (PID: 5001).
[ActionExecutor] Click at coord (400, 300) -> Coordinates verified within screen bounds (1920x1080).
[ActionExecutor] Typing 28 characters to focused window.
[AuditLogger] SHA-256 Signature generated: 7f8a9e2d3c4b... (Audit record appended to JSONL).

Sample Audit Trail Record (JSONL)

{
  "timestamp": 1787332800.12,
  "action_type": "launch_app",
  "target": "notepad",
  "parameters": {"arguments": ["example.txt"]},
  "status": "SUCCESS",
  "approved": true,
  "record_hash": "a4b7c9e1d2f3a4b7c9e1d2f3a4b7c9e1d2f3a4b7c9e1d2f3a4b7c9e1d2f3a4b7"
}

---

Test Report

Desktop Action Agent includes 10 automated unit tests verifying sandbox boundaries and audit immutability:

============================= test session starts =============================
tests/test_actions.py::test_action_executor_bounds_and_typing PASSED     [ 10%]
tests/test_audit_logger.py::test_audit_logger_in_memory_and_disk PASSED  [ 20%]
tests/test_desktop_agent.py::test_desktop_action_agent_workflow PASSED   [ 30%]
tests/test_desktop_agent.py::test_desktop_action_agent_blocks_unauthorized PASSED [ 40%]
tests/test_launcher.py::test_launcher_success_and_blocking PASSED        [ 50%]
tests/test_sandbox_policy.py::test_sandbox_read_only_blocks_mutations PASSED [ 60%]
tests/test_sandbox_policy.py::test_sandbox_allows_whitelisted_apps PASSED [ 70%]
tests/test_sandbox_policy.py::test_sandbox_blocks_destructive_commands PASSED [ 80%]
tests/test_window_manager.py::test_window_manager_listing_and_finding PASSED [ 90%]
tests/test_window_manager.py::test_window_manager_focus PASSED           [100%]
============================= 10 passed in 0.07s ==============================

• Passed: 10 / 10 (100%)
• Test Command: pytest -v

---

Documentation & Roadmap

• Milestone development plan: ROADMAP.md
• Release notes and version history: CHANGELOG.md
• Contribution guidelines: CONTRIBUTING.md
• Security reporting policy: SECURITY.md

---

Part of the DAX Agent Engineering Series

Desktop Action Agent provides the Safe Computer Interaction Tools for the series:

• 🧠 nexus-agent — Intelligence & Local Agent Core
• 💻 micro-coding-agent — Deterministic Code AST & Patch Engine
• 🖥️ desktop-action-agent — Safe Sandboxed Desktop Automation

---

License

Distributed under the MIT License. See LICENSE for details.