Vision-First Desktop Agent (Windows-first)

This is a scaffold for a desktop agent that sees the screen (OCR + optional VLM), plans a short action script, acts via mouse/keyboard, verifies, and speaks back.

Layout

vision_agent/
  main.py
  config.toml
  perception/
  reason/
  act/
  ui/
  assets/
  logs/
scripts/
  mvp_notepad_demo.py

Quick start (Windows PowerShell)

1. Create a virtual environment and install deps:

py -3.11 -m venv .venv
. .venv\Scripts\Activate.ps1
pip install -r requirements.txt

2. Configure the planner (optional, OpenAI):

Option A — environment variable (recommended):

$env:OPENAI_API_KEY = "<your_api_key>"

Option B — local config file (not committed):

Create vision_agent/config.local.toml with:

[planner]
provider = "openai"
model = "gpt-5-nano"
api_key = "<your_api_key>"

If neither the environment variable nor config.local.toml provides an API key, the primary stage also looks for an OPENAI_API_KEY inside .vscode/launch.json.

Option C — VS Code launch config:

The agent also checks .vscode/launch.json for env.OPENAI_API_KEY when the environment variable or local config are absent. This is convenient when running via VS Code's debugger. Add a launch configuration with an env block containing your key.

3. Telemetry & logging

Per step, the loop writes:

logs/sessions/YYYY-MM-DD_HHMMSS/step_00001/
  screen.jpg
  status.json
  plan.json
  action.json
  verify.json
  metrics.json

Toggle a concise console summary via:

[logging]
console_summary = true

4. Run the MVP demo (dry-run preview only):

python .\scripts\mvp_notepad_demo.py

5. Try the main entry (planner w/ OpenAI or local heuristic):

python -m vision_agent.main --goal "Open settings" --preview  # use --live for real clicks

The CLI accepts --goal to skip the prompt and --live/--preview to override run mode.

6. Export a ChatGPT-ready payload (image + OCR JSON embedded):

python .\scripts\capture_to_chatgpt_payload.py

This captures a screenshot, runs OCR (PaddleOCR if available), and writes vision_agent/logs/sessions/payload_<timestamp>.json containing an OpenAI Chat Completions messages array with a data: URL image and OCR JSON in the user content.

Notes:

• OCR prefers PaddleOCR (fast, multilingual). If you've uninstalled it, pytesseract may be used as a fallback.
• If PaddleOCR is installed, no config change is needed; the wrapper auto-detects it.
• pyautogui uses screen coordinates; ensure your DPI scaling is standard while testing.

Security note: Never hardcode API keys in source files. Use environment variables or config.local.toml (ignored by Git) to keep secrets out of version control.

Configuration

Defaults live in vision_agent/config.toml, while a git-ignored vision_agent/config.local.toml can override them. Key sections include:

• [general] for preview mode, step limits, and confirmation thresholds.
• [primary] selecting the provider/model for high-level directions.
• [local_llm] configuring the host and model for the local evaluator.
• [logging] for console summaries.
• [verify] for OCR verification window size and regex usage.

API keys are read from environment variables such as OPENAI_API_KEY or CHATGPT_API_KEY, from [primary].api_key in the config, or, as a last resort, from an OPENAI_API_KEY entry in .vscode/launch.json.

OWL-ViT (optional icons)

To enable open‑vocabulary icon detection on the toolbar ROI, install:

pip install torch torchvision transformers

Then set in vision_agent/config.toml:

[icons]
use_open_vocab = true
region = "top"
fraction = 0.25

If you skip this, the agent will try simple template matching from vision_agent/assets/icons when present.

Preview vs. live clicks

Control via vision_agent/config.toml:

[general]
preview = true          # dry-run with overlay + console only
confirm_threshold = 0.85
max_steps = 8

The executor reads confirm_threshold and will request a Space/Enter confirmation below this fusion score in live mode.

Reasoning pipeline

Primary direction stage

vision_agent.reason.primary.get_primary_directions generates short high-level instructions for each iteration. It uses the provider and model from the configuration and retrieves the API key from environment variables, config files, or .vscode/launch.json. If no key is found, it falls back to a heuristic string.

Local evaluator

vision_agent.reason.local_eval.summarize_and_check runs a small local LLM (via Ollama) to summarize the current observation and judge whether the goal is complete, falling back to heuristics when the model or server is unavailable.

Consensus logic

Recent summaries are stored, and confirm_completion_gpt must report completion three times in a row before the session ends. This consensus step helps prevent single-step hallucinations.

Smoke tests

Convenience scripts under scripts/:

• python scripts/test_ocr.py — capture + OCR; saves to logs/smoke/ocr.
• python scripts/test_planner_local.py — runs local/heuristic planner.
• python scripts/test_planner_openai.py — runs OpenAI planner (needs API key).
• python scripts/test_executor_dryrun.py — runs a safe dry-run plan to exercise overlay and ops.
• python scripts/smoke_checks.py — compiles and imports core modules and runs a quick status.

Next steps

• Wire perception.icons.match_templates and store icons under vision_agent/assets/icons.
• Replace reason/planner.py with a call to a vision-capable LLM (e.g., OpenAI GPT-4o/5-nano) and enforce a strict JSON schema.
• Implement safety confirmations and Esc emergency stop.