feat(desktop): the agent's browser draws on the Code screen after every action

[brcampidelli]

What

The agent's browser draws on the Code screen: after every browser action, the viewport appears under the turn, with the page's address as text beside it. Item 2 of the list audited on 2026-09-16 (the "cockpit").

Why, and what was measured first

The model reads a page as text — the tag tree render_elements prints — and the person read the same text, or watched a second Chromium window beside the app with CHIMERA_BROWSER_HEADLESS off. Neither was watching the agent open the site and type the dates.

Before wiring anything, the cost of a frame (viewport 1280×720, three pages, this machine):

page	PNG	JPEG q55	JPEG q40
example.com	18 KB · 54 ms	13 KB · 24 ms	12 KB · 32 ms
GitHub repo page	103 KB · 198 ms	62 KB · 106 ms	55 KB · 150 ms
Wikipedia article	232 KB · 224 ms	96 KB · 52 ms	81 KB · 33 ms

So: JPEG at quality 55, one frame per browser action, never on a timer. A turn that browsed twenty times costs about a megabyte over its stream; an idle browser costs nothing; a run with no screen captures nothing.

How

• BrowserDriver.frame() -> BrowserFrame | None — the Playwright driver captures the viewport (type="jpeg", quality=55, full_page=False) with the page's URL, title and size.
• BrowserTool.on_frame — set by assemble_registry only when the request has a screen (frame_sink, a late-bound FrameAnnouncer like the approval one). The tool announces after every action, success or failure (a failed click still shows the page it failed on), and checks the sink before asking the driver: a headless run never captures — pinned by a test that counts the driver's frame() calls. A capture that fails is logged and skipped, never a word in the observation the model reads.
• The turn's stream gets event: browser — action, url, title, width, height, base64 jpeg, n (this turn's frame count). Never written to the run log: replay exists for the words a dropped connection lost, not to redraw a page that has moved on; a reopened conversation shows no frame, honestly.
• Desktop: CodeBrowserFrame, onBrowser, the dispatcher's browser branch; BrowserView under the exchange, replaced on every frame (where the browser is), the URL printed as text — a picture of a page is a claim about where the browser is and the address is the part a person can check. Nothing in the panel is interactive: the person is watching, not driving. Ten dictionaries (code.browser.title/frame/alt).

Tests

• Backend (tests/test_the_agents_browser_draws_on_the_code_screen.py, 6): a frame follows every action, success or failure, with the URL of that action; without a sink the driver is never asked; a failing capture never reaches the observation; the announcer drops frames until a screen binds; through assemble_registry the browser gets its sink only when there is one; through the real app the turn streams the frame (base64 round-trips, n == 1, seq set) and the run log holds done and no jpeg.
• Desktop: Code.browser.test.tsx (3) — the picture with the address beside it, the last frame wins and there is one panel, nothing for a turn that never browsed; lib/browser-frame.test.ts (1) — the frame read off the wire reaches onBrowser with every field and no other handler. Sabotage: the dispatcher's browser branch removed → the wire test fails; restored. 154 files, 1102 tests green, tsc clean. OpenAPI byte-identical (SSE frames are not in it).

Full gate on a clean copy in WSL — see the last comment.

Not in this PR

A frame on a timer while the page loads or scrolls (per-action was the measured trade); the browser's frames on the autonomous run's stream (POST /api/runs has no screen sink today); a clickable panel.

🤖 Generated with Claude Code

[brcampidelli]

Full gate on a clean copy in WSL (uv venv, .[dev,desktop]): ruff clean, mypy clean, 6516 passed, 18 skipped, 10 xfailed in 3:02. Desktop: 154 files, 1102 tests, tsc clean. Sabotage: dispatcher's browser branch removed → the wire test fails; restored.