perf: fuse observation into action, and pick snapshot elements by what is on screen

[ishu86]

Two changes to the agent loop, both measured on a live stack rather than estimated.

Act-then-observe

click_element, fill(submit) and navigate now return the settled page with the result, so an agent stops spending one LLM turn per "what did that do?".

Measured: an eval round trip is 11ms and a snapshot 55–145ms, against seconds for the model turn it replaces. The fusion costs +26ms on a navigate. snapshot:false opts out.

The first cut of this was wrong and the live test caught it: it polled readyState, which the old document reports as complete for a beat after a click, so a click that navigated handed back the page the agent had just left. Fixed with the same document-stamp deskclient.navigate already uses. Verified against ground truth: clicking through example.com → iana.org returns the destination page, matching location.href exactly.

Snapshot selection

The 150-element cap was spent in document order, so on a long page it went to whatever the HTML listed first — sidebar and nav chrome the reader had already scrolled past.

Measured on a scrolled Wikipedia article, of the elements actually on screen:

scroll	before	after
top	117/117 (100%)	100%
3000px	47/70 (67%)	100%
9000px	47/63 (75%)	100%

Same 9.5KB payload. A ref is the element's index in the whole walk rather than its position in the shown list, so showing a subset costs nothing and _locate/fill re-derive exactly the same element — the shown numbers just skip. Verified by clicking ref 232, which the old selection could not see at all.

The cap stays at 150. It is now a reach knob, not a coverage one, and raising it is not free because click and navigate each carry a snapshot now: 150 els = 12.8KB, 300 = 25.3KB, and past ~800 the eval result blows the 64KB limit.

Also

• The Drive UI keeps its own tool table and agent prompt, so both are updated — otherwise the one agent shipped in the repo would still be told to re-snapshot after every click.
• Wake logs its two halves separately (measured 0.1s docker, 2–3s Chromium — the tunable one).
• --restore-last-session, so a wake hands back the tabs it had. Verified across a real sleep/wake cycle.
• --renderer-process-limit=8 and Memory Saver for desks that now accumulate tabs.

Measured and deliberately not done

• Screenshot encode: 50ms / 196KB, dominated by the pure-Python row shuffle rather than zlib, and vision tokens bill by dimension not bytes — a Pillow/JPEG path buys nothing here.
• deskd's settle() tail: login-only, and its blast radius is the auth flow. Left alone.
• No demonstrated gain from the two Chromium flags above. They are hedges for long-lived desks, not measured wins — --renderer-process-limit also lets cross-site tabs share a renderer, which matters less here only because --no-sandbox already gave up that isolation.

Merge notes

• No behaviour changes anyone has to opt out of; nothing new runs on a timer.
• API is additive only — snapshot is a new response key, old clients ignore it.
• Refs now skip numbers; checked that nothing reads them by position.
• Existing desk containers won't pick up --restore-last-session until recreated. A no-op for them, not a break.
• Includes dd1d993 (desk look version-stamp) from an earlier session that never reached main — it force-reseeds desktop config once per desk.

Tests

All 20 Python test files and 3 node test files pass; docker compose config validates. New coverage for the fused snapshot (including the navigation race the first cut got wrong) and for refs being numbered by document index rather than list position.

🤖 Generated with Claude Code