Page automation primitives: snapshot/read/click/type/paste + trusted input - #468
Page automation primitives: snapshot/read/click/type/paste + trusted input#468ianu82 wants to merge 3 commits into
Conversation
Two standalone, independently testable layers the bridge builds on: - browser-dom-tools: the scripts executed inside tabs — interactive- element snapshot (indexed, visibility-filtered, with a per-snapshot staleness version), readable-text extraction, click/type by snapshot index (native value setter + input/change events), scroll, and the synthetic ClipboardEvent paste that splits TSV into spreadsheet cells. - browser-input: trusted CDP input for canvas-rendered apps — key/modifier resolution (pure), click/keys/text via webContents.debugger with keyDown so default actions fire.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 319e5861a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const text = (el.innerText || el.value || el.getAttribute('aria-label') | ||
| || el.getAttribute('placeholder') || el.getAttribute('title') || '').trim().slice(0, 120); |
There was a problem hiding this comment.
When a visible <input type="password"> has a saved or typed value, this expression falls through to el.value and serializes the secret as the element text returned by /snapshot to the loopback agent bridge. Hidden inputs are skipped, but password inputs are not, so visiting an autofilled login page exposes credentials; omit or mask values for password fields and prefer labels/placeholders instead.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed — password input values are masked in snapshots (5b86a10).
| const STASH = ${stashRef(stash)}; | ||
| const els = []; | ||
| const refs = []; | ||
| const nodes = document.querySelectorAll('a,button,input,textarea,select,[role],[onclick],summary'); |
There was a problem hiding this comment.
On pages with a very large number of matching nodes, querySelectorAll(...) constructs the full static NodeList before the SCAN_MAX loop guard runs, so the snapshot path still does unbounded selector work and allocation despite the cap. A hostile or just huge page with many buttons/links can still hang the renderer; traverse incrementally, such as with a TreeWalker or chunking, so the cap is enforced before collecting all matches.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed — the walk itself is capped instead of collecting the full NodeList first (5b86a10).
Review follow-up (one P1): a filled password input's value could be serialized into /snapshot text and flow to the agent — passwords now fall back to labels/placeholders. The snapshot also traverses with a TreeWalker so the scan cap binds before any full NodeList is built on hostile pages.
1 participant
What
The two automation layers the bridge builds on (part 2 of 5) — everything that acts on a web page, before any window management exists.
browser-dom-tools.ts— scripts that run inside tabs:el.click(); native value setter withinput/changeevents so React-controlled fields work.browser-input.ts— trusted CDP input for canvas apps that ignore synthetic events: pure key/modifier resolution (with aliases likeleft/return), and click/keys/text dispatched viawebContents.debuggerusingkeyDownso default actions (caret movement, shortcuts) actually fire.Verification
For QA
Still nothing to click in the UI — this is plumbing. The acceptance evidence is in the follow-up PRs that put it in front of you; if you'd like a preview, the final PR's e2e exercises these exact paths.