feat(panel): responsive full-stretch viewport, live nav, fidelity + anti-detection (v0.6.1)

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## v0.6.1
+- **Full-stretch, responsive viewport.** The panel now resizes Chrome's layout viewport to
+  your dock's size (× device-pixel-ratio) over CDP as you resize/expand/collapse it — the page
+  reflows to fill, instead of a fixed landscape box floating in dead space. The canvas fills the
+  stage; a `ResizeObserver` (debounced) keeps them in sync.
+- **The agent's navigation is now visible.** A cross-process navigation tears down Chrome's
+  screencast, so previously you saw the first page load but nothing as the agent moved around or
+  into sub-pages. The stream now **re-arms** on every navigation-complete signal (`loadEventFired`
+  / `frameStoppedLoading` / `frameNavigated` / `navigatedWithinDocument`, debounced) — every page
+  paints.
+- **Higher fidelity.** JPEG quality raised to **80** (was 60) and configurable via
+  `stream_quality`; combined with hi-dpi rendering (deviceScaleFactor up to 2×) the viewport is
+  noticeably crisper.
+- **Anti-detection (`stealth`).** New knobs to help past bot walls (Google, Reddit, Cloudflare):
+  `stealth` drops the `navigator.webdriver` automation flag and, when headless, swaps the
+  "HeadlessChrome" User-Agent for a real desktop one; plus `user_agent` and `browser_args`
+  overrides. Most reliable paired with `headed: true` + a logged-in `profile`. No setting defeats
+  detection entirely. Launch flags are now also applied when a session is started from the panel
+  (the Start button / URL bar), not just from the agent's `browser_open`.
+
 ## v0.6.0
 - **The Browser panel is now a fully interactive, drivable viewport — and it is the ONLY mode.**
   A live **CDP screencast** — event-driven JPEG frames (not a screenshot poll) painted on a

README.md

@@ -45,9 +45,20 @@ point.
   **host and a remote fleet member** alike. A second CDP client attaches to the same Chrome
   agent-browser drives (`agent-browser get cdp-url`); `browser_stream.py` does the bridging.
 
+  The viewport is **full-stretch and responsive** — it resizes Chrome's layout viewport to your
+  dock's size (× device-pixel-ratio) as you expand/collapse it, so the page reflows to fill rather
+  than sitting as a fixed box, and the screencast **re-arms on every navigation** so you see the
+  agent move through pages and sub-pages (not just the first load). Tune sharpness with
+  `stream_quality`.
+
   When no page is open the panel shows a **Start button** (not a dead end). Set `home_url` to a
   page and the panel **auto-opens** it — a homepage — otherwise Start opens `about:blank`.
 
+  **Getting past bot walls** (Google, Reddit, Cloudflare): set `stealth: true` (drops the
+  automation flag + real UA when headless), and — most reliably — `headed: true` with a logged-in
+  Chrome `profile`. `user_agent` / `browser_args` are there for fine control. No setting defeats
+  detection entirely.
+
   **WebSocket auth:** the host's operator-bearer gate is HTTP-only and doesn't cover WS
   handshakes, so the stream self-gates — the panel mints a **single-use ticket** from the gated
   `POST /api/plugins/agent_browser/stream-ticket` and presents it on the WS URL.
@@ -82,7 +93,8 @@ agent_browser:
 |---|---|
 | `tools.py` | the browser tools — subprocess wrappers over the `agent-browser` CLI |
 | `browser_panel.py` | the Browser panel page + routes — the interactive canvas + the gated nav / stream-ticket / WS-stream routes |
-| `browser_stream.py` | the CDP bridge — screencast frames out, input in; the WS ticket auth |
+| `browser_stream.py` | the CDP bridge — screencast frames out, input in, viewport resize + nav re-arm; the WS ticket auth |
+| `runtime.py` | shared launch-flag builder (headed / profile / device / stealth), used by the tools and the panel |
 | `skills/` | the discovery skill (defers to `agent-browser skills get core`) |
 | `workflows/` | declarative browser recipes |
 | `tests/` | the host-free pytest suite (subprocess mocked — no binary needed) |

browser_panel.py

@@ -25,6 +25,7 @@
 from fastapi.responses import HTMLResponse, JSONResponse
 
 from . import browser_stream
+from .runtime import launch_flags
 
 log = logging.getLogger("protoagent.plugins.agent_browser")
 
@@ -58,6 +59,7 @@ def build_panel_data_router(cfg: dict | None):
     cfg = cfg or {}
     binary = str(cfg.get("binary") or "agent-browser")
     timeout = float(cfg.get("timeout_s", 60))
+    quality = int(cfg.get("stream_quality", 80) or 80)
 
     router = APIRouter()
 
@@ -105,7 +107,7 @@ async def on_frame(jpeg: bytes, md: dict):
                 return
 
         try:
-            async with browser_stream.CDPStream(page_ws, on_frame) as cdp:
+            async with browser_stream.CDPStream(page_ws, on_frame, quality=quality) as cdp:
                 await cdp.start_screencast()
                 while True:
                     await cdp.dispatch(await ws.receive_json())
@@ -126,7 +128,9 @@ async def _nav(body: dict = Body(...)):
         if action == "open":
             if not url:
                 return JSONResponse({"ok": False, "error": "url required"})
-            rc, err = await asyncio.to_thread(lambda: _run("open", url))
+            # launch flags (headed/profile/stealth/…) so a session started from the panel
+            # matches one the agent opens.
+            rc, err = await asyncio.to_thread(lambda: _run(*launch_flags(cfg), "open", url))
         elif action in ("back", "forward", "reload"):
             rc, err = await asyncio.to_thread(lambda: _run(action))
         else:
@@ -160,9 +164,8 @@ async def _nav(body: dict = Body(...)):
   .dot{width:7px;height:7px;border-radius:50%;display:inline-block;flex:none;
     background:var(--pl-color-fg-muted,#9aa0aa)}
   .dot.ok{background:#22c55e}.dot.err{background:#ef4444}
-  .stage{height:calc(100% - 38px);position:relative;background:var(--pl-color-bg-inset);
-    display:flex;align-items:flex-start;justify-content:center;overflow:auto}
-  canvas{max-width:100%;display:block;outline:none;touch-action:none}
+  .stage{height:calc(100% - 38px);position:relative;background:var(--pl-color-bg-inset);overflow:hidden}
+  canvas{width:100%;height:100%;display:block;outline:none;touch-action:none}
   #msg{display:none;position:absolute;inset:0;align-items:center;justify-content:center;
     padding:24px;box-sizing:border-box}
   .card{max-width:460px;text-align:center}
@@ -234,7 +237,7 @@ async def _nav(body: dict = Body(...)):
     const ticket=(await r.json()).ticket;
     ws=new WebSocket(wsUrl(ticket));
     ws.binaryType="arraybuffer";
-    ws.onopen=()=>{ connected=true; setStatus("ok","live"); };
+    ws.onopen=()=>{ connected=true; setStatus("ok","live"); sendResize(); };
     ws.onmessage=onMsg;
     ws.onclose=()=>{ connected=false; setStatus("err","offline"); scheduleRetry(); };
     ws.onerror=()=>{ try{ ws.close(); }catch(_){} };
@@ -278,6 +281,15 @@ async def _nav(body: dict = Body(...)):
 cv.addEventListener("keyup",(e)=>{ e.preventDefault();
   send({t:"key",action:"up",key:e.key,code:e.code,keyCode:e.keyCode,...mods(e)}); });
 
+// ── keep the browser viewport matched to the panel (full stretch + responsive) ──
+// The server resizes Chrome's layout viewport to these dims (CDP), so the page reflows
+// to fill the dock and frames come at its shape/DPI — no letterboxed "standard viewport".
+let rzTimer=null;
+function panelSize(){ const r=cv.parentElement.getBoundingClientRect();
+  return { w:Math.round(r.width), h:Math.round(r.height), dpr:Math.min(window.devicePixelRatio||1, 2) }; }
+function sendResize(){ const s=panelSize(); if(s.w>10 && s.h>10) send({t:"resize",w:s.w,h:s.h,dpr:s.dpr}); }
+new ResizeObserver(()=>{ clearTimeout(rzTimer); rzTimer=setTimeout(sendResize, 180); }).observe(cv.parentElement);
+
 // ── boot ONCE — on the handshake (so apiFetch has the bearer for the gated ticket)
 // or an 800ms fallback for a standalone/older host that posts no init. ──────────
 let booted=false;

browser_stream.py

@@ -181,21 +181,52 @@ def resolve_page_target(binary: str, timeout: float = 10.0) -> tuple[str | None,
     return (page, "" if page else "no page target to stream (open a URL first)")
 
 
-class CDPStream:
-    """A minimal async CDP client over one page target: start a screencast, ack
-    frames, and dispatch input. `frame_cb(jpeg_bytes, metadata)` is called for each
-    `Page.screencastFrame`. Requires the ``websockets`` package (a uvicorn extra —
-    already present wherever the host serves WebSockets)."""
+def viewport_metrics(w, h, dpr) -> tuple[int, int, float, int, int]:
+    """Clamp a panel size → ``(css_w, css_h, scale, frame_max_w, frame_max_h)``. The CSS
+    size drives Chrome's layout viewport; scale (device-pixel-ratio, capped at 2) keeps
+    frames crisp on hi-dpi; the frame max caps the screencast so a huge dock can't flood
+    the socket."""
+    cw = max(1, min(int(w or 0), 2048))
+    ch = max(1, min(int(h or 0), 2048))
+    scale = max(1.0, min(float(dpr or 1), 2.0))
+    return cw, ch, scale, min(int(cw * scale), 2560), min(int(ch * scale), 2560)
+
+
+# CDP events that mean "a new document is ready" → re-arm the screencast. The set that
+# actually fires varies by navigation kind, so we listen for all of them (debounced).
+_NAV_DONE = frozenset((
+    "Page.loadEventFired", "Page.frameStoppedLoading",
+    "Page.frameNavigated", "Page.navigatedWithinDocument",
+))
 
-    def __init__(self, page_ws_url: str, frame_cb):
+
+class CDPStream:
+    """A minimal async CDP client over one page target: start a screencast, ack frames,
+    resize the viewport to the panel, and dispatch input. ``frame_cb(jpeg, metadata)``
+    fires per ``Page.screencastFrame``. Requires ``websockets`` (a uvicorn extra).
+
+    Two robustness details the naive version missed:
+    - **Re-arm on navigation.** A cross-process navigation swaps the page's render widget
+      and Chrome silently stops the screencast — so we re-issue ``startScreencast`` on each
+      top-frame ``Page.frameNavigated`` / ``Page.loadEventFired``. Without this you see the
+      first page but nothing as the agent moves around or into sub-pages.
+    - **One writer.** Frame acks + nav re-arms (reader task) and input/resize (request
+      task) both write to the socket, so a lock serializes sends."""
+
+    def __init__(self, page_ws_url: str, frame_cb, quality: int = 80):
         self._url = page_ws_url
         self._frame_cb = frame_cb
+        self._quality = max(1, min(int(quality or 80), 100))
+        self._cast = (1280, 800)      # current screencast max frame size (device px)
         self._ws = None
         self._id = 0
+        self._last_arm = 0.0          # debounce re-arms (multi-frame pages fire many events)
+        self._lock: asyncio.Lock | None = None
         self._reader: asyncio.Task | None = None
 
     async def __aenter__(self):
         import websockets
+        self._lock = asyncio.Lock()
         self._ws = await websockets.connect(self._url, max_size=None, open_timeout=10)
         self._reader = asyncio.create_task(self._read_loop())
         return self
@@ -207,16 +238,34 @@ async def __aexit__(self, *exc):
             await self._ws.close()
 
     async def _send(self, method: str, params: dict | None = None) -> int:
-        self._id += 1
-        await self._ws.send(json.dumps({"id": self._id, "method": method, "params": params or {}}))
-        return self._id
-
-    async def start_screencast(self, max_w: int = 1280, max_h: int = 800, quality: int = 60):
-        await self._send("Page.enable")
-        await self._send("Page.startScreencast", {"format": "jpeg", "quality": quality,
-                         "maxWidth": max_w, "maxHeight": max_h, "everyNthFrame": 1})
+        async with self._lock:  # serialize writes: reader (acks/re-arm) + request task (input/resize)
+            self._id += 1
+            await self._ws.send(json.dumps({"id": self._id, "method": method, "params": params or {}}))
+            return self._id
+
+    async def _arm_cast(self):
+        mw, mh = self._cast
+        await self._send("Page.startScreencast", {"format": "jpeg", "quality": self._quality,
+                         "maxWidth": mw, "maxHeight": mh, "everyNthFrame": 1})
+
+    async def start_screencast(self, max_w: int = 1280, max_h: int = 800):
+        await self._send("Page.enable")   # also enables frameNavigated / loadEventFired for re-arm
+        self._cast = (max_w, max_h)
+        await self._arm_cast()
+
+    async def set_viewport(self, w, h, dpr=1.0):
+        """Resize Chrome's layout viewport to the panel and re-arm the screencast at the
+        matching frame size — so the page reflows to fill, and stays crisp."""
+        cw, ch, scale, mw, mh = viewport_metrics(w, h, dpr)
+        await self._send("Emulation.setDeviceMetricsOverride",
+                         {"width": cw, "height": ch, "deviceScaleFactor": scale, "mobile": False})
+        self._cast = (mw, mh)
+        await self._arm_cast()
 
     async def dispatch(self, msg: dict):
+        if msg.get("t") == "resize":
+            await self.set_viewport(msg.get("w"), msg.get("h"), msg.get("dpr", 1))
+            return
         cmd = input_to_cdp(msg)
         if cmd:
             await self._send(*cmd)
@@ -228,9 +277,23 @@ async def _read_loop(self):
                 m = json.loads(raw)
             except Exception:  # noqa: BLE001
                 continue
-            if m.get("method") == "Page.screencastFrame":
+            method = m.get("method")
+            if method == "Page.screencastFrame":
                 p = m["params"]
                 try:
                     await self._frame_cb(base64.b64decode(p["data"]), p.get("metadata", {}))
                 finally:
                     await self._send("Page.screencastFrameAck", {"sessionId": p["sessionId"]})
+            elif method in _NAV_DONE:
+                # A navigation swaps the render widget and Chrome drops the screencast.
+                # These are the "new document is ready" signals — which one fires varies by
+                # navigation kind (real cross-origin nav emits loadEventFired/frameNavigated;
+                # data:/SPA emit frameStoppedLoading/navigatedWithinDocument), so we cover all
+                # and debounce, since a multi-frame page fires several.
+                now = time.monotonic()
+                if now - self._last_arm > 0.2:
+                    self._last_arm = now
+                    try:
+                        await self._arm_cast()   # bring the screencast back
+                    except Exception:  # noqa: BLE001 — transient during teardown
+                        pass

protoagent.plugin.yaml

@@ -1,6 +1,6 @@
 id: agent_browser
 name: Agent Browser
-version: 0.6.0
+version: 0.6.1
 description: >-
   Browser automation for protoAgent, backed by **agent-browser** (vercel-labs) — a
   fast native-Rust CLI/daemon that drives Chrome over CDP with accessibility-tree
@@ -26,6 +26,8 @@ config:
   home_url: ""              # homepage the panel opens to. When set, the panel auto-opens it if
                             # no browser page is open; the empty state also shows a Start button.
                             # Blank → the Start button opens about:blank (no auto-open).
+  stream_quality: 80        # interactive panel JPEG quality (1–100). Higher = crisper, more data.
+                            # (The panel also renders at your dock's size × device-pixel-ratio.)
   # Runtime / launch options — a curated set passed to `agent-browser open` so you can
   # shape and lock down the browser the agent spins up. Blank/0/false = CLI default.
   headed: false             # show a real browser window instead of headless
@@ -34,15 +36,27 @@ config:
   allowed_domains: ""       # comma-separated navigable-domain whitelist (e.g. "example.com,*.foo.com")
   confirm_actions: ""       # comma-separated action categories that require confirmation
   max_output: 0             # cap page-text output chars (LLM-safety); 0 = CLI default
+  # Anti-detection — help pages that block automated/headless browsers (Google, Reddit,
+  # Cloudflare, …). No setting defeats detection entirely; headed + a logged-in `profile`
+  # is the most reliable combination. Applied at session launch (the first open).
+  stealth: false            # drop the navigator.webdriver automation flag; when headless, also
+                            # swap the "HeadlessChrome" User-Agent for a real desktop Chrome one
+  user_agent: ""            # override the User-Agent (blank → CLI default; stealth may set one)
+  browser_args: ""          # extra Chrome launch args, comma/newline separated
+                            # (e.g. "--disable-blink-features=AutomationControlled")
 
 # Editable in Settings ▸ Plugins (ADR 0019) — the operator knobs above as UI fields.
 settings:
   - { key: home_url, label: "Homepage URL", type: string, description: "The page the interactive panel opens to. When set, it auto-opens if no browser page is open, and the empty state shows a Start button. Blank = a Start button that opens about:blank." }
-  - { key: headed, label: "Headed browser", type: bool, description: "Show a real browser window instead of running headless." }
+  - { key: stealth, label: "Anti-detection (stealth)", type: bool, description: "Help pages that block automated/headless browsers (Google, Reddit, Cloudflare). Drops the navigator.webdriver flag and, when headless, uses a real desktop User-Agent. Most reliable paired with Headed + a logged-in Browser profile. No setting defeats detection entirely." }
+  - { key: headed, label: "Headed browser", type: bool, description: "Show a real browser window instead of running headless. Also far less bot-detectable than headless." }
+  - { key: user_agent, label: "User-Agent override", type: string, description: "Override the browser User-Agent. Blank = CLI default (stealth may set a realistic one when headless)." }
+  - { key: browser_args, label: "Extra Chrome args", type: string, description: "Extra Chrome launch args, comma/newline separated (e.g. --disable-blink-features=AutomationControlled)." }
   - { key: allowed_domains, label: "Allowed domains", type: string, description: "Comma-separated navigable-domain allowlist (e.g. example.com,*.foo.com). Blank = unrestricted." }
   - { key: confirm_actions, label: "Confirm actions", type: string, description: "Comma-separated action categories that require confirmation before the agent runs them." }
-  - { key: profile, label: "Browser profile dir", type: string, description: "Path to a browser profile directory — isolation + persisted auth/cookies across sessions." }
+  - { key: profile, label: "Browser profile dir", type: string, description: "Path to a browser profile directory — isolation + persisted auth/cookies across sessions. Reusing a logged-in Chrome profile is the most reliable way past login walls." }
   - { key: device, label: "Device emulation", type: string, description: "Emulate a device, e.g. \"iPhone 16 Pro\". Blank = desktop." }
+  - { key: stream_quality, label: "Panel stream quality", type: number, description: "Interactive panel JPEG quality (1–100). Higher = crisper, more bandwidth. The panel also renders at your dock's size × device-pixel-ratio." }
   - { key: max_output, label: "Max page-text output (chars)", type: number, description: "Cap characters of page text returned to the model (LLM-safety). 0 = the CLI default." }
   - { key: timeout_s, label: "Command timeout (s)", type: number, description: "Per-command subprocess timeout for the browser tools." }
   - { key: binary, label: "agent-browser binary", type: string, description: "The agent-browser CLI on PATH (override with a pinned absolute path if needed)." }

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agent-browser-plugin"
-version = "0.6.0"
+version = "0.6.1"
 description = "Browser-automation plugin for protoAgent, backed by vercel-labs/agent-browser (tools + skill + workflows + an interactive, drivable browser panel)."
 requires-python = ">=3.11"