Hackbrowser crawl depth: terminates after 3-5 pages on weak models / SPA targets

[badchars]

Problem

Hackbrowser crawls often terminate after only 3-5 pages, especially with:

• Weaker AI models (GPT-4o-mini, Haiku) that fail to generate valid page plans
• SPA targets where navigation is done via onClick handlers, not <a href> tags
• Multi-credential mode where redirect detection produces false positives

Root Causes (from audit)

1. planPage returns empty plan on model failure — navigator.ts:131. When LLM output can't be parsed, { tasks: [] } is returned. No actions executed, no new endpoints discovered.

2. collectDOMLinks only collects <a href> — agent.ts:1592. Misses <form action>, <button>, [data-href], [onclick], SPA router elements. On modern SPAs this finds 0-3 links.

3. ACCESS_DENIED_PATTERNS false positives — agent.ts:1949. Regex /\/(login|signin|...|auth)/i matches legitimate pages like /auth/profile, /auth/settings.

4. MAX_PER_PATH_PATTERN = 5 too aggressive — agent.ts:566. Same pathname + param keys pattern limited to 5 URLs.

5. resolveElement failures silently dropped — agent.ts:1226. Wrong role/label from weak model → all tasks silently skipped.

6. No network-based URL discovery — page.on("request") captures traffic but doesn't feed discovered URLs back into the page queue. JS files, XHR calls, and API endpoints are invisible to the crawler.

Expected Behavior

• Crawl should explore significantly more pages before terminating
• Network traffic (JS, XHR, fetch) should feed back into discovery
• Weak model failures should fall back to mechanical DOM interaction
• Browser should stay useful after scan completion (capture manual browsing)

Related

• Hackbrowser stability: browser dies after ~10 requests, agent unaware #84 (closed) — browser stability fixes (merged)
• Post-crawl capture drain already implemented in stability PR

[badchars]

Hackbrowser Capability Audit — Improvement Plan

Full audit of all 17 source files (~15K LOC) completed. Below is the prioritized roadmap grouped into implementation phases.

---

Phase 1: Discovery Breadth (highest impact — doubles endpoint coverage)

1.1 Rich URL Discovery (collectDOMLinks expansion)

File: agent.ts:1591
Current: Only a[href] — misses most SPA navigation.
Add:

• form[action], button[formaction]
• data-href, data-url, data-src, data-link attributes
• iframe[src] (same-origin only)
• area[href] (image maps)
• link[rel=preload][href], link[rel=prefetch]
• meta[http-equiv=refresh] content URL extraction

1.2 JS Bundle API Endpoint Extraction

New module: js-scanner.ts
What: Intercept JS/JSON responses during crawl, regex-scan for:

• fetch("..."), axios.get/post("..."), $.ajax({url: "..."}), XMLHttpRequest.open("...", "...")
• Route definitions: path: "/users", "/api/v2/" patterns
• GraphQL endpoint URLs (/graphql, /gql)
  Feed: Discovered API paths → enqueueUrl() or a new discoveredEndpoints set forwarded to CyberStrike

1.3 Sitemap / robots.txt Parsing

Where: agent.ts — before BFS loop starts
What: GET /sitemap.xml, GET /robots.txt → parse URLs → enqueueUrl()
Guard: Only at crawl start, max 500 URLs from sitemap, scope-filtered

1.4 Network-Based URL Feedback

Where: agent.ts request interceptor (~line 447)
Current: Captures requests but doesn't feed URLs back to page queue.
Fix: For every captured request URL, call enqueueUrl() if it's a navigable page (HTML content-type, not API/XHR). This alone would dramatically improve SPA coverage — XHR/fetch calls to page URLs get queued.

---

Phase 2: Crawl Intelligence

2.1 Response Body Analysis During Crawl

Where: Request interceptor or new response-analyzer.ts
What: For HTML responses, extract:

• URLs from href, src, action attributes in response HTML
• Error page detection (403/404/500 status + error patterns)
• Technology fingerprinting (X-Powered-By, Server headers, framework signatures)
• Stack traces / debug info flagging
  Feed: New URLs → enqueue. Error pages → skip further planning (save LLM calls). Tech fingerprints → crawl metadata.

2.2 Error / Access-Denied Page Intelligence

Where: agent.ts — before calling planPage()
What: Check HTTP status of the page load. If 4xx/5xx:

• Skip LLM planning (save tokens)
• Log as "error page" in crawl result
• Extract any info leaks (server version, stack trace, internal paths)
• Still capture the request/response for CyberStrike analysis

2.3 WAF / Rate Limit Detection

Where: Request interceptor
What: Detect 429 status, Cloudflare challenge pages, Akamai blocks, AWS WAF responses.

• On 429: exponential backoff (wait 2s, 4s, 8s, up to 30s)
• On WAF block: log warning, try CDP mode suggestion, continue without that URL
• Surface in crawl result so the agent knows coverage was limited

2.4 planPage Failure Escalation

Where: navigator.ts:137
Current: 2 attempts → empty plan → page silently skipped.
Fix:

• After 2 LLM failures, fall back to mechanical DOM interaction: click every enabled button, fill every empty form with defaults, submit
• Log which pages got LLM vs mechanical exploration
• Add planFailures counter to CrawlResult so agent knows

---

Phase 3: Richer Interaction

3.1 Dynamic Content Re-scan (MutationObserver)

Where: scanner.ts + agent.ts
What: After each action (click/fill/submit), wait 500ms and re-scan for new interactive elements that appeared via AJAX.
Implementation:

• Install a MutationObserver before action, collect mutations during/after
• If new interactive elements appeared (not in the pre-action snapshot), re-run collectElements() and send only the delta to the planner as a "follow-up" plan
• Bounded: max 2 re-scans per page to prevent infinite loops

3.2 WebSocket / SSE Capture

New module: ws-capture.ts
What: Use Playwright's CDP session to intercept WebSocket frames:

cdpSession.on('Network.webSocketFrameReceived', ...)
cdpSession.on('Network.webSocketFrameSent', ...)

• Capture WS handshake URL, sent/received frames
• Forward to CyberStrike as a new capture type (needs ingest schema extension)
• SSE: intercept text/event-stream responses

3.3 iframe Content Exploration

Where: agent.ts — after main page exploration
What: For same-origin iframes (page.frames().filter(f => sameOrigin(f.url()))):

• Run collectElements() inside each frame
• Execute tasks via frame-scoped locators
• Capture requests from iframe context

3.4 Multi-Tab / Popup Handling

Where: agent.ts
What: Listen for context.on("page") events.

• New tabs from window.open() or target="_blank" → explore in new page context
• Close popup after exploration, return to main page
• Bounded: max 3 popup follows per page

---

Phase 4: Form & Input Intelligence

4.1 Context-Aware Fill Values

Where: navigator.ts planner prompt + executor.ts
What: Instead of static fill-value table:

• Let the planner generate contextual values based on field labels/constraints
• Add field-type-specific generators: email, phone, date, currency, URL, SKU
• Boundary values: maxlength+1, min-1, 0, negative, very long strings
• The planner already has the constraints — just needs prompt guidance to use them better

4.2 Form Branch Coverage

Where: agent.ts — task execution loop
What: When planner returns multiple form tasks for the same form (different select/radio values — the "conditional forms" prompt already exists), execute each one:

• Save page state (URL + form fields) before first submission
• Navigate back after each submission
• Re-fill with the next branch's values
• Bounded: max 4 branches per form

4.3 localStorage / sessionStorage Extraction

Where: agent.ts — after page exploration, before moving to next
What: page.evaluate(() => ({ localStorage: {...localStorage}, sessionStorage: {...sessionStorage} }))

• Forward to CyberStrike as metadata on the page's captures
• Flag tokens, API keys, user data found in storage

---

Phase 5: Capture Enrichment

5.1 Cookie / Security Header Analysis

Where: Request interceptor or new header-analyzer.ts
What: On first response per domain:

• Extract security headers: CSP, X-Frame-Options, HSTS, X-Content-Type-Options, Permissions-Policy
• Analyze Set-Cookie: HttpOnly, Secure, SameSite, Path, Domain, Expires
• Forward as session-level metadata to CyberStrike