hackbrowser: multi-signal DOM change detection (replace brute-force collectElements scanning)

[badchars]

Problem

explorePageWithAI() calls collectElements() (full DOM scan via page.evaluate() + getComputedStyle() on every div/span/li) twice per task — once before and once after each action. On a page with 20 tasks, this results in 40+ full DOM scans (~8.4s pure scan time), most of which detect no changes.

Current discoverNewElements() only checks seenKeys Set for new role::label combos — it misses:

• CSS-only visibility changes (:checked, :has() selectors)
• Disappeared elements (authorization/conditional UI)
• Delayed async DOM updates (Qwik resumability, RSC streaming, @defer)

Solution

Replace brute-force before/after scanning with a multi-signal change detection module (change-detector.ts):

Signal	What it detects	Cost
MutationObserver	DOM node add/remove/attribute changes	~0ms (native)
Network Activity	AJAX responses that may trigger DOM updates	~0ms (existing interceptor)
Visibility Snapshot	CSS-only state changes (:checked, :has(), details/summary)	~2ms
Element Count Delta	Safety net — catches anything the above miss	~1ms

Decision matrix: ALL signals = 0 → skip scan. ANY signal > 0 → run collectElements().

Expected Impact

• Speed: ~5.5x reduction in scan time (8.4s → ~1.5s per page)
• Quality: Catches CSS-only toggles and disappeared elements that current system misses
• LLM context: Delta-aware re-planning ("3 elements added after clicking Transfer") vs current full re-plan

New Files

• packages/hackbrowser/src/change-detector.ts — multi-signal detection module
• packages/hackbrowser/src/element-tracker.ts — element state & diff management

Framework Coverage

Validated against: React, Vue, Angular, Svelte 5, HTMX, Qwik, Astro, Lit/Web Components, jQuery, AngularJS 1.x, ExtJS, ASP.NET WebForms, JSF. See implementation plan comment below.

Related

• Part of Hackbrowser crawl depth: terminates after 3-5 pages on weak models / SPA targets #87 (hackbrowser crawl depth)
• Branch: hackbrowser-improvement

[badchars]

Implementation Plan: Multi-Signal DOM Change Detection

Architecture

packages/hackbrowser/src/
├── agent.ts              ← main crawl loop (modify)
├── scanner.ts            ← element collection (existing)
├── navigator.ts          ← LLM planner (existing)
├── executor.ts           ← task execution (existing)
├── change-detector.ts    ← NEW: multi-signal change detection
└── element-tracker.ts    ← NEW: element state & diff management

---

Module 1: change-detector.ts

Exports:

• install(page: Page): Promise<void> — set up MutationObserver + network listener
• check(): Promise<ChangeResult> — read all signals, return verdict
• teardown(): Promise<void> — disconnect observer, clean up

ChangeResult type:

interface ChangeResult {
  changed: boolean
  signals: {
    mutations: number        // DOM nodes added/removed/modified
    networkRequests: number  // XHR/fetch responses since last check
    visibilityDelta: number  // interactive elements visibility change count
    elementCountDelta: number // total element count change
  }
  verdict: "skip" | "scan" | "full-scan"
}

Signal implementation details:

Signal 1 — MutationObserver:

page.evaluate(() => {
  window.__cs_mutations = []
  window.__cs_observer = new MutationObserver(muts => {
    for (const m of muts) {
      if (m.type === "childList") {
        window.__cs_mutations.push({ type: "add", count: m.addedNodes.length })
        window.__cs_mutations.push({ type: "remove", count: m.removedNodes.length })
      }
      if (m.type === "attributes") {
        window.__cs_mutations.push({ type: "attr", target: m.target.tagName })
      }
    }
  })
  window.__cs_observer.observe(document.body, {
    childList: true, subtree: true, attributes: true,
    attributeFilter: ["class", "style", "hidden", "disabled", "aria-hidden", "open", "popover"]
  })
})

Note: attributeFilter limits noise — we only care about visibility/state attributes, not data-* or tracking attributes.

Signal 2 — Network Activity:
Tap into hackbrowser's existing request interceptor. Count responses received between install() and check(). Already captured — zero additional cost.

Signal 3 — Visibility Snapshot:

page.evaluate(() => {
  const selectors = 'button,input,select,textarea,a[href],[role="button"],[role="link"],[role="tab"],[role="menuitem"]'
  let visible = 0
  for (const el of document.querySelectorAll(selectors)) {
    const rect = el.getBoundingClientRect()
    if (rect.width > 0 && rect.height > 0) visible++
  }
  return visible
})

Run before and after action. Delta = |after - before|. Catches CSS :has(), :checked ~ sibling, details/summary.

Signal 4 — Element Count Delta:

page.evaluate(() => document.querySelectorAll('button,input,select,textarea,a').length)

Pure safety net. <1ms cost.

Decision matrix:

mutations=0 AND network=0 AND visDelta=0 AND countDelta=0 → "skip"
mutations>50 OR visDelta>10 → "full-scan" (major DOM change)
any signal > 0 → "scan" (targeted scan)

Network-aware wait (replaces fixed 400ms):

async function waitForSettled(page, maxWait = 3000) {
  const start = Date.now()
  let lastActivity = start

  while (Date.now() - lastActivity < 500 && Date.now() - start < maxWait) {
    await page.waitForTimeout(100)
    const pending = await page.evaluate(() => 
      performance.getEntriesByType("resource").filter(e => !e.responseEnd).length
    )
    if (pending > 0 || window.__cs_mutations.length > lastMutCount) {
      lastActivity = Date.now()
    }
  }
}

---

Module 2: element-tracker.ts

Exports:

• snapshot(elements: RawElement[]): void — store current state
• diff(newElements: RawElement[]): ElementDiff — compute delta
• visibilityDelta(page: Page): Promise<number> — quick visibility count

ElementDiff type:

interface ElementDiff {
  added: RawElement[]      // new elements not in previous snapshot
  removed: RawElement[]    // elements that disappeared
  stateChanged: Array<{    // same element, different state
    element: RawElement
    changes: string[]      // ["disabled→enabled", "hidden→visible"]
  }>
}

Key improvement: LLM gets delta context instead of full element dump:

Current: "Here are 205 elements on the page, plan what to do"
New:     "After clicking 'Enable 2FA', 3 elements appeared: 
          OTP input, QR code image, Verify button.
          1 element disappeared: 'Enable 2FA' toggle.
          Plan actions for these new elements."

Less tokens, more context, better plans.

---

Integration: agent.ts changes

Current flow (per task in explorePageWithAI):

L707:  pre = collectElements(page)         // FULL SCAN
L710:  executeTask(task)
L762:  post = collectElements(page)        // FULL SCAN  
L773:  newFound = discoverNewElements(post) // compare with seenKeys
L782:  if (newFound) → re-plan             // LLM call

New flow:

// Once at page exploration start:
detector.install(page)
initial = collectElements(page)            // 1 full scan
tracker.snapshot(initial)

// Per task:
executeTask(task)
result = await detector.check()            // ~5ms multi-signal

if (result.verdict === "skip") continue    // NO SCAN NEEDED

elements = collectElements(page)           // only when needed
delta = tracker.diff(elements)
tracker.snapshot(elements)

if (delta.added.length >= REPLAN_THRESHOLD) {
  // Re-plan with delta context
  rePlan = planDelta(delta)                // smaller prompt, faster
} 
// delta.removed tracked for security analysis

Constants:

• REPLAN_THRESHOLD = 3 — don't re-plan for 1-2 new elements (often just aria attributes)
• MAX_WAIT_SETTLED = 3000 — max wait for network idle
• SETTLED_QUIET_PERIOD = 500 — no activity for 500ms = settled

---

Framework-Specific Validation

Framework	DOM Pattern	Signal that catches it
React/Vue/Angular	conditional render (add/remove nodes)	MutationObserver
jQuery	.show()/.hide() (style attribute)	MutationObserver
HTMX	hx-swap (DOM chunk replacement)	MutationObserver + Network
Qwik	lazy handler download → DOM update	Network + MutationObserver (delayed)
RSC streaming	progressive HTML chunks	Network + MutationObserver
Angular @defer	lazy block hydration	Network + MutationObserver
Svelte 5	synchronous fine-grained updates	MutationObserver
CSS :has()/:checked	visibility change, no DOM mutation	Visibility Snapshot
Closed Shadow DOM	internal changes invisible	Element Count Delta (partial)
Astro islands	hydration (JS only, DOM same)	No DOM change = nothing to scan ✓
Popover API	popover attribute toggle	MutationObserver (attribute)
<dialog>	open attribute toggle	MutationObserver (attribute)
ExtJS	heavy DOM manipulation (100+ nodes)	MutationObserver → full-scan
AngularJS 1.x	ng-show class toggle	MutationObserver (attribute)
ASP.NET UpdatePanel	AJAX partial postback	Network + MutationObserver

Known gap: Closed Shadow DOM internal changes — Element Count Delta catches element count shifts but not attribute-only changes inside closed shadow. Mitigation: these are rare (<3% of targets) and the current system also can't interact with closed shadow DOM elements.

---

Performance Projection

Per-page cost (20 tasks, 5 trigger DOM changes):

Metric	Current	New	Improvement
Full scans	42	7	6x fewer
Scan time	~8.4s	~1.5s	5.6x faster
LLM re-plans	~5 (full context)	~2 (delta context)	fewer + cheaper
CSS-only detection	❌ missed	✅ caught	quality gain
Disappeared elements	❌ missed	✅ tracked	quality gain

---

Commit Plan

1. feat(hackbrowser): add change-detector module with multi-signal DOM tracking
2. feat(hackbrowser): add element-tracker module with snapshot/diff
3. refactor(hackbrowser): integrate change detection into explorePageWithAI loop
4. feat(hackbrowser): add network-aware settled wait (replace fixed 400ms)
5. feat(hackbrowser): delta-aware re-planning for LLM