fix: bypass CSP and fix cross-origin fetch for site adapters

[rayzhux]

Problem

~50% of site adapters fail with TypeError: Failed to fetch when running cross-origin API calls via bb-browser site. Two separate browser security mechanisms cause this:

1. CSP connect-src: Sites like HackerNews, npm, and StackOverflow set Content-Security-Policy headers that restrict which domains fetch() can reach. Adapter eval scripts inherit these restrictions.

2. CORS + credentials: Many adapters use credentials: 'include' for same-origin cookie auth, but this triggers CORS preflight for cross-origin requests. Most API servers don't set Access-Control-Allow-Credentials, causing the browser to reject the response.

Affected adapters (tested)

Adapter	Root cause	After fix
hackernews/top	CSP blocks firebaseio.com	✅
npm/search	CSP blocks registry.npmjs.org	✅
stackoverflow/search	CSP + CORS	✅
github/me	Adapter bug (needs OAuth, not cookies)	❌ separate issue
twitter/*	Needs login (expected)	N/A

Solution

1. Extension: Page.setBypassCSP (cdp-service.ts)

Call Page.setBypassCSP({ enabled: true }) immediately after attaching the debugger to a tab. This disables CSP enforcement for pages loaded while the debugger is attached.

Note: This only affects pages loaded after the command. Tabs already open need a refresh. New tabs created by bb-browser work automatically since the debugger attaches before navigation.

2. CLI: Cross-origin fetch wrapper (site.ts)

Wrap adapter execution with a fetch() interceptor that detects cross-origin requests and downgrades credentials: 'include' to credentials: 'omit'. Same-origin requests are unaffected, preserving cookie-based auth.

Testing

Tested on macOS with Chrome 146 + bb-browser 0.3.0:

bb-browser site hackernews/top 5      # ✅ was: Failed to fetch
bb-browser site npm/search react       # ✅ was: Failed to fetch  
bb-browser site stackoverflow/search async  # ✅ was: Failed to fetch
bb-browser site reddit/posts spez      # ✅ unaffected (same-origin)
bb-browser site duckduckgo/search test # ✅ unaffected (same-origin)

[yan5xu]

Review / 代码审查

Thanks for the PR! The direction is solid — CSP connect-src blocking and CORS credential issues are real pain points for the site adapter ecosystem. The two-pronged approach (CDP CSP bypass + fetch credential downgrade) makes sense. A few issues to address before merging:

感谢这个 PR！方向是对的——CSP connect-src 阻断和 CORS credential 问题确实是 site adapter 生态的真实痛点。CDP 绕过 CSP + fetch credential 降级的双管齐下思路合理。合并前有几个问题需要修：

---

High / 高优先级

1. CSP bypass has no teardown / CSP bypass 没有关闭路径

Page.setBypassCSP({ enabled: true }) is called on attach but never disabled. Long-lived attached tabs will run with CSP permanently disabled, which materially weakens browser security. Consider enabling it only around adapter execution, or at minimum disabling it on detach.

Page.setBypassCSP({ enabled: true }) 在 attach 时开启，但没有对应的关闭。长期 attach 的 tab 会一直处于 CSP 关闭状态。建议只在 adapter 执行期间开启，或至少在 detach 时关闭。

2. Request object inputs not handled / Request 对象未被正确处理

When an adapter calls fetch(new Request(url, { credentials: 'include' })), the code reads input.url but only mutates init. The original Request still carries credentials: 'include'. Fix: clone the Request with overridden credentials:

当 adapter 调用 fetch(new Request(url, { credentials: 'include' })) 时，代码只修改了 init，原始 Request 的 credentials 仍然生效。需要 clone Request 并覆盖：

if (input instanceof Request) {
  input = new Request(input, { credentials: 'omit' });
}

3. Origin comparison is fragile / origin 比较方式不够健壮

url.startsWith('http') && !url.startsWith(__pageOrigin) is a string prefix check, which breaks on case differences, trailing slashes, or origin-like prefixes. Use proper URL parsing:

startsWith 做字符串前缀匹配，在大小写差异、尾部斜杠等情况下会误判。建议用标准 URL 解析：

const parsed = new URL(url, location.href);
if (parsed.origin !== location.origin) { ... }

---

Medium / 中优先级

4. Silent error swallowing / 静默吞掉错误

catch(e) {} around the URL inspection hides real bugs. At least add console.debug for development visibility.

URL 检测部分的 catch(e) {} 会吞掉真实错误，建议至少加个 console.debug 方便调试。

5. Async leakage / 异步泄漏

window.fetch is restored in finally when the adapter resolves, but any fire-and-forget async work spawned by the adapter will run after restoration and may still fail.

adapter resolve 后 finally 里恢复了 window.fetch，但 adapter 内部未 await 的异步请求会在恢复后执行，仍然可能失败。

---

Overall / 总结

Direction is right, implementation needs a few fixes. Main concerns are the CSP lifecycle and the Request object handling. Looking forward to the updated version! 🙏

方向正确，实现上需要几处修复。主要关注 CSP 生命周期和 Request 对象处理。期待更新版本！

[yan5xu]

Thanks for the detailed analysis and testing! The approach of CSP bypass + fetch credentials downgrade is the right direction. A few issues to address:

1. CSP bypass has no lifecycle management (critical)

Page.setBypassCSP({ enabled: true }) is called in ensureAttached() but never disabled — not in detach(), not in cleanupTab(). This means once bb-browser attaches to a tab, CSP is permanently disabled for that tab, even after bb-browser is done. If the user browses to a banking or social media site in that tab, XSS protections are gone.

Fix: At minimum, call Page.setBypassCSP({ enabled: false }) in detach(). Ideally, only enable CSP bypass during adapter execution and restore it afterward.

2. Request object credentials not properly overridden (maintainer already noted)

When adapters call fetch(new Request(url, { credentials: 'include' })), the wrapper only modifies the init object. The browser prioritizes the Request object's own credentials property. Need to clone: new Request(input, { credentials: 'omit' }).

3. Origin comparison is vulnerable (maintainer already noted)

if (url.startsWith('http') && !url.startsWith(__pageOrigin))

This string prefix match can be fooled: if __pageOrigin is https://example.com, then https://example.com.evil.com would be considered same-origin. Use new URL(url, location.href).origin !== location.origin instead.

4. Empty catch silences errors

} catch(e) {}

At least add console.debug('[bb-browser] fetch wrapper error:', e) for debuggability.

Relationship with PR #45

PR #45 solves the same issue by running adapters in Node.js. The two approaches are complementary:

• This PR (fix: bypass CSP and fix cross-origin fetch for site adapters #15): general fix for all adapters, preserves browser context (DOM, cookies)
• PR fix(cli): execute network-capable adapters in Node to bypass CORS #45: optimization for pure-API adapters that don't need browser context

I'd suggest fixing the issues above and merging this as the primary solution. PR #45 can be considered separately as an optional optimization with a new explicit capability flag.

Looking forward to the updated version!

[yan5xu]

Superseded by #131 (daemon unification). Extension will be removed — CSP bypass will be handled at the CDP level.

[yan5xu]

Closing: the Extension has been removed in #132. The daemon connects directly via CDP WebSocket, which is not subject to page CSP restrictions. Cross-origin fetch in site adapters is handled at the CDP level (Runtime.evaluate runs in page context with full access).

[yan5xu]

补充中文说明：Extension 已在 #132 中移除。daemon 直接通过 CDP WebSocket 连接，不受页面 CSP 限制。site adapter 的跨域 fetch 在 CDP 层面处理（Runtime.evaluate 在页面上下文中运行，有完整权限）。