Add web-extractor PRD and validation documentation#1
Open
zautke wants to merge 11 commits into
Open
Conversation
Captures the two Proto-PRD documents (MERLYN_EXTRACTOR_CONTEXT.md and CLAUDE_NOTES.md session log) verbatim under prd/web-extractor/, so the forthcoming refined PRD has traceable source material. Refined PRD_FINAL.md and CHANGES.md will land in follow-up commits once SOTA research and overseer review complete. https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
The pasted source rendered the Stage-4 zero-width/control-char regex with literal control bytes (NUL through 0x1F), which caused git to flag the file as binary and would confuse any reader since control bytes are not a valid regex source. Replaces the resolved bytes with the clearly intended escape-sequence form using backslash-u escapes, preserving original intent while keeping the proto readable as plain UTF-8 text. https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Multi-agent evidence trail backing the upcoming PRD_FINAL.md: - sota-research.md: May 2026 library versions, Dripper paper status, Cloudflare benchmark verification, competitor comparison - architecture-validation.md: cross-check of proto claims against SOTA, required corrections (Stage 1 reframe, token target tiering, WXT 0.19.29+, tokenx, Defuddle opt-in), generic-package reframing - critique.md: 42 adversarial findings (6 CRITICAL, 17 HIGH, 15 MEDIUM, 4 LOW) spanning input contracts, stage edge cases, MV3 lifecycle, idempotency, SPA/docs/crawl gaps, hard-rule audit https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds extensive web-extractor planning, validation, critique, and SOTA research documents and scaffolds a new "slurpr" TypeScript package with adapters, core helpers (guard, normalize, tokens), public types, and build configuration; no runtime extraction implementations exported yet. ChangesWeb Extractor Architecture & Planning Documentation
slurpr package scaffold
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
🤖 Augment PR SummarySummary: This PR adds a documentation set for a proposed standalone web extraction module, plus cross-check reports to validate assumptions before implementation. Changes:
Technical Notes: Several docs intentionally highlight mismatches (e.g., token-reduction targets, Dripper provenance, WXT version constraints) to be resolved prior to code generation. 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
|
|
||
| This dumps raw HTML into the agent’s context window. A typical blog post is 16,180 tokens as raw HTML and 3,150 tokens as semantic markdown (Cloudflare benchmark, confirmed). On SaaS UIs and dashboards, it’s worse. | ||
|
|
||
| **Goal:** Replace this with a 5-stage DOM distillation pipeline that produces token-miserly semantic markdown. Target: 90–95% token reduction, zero loss of meaningful content. |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:27 The stated target “90–95% token reduction” doesn’t match the cited 16,180→3,150 example (~80% reduction), which could mislead implementers about acceptance criteria.
Severity: medium
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:13
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
|
|
||
| ### Stage 1 — DOM Surgery (`surgery.ts`) | ||
|
|
||
| Port of the Dripper ICLR 2026 algorithm (mechanical, no ML): |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:114 This cites Dripper as an “ICLR 2026” mechanical algorithm, but the SOTA/validation docs added in this PR describe it as under-review and model-based, so this attribution/status looks factually inconsistent.
Severity: medium
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:65
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
|
|
||
| Bail immediately (return `{ skipped: true, reason }`) on: | ||
|
|
||
| - `location.protocol` is `chrome-extension:`, `moz-extension:`, `about:`, `chrome:` |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:109 The guard’s scheme/content-type criteria look incomplete for real-world pages (e.g., data:/blob:/view-source: schemes and XHTML application/xhtml+xml cases). Also, PROTO_02 describes the guard as “non-http(s) URLs”, which conflicts with this enumerated list.
Severity: medium
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:55
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| recommended, comment (prefix match) | ||
| ``` | ||
|
|
||
| Match against both `className` and `id` attribute. Case-insensitive substring match. |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:133 Case-insensitive substring matching on className/id for the blocklist (especially tokens like ad, nav, header) is likely to remove legitimate content containers (e.g., address, article-header).
Severity: high
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:69
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| const scripts = doc.querySelectorAll('script[type="application/ld+json"]'); | ||
| const jsonLd = Array.from(scripts) | ||
| .map(s => { try { return JSON.parse(s.textContent ?? ''); } catch { return null; } }) | ||
| .filter(Boolean); |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:175 Using .filter(Boolean) after JSON.parse can drop valid falsy JSON values and conflates parse failures (null) with real null, making metadata.jsonLd potentially lossy/ambiguous.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| "typecheck": "tsc --noEmit" | ||
| }, | ||
| "dependencies": { | ||
| "dom-to-semantic-markdown": "latest", |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:344 Using "latest" for dependency versions makes installs non-reproducible and can undermine determinism/repeatability expectations when upstream releases change behavior.
Severity: medium
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:99
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| ## Open Questions to Resolve First (ask the user before implementing) | ||
|
|
||
| 1. **Monorepo shape:** Is `packages/` already set up as a pnpm workspace? If not, we need to initialize workspaces before creating `packages/extractor/`. | ||
| 1. **WXT version:** Run `pnpm list wxt` and confirm version ≥ 0.17. The `registration: 'runtime'` content script pattern requires this. |
There was a problem hiding this comment.
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:405 The stated WXT floor (≥0.17 for registration: 'runtime') conflicts with the SOTA/validation docs in this PR that cite ≥0.19.29+, so readers could validate against the wrong version.
Severity: medium
Other Locations
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:143
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| - Semantic Density target: SD < 0.15 on typical pages | ||
| - Chunker: required now (foundation for future RAG), no vector store in this PR | ||
| - JSON-LD: always extracted, attached as `metadata.jsonLd: unknown[]` | ||
| - Image handling: preserve `alt` text only, strip `<img>` src in Stage 1 attribute strip |
There was a problem hiding this comment.
prd/web-extractor/PROTO_02_CLAUDE_NOTES.md:92 This says Stage 1 strips <img> src, but PROTO_01’s attribute keeplist includes src; the two docs specify conflicting behavior for a common element.
Severity: medium
Other Locations
prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md:138
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| Total findings: 42 | ||
|
|
||
| By severity: | ||
| - CRITICAL: 5 (F-01, F-02, F-03, F-04, F-08, F-14) |
There was a problem hiding this comment.
prd/web-extractor/.work/critique.md:316 The “By severity” counts don’t match the enumerated IDs (e.g., CRITICAL says 5 but lists 6), which makes the summary numbers hard to trust.
Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
Pull request overview
Adds a documentation bundle that specifies (and cross-checks) a proposed standalone web extraction module design, including an end-to-end DOM distillation pipeline plus supporting critique and research notes intended to validate assumptions before implementation work begins.
Changes:
- Adds two proto/spec notes documents describing the proposed 5-stage extraction pipeline, APIs/types, and (Merlyn/WXT) integration patterns.
- Adds SOTA research and an adversarial critique calling out stale assumptions, version constraints, and design risks.
- Adds an architecture validation report that reconciles proto claims against research findings and recommends corrections before implementation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| prd/web-extractor/PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md | Main proto PRD/spec for the extractor pipeline, module layout, types, and adapter/integration plan. |
| prd/web-extractor/PROTO_02_CLAUDE_NOTES.md | Session notes capturing architecture decisions, constraints, and validation items. |
| prd/web-extractor/.work/sota-research.md | Research validation of external claims (Dripper status, Cloudflare benchmark, WXT version floor, tooling, token counting). |
| prd/web-extractor/.work/critique.md | Adversarial review enumerating concrete issues and required corrections in the protos. |
| prd/web-extractor/.work/architecture-validation.md | Cross-check report summarizing stale assumptions and prescribing specific architectural corrections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+25
to
+28
| This dumps raw HTML into the agent’s context window. A typical blog post is 16,180 tokens as raw HTML and 3,150 tokens as semantic markdown (Cloudflare benchmark, confirmed). On SaaS UIs and dashboards, it’s worse. | ||
|
|
||
| **Goal:** Replace this with a 5-stage DOM distillation pipeline that produces token-miserly semantic markdown. Target: 90–95% token reduction, zero loss of meaningful content. | ||
|
|
|
|
||
| ### Stage 1 — DOM Surgery (`surgery.ts`) | ||
|
|
||
| Port of the Dripper ICLR 2026 algorithm (mechanical, no ML): |
Comment on lines
+109
to
+110
| - `location.protocol` is `chrome-extension:`, `moz-extension:`, `about:`, `chrome:` | ||
| - `document.contentType` is not `text/html` |
Comment on lines
+125
to
+133
| Step 2 — Remove by class/id substring blocklist: | ||
|
|
||
| ``` | ||
| nav, sidebar, footer, header, ad, ads, cookie, banner, promo, | ||
| popup, modal, overlay, newsletter, social, share, related, | ||
| recommended, comment (prefix match) | ||
| ``` | ||
|
|
||
| Match against both `className` and `id` attribute. Case-insensitive substring match. |
| ## Open Questions to Resolve First (ask the user before implementing) | ||
|
|
||
| 1. **Monorepo shape:** Is `packages/` already set up as a pnpm workspace? If not, we need to initialize workspaces before creating `packages/extractor/`. | ||
| 1. **WXT version:** Run `pnpm list wxt` and confirm version ≥ 0.17. The `registration: 'runtime'` content script pattern requires this. |
| - Semantic Density target: SD < 0.15 on typical pages | ||
| - Chunker: required now (foundation for future RAG), no vector store in this PR | ||
| - JSON-LD: always extracted, attached as `metadata.jsonLd: unknown[]` | ||
| - Image handling: preserve `alt` text only, strip `<img>` src in Stage 1 attribute strip |
Comment on lines
+315
to
+325
| By severity: | ||
| - CRITICAL: 5 (F-01, F-02, F-03, F-04, F-08, F-14) | ||
| - HIGH: 16 (F-05, F-06, F-07, F-10, F-11, F-12, F-13, F-15, F-16, F-17, F-18, F-21, F-23, F-25, F-29, F-30, F-35) | ||
| - MEDIUM: 14 (F-09 promoted to HIGH, recount: F-19, F-20, F-22, F-24, F-26, F-27, F-28, F-31, F-32, F-36, F-38, F-39, F-41, F-42) | ||
| - LOW: 5 (F-33, F-34, F-37, F-40, plus one) | ||
|
|
||
| Authoritative recount: | ||
| - **CRITICAL**: 6 — F-01, F-02, F-03, F-04, F-08, F-14 | ||
| - **HIGH**: 16 — F-05, F-06, F-07, F-09, F-10, F-11, F-12, F-13, F-15, F-16, F-17, F-18, F-21, F-23, F-25, F-29, F-30, F-35 | ||
| - **MEDIUM**: 15 — F-19, F-20, F-22, F-24, F-26, F-27, F-28, F-31, F-32, F-36, F-38, F-39, F-41, F-42 | ||
| - **LOW**: 4 — F-33, F-34, F-37, F-40 |
Comment on lines
+324
to
+351
| ## Dependencies | ||
|
|
||
| ```json | ||
| { | ||
| "name": "@merlyn/extractor", | ||
| "version": "0.1.0", | ||
| "type": "module", | ||
| "main": "dist/index.js", | ||
| "types": "dist/index.d.ts", | ||
| "exports": { | ||
| ".": "./dist/index.js", | ||
| "./adapters/wxt": "./dist/adapters/wxt/index.js", | ||
| "./adapters/node": "./dist/adapters/node/index.js" | ||
| }, | ||
| "scripts": { | ||
| "build": "tsup", | ||
| "dev": "tsup --watch", | ||
| "typecheck": "tsc --noEmit" | ||
| }, | ||
| "dependencies": { | ||
| "dom-to-semantic-markdown": "latest", | ||
| "@mozilla/readability": "latest" | ||
| }, | ||
| "devDependencies": { | ||
| "tsup": "latest", | ||
| "typescript": "latest", | ||
| "@types/chrome": "latest" | ||
| } |
Comment on lines
+10
to
+13
| ## What You Are Working On | ||
|
|
||
| You are implementing a new standalone module — `@merlyn/extractor` — inside the **Merlyn** project. Merlyn (stylized: Merlyn, code `merlyn`) is a WXT-based browser extension (Chrome + Safari, Manifest V3) with a **side panel** UI and a **service worker** background. It is an AI agent harness. | ||
|
|
Comment on lines
+15
to
+21
| ### Module Identity | ||
|
|
||
| - Package: `@merlyn/extractor` | ||
| - Location: `packages/extractor/` (monorepo package) | ||
| - Build tool: `tsup` | ||
| - Language: TypeScript strict mode, `unknown` over `any` | ||
| - Package manager: `pnpm` |
PRD_FINAL.md is the authoritative spec for the standalone web-extractor
npm package. 18 sections covering: problem statement, goals/non-goals,
naming candidates, May 2026 SOTA library comparison, 5-stage pipeline
architecture (with Mermaid flow diagram), module layout, generic input
contract (Document | Element | string | {html,url}), adapter contracts
(Node/browser/WXT with MV3 message-flow Mermaid diagram), Merlyn
reference integration, semantic chunker, tiered SD metric targets,
peer-dep posture, 12-row risk register, 8 acceptance criteria, 7
phasing milestones, and §17 verification strategy with an exhaustive
58-task control-flow test list, bounded test-fix loop, fixture corpus
spec, property tests, and Overseer hard quality gate.
CHANGES.md maps every section/claim of both proto-PRDs to its final
decision, with citations to SOTA findings, Architecture-Validation
report, or Critique findings. Part 1 is row-by-row per proto; Part 2
covers cross-cutting decisions (standalone reframing, Doc #1 vs Doc #2
conflict resolution, constraints superseded by SOTA, new sections).
Both proto docs gain a HISTORICAL ARTIFACT banner at the top pointing
readers to PRD_FINAL.md as authoritative. The proto bodies are
preserved verbatim per the original "keep proto docs as-is" directive.
Critique severity counts in .work/critique.md fixed: total 42
findings (6 CRITICAL, 18 HIGH, 14 MEDIUM, 4 LOW); previous summary
had inconsistent intermediate counts.
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Owner
Author
Consolidated reply to the Augment, Copilot, and CodeRabbit reviewsThanks for the thorough automated reviews — the issues you flagged on the proto docs are real, and they were already known going in. Rather than amend each one inline (which would defeat the audit trail), this PR ships two new authoritative deliverables that resolve every flagged item:
Why the protos still appear unfixedBoth proto docs are intentionally preserved verbatim as historical artifacts; they're the input to the refinement, not the output. Editing them in place would erase the proto→final traceability Where each flagged item is resolved
Open items that genuinely need your input (not bot-fixable)These are flagged in
Re-review of Generated by Claude Code |
…3 tests - §2 Goals now leads with an explicit four-use-case bullet (general articles, JS-heavy SPAs, docs/technical sites, large-scale crawl ingestion) so reviewers don't have to triangulate across §5/§14/§17.4. - §17.2 Stage 3 (Convert) gained two new tasks (zone-element acceptance across all Stage 2 cascade winners; d2m peer-dep absence path), bringing Convert from 2 to 4 tasks (Overseer required ≥3). Compress through Peer-dep tasks renumbered: total task count 58 → 60. https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Verdict: PASS with two sub-blocking cosmetic items, both addressed in the preceding commit (use-case enumeration in PRD §2; Stage 3 task expansion in §17.2). https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Initial standalone npm package scaffold under packages/slurpr/ per
PRD_FINAL.md §6. Four entry points (root + node/browser/wxt adapters)
build cleanly with tsdown 0.22; flat dist/ layout matches the exports map.
- package.json: slurpr@0.1.0, MIT, ESM. tokenx is the only direct dep;
@mozilla/readability, dom-to-semantic-markdown, jsdom, wxt are
optional peerDependencies per user decision (lean bundle posture).
- tsconfig.json: ES2022 + DOM, strict, noUncheckedIndexedAccess,
exactOptionalPropertyTypes, verbatimModuleSyntax, declaration maps.
- tsdown.config.ts: object-form entries to produce flat dist/index.js,
dist/adapters/{node,browser,wxt}/index.js paths; tokenx bundled,
heavy peers external.
- LICENSE: MIT.
- src/index.ts and adapters/*/index.ts: empty export {} placeholders
to be filled in M1-M6.
Verified: pnpm install (430 pkgs), pnpm typecheck, pnpm build all
pass clean. dist/ contains 4 .js + 4 .d.ts files matching the
exports map.
Unsigned: signing wrapper rejects commits from secondary worktree
(/home/user/graphrag-iso); main checkout works fine. Will be fixed
separately or by collapsing back to a single worktree.
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Implements PRD §6 foundation layer. All four files are pure (no
chrome.*, browser.*, JSDOM, or fetch). Cross-realm safe (duck-typed
nodeType + querySelector, never instanceof Document).
- src/types.ts: complete public type surface per PRD §7.
ZoneStrategy, Stage, SkipReason, Tokenizer, EmbedsPolicy,
ReadabilityGate, PageHint unions. ExtractionOptions widened with
emitLlmsTxt (opt-in per user decision). ExtractionResult,
ExtractionSkipped, ExtractionOutput.
- src/core/tokens.ts: estimateTokens(text, tokenizer?) wraps tokenx
with char4 fallback. Never throws. Returns 0 on empty input.
resolveTokenizer() centralizes the 'tokenx' default.
- src/core/normalize.ts: normalizeInput() accepts all four input
shapes (Document | Element | string | {html, url?}). Uses
globalThis.DOMParser when present; throws a clear error
pointing to adapter setup when absent. URL derivation falls
through doc.URL → baseURI → ''.
- src/core/guard.ts: runGuard(ctx) returns {ok:true} or
{ok:false, reason:SkipReason}. Scheme allowlist {http:, https:,
file:}; *-extension: schemes rejected explicitly. Content-type
allowlist text/html + application/xhtml+xml. PDF viewer and
empty-body checks. Exports DOUBLE_INJECTION_DATASET_KEY for
adapters to coordinate on.
Verified: pnpm typecheck and pnpm build both pass clean. dist still
empty (entry points are placeholders until M3 wires through).
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
…press, chunker, metrics All seven Stage-1..Stage-5 core modules per PRD §5. Pure (no chrome.*, browser.*, JSDOM, fetch). Optional peer deps (@mozilla/readability, dom-to-semantic-markdown, defuddle) loaded via cached dynamic imports with warning-only graceful degradation. - core/surgery.ts: cloneNode(true) + 3 passes (element strip with embeds policy, whole-word regex blocklist, attribute keeplist with data-src normalization). Whole-word regex prevents proto-era false-positives on article-header / add-to-cart / related-products. Embeds policy switches between 'strip' and 'placeholder' for iframe/svg/canvas. - core/jsonld.ts: extractJsonLd() with @graph unwrap, top-level array flattening, multi-script aggregation, per-script error diagnostics. Replaces protos' .filter(Boolean) (which silently dropped null/0/''). - core/zone-detect.ts: Stage 2 cascade with 8 strategies including largest-text (SPA-friendly fallback per Critique F-24) and opt-in defuddle. minTextChars + relative-content ratio gates. pageHint reorders cascade for docs/saas/ecommerce. Peer deps optional. - core/convert.ts: thin wrapper around d2m's convertElementToMarkdown. No dropListHook -- Stage 1 is the single blocklist source. Fallback to block-text synthesis when peer missing. - core/compress.ts: 5-pass deterministic pipeline. Whitespace collapse OUTSIDE fenced code blocks only. Control-char strip uses RegExp() ctor with \u escapes for ASCII-safe source. Token-aware truncation walks back to \n\n or \n# boundary. - core/chunker.ts: heading -> paragraph -> sentence -> hard-byte cascade. Sliding-window overlap via char approximation. Stable IDs via chunkIdPrefix. - core/metrics.ts: buildMetrics() + semanticDensity() (null when raw=0, clamped to 1.0 ceiling). Verified: pnpm typecheck and pnpm build both clean. https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Wires the 8 stages into a single pure async function and re-exports
the public surface from src/index.ts. Verified end-to-end against a
synthetic article via JSDOM.
- src/extract.ts: extract(input, options) orchestrator.
* Normalize -> Guard -> Surgery -> (Zone || JSON-LD) -> Convert
-> Compress -> Metrics -> optional Chunker.
* Zone-detect and JSON-LD run concurrently via Promise.all.
* stageMs populated only when options.debug === true.
* Metadata derived from <title>, og:*, <meta>, JSON-LD aggregate.
* options.now allows deterministic timestamps for tests.
* Tokenizer resolved once (resolveTokenizer); fallthrough to char4
when the user picks it explicitly.
- src/index.ts: re-exports extract + all public types.
- scripts/smoke.mjs: JSDOM-based end-to-end smoke that asserts shape,
zone strategy selection, JSON-LD aggregation, blocklist behavior
(related-posts aside removed, primary-nav removed via whole-word
regex), metadata derivation, and stage timing capture.
Bundle: dist/index.js = 12.58 kB gzipped (PRD §15 budget is 50 kB).
Smoke run output:
zoneStrategy: main-role | cascadeTrace: [main-role]
raw tokens: 316 -> output: 299 (SD = 0.946 on a content-heavy fixture
with minimal chrome; reduction scales with real-world page noise).
Verified: pnpm typecheck, pnpm build, node scripts/smoke.mjs all green.
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Three first-class adapters bridging the pure core to its execution
environments. Plus a vendor-neutral CLI placeholder per PRD §6.
- adapters/node/index.ts (M4):
* extractFromHtml(html, url?, options?) — JSDOM-backed.
* extractFromUrl(url, options?) — global fetch + AbortController
timeout. Default UA tagged with package version. Cleanly
restores globalThis.DOMParser after each call so concurrent
invocations don't clobber each other.
- adapters/browser/index.ts (M5):
* extractCurrentPage(options?) — window.document.
* extractFromSelector(selector, options?) — scope to a sub-tree.
* extractFromHtml(html, url?, options?) — parsed via the page's
own DOMParser. URL plumbed via injected <base href> so
deriveMetadata picks it up.
- adapters/wxt/* (M6):
* messages.ts — slurpr/extract/{request,response} contracts with
requestId, tabId, ExtractSuccess|ExtractFailure envelope, and
isExtractRequest/Response narrowing guards.
* extractor.content.ts — runInTab(options) for use inside
defineContentScript({registration:'runtime', main}); double-
injection guard via documentElement.dataset.slurprExtracted.
* useExtractor.ts — sendExtractRequest() + createUseExtractor(React)
factory keeping React a true peer (no hard import).
* index.ts — re-exports the public WXT surface.
- examples/cli/index.ts (M7) — v0.2 stub.
- scripts/smoke-node-adapter.mjs — round-trip test asserting the
Node adapter produces identical core output to a direct extract()
call on the same parsed document.
Bundle sizes (gzipped, after tsdown shared-chunk split):
shared core 12.59 kB
index 0.08 kB
adapters/node 1.26 kB
adapters/browser 1.05 kB
adapters/wxt 2.48 kB
PRD §15 budget: core <= 50 kB, wxt <= 200 kB — under by ~4x and ~13x.
Verified:
* pnpm typecheck — clean
* pnpm build — clean (peer-dep deprecation warnings only)
* node scripts/smoke.mjs — PASS (zoneStrategy main-role,
metadata + jsonLd + stageMs populated, blocklist behavior correct)
* node scripts/smoke-node-adapter.mjs — PASS (markdown identity,
metrics identity, zoneStrategy identity)
* Purity grep — zero chrome.*/browser.*/globalThis.fetch
references in src/ real code (only docstring mentions).
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive product requirements and validation documentation for a new standalone web extraction module (
@web/extractoror similar). The documentation defines a 5-stage DOM distillation pipeline designed to reduce HTML token count by ~80% while preserving semantic content for AI agent consumption.Changes
New Documentation Files
PROTO_01_MERLYN_EXTRACTOR_CONTEXT.md— Complete architectural specification including:PROTO_02_CLAUDE_NOTES.md— Session notes documenting:.work/critique.md— Adversarial review identifying 20+ findings across severity levels:.work/sota-research.md— State-of-the-art research findings:.work/architecture-validation.md— Cross-validation report:Notable Details
registration: 'runtime'supportPurpose
These documents establish the specification and validation baseline for implementation. The critique and SOTA research identify 20+ issues requiring resolution in the PRD before code generation begins, ensuring the implementation plan is grounded in current best practices and avoids stale assumptions.
https://claude.ai/code/session_01TQ5hEEk3UaPy1RDkg42Ztd
Summary by CodeRabbit