Conversation
Add runtime_checkable Protocols (ResearchProvider, WebGrounder, Synthesizer, CitationVerifier, CommunitySource), their null implementations, and build_providers() credential-gated wiring. Real providers are lazy-imported inside gated branches so the module imports and the all-nulls path works before those providers exist; missing credential or missing module degrades to a null carrying a reason that names what is absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fix round for Task 3. - deep_research: providers.perplexity -> providers.deep_research (C1) - synthesizers: providers.synthesis -> providers.synthesizers (C2) - linkedin: community.linkedin.LinkedInSource -> providers.linkedin.LinkedInEnricher (C3) - community sources: per-module imports -> single community.adapters module (C4) Restructure _build_community_sources so keyless adapters (Reddit fallback, HackerNews, Bluesky, GitHub unauth) are attempted unconditionally and activate at any tier, while YouTube/X stay credential-gated; the whole layer degrades to skip when the adapters module is absent. Add a test pinning that degrade-to-skip behavior against a mocked-missing module. Verifier stays FirecrawlVerifier in verification.verifier (I1 confirmed binding). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the internal research-pipeline trigger classifier to the public package, minus all vertical-specific patterns. classify(topic, profile) returns a Classification (query_type PERSON/PRODUCT/COMPARISON/THEMATIC/ FACTUAL, suggested_depth, flags, cost_estimate) using pure rules — <10ms, zero network. Company/team/competitor lists are replaced by a pluggable DomainProfile loaded from user YAML via POLYSEARCH_DOMAIN_PROFILE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add LinkedInEnricher (ScrapeCreators, credential-gated on scrapecreators_api_key) that normalizes a public LinkedIn profile into a single SourceResult, surfacing only unmasked fields and flagging masked employment/education instead of emitting asterisks. Inactive without a key (no network, reason set); API/transport errors return None with the cause surfaced on reason. Add the runtime_checkable PersonContextHook protocol to providers.base — the public seam for registering a private person-context source via run_research(..., person_hook=...); default None. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts, domain filters, search API Ports the A1-upgraded internal perplexity_direct module to the public package: chat-completions Sonar path (parallel sub-question fan-out, citation parsing, cost/token tracking, tenacity retry, <think> stripping) plus current-API features (search_domain_filter, search_recency_filter, structured-output response_format). Adds the flat-rate Search API wrapper returning SourceResult, and PerplexityProvider adapting research() to the ResearchProvider protocol (LayerOutput). Models sourced from Settings; single _acquire() rate-limit seam left as a no-op TODO for Task 2.5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… example
Schema YAML now drives extraction directly: each schema declares a domain
and a list of {name, regex, context_window} patterns that the generic engine
compiles and runs, replacing the internal design's hard-coded per-domain
extractors. Ships the federalregister.gov schema (docket numbers, effective
dates, CFR cites) plus a loader that merges the bundled config dir with a
user directory via POLYSEARCH_SCHEMA_DIR (user extends/overrides built-ins).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…HECKING The linkedin slot held CommunitySource | None but LinkedInEnricher exposes enrich, not CommunitySource.search — a misleading annotation. Tighten it to LinkedInEnricher | None behind a TYPE_CHECKING guard so the runtime lazy import in _build_linkedin (and the credential-gated wiring) is preserved with no import cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the internal source-authority classifier to the public package as polysearch.sources.authority, backed by a scrubbed, generic config/domain_tiers.yaml (12,382 domains regenerated from the internal merged credibility map). - classify_url(url, *, published_date=None) -> Tier is the public entry and folds in the undated downgrade (HIGH->MEDIUM, MEDIUM->LOW). - classify_domain returns the underlying (tier, reason) pair; subdomain- aware matching, path downgrades, and github/huggingface primary-artifact path upgrades (Appendix A path_rules). - BLOCKED is a hard exclude (subdomain-aware, never softened by a path rule, never in high_tier_domains()); SME tier dropped. - Unknown domains logged under the injected Settings.output_dir, never a package-relative path; POLYSEARCH_DOMAIN_TIERS overrides the YAML path. - Public YAML strips every internal comment, drops SME, keeps the BLOCKED group and curated-core-first HIGH ordering (recovery allowlist), cites only public datasets (DQR, WP:RSP, CRED-1, Iffy, Tranco, Majestic), and carries zero vertical-specific / banned strings (guarded by a test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fixes for task 11: - Remove the "skip" depth value; Depth is quick/standard/deep only. Definitional (FACTUAL) queries now map to "quick", so --depth=skip can never be emitted (the public CLI would reject it). - Restrict Classification.flags to public-CLI-valid flags (today just --depth=). Drop --domain-context/--competitive/--recency/--x-handle/ --github-user/--github-repo. Their signals stay as typed fields: domain_related, competitive_mode, time_sensitive already existed; add typed x_handle/github_user/github_repo for the extracted references. - Assign _DEEP_RE.search(topic) once in _suggest_depth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_schema now compiles every pattern regex at load time and raises SchemaError naming the file, domain, and pattern name on a bad regex (instead of a bare re.error surfacing mid-extract). A schema file missing the required 'domain' key likewise raises SchemaError naming the file. Adds tests for the bad-regex path, the missing-domain path, and bad-regex surfacing through load_schemas() via POLYSEARCH_SCHEMA_DIR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add path_rules.downgrades_by_domain so contributor-network and native- advertising path patterns downgrade a source only on the named host, without a bare substring demoting unrelated domains. - forbes.com: /sites/ (contributor network) + /brandvoice/ (native ads); linkedin.com: /pulse/ (Appendix A rule 6). All force LOW on that host. - The global flat downgrade list is unchanged. classify_domain applies the scoped downgrade after the global one and before the upgrade; BLOCKED stays exempt from all path rules. - Guards the false positive the flat list couldn't: nih.gov/sites/default/ files/... stays HIGH while forbes.com/sites/... becomes LOW. Tests: forbes /sites/->LOW, forbes editorial->MEDIUM, nih /sites/ stays HIGH, linkedin /pulse/->LOW. Full suite 207 passed; YAML carries zero banned strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the shared file-ledger rate limiter to the public package. Each provider gets a sliding 60s window of request timestamps persisted under ~/.cache/polysearch/ratelimit/<provider>.json, guarded by fcntl.flock so concurrent processes share one budget. record_429() stamps a blocked_until floor honoring Retry-After, visible to every sibling process. Public deltas vs the internal module: ledger dir under ~/.cache/polysearch, POLYSEARCH_RPM_<PROVIDER> env overrides, and the release provider table (youtube exempt as quota-based). Degrades to in-process-only limiting on lock timeout / cache-dir failure; resets a corrupt state file in place. Tests are hermetic (injected clock + sleep, temp state dir) and an autouse conftest fixture points the default ledger at a temp dir so no test writes to the real ~/.cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the rate limiter into the provider call sites. perplexity.py's no-op
_acquire seam now delegates to ratelimit.acquire("perplexity") (single seam,
patchable by attribute), and linkedin.py routes its ScrapeCreators httpx call
through acquire("scrapecreators").
Move the primary Perplexity model's per-token pricing into Settings
(perplexity_price_in/out, defaulting to real sonar-pro rates $3/$15 per 1M
tokens, POLYSEARCH_PERPLEXITY_PRICE_* overridable). PerplexityProvider threads
these into _estimate_cost via a per-run pricing override; the deep/reasoning
model retains the module _PRICING table.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire record_429 into the provider error paths so a live 429 is visible to sibling processes via the file ledger, not just proactive pacing (plan Task 2.5 spec). perplexity's tenacity retry path records the 429 (with parsed Retry-After) before re-raising for backoff; linkedin's error handling records it on an HTTPStatusError 429. Add ratelimit.parse_retry_after for the shared numeric-seconds header parse. Test: a 429-with-Retry-After from a provider call pushes the next slot out for a second limiter instance sharing the ledger; plus a fast guard that perplexity's retry path records the 429. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FirecrawlGrounder (WebGrounder) grounds a topic in live web results: pluggable discovery per settings.discovery_backend (Perplexity Search default, Brave when keyed, Firecrawl /v2/search), Firecrawl-scraped top-K via the resilient fallback chain, source-authority tier tagging with BLOCKED-source drop, undated + paywall downgrades, and >12-token query trim. sources/scrape.py ports the Firecrawl -> httpx -> Playwright(optional) fallback chain with FETCH_BLOCKED vs URL_DEAD distinction, modernized onto the firecrawl-py v3+ AsyncFirecrawl.scrape API (Document return + typed status_code-bearing errors) from the legacy scrape_url dict shape. playwright ships as the [browser] optional extra.
… through Perplexity discovery ground() now mirrors synthesis parity (snippet or markdown[:600], whitespace collapsed) so a scraped source with a thin/empty discovery snippet still carries the content synthesis reads. Perplexity Search API supports search_recency_filter (hour/day/week/month/ year) on POST /search, so search() gains an optional recency passthrough and the grounder threads its recency into the default (Perplexity) backend — closing the earlier gap where only Brave time-filtered discovery.
…ort-circuit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + relevance gate, adapted from last30days (MIT) Six CommunitySource adapters in community/adapters.py (Reddit, HackerNews, Bluesky, GitHub, X, YouTube), each returning tier-COMMUNITY SourceResults with engagement and layer=source-slug, credentials read from Settings, failure- isolated (never raise; cause on last_error). A8 hardening: Reddit is OAuth-first (records the served tier on last_path), degrading through browser-UA public JSON to ScrapeCreators; Bluesky sends the realistic browser UA on every request and treats 403 as adapter-degrade. community/fusion.py: URL-canonical dedupe + recency-window drop + per-source engagement z-score / recency-decay ranking over SourceResult. community/filter.py: topic-keyword relevance gate; suppresses the layer with a Pipeline Decisions note when >70% of items are off-topic. Wires build_providers to activate the keyless set (Reddit/HN/Bluesky/GitHub) at any tier and add YouTube/X only when keyed; revisits the two stale no-keys wiring tests accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h verified citations
Async Chat Completions API (POST /v1/async/sonar, poll GET /v1/async/sonar/{id})
— the F-1-corrected contract; /v1/agent rejects sonar-deep-research with 400.
DeepResearchProvider adapts the long-form answer + citations into a LayerOutput
of SourceResult so synthesis/verification cover its citations like any layer.
Submit is rate-limited (acquire perplexity); polls are exempt but record_429.
Failure/timeout/FAILED isolate to LayerOutput.error, never raise.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A hostile or malformed Retry-After (e.g. 86400) was written straight into the shared ledger's blocked_until, and acquire() slept the entire remaining wait in one uncapped call — so one poisoned entry could pin a provider's cross-process bucket for hours across every caller, and an old poisoned blocked_until was slept through without re-checking. - record_429 clamps the backoff to _MAX_BACKOFF_SECONDS (300s). - acquire/acquire_sync cap each sleep at one WINDOW, then re-read the ledger, so a longer block is re-checked periodically and proceeds as soon as it clears. Normal within-window waits (<= one window) are unchanged — still a single sleep. Tests: record_429(86400) -> blocked_until clamped to now+300; acquire against a hand-poisoned ledger takes several window-capped sleeps and proceeds. Suite: 485 passed, 1 live skipped.
Add evals/ (excluded from the package build): a 12-task suite across six categories, a sweep runner targeting the public CLI or an env-supplied internal command, model-free programmatic metrics (verification rate, tier mix, dead links, key-fact coverage, refinement rounds, placeholder leaks, cost/duration), a gpt-5.4-nano LLM judge with a URL-free structured-output schema, and a documented release-gate. Failures are isolated per task as ERROR rows; judge parse failures never crash the sweep. Fully-mocked unit tests, no live calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The flat REFINEMENT_CEILING=4 only matched the deep profile, so a standard-depth run of 3 refinement iterations passed the gate even though the standard cap is 2. Read the ceiling per task from polysearch.config.DEPTH_PROFILES[depth].max_refinement_iterations (quick 0 / standard 2 / deep 4); --depth-override shifts the cap since it mutates the task depth before the sweep. RunMetrics now carries refinement_ceiling and refinement_within_ceiling; the gate flags any run over its own depth's cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…imeout, persistence Round 1 debugged the ruler, not the pipeline. Five fixes: F1 Internal-report adapter (evals/report_adapter.py): the internal PipelineReport uses different top-level field names (total_cost_usd, duration_sec, tiered sources spread across top-level lists, refinement_traces/followup_queries, synthesis object) than the public schema. normalize_report() maps both shapes into one. A field absent from BOTH shapes yields None + a per-row warning, never a silent zero — silent zeros ($0, 0s, tier 0.00, refine 0/exp) poisoned the whole internal round. Sources are collected from any tiered top-level list (no internal field names hardcoded, so the leakage gate stays green). F2 Coverage extraction: key-fact coverage now matches the collected report .md (falling back to the synthesis body only if no md), sentence-localized with token_set_ratio >= 70 so paraphrase is caught but scattered words are not. Decimal-safe sentence splitter keeps "3.75%" whole. None (not 0) when no text. F3 Verification granularity: the GATED metric is claim-level claims_supported/claims_total (both shapes carry it); the pair-level verified_ok/total_citations is kept as an ungated secondary column. A report supporting 16/16 claims no longer scores 0.12 because 25/200 pairs verified. F4 Timeout: default raised to 2700s, overridable via POLYSEARCH_EVAL_TIMEOUT_SEC. F5 Artifact persistence: per-task md+json persist under results/<label>/<target>/<task_id>/ (slug-matched collection so a shared output dir never returns the wrong task's report); the row records report_path. Gate + scoreboard are now None-safe: means skip missing metrics, a metric that is None for every run fails the gate (a poisoned round can't pass on empty data), and per-row warnings surface in the scoreboard. README + tests updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add evals/rescore.py to re-score existing report artifacts without re-running targets — needed to rescore round 1 with the fixed metrics, and useful every future round. Finds each task's report by topic slug (flat md+json pairs OR per-task subdirs), recomputes metrics via the adapter, optionally re-judges, and writes a scoreboard with the same builders/gate as the live sweep. A task with no matching artifact is SKIPPED, not ERROR. Refactors the shared report-loading seam out of run_evals (_match_report_json + _load_report; _collect_report behavior unchanged) so rescore reuses it with no logic duplication. Adds a SKIPPED status to the scoreboard summary/renderer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Large internal reports (tens of thousands of tokens) routinely hit the judge model's tokens-per-minute ceiling during a sweep — a transient 429 that was wasting the whole task's run as an ERROR. judge_report now retries a rate-limited call up to JUDGE_MAX_RETRIES (5) with backoff, honoring the server's suggested wait when given, else capped exponential. Non-rate-limit errors are not retried. sleep_fn is injectable so tests never actually wait. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1. Strip the pipeline's own process/audit sections from the md before judging.
The judge was scoring citation_accuracy off our self-audit ("Citation
Integrity" / "Failed citations" listing NUMBER_MISMATCH tallies), which is
process, not end state. strip_process_sections() excises process/audit H2
sections (Pipeline Decisions/Stats/Errors, Refinement Trace, Citation
Integrity, Style Audit) and audit H3 subsections (Excluded lists, Failed
citations) by safe keyword match — works for both report shapes, keeps
synthesis + sources-by-tier. build_judge_prompt strips by default;
--judge-full-md is the escape hatch.
2. Replace the hand-rolled 429 retry with tenacity: retry on rate-limit (429)
AND 5xx/connection/timeout, exponential backoff, up to JUDGE_MAX_RETRIES (5),
judge ERROR only after retries exhausted. Add inter-call pacing in the
production judge (make_default_judge) via judge_spacing_sec()
(POLYSEARCH_EVAL_JUDGE_RPS or 0.5s default) so a multi-task sweep doesn't
burst the org limit. Pacing lives in the sweep-side factory, so tests never
wait. --judge-full-md threaded through run_evals + rescore.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claim extraction attaches the whole corpus URL list to every claim, so the verifier expanded to claims × all_urls (claim, url) pairs and fetched each page once PER citing claim — the same ~40-70 unique URLs scraped 25-53× each. That made verification 83-92% of wall-clock and 89-97% of cost on public runs (20-30+ min, $3-7.61; 4 tasks timed out at 30 min). Split _verify_one into _scrape_one (network, once per unique URL) and _score_pair (CPU scoring against the shared fetch). Budget now caps unique URLs (tier-sorted) not pairs; cost counts unique scrapes. Per-pair _ScoredResult, claim-level rollup, and status precedence are unchanged, so verification coverage holds or improves. Validation (fed-rate topic, standard depth; timed out >30min in r1): 179.9s / $0.373, 79 unique scrapes for 1401 pairs, 53/56 claims supported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…om cost Review follow-up to the scrape-dedup fix. The score phase gathered all (claim, url) pairs with no concurrency bound; the per-pair embedding fallback makes an OpenAI call, so with O(claims × sources) pairs a large run could burst embedding requests into a 429 storm. Bound the score phase with a semaphore at the same max_concurrency the scrape phase uses (scoring is CPU + an occasional embedding call, so the bound is cheap). Also: a scrape that raised out of the chain never completed a billable fetch, so exclude those entries from total_cost. Adds a test pinning the bound: 30 pairs forced through the embedding fallback, concurrent embedding calls never exceed max_concurrency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aware excerpt budget Rework the synthesis prompt so Key Findings state concrete figures/dates/names/ ranges (with units and the source whose value was used) rather than exec-summary allusions, and require internal consistency (prose numbers must match the report's own lists/counts) to close the r1 P2 coverage gap and the P3 'cut twice vs three cuts' self-inconsistency. Add a config-gated per-source excerpt budget (synthesis_excerpt_chars, default 1200) applied to HIGH/MEDIUM-tier sources so authoritative material is not truncated to 600 chars before synthesis sees it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_claims now accepts (url, snippet) sources and attributes each claim only to sources whose snippet plausibly supports it (content-word overlap or literal figure match), falling back to the corpus-wide URL list when nothing relates. Previously every claim carried the entire corpus URL list, which polluted a claim's citations with off-topic pages (e.g. FDA/CDC pages listed as HIGH sources for a fed-rate report) and coarsened verification pairing. Rollup semantics are unchanged (still one scored result per claim/url pair, same claim-level rollup); the orchestrator wires layer snippets into the synthesis and answer extraction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a scraped authoritative page has a thin/empty discovery snippet, backfill from its markdown up to the configured wide budget (synthesis_excerpt_chars) for HIGH/MEDIUM tiers instead of the 600-char base — measured HIGH-tier discovery snippets in a real run were a median of 36 chars while the figures live in the scraped body. Non-authoritative tiers and the preserve-existing-snippet contract are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe rescore showed the key-fact coverage metric matches abstract fact phrasings (direction of change, count of events, most-recent date) that a value-only report misses. Instruct synthesis to answer each implied dimension in plain answer- language — current value, trend direction and size, event counts, leading entity, most-recent date — and to state explicitly when the material does not support one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round 2 showed key-fact coverage reading 0.000 on reports that demonstrably state the facts. A hand-checked case (comparison-postgres- mysql-oltp, "concurrency and MVCC handling differences" against a sentence that ties PostgreSQL's concurrent-write handling to MVCC) scored 45 under exact-token token_set_ratio matched against a single sentence: "concurrent" != "concurrency", "differences" absent, and short facts got diluted by connective words. Match every fact against 1-3 sentence windows (best window wins) after markdown-noise stripping, a small suffix stripper (s/es/ies->y/ing/ed/ tion(s)), and fact-side-only stopword dropping (and/the/of/for/each/ to). Calibrating against hand-checked real and synthetic cases moves KEY_FACT_MATCH_THRESHOLD from 70 to 45: covered facts land in the 50s-70s once normalized, absent facts land in the 20s-40s, with clear margin between the two. Added calibration tests cover the morphological-variant case, a same-text unrelated fact staying uncovered, a stopword-heavy fact not false-positiving on generic text (would have at 48 without the fact-side stopword drop), a fact split across a sentence window, and the actual r2 comparison-postgres-mysql-oltp artifact (facts 1 and 4 now covered, fact 2 correctly still not). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-oltp "replication and high-availability options for each" and "typical workloads each is better suited to" were among the worst abstract/meta-word-heavy fact phrasings in the suite — hard for any fuzzy matcher to pin to concrete report text regardless of threshold. Rephrase into concrete, noun-heavy wording a real report would actually state (streaming replication/group replication/failover; read-heavy/write-heavy/high-concurrency OLTP patterns), independent of the coverage-metric fix. Topic and facts stay fully generic per the leakage gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One paragraph on why the coverage metric changed: the broken-ruler defect, the normalization + windowing fix, the threshold move from 70 to 45 with calibration evidence, the tasks.yaml rephrasing, and the r1/r1-rescored rescore limitation (no per-task artifacts survive in this repo, only aggregate scoreboards). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Calibrating the lowered threshold against the full real artifact surfaced a genuine false positive: the report's own bibliography heading "### High (primary, peer-reviewed, official) (22)" shares the word "high" with the rephrased "high availability" fact and scored 45.28 — just clearing the calibrated 45 threshold — even though the report never actually discusses replication or HA. Section/tier-bucket headings are document structure, not stated content, and must never stand in for a fact regardless of incidental vocabulary overlap. _sentence_units now drops lines matching a markdown heading pattern before they enter the fact-matching window pool. Re-verified against the full 4-fact comparison-postgres-mysql-oltp set: facts 1 and 4 pass, fact 2 (replication) correctly fails again, fact 3 passes on a genuine partial match (indexing/query performance content is present). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…x fix
Fix round on task r3b's coverage-metric rescore (review findings):
CRITICAL — numeric guard. A fact carrying a number ("...approximately
18.8 percent") could score COVERED against a window that matches on
non-numeric vocabulary alone while stating a different (or no) number
— caught on a private sweep where "18.8" appeared nowhere in the
report. _key_fact_coverage now requires a covering window's raw text
to contain at least one of the fact's numbers (comma/trailing-zero-
tolerant, unit-agnostic so 18.8 ~ 18.8% ~ "18.8 percent", $200 ~ "200
dollars", 1,500 ~ 1500) in addition to clearing the text-similarity
threshold. Self-contained (_number_match_pattern), not imported from
polysearch.verification.verifier — that module pulls in the verifier's
httpx/Playwright/Firecrawl stack, which the eval harness has no other
reason to depend on; the regex-building approach is adapted from it.
IMPORTANT — non-synthesis section exclusion. The match pool still
included pipeline scaffolding: Refinement-Trace follow-up-query echoes
(maximally keyword-dense by construction) and Sources-by-Tier
bibliography entries, both of which could spuriously satisfy a fact
through incidental vocabulary overlap rather than a stated finding.
_strip_non_synthesis_sections drops entire H2 sections (Pipeline
Decisions/Errors/Stats, Refinement Trace, Citation Integrity, Sources
by *) before sentence-splitting, verified against the heading text of
every real report artifact on hand (both public and internal shapes
share this trailing structure). Broader than evals/judge.py's
strip_process_sections, which deliberately keeps the source list for
the judge's citation-accuracy scoring — coverage has no use for it.
MINOR — the suffix stripper's blanket "-es -> strip 2 chars" rule
turned "codes"/"types" into "cod"/"typ" instead of "code"/"type".
Now only strips both letters when the singular needs the sibilant
vowel back ("boxes" -> "box", "watches" -> "watch"); otherwise strips
only the "-s" ("codes" -> "code").
Re-verified against both real r2 public artifacts after all three
fixes: comparison-postgres-mysql-oltp facts 1/3/4 still pass, fact 2
(replication) still correctly fails; contested-four-day-workweek all
4 facts still pass. Full suite 580 passed, 1 skipped (base 566 + 14
new tests across both fix rounds). Leakage test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tests
Reviewer specifics on the numeric guard (task r3b, second review pass):
1a. Second root cause: _normalize_for_match tokenized on [a-z0-9]+,
which split "18.8" into "18"/"8" before either the text-similarity
score or the numeric guard ever saw it. _TOKEN_RE now tries a decimal
pattern (\d+\.\d+) before the plain alnum-run alternative, so decimal
number spans survive whole.
1b. Unit/magnitude awareness. The previous round's guard was fully
unit-agnostic (checked only the bare digit value), which left a real
gap: a fact stating "$1.2 million" could be satisfied by an unrelated
"1.2 percent" anywhere in the window — a factor of a million is not a
coincidental rounding difference. _fact_required_numbers now detects
%/$/magnitude (million/billion/thousand, or an attached M/B/K) context
around each number in the fact and, when present, additionally
requires a same-class marker (symbol or spelled-out word, either
counts) somewhere in the covering window. Also switched from "at least
one number matches" to "every number matches" per the reviewer's
explicit spec — a fact citing two figures needs both present, not one.
2a. Confirmed via test that the tier-label leak already doesn't
survive in bullet form ("- Low (opinion, marketing, unverified) (3)"),
since the section-exclusion fix from the prior commit removes the
whole H2 section body, not just heading lines.
6 new tests: decimal-tokenization regression, wrong-number-present vs
missing-number-entirely (the reviewer's two distinct required cases),
bullet-form tier-label regression, magnitude/unit-awareness (the gap
found while implementing 1b), and multi-number "requires each" check.
Full suite 586 passed, 1 skipped (base 566 + 20 new tests across all
three fix rounds). Leakage test green. Re-verified both real r2 public
artifacts once more: no change from the previous round's results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round 2's public eval sweep still timed out on two run classes: technical-webgpu-adoption (deep depth) and recent-ai-model-releases (RECENT/standard). deep_research_timeout_s (3600s) legitimately outlives the eval ceiling (2700s), and community adapters (Reddit's fallback chain, rate-limiter 429 backoff) had no per-call bound so retries could stack past 300s and stall the whole fused layer. - run_research gains an optional time_budget_s (Settings + POLYSEARCH_TIME_BUDGET_S + CLI --time-budget). When set, every first-pass layer shares one remaining-budget snapshot; a layer that would exceed it is cancelled via asyncio.wait_for and recorded in pipeline_errors with duration_ms populated, never a hard timeout with zero output. Refinement is skipped (with a note) once the budget is already spent. - DeepResearchProvider.research accepts time_budget_s and polls with min(deep_research_timeout_s, remaining budget). - Community adapters get a per-adapter timeout (community_adapter_timeout_s, default 60s), always applied independent of the global budget, so one blocked source can't stall fusion; siblings' results still land. - run_evals.py exports POLYSEARCH_TIME_BUDGET_S at ceiling-120s to both targets so a run self-limits and reports partial results instead of being killed with nothing. - CLI logs each layer's duration to stderr after a run, for future timeout diagnosis. All new behavior is TDD'd with mocked asyncio (no live API calls); full suite green (605 passed, 1 skipped, up from 586/1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…loop Review of the time-budget work (task r3c) found the fix was incomplete: first_pass_budget handed the FULL remaining budget to the first pass, so when it legitimately consumed most of the budget, synthesis/verification/ recovery ran inside only the harness's ~120s outer slack — if any of them exceeded that, the subprocess got SIGKILLed before write_report ran, the exact zero-output failure this task targeted. Separately, refinement's skip check used an exact `refinement_budget == 0.0` comparison, so any small positive remainder (a few seconds) was treated as "plenty of time" and let a deep-profile 4-iteration loop start with no wall-clock bound on its own duration. - The first pass now gets only _FIRST_PASS_BUDGET_FRACTION (80%) of whatever's remaining, reserving a tail slice for what follows. - Added _bounded_await(), a shared asyncio.wait_for-based enforcement point: synthesis, verification, and the recovery pass (+ its re-verify) are each now bounded by the actual remaining budget at that point, converting a timeout into a pipeline_errors entry instead of letting the phase run unbounded. - Refinement's skip check now uses a _MIN_REFINEMENT_BUDGET_S (30s) floor instead of an exact-zero comparison, and the loop itself (once started) is wrapped in the same bounded-await so it can't outlive the budget mid-iteration either. TDD: 7 new integration tests cover the full-slice-consumed-but-report- still-written invariant, individual cancellation of synthesis/ verification/recovery, the floor-vs-exact-zero distinction, and bounding an already-started refinement loop. Full suite green across 3 repeated runs (612 passed, 1 skipped, up from 605/1); leakage test green before commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… allowlist The recovery pass forced domain_filter=high_tier_domains()[:20] on every re-sourcing query regardless of topic, so an off-topic report (e.g. PostgreSQL internals) got NIH/BLS/SEC citations injected into its HIGH bucket. The anti-fabrication prompt still asks for primary sources, but the API-level domain force is gone; recovered results are now gated against the claim they were re-sourced for (content-word overlap >= 0.30, or a literal numeric-figure match — reusing extractors/claims.py's localization scoring). fix(report): trim uncited sources from tier buckets to a collapsed count note "Sources by Quality Tier" listed every source a claim-verifiable layer collected, whether or not any claim actually cited it. Sources never attributed to a checked claim now collapse into a one-line "N additional sources consulted but not cited" note. Grounding/community/linkedin layers (never individually claim-verified) and runs where verification didn't run at all are exempt from the trim.
…ifacts absent (CI)
milock
added a commit
that referenced
this pull request
Jul 17, 2026
polysearch v1.0.0 — full pipeline, plugin packaging, eval harness
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First stable release. Everything the README promises, built and evaluated:
Dist:
polysearch-ai· import/CLI:polysearch· version 1.0.0 across package/plugin/marketplace (lockstep-tested).🤖 Generated with Claude Code