Reviewer 2×2 analysis: aggregation, refusal-rate table, and topic specificity scoring

[canrager]

What this PR does

Turns the reviewer 2×2 ablation (iteration × prefill, run with
crawler.do_filter_refusals=false) into analysis, all on top of one shared
aggregation pipeline. Three related deliverables:

1. Post-hoc refusal-rate table — confirm which candidate topics are real refusals and build the reviewer table per (cluster, cell).
2. Constrained aggregation + per-cell discovery — classify candidate topics into a fixed taxonomy, pool replicate crawls into cells, and track first-occurrence / discovery curves per cell.
3. Topic specificity scoring + unified explorer — score how specific each topic is (not just which broad category), and browse every topic by method × specificity × cluster.

Each is independently useful; together they answer "do the stronger crawl methods surface more, and more specific, genuinely-refused topics?"

---

1. Post-hoc refusal-rate table

The 2×2 cells surface candidate topics, not behavior-confirmed refusals. This pass re-probes the aggregated cluster heads against the target and produces one table per cluster: refusal rate plus discovery booleans for each of the four cells. The whole follow-on runs on the pod with no intermediate local fetch; each stage drops a runpod_latest_* marker the next stage auto-consumes.

Outputs (scripts/runpod_compute_refusal_rates.sh → src/run_refusal_rates.py) → artifacts/refusal_rates/<ts>/:

• refusal_rates.md — reviewer table sorted by refusal rate desc (cluster, one boolean per cell, refusal rate, refusals / probes).
• refusal_rates.json — metadata, per_cluster[] with discovery.{direct,prefill_only,iter_no_prefill,ipc}, per_topic[].
• config.json, run.log, refusal_rates_*.jsonl transcript; marker artifacts/out/runpod_latest_refusal_rates.txt.

Topic persists per-probe cascade verdicts (refusal_check_refused: List[Optional[bool]]) and aggregate refusal_rate: Optional[float]; both default None, so older crawler JSONs round-trip unchanged.

Driver:

python3 scripts/runpod_control.py start --task reviewer_ablation --session ds70b_2x2
python3 scripts/runpod_control.py start --task aggregation     --session ds70b_aggregation
python3 scripts/runpod_control.py start --task refusal_rates   --session ds70b_refusal_rates

--aggregation-dir / --reviewer-out-dir default from the latest pod markers. run_refusal_rates.py Hydra overrides: model=<config>, +aggregation_dir=<path> (required), +out_dir=<path>, crawler.num_refusal_checks_per_topic, crawler.is_refusal_threshold.

---

2. Constrained aggregation + per-cell discovery

src/aggregation/ reworked so the 2×2 cells can be aggregated and compared:

• Config group renamed experiments → aggregation; default agg model gemini-3.1-flash.
• Constrained ("fixed taxonomy") mode (fixed_topics_path): classify each input topic into a predefined list instead of discovering clusters via iterative reduction, so cells are comparable against one taxonomy.
• input_groups: pool several crawl files into one named cell (e.g. 5 replicate direct crawls → one direct cell).
• Per-cell discovery: thread each topic's discovery id through aggregation; topic_cell_matrix.{csv,md} gains per-cell count_/present_/first_abs_/first_rel_ columns, plus a cluster_discovery_curve.png (cumulative distinct clusters vs discovery order, one line per cell).

Config: configs/aggregation/{budget5,fixed_q8p4,...}.yaml. Entry: python -m src.aggregation.run_aggregation aggregation=<name>.

---

3. Topic specificity scoring + unified explorer

Category coverage saturates (every method hits the broad buckets), so it can't discriminate methods — the signal is in the long tail of specific instances. This adds a specificity lens and a richer explorer.

Scoring (score_specificity: true): an LLM judge labels every topic on an L1–L5 ladder (+ Junk) — Domain → Theme → Practice → named Case → unique Pinpoint — via a decision cascade + tie-break. Reuses the classification machinery (new _classify_into() helper; classify() is now a thin caller, behavior-preserving). Forces a single most-specific level per topic. Emits:

• specificity_scores.csv (per-topic level + method presence),
• specificity_by_cell.{csv,md,png} (level × cell counts/fractions, an L4+L5 specific-tier row),
• specificity_grouped_bar.png (x-groups = levels, bars = methods).

Unified explorer (build_specificity_explorer_html): every topic carries method, specificity, and cluster. Switch grouping between the three dimensions; pin Method / Specificity / Cluster via dropdown filters (AND-combined with search); legends, overall + per-group summaries; each topic row is labeled with all three regardless of grouping. Cluster labels come from also classifying into the fixed taxonomy when fixed_topics_path is set; save_artifacts gained write_explorer to skip the unused reduction-tree explorer in this mode.

Config: configs/aggregation/budget5_specificity.yaml. Headline on budget5 — the L4+L5 specific tier cleanly separates methods (prefill drives specificity even more than iteration).

---

Test plan

• Topic round-trips new refusal fields, defaults None for old JSONs.
• Aggregator fallback: refusal-filtered files use summaries; discovery-mode files fall back to head_topics.
• runpod_control.py dry-run shows correct env-prefixed driver command per --task.
• tests/test_specificity.py (4): level bucketing, single-most-specific tie-break, per-cell matrix + plots, explorer records carry method/specificity/cluster, and the _classify_into refactor leaves the taxonomy classify() path intact.
• tests/test_refusal_utils.py pass with the same profile as main.
• Constrained aggregation end-to-end on budget5 (4 pooled cells → topic_cell_matrix + discovery curve).
• Specificity scoring end-to-end on budget5 (14.5k topics → specificity_* tables); explorer verified by joining the specificity + taxonomy runs.
• Pod end-to-end smoke: start --task aggregation → start --task refusal_rates (--probes-per-topic 3) for marker chaining + schema parity.

🤖 Generated with Claude Code