Add generic cascade rule engine + τ=0.80 default + RouterArena context

[doramirdor]

Summary

Lands the verifier-gated cascade, the heuristic post-hoc verifier, and a
generic data-driven cascade rule engine in the free / MIT core. Same
architecture Nadir uses on its currently-projected #5 RouterArena
submission and its RouterBench AUROC-0.961 / ECE-0.016 numbers — minus
the trained DeBERTa verifier and the trained classifier artifact, both
of which remain proprietary to Nadir Pro. The open-source surface is
the routing topology and the rule engine on top of it; users can swap
the heuristic verifier for their own implementation while keeping the
same dispatch contract.

Three commits, ordered for review:

1. feat(cascade): verifier-gated cascade + heuristic verifier + rule engine

   • nadirclaw/cascade.py — cheap-first dispatch, fail-open verifier
     errors, kill switch after 3 consecutive errors, default τ=0.80.
   • nadirclaw/heuristic_verifier.py — rule-based (regex + stdlib),
     ~1 ms / call. Detects refusals, hard-min length, ratio truncation,
     uncertainty, JSON parse failure.
   • nadirclaw/cascade_rules/ — declarative YAML rule engine.
     Conditions: substring / regex / prompt-length / classifier-
     confidence. Actions: force_escalate, force_cheap,
     set_threshold (stacks, max wins), set_max_tokens (stacks, max
     wins for safer routing-side default). TTL + mtime hot-reload
     cache so operators can edit a profile YAML on disk and see the
     new policy within 30 s without a restart.
   • Bundled default.yaml encodes the legacy force-escalate
     patterns and per-domain thresholds for code / summarisation.
   • pyproject.toml — PyYAML is optional via new cascade-rules
     extra; load_inline works without it.
   • 64 new tests covering parsing, priority ordering, stacking,
     malformed-rule rejection, hot-reload, and cascade integration.

2. chore(verifier): contamination audit utility for benchmark reproducibility

   • verifier/contamination_audit.py — standalone, stdlib-only CLI +
     library. Given any benchmark file(s) + corpus file(s), computes
     NFC + casefold + SHA-256 hashes and reports overlap. Exits 0 on
     zero overlap, 2 on any overlap, 1 on missing inputs — drop-in CI
     gate. Supports .jsonl / .json / .txt.
   • Reproduces the audit numbers behind Nadir's published held-out
     claims (RouterBench 0/36,481; RouterArena sub_10 0/809;
     RouterArena full 0/8,399).
   • 9 new tests.

3. docs: MODEL_CARD for wide_deep_asym_v3 + README benchmarks section

   • MODEL_CARD.md — wide-and-deep asymmetric architecture, training
     corpus, contamination posture, held-out numbers, per-domain
     verifier AUROC variance that motivates the default rule profile,
     τ-sweep table, explicit Pro-vs-OSS split.
   • README.md — new "Benchmarks" section under "Why NadirClaw" with
     RouterBench (AUROC 0.961, ECE 0.016, 98.3% quality preserved at
     τ=0.80) and RouterArena (sub_10 composite 0.7118, projected
     Add resilient routing, fallback handling, and setup/admin management to server API #5) numbers, plus the zero-overlap contamination table. Links to
     the open RouterArena submission PR
     (RouteWorks/RouterArena#112).
   • New cascade + rule-engine bullets in the Features list.

Context

• RouterArena sub_10 — composite score 0.7118, projected Add resilient routing, fallback handling, and setup/admin management to server API #5
  on the public leaderboard. Submission PR (live, under review):
  Add Nadir router (verifier-gated cascade + cost-min baseline) RouteWorks/RouterArena#112
• RouterBench held-out (n=11,420) — AUROC 0.961, ECE 0.016.
  At τ=0.80: 98.3% of always-Opus quality preserved, 1.7%
  catastrophic-downgrade rate, ~60% cost reduction vs always-Opus.
• Contamination audit — zero overlap between Nadir's classifier
  training corpus and either RouterBench 0shot (0 of 36,481) or
  either RouterArena split (0 of 809 / 0 of 8,399). Reproducible from
  this PR with verifier/contamination_audit.py.

What is NOT in this PR

By design — these stay on the Nadir Pro side and are not portable as
open-source:

• The trained wide_deep_asym_v3.pt classifier artifact (binary
  centroid and DistilBERT classifiers stay in NadirClaw as before).
• The trained DeBERTa-v3-small cross-encoder verifier (the rule-based
  heuristic verifier shipped here mirrors the interface, ~0.60 AUROC).
• Per-tenant Supabase wiring and the production /v1/route_only
  endpoint.
• Internal labeled-data shards used to train the classifier and
  benchmark-specific rule profiles (routerarena_v3.yaml etc.).

Backwards compatibility

• Cascade(cheap_call=..., expensive_call=...) without a rule_engine
  argument behaves exactly as before — the engine is opt-in.
• DEFAULT_ACCEPTANCE_THRESHOLD ships as 0.80 (was 0.5 in the
  draft cascade module locally). Pass threshold=0.5 to restore the
  more permissive cut.
• A buggy custom rule engine cannot fail the request: evaluate()
  exceptions are swallowed and the cascade falls through to the
  verifier path.

Test plan

• pytest -q --ignore=tests/test_e2e.py — 678 passed
• pytest tests/test_e2e.py -q — 38 passed
• pytest tests/test_cascade_rule_engine.py tests/test_heuristic_verifier.py tests/test_contamination_audit.py -q — 53 passed
• Default profile loads and force-escalates on ```python,
  ```javascript, ```typescript, def ,
  function , and summarize the following / summarize this.
• Threshold-stacking rules raise the verifier bar; only the strictest matched threshold wins.
• Hot-reload picks up file changes via mtime invalidation.
• Contamination audit returns exit-code 0 on zero overlap, 2 on overlap, 1 on missing inputs.

[doramirdor]

Two follow-up commits on this branch closing the two open gaps from the PR audit:

1326016 — bundle the trained wide_deep_asym_v3 checkpoint

The MODEL_CARD described the architecture but the .pt weights were Pro-only, so OSS users got just the heuristic verifier. Bundling fixes that:

• nadirclaw/models/wide_deep_asym_v3.pt (905 KB, λ=3 asym CE loss)
• nadirclaw/models/wide_deep_sym_v3.pt (905 KB, plain CE — recovers correct simple-class behaviour under argmax decoding)
• nadirclaw/wide_deep_classifier.py — singleton-cached loader with argmax + cost-sensitive decoders
• nadirclaw/structural_features.py — 33-d feature extractor (pure regex)
• pyproject.toml — models/*.pt added to package-data so the weights ship in the wheel
• tests/test_wide_deep_classifier.py — 10 integration tests loading the real bundled weights

Picked Option A (bundle) over the HF download or "ship the recipe" alternatives because the file is tiny and the MODEL_CARD already documented the architecture, so shipping the weights closes the loop with the published benchmark numbers. Weights are MIT-licensed alongside the rest of the package.

from nadirclaw.wide_deep_classifier import get_wide_deep_classifier
clf = get_wide_deep_classifier(checkpoint_variant="asym", decision_rule="cost_sensitive", cost_lambda=20.0)
result = clf.classify("Your prompt")

c0fd1b6 — multi-provider routing profile + reproducibility doc

When the model menu spans Gemini + OpenAI + Anthropic + Llama-class fallback, the default profile (calibrated for an Anthropic-only ladder) doesn't model: refusal-style drift, CoT ability gaps, JSON-wrapping inconsistency, or length-control drift on summarisation.

• nadirclaw/cascade_rules/profiles/multi_provider.yaml — 12 rules covering those four failure modes
• docs/multi-provider-routing.md — learnings writeup + reproducibility recipe for running NadirClaw's classifier + rule engine over cached benchmark responses (RouterArena's ./cached_results/ pattern) with no live API calls
• 4 new tests in tests/test_cascade_rule_engine.py

Test status: PR #59's full PR-relevant suite (tests/test_cascade_rule_engine.py + tests/test_heuristic_verifier.py + tests/test_contamination_audit.py) plus the new tests/test_wide_deep_classifier.py = 67 passed, 0 failed.