Skip to content

v0.19: TrainedVerifier (RouterArena 0.7358 snapshot)#61

Merged
doramirdor merged 2 commits into
mainfrom
feature/trained-verifier-v0.19
May 29, 2026
Merged

v0.19: TrainedVerifier (RouterArena 0.7358 snapshot)#61
doramirdor merged 2 commits into
mainfrom
feature/trained-verifier-v0.19

Conversation

@doramirdor
Copy link
Copy Markdown
Collaborator

Summary

Ships the trained DeBERTa-v3-small cross-encoder verifier as an MIT-licensed snapshot, fulfilling the public reproducibility commitment made in RouterArena PR #112:

The exact trained snapshot used here will be released under MIT as `NadirRouter/cascade-verifier-v1` in NadirClaw v0.14 within 14 days of this PR opening, so any reviewer can reproduce 0.7358 end-to-end.

(Numbered v0.19 because NadirClaw shipped 0.14 through 0.18 in the interim; same content, same intent.)

What lands

  • `nadirclaw/trained_verifier.py` — `TrainedVerifier` class. Same `score(prompt, cheap_answer, expect_json=...)` signature and return shape as `HeuristicVerifier`. Lazy load on first call, optional preload, CPU/CUDA/MPS via `device=` constructor arg.
  • `tier_config/profiles/n2_trained.yaml` — N=2 profile that routes verifier decisions through the trained model. `n2_default` stays on the heuristic, so users without the transformer stack keep working unchanged.
  • `CascadeConfig` schema — new `verifier` ("heuristic" | "trained") and `verifier_model` fields. Validated; typos fail fast. Defaults preserve v0.18 behaviour.
  • `NTierCascade.init` — auto-instantiates the trained verifier when the loaded profile specifies it. Lazy import so the heuristic-only path never touches torch.
  • Optional extras: `pip install nadirclaw[trained]` pulls `transformers>=4.40` and `torch>=2.0`. Heuristic-only install stays zero-extra-deps.
  • README: new "Trained verifier" section under "N-tier routing" — install, activation, what is and is not released.
  • Tests: 9 new tests under `tests/test_trained_verifier.py`. Full suite 773/773 passing locally.

Weights release

The HuggingFace upload to `NadirRouter/cascade-verifier-v1` is pending — the local environment running this PR did not have HuggingFace auth, so I am landing the code path now and will push the weights manually right after. The local weights are at `getnadir.dev/verifier/weights/best/` (DeBERTa-v3-small, 6 layers, INT8 quantised `verifier_int8.pt` plus full FP32 `model.safetensors` and tokenizer).

Once the HF repo is live the `TrainedVerifier` constructor reaches it transparently — no code change needed.

Backward compatibility

  • `n2_default` keeps the heuristic verifier. Users who do not opt into the trained extras see zero behavioural change.
  • Existing `Cascade` two-tier API is untouched.
  • `pip install nadirclaw` (no extras) still installs with no new dependencies.

What's NOT released

Per the RouterArena commitment we are releasing the frozen snapshot, not the pipeline:

OSS (this PR) Pro
Frozen verifier weights (v1) YES (MIT) YES
Training pipeline + corpus NO YES
Adaptive retraining loop NO YES
Custom-routed quality scoring NO YES

Test plan

  • `python -m pytest tests/ -q` — 773 passed locally
  • `n2_trained` profile parses and resolves to the trained verifier
  • `n2_default` still resolves to the heuristic (backward compat)
  • Empty cheap_answer short-circuits to 0.0 without invoking the model
  • Interface parity with `HeuristicVerifier` (same kwargs, same return shape)
  • Schema rejects unknown verifier names (typo guard)
  • CI green on Python 3.10 / 3.11 / 3.12 (will verify in this PR)
  • Manual smoke: load real weights from HF once published, verify a single `score()` call returns a float in [0, 1]

Repo: https://github.com/NadirRouter/NadirClaw
Service: https://getnadir.com

Nadir added 2 commits May 28, 2026 21:51
- TrainedVerifier loads NadirRouter/cascade-verifier-v1 from
  HuggingFace (or a local HF cache). Same interface shape as
  HeuristicVerifier — score(prompt, cheap_answer, expect_json=...)
  returns a TrainedScore with .score / .accepted / .threshold /
  .reasons / .to_dict().
- New n2_trained profile uses the trained verifier; n2_default
  stays on the heuristic so users who do not want the transformer
  stack pay nothing for it.
- CascadeConfig schema: new `verifier` and `verifier_model` fields.
  Validated against {"heuristic", "trained"} so typos fail fast.
  Defaults preserve v0.18 behaviour.
- NTierCascade auto-instantiates TrainedVerifier when the loaded
  profile specifies verifier: trained. Lazy import keeps the
  heuristic-only path free of transformers/torch.
- Optional install: pip install nadirclaw[trained] pulls
  transformers>=4.40 and torch>=2.0.
- README: new "Trained verifier" section explains install,
  activation (NADIRCLAW_TIERS_PROFILE=n2_trained), and what is and
  is not released (frozen weights MIT; training pipeline and
  adaptive retraining remain Pro-only).
- 9 new tests; full suite 773/773 passing.

This is the frozen snapshot used in RouterArena PR #112
(arena_F 0.7358). Closes the 14-day reproducibility commitment in
RouteWorks/RouterArena#112

Training pipeline and adaptive retraining loop remain proprietary
to Nadir Pro; only the frozen weights are released.

Repo: https://github.com/NadirRouter/NadirClaw
Service: https://getnadir.com
@doramirdor doramirdor merged commit 3c65a5e into main May 29, 2026
3 checks passed
@doramirdor doramirdor deleted the feature/trained-verifier-v0.19 branch May 29, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant