Live LLM judge on the Lovable gateway (cheap model) wired into the adversarial harness - #33
Merged
Merged
Conversation
- judge.server.ts: createLlmJudge/getLlmJudgeOrNull — rubric-driven pass/fail judge via getGatewayModel (default google/gemini-2.5-flash), server-only, injection-resistant system prompt, structured Output schema - judge.ts: gradeWithJudge mode-safe orchestrator (deterministic fallback on judge error/absence) + rubricFromExpectations - runner.ts: runAdversarialSuite accepts judge/judgeMode; strict ensemble can only catch missed failures, records judge overrides in the trace - tests: +6 cases (gradeWithJudge fallbacks, rubric mapping) — 14 judge / 26 total green - docs: mark live judge as shipped
- runner.ts: runAdversarialSuite returns judge_calibration (agreement + Cohen's kappa between the LLM judge and the deterministic grader) - calibrate.server.ts: admin server fn runAdversarialWithJudge runs the suite with the live judge against a published package and inserts an adversarial_runs row including judge_model/cases/overrides/agreement/kappa - migration: additive judge_* columns on adversarial_runs (drift signal the Trust Score pipeline can read) - docs updated
- calibrate.server.ts: getJudgeCalibration server fn — per-package kappa/agreement history from adversarial_runs, lowest kappa first (drift surfaced) - admin.calibration.tsx: dashboard with kappa tone (Landis & Koch bands), agreement, override count, kappa sparkline, and an inline 'run judged eval' trigger per slug - admin nav: add 'Judge calibration' link
- recalibrateJudgeAgainstGolden server fn: judges active package_golden_cases references against human label_pass, computes agreement + Cohen's kappa, persists to package_evaluations.judge_calibration - admin.calibration.tsx: 'Recalibrate vs golden' action surfacing kappa/agreement/false-pass - docs: scheduling guidance (Vercel Cron / GH Action drives the Node-side judge)
…rate - runner.ts: compute judge_calibration via judgeCalibration() (single source of truth for agreement/kappa) instead of an inline loop + cohenKappa - calibrate.server.ts: adversarial_runs.model now records the skill-under-test model (describeGatewayConfig().defaultModel), kept distinct from judge_model
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #32. Plugs a real model behind the
JudgeFnintroduced earlier — no new infra to host: it reuses the existing AI gateway (getGatewayModel, defaults to the Lovable Cloud gateway) with a cheap model.What's new
src/lib/adversarial/judge.server.ts—createLlmJudge()/getLlmJudgeOrNull(): a rubric-driven pass/fail judge callinggetGatewayModel("google/gemini-2.5-flash")with a structuredOutputschema. Server-only (*.server.ts), so the key never reaches the client. System prompt is injection-resistant (judges the OUTPUT, never follows instructions inside input/output) and conservative on safety.src/lib/adversarial/judge.ts— addedgradeWithJudge, a mode-safe orchestrator that falls back to the deterministic grader when the judge errors or no gateway is configured (so the harness never crashes or flips a verdict on judge failure), plusrubricFromExpectationsto derive a semantic rubric from a case's existing expectations.src/lib/adversarial/runner.ts—runAdversarialSuitenow acceptsjudge/judgeMode. In the default strict mode the judge can only catch failures the substring checks missed — it never lowers the safety bar — and any override is recorded in the failure trace.Why this design
createLlmJudge({ model }).getLlmJudgeOrNull()returns null and the harness runs deterministic-only.Tests
tests/adversarial-judge.test.mjsnow 14 cases (addedgradeWithJudgefallback + rubric mapping); 26 green across the dependency-free TS suites. The live gateway call itself isn't unit-tested (needs an API key); it's a thin wrapper over the samegenerateText+Output.objectpatternpipelines.server.tsalready uses.Note
runAdversarialSuitehas no production caller yet — this lands the seam + the judge so a server function can opt in with one argument. Couldn't run typecheck/build here (node_modulesabsent); unit tests pass undernode --experimental-strip-types.https://claude.ai/code/session_01BnYrQNoKNNxPCSmV2suWna
Generated by Claude Code