Skip to content

feat: phase 6 — metric catalogue + threshold scoring - #8

Merged
barancan merged 1 commit into
mainfrom
feature/phase-6-metrics
Jul 2, 2026
Merged

feat: phase 6 — metric catalogue + threshold scoring#8
barancan merged 1 commit into
mainfrom
feature/phase-6-metrics

Conversation

@barancan

@barancan barancan commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • Metric catalogue replaces the 7 starter chips with 12 categorised entries (format, correctness, safety, performance, rag, quality), each carrying determinism, default_threshold, and the requirement text used to toggle it
  • Threshold scoringCheckResult gains a threshold field; from_raw() overrides passed = score >= threshold when both are present; threshold flows through to CaseResult.checks JSON for display in transcripts
  • Generator — keyword detection expanded to cover toxicity (≥0.90), hallucination (≥0.85), faithfulness / answer relevance / task completion (≥0.80); thresholds embedded into check with dict at generation time so they're editable via the existing PATCH endpoint
  • Wizard Step 1 — chips grouped by category with section headers; active chip colour matches determinism (blue = deterministic, purple = stochastic); stochastic chips show (≥0.80) threshold hint
  • Wizard Step 2 preview — judge check rows show threshold next to the assertion
  • Transcript — score vs threshold display (0.72 < 0.80) was stubbed in Phase 5 and is now live

Test plan

  • pytest -q149 passed
  • Open /pipelines/new — Step 1 chips should be grouped by category (format, safety, rag, etc.) with section labels; stochastic chips should be purple when active and show threshold hints
  • Select "RAG faithfulness" chip — requirement text should appear in the textarea
  • Generate a pipeline with a toxicity/faithfulness requirement — confirm the generated check with dict contains threshold on the review page
  • Run a pipeline with a graded judge check — confirm transcript shows score ≥ threshold or score < threshold format
  • Verify deterministic chips (Valid JSON, No PII) show no threshold hint

🤖 Generated with Claude Code

Spec / model layer
- CheckSpec gains optional threshold field (min passing score 0-1)
- Case gains optional context field (RAG retrieved passages)

Check result layer
- CheckResult gains threshold field; stored in CaseResult.checks JSON
- from_raw(): when threshold set and score present, overrides passed = score >= threshold
- registry.run_check(): extracts threshold from spec.with_['threshold'] or spec.threshold

Generator
- _JUDGE_KEYWORDS table: toxicity 0.90, hallucination 0.85, faithfulness/relevance/
  completion 0.80 — matched by regex against requirements text
- intents_to_spec(): embeds threshold in check with dict for judge intents
- _INTENT_PROMPT updated with full catalogue and threshold guidance
- Additional keyword detection: pii_absent, and expanded judge patterns

Server
- _STARTER_TEMPLATES → _METRIC_CATALOGUE (12 entries with determinism/category/
  default_threshold; passed to template as metric_catalogue)
- POST /pipelines/preview now returns threshold per check intent

Wizard (pipeline_new.html)
- Step 1 chips grouped by category with section headers
- Active chip colour: blue for deterministic, purple for stochastic
- Stochastic chips show threshold hint (≥0.80)
- Step 2 preview card shows threshold next to judge check assertions

Transcript (_transcript.html) already had score/threshold display from Phase 5

6 new tests in tests/test_phase6.py; all 149 tests passing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@barancan
barancan merged commit 681c0be into main Jul 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant