Scope
US3 MVP — declarative prompt configuration: every judge-backed evaluator accepts custom template, few-shot examples, system prompt, output schema, use-reasoning flag, and feedback key without code change. Prompt version is always recorded in results.
Priority: P1 (MVP)
Tasks
Acceptance
- A consumer can override the built-in
CorrectnessEvaluator prompt with a custom template + few-shot examples via builder methods only; no evaluator code is modified.
- The custom template's version string appears in the resulting
EvalMetricResult::details.
- Bumping a prompt version (
_v0 → _v1) is explicit; old version remains accessible.
- Missing or misspelled template variables surface at evaluator construction time.
References
- Spec FR-010, FR-011, Success criterion SC-005
Depends on
#750 (evaluators must exist to override).
Scope
US3 MVP — declarative prompt configuration: every judge-backed evaluator accepts custom template, few-shot examples, system prompt, output schema, use-reasoning flag, and feedback key without code change. Prompt version is always recorded in results.
Priority: P1 (MVP)
Tasks
eval/tests/us3_custom_prompt_test.rs—.with_prompt(custom)replaces built-in, few-shot injection order, version bump is explicit opt-in, missing variable is construction-time error.with_prompt(),.with_few_shot(),.with_system_prompt(),.with_output_schema(),.with_use_reasoning(),.with_feedback_key()— all route throughJudgeEvaluatorConfigprompt_versionrecorded in everyEvalMetricResult::details(re-verification of T056 in context of all evaluators)correctness_v0(built-in) andcorrectness_v1(custom) both resolvable; results distinguish them per-metricAcceptance
CorrectnessEvaluatorprompt with a custom template + few-shot examples via builder methods only; no evaluator code is modified.EvalMetricResult::details._v0→_v1) is explicit; old version remains accessible.References
Depends on
#750 (evaluators must exist to override).