Goal
Turn the already-implemented runtime measurement/statistics owners into an end-to-end repeatability workflow without inventing a confidence score: repeated executions of the exact same frozen experiment expose quality/runtime variability, denominators and failures so the user can judge how stable the evidence is.
Canonical semantics
- A repeatability cohort is exact
ExecutionFingerprint.fingerprint_id equality. Do not pool merely similar runs.
performance/single_request.py remains the owner of client-boundary request timing; evaluation must not create a second inference call just to measure it.performance/statistics.py::summarize_distribution remains the owner of mean/median/stddev/CV and qualified p90/p95.- Run/sample failures and cancellations remain explicit denominators and are never converted to zero-valued measurements.
- No universal confidence score, stability threshold, PASS/FAIL verdict or statistical significance claim is introduced by this slice.
- Representative thermal/load/repeatability claims remain
REAL_ENVIRONMENT evidence.
VALUE-05A — Canonical measured sample evidence
Wire the existing single-request measurement protocol into the normal evaluation inference path so successful SampleExecution evidence retains available client-provenance measurements (for example total latency and trustworthy token throughput) from the same inference used for evaluation. Preserve evidence-rich capture and one inference call per sample.
VALUE-05B — Exact-fingerprint repeatability projection
Add a versioned application/API read model for a retained Run that:
- finds all completed Runs with the same exact fingerprint;
- reports total/succeeded/failed/cancelled run counts;
- reports total/succeeded/failed/cancelled sample-attempt counts;
- projects per-run quality aggregate values and per-run means of matching client performance measurements;
- uses
summarize_distribution across those per-run values;
- keeps percentile qualification and insufficient-repeat/unavailable states explicit.
VALUE-05C — Run Detail repeatability UX
Expose the repeatability projection in Run Detail with progressive disclosure. Show exact-fingerprint cohort size, denominators/failures and distribution evidence; do not imply a winner or verdict. Add deterministic browser/API coverage for available, insufficient-repeat and failure-denominator states.
VALUE-05D — EVID-002 representative acceptance
After final software convergence and fresh exact-head PRE_REAL, run controlled repeated real executions through Local LLM Server on representative hardware. Retain warmup/load assumptions, variability, failures and denominators. Hosted/fixture CI cannot satisfy this claim.
Acceptance
- same experiment means exact fingerprint equality;
- measurements come from the existing performance owner and the same inference call;
- no frontend-authored statistical semantics;
- p90/p95 remain qualified by the existing statistics owner;
- failed/cancelled runs and samples remain visible in denominators;
- old Runs without client measurements remain valid and render unavailable rather than fabricated;
- deterministic software acceptance makes no representative device/thermal claim.
Goal
Turn the already-implemented runtime measurement/statistics owners into an end-to-end repeatability workflow without inventing a confidence score: repeated executions of the exact same frozen experiment expose quality/runtime variability, denominators and failures so the user can judge how stable the evidence is.
Canonical semantics
ExecutionFingerprint.fingerprint_idequality. Do not pool merely similar runs.performance/single_request.pyremains the owner of client-boundary request timing; evaluation must not create a second inference call just to measure it.performance/statistics.py::summarize_distributionremains the owner of mean/median/stddev/CV and qualified p90/p95.REAL_ENVIRONMENTevidence.VALUE-05A — Canonical measured sample evidence
Wire the existing single-request measurement protocol into the normal evaluation inference path so successful
SampleExecutionevidence retains available client-provenance measurements (for example total latency and trustworthy token throughput) from the same inference used for evaluation. Preserve evidence-rich capture and one inference call per sample.VALUE-05B — Exact-fingerprint repeatability projection
Add a versioned application/API read model for a retained Run that:
summarize_distributionacross those per-run values;VALUE-05C — Run Detail repeatability UX
Expose the repeatability projection in Run Detail with progressive disclosure. Show exact-fingerprint cohort size, denominators/failures and distribution evidence; do not imply a winner or verdict. Add deterministic browser/API coverage for available, insufficient-repeat and failure-denominator states.
VALUE-05D — EVID-002 representative acceptance
After final software convergence and fresh exact-head PRE_REAL, run controlled repeated real executions through Local LLM Server on representative hardware. Retain warmup/load assumptions, variability, failures and denominators. Hosted/fixture CI cannot satisfy this claim.
Acceptance