Run date: 2026-08-23
Release: 0.4.0b1
MemCoder includes a reproducible provider-free evaluation for the decision intervention layer introduced in Beta 4. Run it from the repository root:
python scripts/run_beta4_evaluation.pyThe harness runs 24 matched tasks through the real lifecycle and retrieval paths: 16 prompts where verified field-validation guidance should be retrieved and eight unrelated prompts where MemCoder should abstain.
| Gate | Result |
|---|---|
| Matched tasks | 24 |
| Retrieval precision | 100% |
| Correct abstention | 100% |
| Harmful transfer rate | 0% |
| Host-blocking failures | 0 |
| Intervention action-change rate | 100% |
| Median token dividend | 150 tokens |
| P95 lifecycle latency | 105.17 ms |
All declared controlled gates passed in the recorded run. Latency varies by machine, so reruns should be treated as the source of truth for a new host.
- verified guidance can change the compact decision supplied to a host;
- unrelated work receives a safe abstention instead of forced transfer;
- the measured lifecycle remains provider-free and fail-open; and
- host budget and token-accounting gates remain observable.
This harness does not measure independent developer rework, end-to-end coding correctness, or broad performance across real repositories. Those claims still require matched external host runs under the real-project protocol. MemCoder therefore reports this as controlled release evidence, not a universal productivity claim.