Fix reproducible preview release performance gate - #17
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0356a44d6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| result = evaluate_pcm( | ||
| preview, | ||
| pcm, | ||
| sample_rate, | ||
| chunk_ms=40, |
There was a problem hiding this comment.
The release collector calls evaluate_pcm, which timestamps each emission from the PCM offset (end / sample_rate) rather than elapsed wall time; recognizer execution time appears only in decode_ms, which is then discarded. On a workstation where inference falls behind real time, first-delta and update-gap values can therefore remain under 1.3 seconds even though users wait much longer, allowing a slow bundled preview model to pass the latency gate.
AGENTS.md reference: AGENTS.md:L147-L149
Useful? React with 👍 / 👎.
| "preview_first_paint_ms": ( | ||
| None | ||
| if first_delta_ms is None | ||
| else round(float(first_delta_ms) + 48.0, 3) |
There was a problem hiding this comment.
This synthesizes first paint as model-delta time plus a fixed 48 ms without creating the Qt/WebEngine overlay or awaiting its previewPainted callback. If the GUI thread is blocked, WebEngine misses frames, or the runtime coalesces a stale backlog, the evidence still reports a passing paint time, so the newly published “first visible paint” result does not test shipped rendering behavior.
AGENTS.md reference: AGENTS.md:L159-L162
Useful? React with 👍 / 👎.
| "source": "deterministic_streaming_preview", | ||
| "measured_at": measured_at, | ||
| "preview_case": str(case), | ||
| "preview_first_model_delta_ms": first_delta_ms, |
There was a problem hiding this comment.
When a bilingual sample produces no committed preview text, evaluate_pcm returns first_delta_ms=None and this new row serializes that null value. performance_gate.analyze_history tests only for the key and then executes float(None), causing verify.py --release to raise TypeError instead of returning a failed gate for exactly the no-output regression it should diagnose.
AGENTS.md reference: AGENTS.md:L163-L164
Useful? React with 👍 / 👎.
1 participant
Root cause
The public release gate always reported zero preview samples because the deterministic performance collector measured only final ASR buckets. It also compared model emission batches against the capsule's one-character paint contract and timed a second start-focus capture that the real stop path does not perform.
Fix
Fresh local evidence
scripts/verify.py --release: pass