Add Reson8 turns STT service + median-of-N WER scoring - #21
Open
ThomasKluiters wants to merge 1 commit into
Open
Add Reson8 turns STT service + median-of-N WER scoring#21ThomasKluiters wants to merge 1 commit into
ThomasKluiters wants to merge 1 commit into
Conversation
- services_custom/reson8_stt.py: Reson8 WSS Turns API client — streams PCM and maps turn_start/turn_end_candidate/turn_continuation/turn_end to Pipecat interim/final frames; TTFS measured to the last candidate. Native endpointing via the min/max_patience query params (0.2/0.2). - services.py / models.py / config.py / env.example: register the reson8 service (model "Resonant-1 Turns"). - evaluation/semantic_wer.py + cli/wer.py: --passes N (default 1) scores each sample N times and keeps the median-by-total-errors pass, suppressing the Claude scorer's normalization non-determinism (~±0.1pp). No-op at the default. - README + Pareto plots: Reson8 leaderboard entry — 1.10% pooled WER, 85.6% perfect, 326ms median TTFS (1000 samples) — on the accuracy/latency frontier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
markbackman
reviewed
Jun 14, 2026
markbackman
left a comment
markbackman
left a comment
Contributor
There was a problem hiding this comment.
Thanks for contributing! Two comments.
| @@ -0,0 +1,346 @@ | |||
| """Reson8 speech-to-text service implementation. | |||
Contributor
There was a problem hiding this comment.
This repo shouldn't carry code for any custom service. If you have a service to test, please provide a PyPI package for a Pipecat service that can be run directly.
| | NVIDIA | Nemotron 3.0 ASR (en) | 100.0% | 76.1% | 1.90% | 1.95% | 221ms | 238ms | 252ms | | ||
| | NVIDIA | Nemotron 3.5 ASR (multilingual) | 99.6% | 62.0% | 4.54% | 4.58% | 236ms | 253ms | 266ms | | ||
| | OpenAI | gpt-4o-transcribe | 99.3% | 75.9% | 3.24% | 3.06% | 637ms | 965ms | 1655ms | | ||
| | Reson8 | Resonant-1 Turns | 99.9% | 85.6% | 1.39% | 1.10% | 326ms | 639ms | 1156ms | |
Contributor
There was a problem hiding this comment.
One of the benchmark maintainers will have to independently run the integration to validate the results.
Author
|
@markbackman Would you prefer the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds the Reson8 as a provider, specifically it adds the turns endpoint, which is designed for eager turn detection, though, using a comparable technique as Soniox's VAD flush behaviour, we implement a comparable approach through configuration.
Additionally, Claude was giving us variable results, initially giving us a pooled WER of 1.09%; as I doubted this result I re-ran it and obtained 1.19%. This PR allow us to take the median result from Claude, I observed that 9 passes was sufficient to cover most variance.