From ef5439e201005783fd41592a8fc620aa5a9c3363 Mon Sep 17 00:00:00 2001 From: Freddy Martinez Garcia Date: Thu, 17 Sep 2026 17:58:43 -0300 Subject: [PATCH 1/3] QVAC-25109 feat[asr-ggml]: propagate Unified RNN-T cache-aware streaming speech-cpp now drives parakeet-unified-en-0.6b through its native cache-aware encoder instead of re-encoding a sliding window, so the chunk cadence selects a trained operating point rather than an arbitrary window. Give the model its own default of 560 ms, one of the trained chunks, instead of the generic 2000 ms that the engine would snap down anyway, and document the accepted chunk and right-lookahead values on the driver and the parakeet options. The addon keeps forwarding out-of-set values untouched so validation stays in speech-cpp. --- packages/asr-ggml/CHANGELOG.md | 18 ++++++++++--- packages/asr-ggml/README.md | 4 +-- .../parakeet/ParakeetConfig.hpp | 1 + .../parakeet/ParakeetModel.cpp | 2 ++ .../tests/parakeet_test_config_and_audio.cpp | 27 +++++++++++++++++++ .../asr-ggml/engines/parakeet/driver.d.ts | 12 ++++++--- .../asr-ggml/engines/parakeet/parakeet.d.ts | 5 +++- .../asr-ggml/src/engines/parakeet/driver.ts | 12 ++++++--- .../asr-ggml/src/engines/parakeet/parakeet.ts | 5 +++- packages/asr-ggml/vcpkg.json | 8 +++--- 10 files changed, 76 insertions(+), 18 deletions(-) diff --git a/packages/asr-ggml/CHANGELOG.md b/packages/asr-ggml/CHANGELOG.md index aa2898d63a..32e27e54bc 100644 --- a/packages/asr-ggml/CHANGELOG.md +++ b/packages/asr-ggml/CHANGELOG.md @@ -14,12 +14,22 @@ restarts at `0.1.0`; the two pre-merge histories are preserved verbatim as ## [Unreleased] +### Added + +- Cache-aware streaming for `parakeet-unified-en-0.6b`. The engine keeps + per-layer attention and convolution caches across steps instead of + re-encoding a sliding window, so `streamingChunkMs` now selects a trained + operating point: 80, 160, 560, or 1040 ms, with `streamingRightLookaheadMs` + at 0, 80, 160, 240, 320, 560, or 1040 ms. Values outside those sets snap + down to the nearest trained one. Omitting `streamingChunkMs` now yields + 560 ms for this model instead of the generic 2000 ms. + ### Changed -- Raise the `speech-cpp` floor to `2026-09-16`, keeping the speech packages on - one engine stack. The pinned engine adds an optional Apple-only Core ML - sidecar for the Sortformer diarization encoder; the prebuilds keep it - disabled, so published behavior is unchanged. +- Raise the `speech-cpp` floor to `2026-09-17`, keeping the speech packages on + one engine stack. The pinned engine adds cache-aware streaming for the + Unified RNN-T model, on top of the optional Apple-only Core ML sidecar for + the Sortformer diarization encoder that the prebuilds keep disabled. - Add Whisper `contextParams["main-gpu"]` / `contextParams.main_gpu` selection for raw ggml registry indices plus `dedicated` and `integrated` classes. The selector is mutually exclusive with `gpu_device`, does not enable GPU by diff --git a/packages/asr-ggml/README.md b/packages/asr-ggml/README.md index 08453e2680..4c1f574f8f 100644 --- a/packages/asr-ggml/README.md +++ b/packages/asr-ggml/README.md @@ -74,7 +74,7 @@ GGUF metadata** — there is no `modelType` to pass. |---------|-----------|---------|-------------:|-------| | **CTC** (`parakeet-ctc-0.6b`) | English | argmax CTC | ~700 MiB | Fast, no punctuation/capitalization | | **TDT** (`parakeet-tdt-0.6b-v3`) | ~25 | RNN-T greedy + duration | ~715 MiB | Recommended default; PnC + language auto-detect | -| **Unified** (`parakeet-unified-en-0.6b`) | English | RNN-T | ~715 MiB | One checkpoint for batch and low-latency streaming; PnC | +| **Unified** (`parakeet-unified-en-0.6b`) | English | RNN-T | ~715 MiB | One checkpoint for batch and cache-aware streaming at 80/160/560/1040 ms; PnC | | **EOU** (`parakeet-eou-120m-v1`) | English | RNN-T greedy + `` | ~132 MiB | Streaming-trained; native end-of-turn token | | **Indic Conformer CTC** (`indic-conformer-ctc`) | Indic aggregate | argmax CTC + language mask | ~701 MiB | Multilingual Indic; set `parakeetConfig.language` (e.g. `"hi"`) | | **Sortformer v1** (`sortformer-4spk-v1`) | n/a | Diarization head (sliding history) | ~141 MiB | 4-speaker. Default for **offline** diarization | @@ -95,7 +95,7 @@ language coverage, translation, and diarization. | If you need… | Use this model | Notes | | --- | --- | --- | | Default multilingual / English ASR (batch or duplex stream) | `parakeet-tdt-0.6b-v3` (q8_0 GGUF) | Recommended Parakeet default: ~25 languages, punctuation/capitalization, language auto-detect, low-latency streaming. | -| English batch and low-latency streaming with one checkpoint | `parakeet-unified-en-0.6b` | Standard RNN-T with punctuation and capitalization; use when multilingual TDT or native EOU tokens are not required. | +| English batch and low-latency streaming with one checkpoint | `parakeet-unified-en-0.6b` | Standard RNN-T with punctuation and capitalization; use when multilingual TDT or native EOU tokens are not required. Streaming uses the native cache-aware encoder: `streamingChunkMs` accepts 80, 160, 560, or 1040 and `streamingRightLookaheadMs` 0, 80, 160, 240, 320, 560, or 1040, both snapped down to the nearest trained value. Defaults to 560 ms. | | Native end-of-turn for conversational / duplex English | `parakeet-eou-120m-v1` | Emits ``; smallest Parakeet (~132 MiB). Pair with TDT when you need broader language coverage *and* EOU. | | Fast English-only, no punctuation | `parakeet-ctc-0.6b` | Lowest decode cost in the Parakeet family; no PnC. | | Indic-language ASR (Hindi and other Indic ids) | `indic-conformer-ctc` | Pass `parakeetConfig.language` (e.g. `"hi"`). Same Parakeet engine; GGUF lives under `indic_conformer/` in the registry. | diff --git a/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetConfig.hpp b/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetConfig.hpp index 42ccb123cd..6bcaa56c50 100644 --- a/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetConfig.hpp +++ b/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetConfig.hpp @@ -9,6 +9,7 @@ namespace qvac::asrggml::parakeet { struct ParakeetConfig { static constexpr int DEFAULT_STREAMING_CHUNK_MS = 2000; static constexpr int DEFAULT_NEMOTRON_STREAMING_CHUNK_MS = 320; + static constexpr int DEFAULT_UNIFIED_STREAMING_CHUNK_MS = 560; static constexpr int DEFAULT_STREAMING_HISTORY_MS = 30000; static constexpr int DEFAULT_STREAMING_SPK_CACHE_LEN = 188; static constexpr int DEFAULT_STREAMING_FIFO_LEN = 188; diff --git a/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetModel.cpp b/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetModel.cpp index 9bf5b45442..0f0f3d56d2 100644 --- a/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetModel.cpp +++ b/packages/asr-ggml/addon/src/model-interface/parakeet/ParakeetModel.cpp @@ -421,6 +421,8 @@ int ParakeetModel::resolveStreamingChunkMs( return configuredChunkMs; if (modelType == ModelType::NEMOTRON) return ParakeetConfig::DEFAULT_NEMOTRON_STREAMING_CHUNK_MS; + if (modelType == ModelType::RNNT) + return ParakeetConfig::DEFAULT_UNIFIED_STREAMING_CHUNK_MS; return ParakeetConfig::DEFAULT_STREAMING_CHUNK_MS; } diff --git a/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp b/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp index a5d19572e2..e18b8c1815 100644 --- a/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp +++ b/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp @@ -1,4 +1,5 @@ #include +#include #include #include @@ -56,6 +57,12 @@ TEST(ParakeetStreamingGetters, ResolveDefaultsByDetectedModelType) { EXPECT_EQ( nemotron.getStreamingChunkMs(), ParakeetConfig::DEFAULT_NEMOTRON_STREAMING_CHUNK_MS); + + c.modelType = ModelType::RNNT; + ParakeetModel unified(c); + EXPECT_EQ( + unified.getStreamingChunkMs(), + ParakeetConfig::DEFAULT_UNIFIED_STREAMING_CHUNK_MS); } TEST(ParakeetStreamingGetters, HonourPositiveOverrides) { @@ -79,6 +86,26 @@ TEST(ParakeetStreamingGetters, PreserveNemotronOperatingPointOverrides) { ParakeetModel::resolveStreamingChunkMs(ModelType::NEMOTRON, 2000), 2000); } +TEST(ParakeetStreamingGetters, PreserveUnifiedOperatingPointOverrides) { + for (const int chunkMs : {80, 160, 560, 1040}) { + EXPECT_EQ( + ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, chunkMs), + chunkMs); + } + + // speech-cpp snaps untrained values down to the nearest trained chunk; + // the addon must forward them untouched so that decision stays in one place. + EXPECT_EQ(ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 1000), 1000); + EXPECT_EQ(ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 2000), 2000); +} + +TEST(ParakeetStreamingGetters, UnifiedDefaultIsATrainedOperatingPoint) { + const int resolved = ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 0); + EXPECT_EQ(resolved, ParakeetConfig::DEFAULT_UNIFIED_STREAMING_CHUNK_MS); + const std::vector trained = {80, 160, 560, 1040}; + EXPECT_NE(std::find(trained.begin(), trained.end(), resolved), trained.end()); +} + TEST(ParakeetPreprocessAudio, S16LeHandlesRangeExtremes) { std::vector raw = {0x00, 0x00, 0x00, 0x80, 0xFF, 0x7F}; auto out = ParakeetModel::preprocessAudioData(raw, "s16le"); diff --git a/packages/asr-ggml/engines/parakeet/driver.d.ts b/packages/asr-ggml/engines/parakeet/driver.d.ts index 4897feac0e..e46f2c062e 100644 --- a/packages/asr-ggml/engines/parakeet/driver.d.ts +++ b/packages/asr-ggml/engines/parakeet/driver.d.ts @@ -33,8 +33,10 @@ export interface ParakeetConfig { */ streaming?: boolean; /** - * Streaming chunk cadence. Defaults to 320 ms for Nemotron and 2000 ms for - * existing models. Nemotron supports 80, 160, 320, 560, or 1120 ms. + * Streaming chunk cadence. Defaults to 320 ms for Nemotron, 560 ms for the + * Unified RNN-T model, and 2000 ms for existing models. Nemotron supports + * 80, 160, 320, 560, or 1120 ms; Unified RNN-T supports 80, 160, 560, or + * 1040 ms and snaps any other value down to the nearest trained chunk. */ streamingChunkMs?: number; /** Sortformer rolling-history window in ms (default: 30000). */ @@ -45,7 +47,11 @@ export interface ParakeetConfig { streamingEnergyVad?: boolean; /** ASR encoder left-context window in milliseconds. */ streamingLeftContextMs?: number; - /** ASR encoder right-lookahead window in milliseconds. */ + /** + * ASR encoder right-lookahead window in milliseconds. Unified RNN-T + * cache-aware streaming supports 0, 80, 160, 240, 320, 560, or 1040 ms and + * snaps any other value down to the nearest trained right context. + */ streamingRightLookaheadMs?: number; /** Enable v2.1 Sortformer AOSC speaker-cache streaming (default: true). */ streamingSpkCacheEnable?: boolean; diff --git a/packages/asr-ggml/engines/parakeet/parakeet.d.ts b/packages/asr-ggml/engines/parakeet/parakeet.d.ts index 41a3264977..7f69126427 100644 --- a/packages/asr-ggml/engines/parakeet/parakeet.d.ts +++ b/packages/asr-ggml/engines/parakeet/parakeet.d.ts @@ -15,7 +15,10 @@ export interface ParakeetConfigurationParams { /** Indic CTC language id or Nemotron locale alias; empty selects auto. */ language?: string; streaming?: boolean; - /** Model-specific when omitted: Nemotron 320 ms, existing models 2000 ms. */ + /** + * Model-specific when omitted: Nemotron 320 ms, Unified RNN-T 560 ms, + * existing models 2000 ms. + */ streamingChunkMs?: number; streamingHistoryMs?: number; streamingEmitPartials?: boolean; diff --git a/packages/asr-ggml/src/engines/parakeet/driver.ts b/packages/asr-ggml/src/engines/parakeet/driver.ts index 97219bf3ce..59f3abb330 100644 --- a/packages/asr-ggml/src/engines/parakeet/driver.ts +++ b/packages/asr-ggml/src/engines/parakeet/driver.ts @@ -60,8 +60,10 @@ export interface ParakeetConfig { */ streaming?: boolean; /** - * Streaming chunk cadence. Defaults to 320 ms for Nemotron and 2000 ms for - * existing models. Nemotron supports 80, 160, 320, 560, or 1120 ms. + * Streaming chunk cadence. Defaults to 320 ms for Nemotron, 560 ms for the + * Unified RNN-T model, and 2000 ms for existing models. Nemotron supports + * 80, 160, 320, 560, or 1120 ms; Unified RNN-T supports 80, 160, 560, or + * 1040 ms and snaps any other value down to the nearest trained chunk. */ streamingChunkMs?: number; /** Sortformer rolling-history window in ms (default: 30000). */ @@ -72,7 +74,11 @@ export interface ParakeetConfig { streamingEnergyVad?: boolean; /** ASR encoder left-context window in milliseconds. */ streamingLeftContextMs?: number; - /** ASR encoder right-lookahead window in milliseconds. */ + /** + * ASR encoder right-lookahead window in milliseconds. Unified RNN-T + * cache-aware streaming supports 0, 80, 160, 240, 320, 560, or 1040 ms and + * snaps any other value down to the nearest trained right context. + */ streamingRightLookaheadMs?: number; /** Enable v2.1 Sortformer AOSC speaker-cache streaming (default: true). */ streamingSpkCacheEnable?: boolean; diff --git a/packages/asr-ggml/src/engines/parakeet/parakeet.ts b/packages/asr-ggml/src/engines/parakeet/parakeet.ts index 59d8662529..a9032404e8 100644 --- a/packages/asr-ggml/src/engines/parakeet/parakeet.ts +++ b/packages/asr-ggml/src/engines/parakeet/parakeet.ts @@ -37,7 +37,10 @@ export interface ParakeetConfigurationParams { /** Indic CTC language id or Nemotron locale alias; empty selects auto. */ language?: string; streaming?: boolean; - /** Model-specific when omitted: Nemotron 320 ms, existing models 2000 ms. */ + /** + * Model-specific when omitted: Nemotron 320 ms, Unified RNN-T 560 ms, + * existing models 2000 ms. + */ streamingChunkMs?: number; streamingHistoryMs?: number; streamingEmitPartials?: boolean; diff --git a/packages/asr-ggml/vcpkg.json b/packages/asr-ggml/vcpkg.json index aa0f671f80..ec0e16fc5e 100644 --- a/packages/asr-ggml/vcpkg.json +++ b/packages/asr-ggml/vcpkg.json @@ -17,7 +17,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-16", + "version>=": "2026-09-17", "default-features": false, "features": [ "whisper", @@ -29,7 +29,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-16", + "version>=": "2026-09-17", "default-features": false, "features": [ "whisper", @@ -41,7 +41,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-16", + "version>=": "2026-09-17", "default-features": false, "features": [ "whisper", @@ -66,7 +66,7 @@ "dependencies": [ { "name": "speech-cpp", - "version>=": "2026-09-16", + "version>=": "2026-09-17", "default-features": false, "features": [ "cuda" From 71bf90308e3de34968dc1a2baab4586e5f5a8286 Mon Sep 17 00:00:00 2001 From: Freddy Martinez Garcia Date: Thu, 17 Sep 2026 19:11:11 -0300 Subject: [PATCH 2/3] QVAC-25109 style[asr-ggml]: format the new parakeet streaming tests --- .../addon/tests/parakeet_test_config_and_audio.cpp | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp b/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp index e18b8c1815..1122e50fa4 100644 --- a/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp +++ b/packages/asr-ggml/addon/tests/parakeet_test_config_and_audio.cpp @@ -1,5 +1,5 @@ -#include #include +#include #include #include @@ -95,12 +95,15 @@ TEST(ParakeetStreamingGetters, PreserveUnifiedOperatingPointOverrides) { // speech-cpp snaps untrained values down to the nearest trained chunk; // the addon must forward them untouched so that decision stays in one place. - EXPECT_EQ(ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 1000), 1000); - EXPECT_EQ(ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 2000), 2000); + EXPECT_EQ( + ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 1000), 1000); + EXPECT_EQ( + ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 2000), 2000); } TEST(ParakeetStreamingGetters, UnifiedDefaultIsATrainedOperatingPoint) { - const int resolved = ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 0); + const int resolved = + ParakeetModel::resolveStreamingChunkMs(ModelType::RNNT, 0); EXPECT_EQ(resolved, ParakeetConfig::DEFAULT_UNIFIED_STREAMING_CHUNK_MS); const std::vector trained = {80, 160, 560, 1040}; EXPECT_NE(std::find(trained.begin(), trained.end(), resolved), trained.end()); From db9c63e752a019a9cda78f3b47ba084af02ae192 Mon Sep 17 00:00:00 2001 From: Freddy Martinez Garcia Date: Fri, 18 Sep 2026 15:20:47 -0300 Subject: [PATCH 3/3] QVAC-25109 bump[asr]: require the cache-aware streaming speech-cpp revision (2026-09-18#1) The 2026-09-18#1 registry revision pins qvac-fabric-speech.cpp master at 6380471c, the first published version carrying the Unified RNN-T cache-aware streaming that this package's defaults rely on. --- packages/asr-ggml/CHANGELOG.md | 2 +- packages/asr-ggml/vcpkg.json | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/asr-ggml/CHANGELOG.md b/packages/asr-ggml/CHANGELOG.md index 32e27e54bc..1827447f0a 100644 --- a/packages/asr-ggml/CHANGELOG.md +++ b/packages/asr-ggml/CHANGELOG.md @@ -26,7 +26,7 @@ restarts at `0.1.0`; the two pre-merge histories are preserved verbatim as ### Changed -- Raise the `speech-cpp` floor to `2026-09-17`, keeping the speech packages on +- Raise the `speech-cpp` floor to `2026-09-18#1`, keeping the speech packages on one engine stack. The pinned engine adds cache-aware streaming for the Unified RNN-T model, on top of the optional Apple-only Core ML sidecar for the Sortformer diarization encoder that the prebuilds keep disabled. diff --git a/packages/asr-ggml/vcpkg.json b/packages/asr-ggml/vcpkg.json index ec0e16fc5e..cc7fdfd8a0 100644 --- a/packages/asr-ggml/vcpkg.json +++ b/packages/asr-ggml/vcpkg.json @@ -17,7 +17,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-17", + "version>=": "2026-09-18#1", "default-features": false, "features": [ "whisper", @@ -29,7 +29,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-17", + "version>=": "2026-09-18#1", "default-features": false, "features": [ "whisper", @@ -41,7 +41,7 @@ }, { "name": "speech-cpp", - "version>=": "2026-09-17", + "version>=": "2026-09-18#1", "default-features": false, "features": [ "whisper", @@ -66,7 +66,7 @@ "dependencies": [ { "name": "speech-cpp", - "version>=": "2026-09-17", + "version>=": "2026-09-18#1", "default-features": false, "features": [ "cuda"