diff --git a/AGENTS.md b/AGENTS.md index e4eb1b7..7e9c808 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,7 +68,7 @@ ONNX models hosted on HuggingFace under [`aufklarer/`](https://huggingface.co/au org. INT8 quantized by default. - `soniqo/Silero-VAD-v5-ONNX` — VAD -- `soniqo/Parakeet-TDT-v3-ONNX` — STT (114 languages, 8192 BPE vocab) +- `soniqo/Parakeet-TDT-v3-ONNX` — STT (25 European languages, 8192 BPE vocab) - `soniqo/Kokoro-82M-ONNX` — TTS + phonemizer dicts + voice embeddings - `soniqo/Pocket-TTS-100M-ONNX-INT8` — streaming English TTS, fixed Alba voice - `soniqo/DeepFilterNet3-ONNX` — noise enhancer diff --git a/README.md b/README.md index 8047772..e615d18 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ On-device speech SDK for Android, powered by [ONNX Runtime](https://onnxruntime.ai) and [speech-core](https://github.com/soniqo/speech-core). -Low-memory streaming speech recognition (25 languages by default, 114-language TDT optional), text-to-speech, voice activity detection, and noise cancellation — all running locally. No cloud APIs, no data leaves the device. +Low-memory streaming speech recognition and an optional larger TDT model, both covering 25 European languages, plus text-to-speech, voice activity detection, and noise cancellation — all running locally. No cloud APIs, no data leaves the device. **[📚 Android Documentation](https://soniqo.audio/getting-started/android)** @@ -28,7 +28,8 @@ This repo is the **Android packaging**: Kotlin SDK, JNI bridge, demo app. The C+ | Model | Task | Download | Peak memory | Languages | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/guides/dictate) | Streaming STT + end-of-utterance (default) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/guides/parakeet/android) | Broad-coverage STT (optional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/guides/parakeet/android) | Broad-coverage STT (optional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 25 European | +| [Nemotron-3.5 multilingual](https://soniqo.audio/guides/nemotron) | Prompt-conditioned streaming STT (optional) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | not yet measured | 100+ (including zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | Offline STT + translation (optional) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/guides/kokoro/android) | Text-to-speech (default) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Streaming text-to-speech (optional, fixed Alba voice) | ~126 MB | not yet measured | English | @@ -42,11 +43,27 @@ Models are downloaded automatically on first launch via `ModelManager.ensureMode `SpeechConfig()` defaults to `SttModel.PARAKEET_EOU` and `TtsModel.KOKORO_SHORT_TURN` to keep SDK integrations and the system recognizer on the low-memory Android path. The demo app opts into `SttModel.PARAKEET` so its echo and dictation -screens exercise the larger 114-language TDT model. +screens exercise the larger 25-European-language TDT model. -For language-focused recognition, use `SpeechConfig(sttModel = SttModel.PARAKEET, -languageHints = listOf("en", "fr"))`. Set `language = "en"` when you want a -single concrete language instead of a shortlist. +Both Parakeet models always detect the language automatically: neither accepts +`language` or `languageHints`, and neither supports Chinese. To select one +language, including Mandarin, use the prompt-conditioned Nemotron backend: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // or "zh-TW" +) +``` **Supertonic-3** is an opt-in higher-quality multilingual TTS — select it with `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` (requires the LiteRT backend). The host runs its four @@ -158,7 +175,7 @@ The [`app/`](app/) module is a minimal voice assistant demo with: - Echo mode: transcribes speech and synthesizes it back (no LLM) - Dictation mode: streaming partial results - Voice overlay: a floating mic button that dictates into any app -- 114-language Parakeet TDT STT in the echo and dictation screens +- 25-European-language Parakeet TDT STT in the echo and dictation screens - `SpeechRecognizer` test screen — exercises the system-wide voice input path - Chat bubble UI with STT/TTS latency display diff --git a/README_de.md b/README_de.md index 133aa2b..e6f9f16 100644 --- a/README_de.md +++ b/README_de.md @@ -4,7 +4,7 @@ On-Device Speech-SDK für Android, basierend auf [ONNX Runtime](https://onnxruntime.ai) und [speech-core](https://github.com/soniqo/speech-core). -Speicherarme Streaming-Spracherkennung (standardmäßig 25 Sprachen, optionales TDT mit 114 Sprachen), Text-to-Speech, Sprachaktivitätserkennung und Rauschunterdrückung — alles lokal ausgeführt. Keine Cloud-APIs, keine Daten verlassen das Gerät. +Speicherarme Streaming-Spracherkennung und ein optionales größeres TDT-Modell, beide für 25 europäische Sprachen, dazu Text-to-Speech, Sprachaktivitätserkennung und Rauschunterdrückung — alles lokal ausgeführt. Keine Cloud-APIs, keine Daten verlassen das Gerät. **[📚 Android-Dokumentation](https://soniqo.audio/de/getting-started/android)** @@ -28,7 +28,8 @@ Dieses Repo ist das **Android-Packaging**: Kotlin-SDK, JNI-Bridge, Demo-App. Die | Modell | Aufgabe | Download | Spitzen-Speicher | Sprachen | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/de/guides/dictate) | Streaming-STT + EOU (Standard) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/de/guides/parakeet/android) | Breite STT-Abdeckung (optional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/de/guides/parakeet/android) | Breite STT-Abdeckung (optional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 GB | 25 europäische | +| [Nemotron-3.5 mehrsprachig](https://soniqo.audio/de/guides/nemotron) | Prompt-konditioniertes Streaming-STT (optional) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | noch nicht gemessen | über 100 (einschließlich zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | Offline-STT + Übersetzung (optional) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/de/guides/kokoro/android) | Text-to-Speech (Standard) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Streaming-Text-to-Speech (optional, feste Alba-Stimme) | ~126 MB | noch nicht gemessen | Englisch | @@ -39,9 +40,25 @@ Dieses Repo ist das **Android-Packaging**: Kotlin-SDK, JNI-Bridge, Demo-App. Die Modelle werden beim ersten Start automatisch über `ModelManager.ensureModels()` heruntergeladen. -`SpeechConfig()` verwendet standardmäßig `SttModel.PARAKEET_EOU` und `TtsModel.KOKORO_SHORT_TURN`, damit SDK-Integrationen und Systemerkennung den speicherarmen Android-Pfad nutzen. Die Demo-App wählt `SttModel.PARAKEET`, sodass Echo- und Diktieransicht das größere TDT-Modell mit 114 Sprachen verwenden. +`SpeechConfig()` verwendet standardmäßig `SttModel.PARAKEET_EOU` und `TtsModel.KOKORO_SHORT_TURN`, damit SDK-Integrationen und Systemerkennung den speicherarmen Android-Pfad nutzen. Die Demo-App wählt `SttModel.PARAKEET`, sodass Echo- und Diktieransicht das größere TDT-Modell für 25 europäische Sprachen verwenden. -Für sprachfokussierte Erkennung verwende `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`. Setze `language = "en"`, wenn genau eine Sprache fest vorgegeben werden soll. +Beide Parakeet-Modelle erkennen die Sprache immer automatisch: Sie akzeptieren weder `language` noch `languageHints` und unterstützen kein Chinesisch. Um eine Sprache einschließlich Mandarin festzulegen, verwende das Prompt-konditionierte Nemotron-Backend: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // oder "zh-TW" +) +``` **Supertonic-3** ist ein optionales, höherwertiges mehrsprachiges TTS — wähle es mit `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` aus (erfordert das LiteRT-Backend). Der Host führt seine vier nicht-autoregressiven Flow-Matching-Graphen mit 44,1 kHz auf dem Gerät aus; das Front-End ist G2P-frei (NFKD + Unicode-Index — kein Phonemizer), sodass alle 31 Sprachen über einen einzigen Pfad laufen. @@ -101,7 +118,7 @@ Das Modul [`app/`](app/) ist eine minimale Sprachassistenten-Demo mit: - Echo-Modus: transkribiert Sprache und synthetisiert sie zurück (kein LLM) - Diktiermodus: Streaming-Teilergebnisse - Sprach-Overlay: schwebende Mikrofon-Schaltfläche zum Diktieren in jede App -- Parakeet TDT STT mit 114 Sprachen in Echo- und Diktieransicht +- Parakeet TDT STT für 25 europäische Sprachen in Echo- und Diktieransicht - `SpeechRecognizer`-Testbildschirm — übt den systemweiten Spracheingabepfad aus - Chat-Bubble-UI mit STT/TTS-Latenzanzeige diff --git a/README_es.md b/README_es.md index fd11881..51b8e46 100644 --- a/README_es.md +++ b/README_es.md @@ -4,7 +4,7 @@ SDK de voz en el dispositivo para Android, impulsado por [ONNX Runtime](https://onnxruntime.ai) y [speech-core](https://github.com/soniqo/speech-core). -Reconocimiento de voz en streaming de baja memoria (25 idiomas por defecto; TDT de 114 idiomas opcional), texto a voz, detección de actividad de voz y cancelación de ruido — todo ejecutándose localmente. Sin APIs en la nube, ningún dato sale del dispositivo. +Reconocimiento de voz en streaming de baja memoria y un modelo TDT opcional más grande, ambos para 25 idiomas europeos, además de texto a voz, detección de actividad de voz y cancelación de ruido — todo ejecutándose localmente. Sin APIs en la nube, ningún dato sale del dispositivo. **[📚 Documentación de Android](https://soniqo.audio/es/getting-started/android)** @@ -28,7 +28,8 @@ Este repositorio es el **empaquetado para Android**: SDK de Kotlin, puente JNI, | Modelo | Tarea | Descarga | Pico de memoria | Idiomas | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/es/guides/dictate) | STT en streaming + EOU (por defecto) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/es/guides/parakeet/android) | STT de cobertura amplia (opcional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/es/guides/parakeet/android) | STT de cobertura amplia (opcional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 25 europeos | +| [Nemotron-3.5 multilingüe](https://soniqo.audio/es/guides/nemotron) | STT en streaming condicionado por prompt (opcional) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | aún no medido | más de 100 (incluido zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | STT sin conexión + traducción (opcional) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/es/guides/kokoro/android) | Texto a voz (por defecto) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Texto a voz en streaming (opcional, voz Alba fija) | ~126 MB | aún no medido | Inglés | @@ -39,9 +40,25 @@ Este repositorio es el **empaquetado para Android**: SDK de Kotlin, puente JNI, Los modelos se descargan automáticamente al primer inicio vía `ModelManager.ensureModels()`. -`SpeechConfig()` usa `SttModel.PARAKEET_EOU` y `TtsModel.KOKORO_SHORT_TURN` por defecto para mantener las integraciones del SDK y el reconocedor del sistema en la ruta Android de baja memoria. La app demo opta por `SttModel.PARAKEET` para que las pantallas de eco y dictado usen el modelo TDT más grande de 114 idiomas. +`SpeechConfig()` usa `SttModel.PARAKEET_EOU` y `TtsModel.KOKORO_SHORT_TURN` por defecto para mantener las integraciones del SDK y el reconocedor del sistema en la ruta Android de baja memoria. La app demo opta por `SttModel.PARAKEET` para que las pantallas de eco y dictado usen el modelo TDT más grande para 25 idiomas europeos. -Para reconocimiento enfocado por idioma, usa `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`. Define `language = "en"` si quieres fijar un único idioma. +Ambos modelos Parakeet siempre detectan el idioma automáticamente: no aceptan `language` ni `languageHints` y no son compatibles con chino. Para fijar un idioma, incluido el mandarín, usa el backend Nemotron condicionado por prompt: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // o "zh-TW" +) +``` **Supertonic-3** es un TTS multilingüe opcional de mayor calidad — selecciónalo con `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` (requiere el backend LiteRT). El host ejecuta sus cuatro grafos de flow-matching no autorregresivos en el dispositivo a 44.1 kHz; el front-end es G2P-free (NFKD + índice Unicode — sin fonemizador), por lo que los 31 idiomas pasan por una sola ruta. @@ -101,7 +118,7 @@ El módulo [`app/`](app/) es una demo mínima de asistente de voz con: - Modo eco: transcribe la voz y la sintetiza de vuelta (sin LLM) - Modo dictado: resultados parciales en streaming - Superposición de voz: botón de micrófono flotante para dictar en cualquier app -- STT Parakeet TDT de 114 idiomas en las pantallas de eco y dictado +- STT Parakeet TDT para 25 idiomas europeos en las pantallas de eco y dictado - Pantalla de prueba `SpeechRecognizer` — ejercita la ruta de entrada de voz a nivel de sistema - UI de burbujas de chat con visualización de latencia STT/TTS diff --git a/README_fr.md b/README_fr.md index 1a88307..e0d3b77 100644 --- a/README_fr.md +++ b/README_fr.md @@ -4,7 +4,7 @@ SDK vocal sur appareil pour Android, propulsé par [ONNX Runtime](https://onnxruntime.ai) et [speech-core](https://github.com/soniqo/speech-core). -Reconnaissance vocale en streaming à faible mémoire (25 langues par défaut, TDT 114 langues en option), synthèse vocale, détection d'activité vocale et suppression de bruit — tout fonctionne en local. Aucune API cloud, aucune donnée ne quitte l'appareil. +Reconnaissance vocale en streaming à faible mémoire et modèle TDT plus grand en option, tous deux pour 25 langues européennes, plus synthèse vocale, détection d'activité vocale et suppression de bruit — tout fonctionne en local. Aucune API cloud, aucune donnée ne quitte l'appareil. **[📚 Documentation Android](https://soniqo.audio/fr/getting-started/android)** @@ -28,7 +28,8 @@ Ce dépôt fournit le **packaging Android** : SDK Kotlin, pont JNI, application | Modèle | Tâche | Téléchargement | Mémoire max | Langues | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/fr/guides/dictate) | STT streaming + EOU (défaut) | [153 Mo](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 Mo | 25 | -| [Parakeet TDT v3](https://soniqo.audio/fr/guides/parakeet/android) | STT large couverture (optionnel) | [891 Mo](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 Go | 114 | +| [Parakeet TDT v3](https://soniqo.audio/fr/guides/parakeet/android) | STT large couverture (optionnel) | [891 Mo](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 Go | 25 langues européennes | +| [Nemotron-3.5 multilingue](https://soniqo.audio/fr/guides/nemotron) | STT streaming conditionné par prompt (optionnel) | [~721 Mo](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | pas encore mesuré | plus de 100 (dont zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | STT hors ligne + traduction (optionnel) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/fr/guides/kokoro/android) | Synthèse vocale (défaut) | [330 Mo](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 Mo | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Synthèse vocale streaming (optionnel, voix Alba fixe) | ~126 Mo | pas encore mesuré | Anglais | @@ -39,9 +40,25 @@ Ce dépôt fournit le **packaging Android** : SDK Kotlin, pont JNI, application Les modèles sont téléchargés automatiquement au premier lancement via `ModelManager.ensureModels()`. -`SpeechConfig()` utilise `SttModel.PARAKEET_EOU` et `TtsModel.KOKORO_SHORT_TURN` par défaut afin que les intégrations SDK et le service de reconnaissance système restent sur le chemin Android à faible mémoire. L'application de démo sélectionne `SttModel.PARAKEET` pour que les écrans écho et dictée utilisent le modèle TDT plus grand à 114 langues. +`SpeechConfig()` utilise `SttModel.PARAKEET_EOU` et `TtsModel.KOKORO_SHORT_TURN` par défaut afin que les intégrations SDK et le service de reconnaissance système restent sur le chemin Android à faible mémoire. L'application de démo sélectionne `SttModel.PARAKEET` pour que les écrans écho et dictée utilisent le modèle TDT plus grand couvrant 25 langues européennes. -Pour une reconnaissance centrée sur certaines langues, utilisez `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`. Définissez `language = "en"` pour fixer une seule langue. +Les deux modèles Parakeet détectent toujours la langue automatiquement : aucun n'accepte `language` ou `languageHints`, et aucun ne prend en charge le chinois. Pour sélectionner une langue, y compris le mandarin, utilisez le backend Nemotron conditionné par prompt : + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // ou "zh-TW" +) +``` **Supertonic-3** est une synthèse vocale multilingue de meilleure qualité, activable en option — sélectionnez-la avec `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` (nécessite le backend LiteRT). L'hôte exécute ses quatre graphes de flow-matching non autorégressifs en local à 44,1 kHz ; le front-end est G2P-free (NFKD + index Unicode — aucun phonémiseur), de sorte que les 31 langues passent par un seul chemin. @@ -101,7 +118,7 @@ Le module [`app/`](app/) est une démo minimale d'assistant vocal avec : - Mode écho : transcrit la voix et la synthétise en retour (sans LLM) - Mode dictée : résultats partiels en streaming - Superposition vocale : bouton micro flottant pour dicter dans n'importe quelle app -- STT Parakeet TDT à 114 langues dans les écrans écho et dictée +- STT Parakeet TDT couvrant 25 langues européennes dans les écrans écho et dictée - Écran de test `SpeechRecognizer` — exerce le chemin d'entrée vocale à l'échelle du système - Interface de bulles de chat avec affichage de la latence STT/TTS diff --git a/README_hi.md b/README_hi.md index 60517e5..6ba7323 100644 --- a/README_hi.md +++ b/README_hi.md @@ -4,7 +4,7 @@ Android के लिए ऑन-डिवाइस स्पीच SDK, [ONNX Runtime](https://onnxruntime.ai) और [speech-core](https://github.com/soniqo/speech-core) द्वारा संचालित। -कम-मेमोरी स्ट्रीमिंग स्पीच रिकग्निशन (डिफ़ॉल्ट 25 भाषाएँ, 114-भाषा TDT वैकल्पिक), टेक्स्ट-टू-स्पीच, वॉयस एक्टिविटी डिटेक्शन, और शोर रद्दीकरण — सभी स्थानीय रूप से चलते हैं। कोई क्लाउड API नहीं, कोई डेटा डिवाइस से बाहर नहीं जाता। +कम-मेमोरी स्ट्रीमिंग स्पीच रिकग्निशन और वैकल्पिक बड़ा TDT मॉडल, दोनों 25 यूरोपीय भाषाओं के लिए, साथ में टेक्स्ट-टू-स्पीच, वॉयस एक्टिविटी डिटेक्शन और शोर रद्दीकरण — सभी स्थानीय रूप से चलते हैं। कोई क्लाउड API नहीं, कोई डेटा डिवाइस से बाहर नहीं जाता। **[📚 Android दस्तावेज़](https://soniqo.audio/hi/getting-started/android)** @@ -28,7 +28,8 @@ Android के लिए ऑन-डिवाइस स्पीच SDK, [ONNX Ru | मॉडल | कार्य | डाउनलोड | पीक मेमोरी | भाषाएँ | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/hi/guides/dictate) | स्ट्रीमिंग STT + EOU (डिफ़ॉल्ट) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/hi/guides/parakeet/android) | व्यापक STT (वैकल्पिक) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/hi/guides/parakeet/android) | व्यापक STT (वैकल्पिक) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 25 यूरोपीय | +| [Nemotron-3.5 बहुभाषी](https://soniqo.audio/hi/guides/nemotron) | प्रॉम्प्ट-कंडीशन्ड स्ट्रीमिंग STT (वैकल्पिक) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | अभी मापा नहीं गया | 100+ (zh सहित) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | Offline STT + translation (optional) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/hi/guides/kokoro/android) | टेक्स्ट-टू-स्पीच (डिफ़ॉल्ट) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | स्ट्रीमिंग टेक्स्ट-टू-स्पीच (वैकल्पिक, स्थिर Alba आवाज़) | ~126 MB | अभी मापा नहीं गया | अंग्रेज़ी | @@ -39,9 +40,25 @@ Android के लिए ऑन-डिवाइस स्पीच SDK, [ONNX Ru मॉडल पहले लॉन्च पर `ModelManager.ensureModels()` के माध्यम से स्वचालित रूप से डाउनलोड होते हैं। -`SpeechConfig()` डिफ़ॉल्ट रूप से `SttModel.PARAKEET_EOU` और `TtsModel.KOKORO_SHORT_TURN` इस्तेमाल करता है, ताकि SDK इंटीग्रेशन और सिस्टम रिकग्नाइज़र कम-मेमोरी Android पथ पर रहें। डेमो ऐप `SttModel.PARAKEET` चुनता है, इसलिए इको और डिक्टेशन स्क्रीन बड़े 114-भाषा TDT मॉडल का उपयोग करती हैं। +`SpeechConfig()` डिफ़ॉल्ट रूप से `SttModel.PARAKEET_EOU` और `TtsModel.KOKORO_SHORT_TURN` इस्तेमाल करता है, ताकि SDK इंटीग्रेशन और सिस्टम रिकग्नाइज़र कम-मेमोरी Android पथ पर रहें। डेमो ऐप `SttModel.PARAKEET` चुनता है, इसलिए इको और डिक्टेशन स्क्रीन 25 यूरोपीय भाषाओं वाले बड़े TDT मॉडल का उपयोग करती हैं। -भाषा-केंद्रित रिकग्निशन के लिए `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))` इस्तेमाल करें। केवल एक भाषा तय करनी हो तो `language = "en"` सेट करें। +दोनों Parakeet मॉडल हमेशा भाषा अपने-आप पहचानते हैं: वे `language` या `languageHints` स्वीकार नहीं करते और चीनी का समर्थन नहीं करते। मंदारिन सहित किसी एक भाषा को चुनने के लिए प्रॉम्प्ट-कंडीशन्ड Nemotron बैकएंड इस्तेमाल करें: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // या "zh-TW" +) +``` **Supertonic-3** एक ऑप्ट-इन उच्च-गुणवत्ता वाला बहुभाषी TTS है — इसे `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` के साथ चुनें (LiteRT बैकएंड आवश्यक)। होस्ट इसके चार नॉन-ऑटोरिग्रेसिव फ़्लो-मैचिंग ग्राफ़ ऑन-डिवाइस 44.1 kHz पर चलाता है; फ़्रंट-एंड G2P-free है (NFKD + Unicode इंडेक्स — कोई फ़ोनेमाइज़र नहीं), इसलिए सभी 31 भाषाएँ एक ही पथ से होकर गुजरती हैं। @@ -101,7 +118,7 @@ cd speech-android - इको मोड: स्पीच को ट्रांसक्राइब करता है और इसे वापस सिंथेसाइज़ करता है (कोई LLM नहीं) - डिक्टेशन मोड: स्ट्रीमिंग आंशिक परिणाम - वॉयस ओवरले: किसी भी ऐप में बोलकर लिखने के लिए फ़्लोटिंग माइक बटन -- इको और डिक्टेशन स्क्रीन में 114-भाषा Parakeet TDT STT +- इको और डिक्टेशन स्क्रीन में 25 यूरोपीय भाषाओं वाला Parakeet TDT STT - `SpeechRecognizer` टेस्ट स्क्रीन — सिस्टम-वाइड वॉयस इनपुट पथ का परीक्षण करता है - STT/TTS विलंबता प्रदर्शन के साथ चैट बबल UI diff --git a/README_ja.md b/README_ja.md index 524af83..546f98c 100644 --- a/README_ja.md +++ b/README_ja.md @@ -4,7 +4,7 @@ [ONNX Runtime](https://onnxruntime.ai) と [speech-core](https://github.com/soniqo/speech-core) を活用した、Android 向けオンデバイス音声 SDK。 -低メモリのストリーミング音声認識(既定 25 言語、114 言語 TDT は任意)、テキスト読み上げ、音声活動検出、ノイズキャンセリング — すべてローカルで動作。クラウド API 不要、データはデバイスから外に出ません。 +低メモリのストリーミング音声認識と、任意で選べるより大きな TDT モデル(どちらもヨーロッパの 25 言語に対応)、テキスト読み上げ、音声活動検出、ノイズキャンセリング — すべてローカルで動作。クラウド API 不要、データはデバイスから外に出ません。 **[📚 Android ドキュメント](https://soniqo.audio/ja/getting-started/android)** @@ -28,7 +28,8 @@ | モデル | タスク | ダウンロード | ピークメモリ | 言語 | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/ja/guides/dictate) | ストリーミング STT + EOU(既定) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/ja/guides/parakeet/android) | 広範囲 STT(任意) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/ja/guides/parakeet/android) | 広範囲 STT(任意) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | ヨーロッパの 25 言語 | +| [Nemotron-3.5 多言語](https://soniqo.audio/ja/guides/nemotron) | プロンプト条件付きストリーミング STT(任意) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | 未計測 | 100 以上(zh を含む) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | オフライン STT + 翻訳(任意) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/ja/guides/kokoro/android) | テキスト読み上げ(既定) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8(en、fr、es、it、pt、hi、ja、zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | ストリーミング音声合成(任意、固定 Alba 音声) | ~126 MB | 未計測 | 英語 | @@ -39,9 +40,25 @@ モデルは初回起動時に `ModelManager.ensureModels()` 経由で自動ダウンロードされます。 -`SpeechConfig()` は `SttModel.PARAKEET_EOU` と `TtsModel.KOKORO_SHORT_TURN` を既定にして、SDK 組み込みとシステム認識サービスを低メモリ Android パスで動かします。デモアプリは `SttModel.PARAKEET` を選択し、エコー画面とディクテーション画面でより大きい 114 言語 TDT モデルを使います。 +`SpeechConfig()` は `SttModel.PARAKEET_EOU` と `TtsModel.KOKORO_SHORT_TURN` を既定にして、SDK 組み込みとシステム認識サービスを低メモリ Android パスで動かします。デモアプリは `SttModel.PARAKEET` を選択し、エコー画面とディクテーション画面でヨーロッパの 25 言語に対応する、より大きな TDT モデルを使います。 -言語を絞った認識には `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))` を使います。単一の言語に固定したい場合は `language = "en"` を設定します。 +どちらの Parakeet モデルも常に言語を自動検出します。`language` と `languageHints` は受け付けず、中国語にも対応していません。中国語を含む単一言語を指定するには、プロンプト条件付き Nemotron バックエンドを使用します: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // または "zh-TW" +) +``` **Supertonic-3** はオプトインの高品質な多言語 TTS です — `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` で選択します(LiteRT バックエンドが必要)。ホストはその 4 つの非自己回帰 flow-matching グラフを 44.1 kHz でオンデバイス実行します。フロントエンドは G2P-free(NFKD + Unicode インデックス — phonemizer なし)なので、31 言語すべてが単一のパスを通ります。 @@ -101,7 +118,7 @@ cd speech-android - エコーモード:音声を文字起こしして合成し直す(LLM なし) - ディクテーションモード:ストリーミング部分結果 - 音声オーバーレイ:任意のアプリに口述入力できるフローティングマイクボタン -- エコー画面とディクテーション画面で 114 言語 Parakeet TDT STT を使用 +- エコー画面とディクテーション画面でヨーロッパの 25 言語に対応する Parakeet TDT STT を使用 - `SpeechRecognizer` テスト画面 — システム全体の音声入力パスを実行 - STT/TTS のレイテンシ表示付きチャットバブル UI diff --git a/README_ko.md b/README_ko.md index 7f1bb80..855788a 100644 --- a/README_ko.md +++ b/README_ko.md @@ -4,7 +4,7 @@ [ONNX Runtime](https://onnxruntime.ai)와 [speech-core](https://github.com/soniqo/speech-core) 기반의 Android용 온디바이스 음성 SDK. -저메모리 스트리밍 음성 인식(기본 25개 언어, 114개 언어 TDT는 선택), 텍스트 음성 변환, 음성 활동 감지, 노이즈 캔슬링 — 모두 로컬에서 실행됩니다. 클라우드 API도, 디바이스 외부로 전송되는 데이터도 없습니다. +저메모리 스트리밍 음성 인식과 선택 가능한 대형 TDT 모델(둘 다 유럽 25개 언어 지원), 텍스트 음성 변환, 음성 활동 감지, 노이즈 캔슬링 — 모두 로컬에서 실행됩니다. 클라우드 API도, 디바이스 외부로 전송되는 데이터도 없습니다. **[📚 Android 문서](https://soniqo.audio/ko/getting-started/android)** @@ -28,7 +28,8 @@ | 모델 | 작업 | 다운로드 | 피크 메모리 | 언어 | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/ko/guides/dictate) | 스트리밍 STT + EOU(기본) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/ko/guides/parakeet/android) | 광범위 STT(선택) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/ko/guides/parakeet/android) | 광범위 STT(선택) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 유럽 25개 언어 | +| [Nemotron-3.5 다국어](https://soniqo.audio/ko/guides/nemotron) | 프롬프트 조건부 스트리밍 STT(선택) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | 미측정 | 100개 이상(zh 포함) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | 오프라인 STT + 번역 (선택) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/ko/guides/kokoro/android) | 텍스트 음성 변환(기본) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8(en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | 스트리밍 음성 합성(선택, 고정 Alba 음성) | ~126 MB | 미측정 | 영어 | @@ -39,9 +40,25 @@ 모델은 `ModelManager.ensureModels()`를 통해 첫 실행 시 자동으로 다운로드됩니다. -`SpeechConfig()`는 `SttModel.PARAKEET_EOU`와 `TtsModel.KOKORO_SHORT_TURN`를 기본값으로 사용해 SDK 통합과 시스템 인식 서비스를 저메모리 Android 경로에서 실행합니다. 데모 앱은 `SttModel.PARAKEET`를 선택해 에코와 받아쓰기 화면에서 더 큰 114개 언어 TDT 모델을 사용합니다. +`SpeechConfig()`는 `SttModel.PARAKEET_EOU`와 `TtsModel.KOKORO_SHORT_TURN`를 기본값으로 사용해 SDK 통합과 시스템 인식 서비스를 저메모리 Android 경로에서 실행합니다. 데모 앱은 `SttModel.PARAKEET`를 선택해 에코와 받아쓰기 화면에서 유럽 25개 언어를 지원하는 더 큰 TDT 모델을 사용합니다. -언어 중심 인식에는 `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`를 사용하세요. 하나의 언어로 고정하려면 `language = "en"`을 설정하세요. +두 Parakeet 모델은 항상 언어를 자동 감지합니다. `language`나 `languageHints`를 받지 않으며 중국어도 지원하지 않습니다. 중국어를 포함한 하나의 언어를 지정하려면 프롬프트 조건부 Nemotron 백엔드를 사용하세요: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // 또는 "zh-TW" +) +``` **Supertonic-3**는 옵트인 방식의 더 높은 품질의 다국어 TTS입니다 — `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)`로 선택하세요(LiteRT 백엔드가 필요합니다). 호스트는 4개의 비자기회귀(non-autoregressive) flow-matching 그래프를 44.1 kHz로 온디바이스에서 실행합니다. 프런트엔드는 G2P-free(NFKD + 유니코드 인덱스 — 음소 변환기 없음)이므로 31개 언어 모두 하나의 경로를 통과합니다. @@ -101,7 +118,7 @@ cd speech-android - 에코 모드: 음성을 전사하고 다시 합성(LLM 없음) - 받아쓰기 모드: 스트리밍 부분 결과 - 음성 오버레이: 어떤 앱에나 받아쓸 수 있는 플로팅 마이크 버튼 -- 에코와 받아쓰기 화면에서 114개 언어 Parakeet TDT STT 사용 +- 에코와 받아쓰기 화면에서 유럽 25개 언어를 지원하는 Parakeet TDT STT 사용 - `SpeechRecognizer` 테스트 화면 — 시스템 전체 음성 입력 경로 실행 - STT/TTS 지연 시간 표시가 있는 채팅 버블 UI diff --git a/README_pt.md b/README_pt.md index e94ff3c..2f4828f 100644 --- a/README_pt.md +++ b/README_pt.md @@ -4,7 +4,7 @@ SDK de voz no dispositivo para Android, baseado em [ONNX Runtime](https://onnxruntime.ai) e [speech-core](https://github.com/soniqo/speech-core). -Reconhecimento de fala em streaming com baixa memória (25 idiomas por padrão, TDT de 114 idiomas opcional), texto para fala, detecção de atividade vocal e cancelamento de ruído — tudo executado localmente. Sem APIs em nuvem, nenhum dado sai do dispositivo. +Reconhecimento de fala em streaming com baixa memória e um modelo TDT maior opcional, ambos para 25 idiomas europeus, além de texto para fala, detecção de atividade vocal e cancelamento de ruído — tudo executado localmente. Sem APIs em nuvem, nenhum dado sai do dispositivo. **[📚 Documentação Android](https://soniqo.audio/pt/getting-started/android)** @@ -28,7 +28,8 @@ Este repositório é o **empacotamento Android**: SDK Kotlin, ponte JNI, app de | Modelo | Tarefa | Download | Pico de memória | Idiomas | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/pt/guides/dictate) | STT em streaming + EOU (padrão) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/pt/guides/parakeet/android) | STT de ampla cobertura (opcional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/pt/guides/parakeet/android) | STT de ampla cobertura (opcional) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 GB | 25 europeus | +| [Nemotron-3.5 multilíngue](https://soniqo.audio/pt/guides/nemotron) | STT em streaming condicionado por prompt (opcional) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | ainda não medido | mais de 100 (incluindo zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | STT offline + tradução (opcional) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/pt/guides/kokoro/android) | Texto para fala (padrão) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Texto para fala em streaming (opcional, voz Alba fixa) | ~126 MB | ainda não medido | Inglês | @@ -39,9 +40,25 @@ Este repositório é o **empacotamento Android**: SDK Kotlin, ponte JNI, app de Os modelos são baixados automaticamente no primeiro lançamento via `ModelManager.ensureModels()`. -`SpeechConfig()` usa `SttModel.PARAKEET_EOU` e `TtsModel.KOKORO_SHORT_TURN` por padrão para manter integrações do SDK e o reconhecedor do sistema no caminho Android de baixa memória. O app demo seleciona `SttModel.PARAKEET` para que as telas de eco e ditado usem o modelo TDT maior de 114 idiomas. +`SpeechConfig()` usa `SttModel.PARAKEET_EOU` e `TtsModel.KOKORO_SHORT_TURN` por padrão para manter integrações do SDK e o reconhecedor do sistema no caminho Android de baixa memória. O app demo seleciona `SttModel.PARAKEET` para que as telas de eco e ditado usem o modelo TDT maior com 25 idiomas europeus. -Para reconhecimento focado em idiomas, use `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`. Defina `language = "en"` quando quiser fixar um único idioma. +Os dois modelos Parakeet sempre detectam o idioma automaticamente: nenhum aceita `language` ou `languageHints`, e nenhum oferece suporte a chinês. Para selecionar um idioma, incluindo mandarim, use o backend Nemotron condicionado por prompt: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // ou "zh-TW" +) +``` O **Supertonic-3** é um TTS multilíngue opcional de maior qualidade — selecione-o com `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` (requer o backend LiteRT). O host executa seus quatro grafos de flow-matching não autorregressivos no dispositivo a 44,1 kHz; o front-end é G2P-free (NFKD + índice Unicode — sem phonemizer), de modo que todos os 31 idiomas seguem um único caminho. @@ -101,7 +118,7 @@ O módulo [`app/`](app/) é uma demo mínima de assistente de voz com: - Modo eco: transcreve a fala e a sintetiza de volta (sem LLM) - Modo ditado: resultados parciais em streaming - Sobreposição de voz: botão de microfone flutuante para ditar em qualquer app -- STT Parakeet TDT de 114 idiomas nas telas de eco e ditado +- STT Parakeet TDT com 25 idiomas europeus nas telas de eco e ditado - Tela de teste `SpeechRecognizer` — exercita o caminho de entrada de voz em todo o sistema - UI de bolhas de chat com exibição de latência STT/TTS diff --git a/README_ru.md b/README_ru.md index a6d919e..7543c9c 100644 --- a/README_ru.md +++ b/README_ru.md @@ -4,7 +4,7 @@ Локальный речевой SDK для Android, основанный на [ONNX Runtime](https://onnxruntime.ai) и [speech-core](https://github.com/soniqo/speech-core). -Потоковое распознавание речи с низким потреблением памяти (25 языков по умолчанию, TDT на 114 языков опционально), синтез речи, определение голосовой активности и шумоподавление — всё работает локально. Никаких облачных API, никакие данные не покидают устройство. +Потоковое распознавание речи с низким потреблением памяти и опциональная более крупная TDT-модель, обе для 25 европейских языков, а также синтез речи, определение голосовой активности и шумоподавление — всё работает локально. Никаких облачных API, никакие данные не покидают устройство. **[📚 Документация Android](https://soniqo.audio/ru/getting-started/android)** @@ -28,7 +28,8 @@ | Модель | Задача | Загрузка | Пиковая память | Языки | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/ru/guides/dictate) | Потоковый STT + EOU (по умолчанию) | [153 МБ](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 МБ | 25 | -| [Parakeet TDT v3](https://soniqo.audio/ru/guides/parakeet/android) | STT с широким покрытием (опционально) | [891 МБ](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 ГБ | 114 | +| [Parakeet TDT v3](https://soniqo.audio/ru/guides/parakeet/android) | STT с широким покрытием (опционально) | [891 МБ](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1,1-1,3 ГБ | 25 европейских | +| [Nemotron-3.5 multilingual](https://soniqo.audio/ru/guides/nemotron) | Потоковый STT с управлением через промпт (опционально) | [~721 МБ](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | ещё не измерено | более 100 (включая zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | Офлайн STT + перевод (опционально) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/ru/guides/kokoro/android) | Синтез речи (по умолчанию) | [330 МБ](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 МБ | 8 (en, fr, es, it, pt, hi, ja, zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | Потоковый синтез речи (опционально, фиксированный голос Alba) | ~126 МБ | ещё не измерено | Английский | @@ -39,9 +40,25 @@ Модели загружаются автоматически при первом запуске через `ModelManager.ensureModels()`. -`SpeechConfig()` по умолчанию использует `SttModel.PARAKEET_EOU` и `TtsModel.KOKORO_SHORT_TURN`, чтобы интеграции SDK и системный распознаватель работали по низкопамятному Android-пути. Демо-приложение выбирает `SttModel.PARAKEET`, поэтому экраны эха и диктовки используют более крупную TDT-модель на 114 языков. +`SpeechConfig()` по умолчанию использует `SttModel.PARAKEET_EOU` и `TtsModel.KOKORO_SHORT_TURN`, чтобы интеграции SDK и системный распознаватель работали по низкопамятному Android-пути. Демо-приложение выбирает `SttModel.PARAKEET`, поэтому экраны эха и диктовки используют более крупную TDT-модель для 25 европейских языков. -Для распознавания с фокусом на языки используйте `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`. Задайте `language = "en"`, если нужно зафиксировать один язык. +Обе модели Parakeet всегда определяют язык автоматически: ни одна не принимает `language` или `languageHints`, и ни одна не поддерживает китайский. Чтобы выбрать один язык, включая мандаринский китайский, используйте управляемый промптом backend Nemotron: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // или "zh-TW" +) +``` **Supertonic-3** — это опциональный многоязычный TTS повышенного качества: выберите его через `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` (требуется бэкенд LiteRT). Хост выполняет его четыре неавторегрессионных flow-matching-графа на устройстве на частоте 44,1 кГц; фронтенд работает G2P-free (NFKD + индекс Unicode — без фонемизатора), поэтому все 31 язык проходят через один путь. @@ -101,7 +118,7 @@ cd speech-android - Эхо-режим: транскрибирует речь и синтезирует её обратно (без LLM) - Режим диктовки: потоковые частичные результаты - Голосовой оверлей: плавающая кнопка микрофона для диктовки в любом приложении -- Parakeet TDT STT на 114 языков в экранах эха и диктовки +- Parakeet TDT STT для 25 европейских языков в экранах эха и диктовки - Тестовый экран `SpeechRecognizer` — задействует системный путь голосового ввода - UI с пузырями чата и отображением задержки STT/TTS diff --git a/README_zh.md b/README_zh.md index 2e1663c..664f607 100644 --- a/README_zh.md +++ b/README_zh.md @@ -4,7 +4,7 @@ 适用于 Android 的设备端语音 SDK,基于 [ONNX Runtime](https://onnxruntime.ai) 和 [speech-core](https://github.com/soniqo/speech-core) 构建。 -低内存流式语音识别(默认 25 种语言,可选 114 语言 TDT)、文本转语音、语音活动检测和噪声消除——全部在本地运行。无需云端 API,数据不会离开设备。 +低内存流式语音识别和可选的大型 TDT 模型(两者均覆盖 25 种欧洲语言),以及文本转语音、语音活动检测和噪声消除——全部在本地运行。无需云端 API,数据不会离开设备。 **[📚 Android 文档](https://soniqo.audio/zh/getting-started/android)** @@ -28,7 +28,8 @@ | 模型 | 任务 | 下载大小 | 峰值内存 | 语言 | | --- | --- | --- | --- | --- | | [Parakeet-EOU 120M](https://soniqo.audio/zh/guides/dictate) | 流式 STT + 端点检测(默认) | [153 MB](https://huggingface.co/soniqo/Parakeet-EOU-120M-ONNX-INT8) | 232 MB | 25 | -| [Parakeet TDT v3](https://soniqo.audio/zh/guides/parakeet/android) | 广覆盖 STT(可选) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 114 | +| [Parakeet TDT v3](https://soniqo.audio/zh/guides/parakeet/android) | 广覆盖 STT(可选) | [891 MB](https://huggingface.co/soniqo/Parakeet-TDT-v3-ONNX) | ~1.1-1.3 GB | 25 种欧洲语言 | +| [Nemotron-3.5 多语言](https://soniqo.audio/zh/guides/nemotron) | 提示词条件流式 STT(可选) | [~721 MB](https://huggingface.co/soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-LiteRT-INT8) | 尚未测量 | 100+(包括 zh) | | [Canary 180M Flash](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | 离线 STT + 翻译(可选) | [273 MB](https://huggingface.co/soniqo/Canary-180M-Flash-ONNX) | ~780 MB | 4 (en, de, es, fr) | | [Kokoro 82M](https://soniqo.audio/zh/guides/kokoro/android) | 文本转语音(默认) | [330 MB](https://huggingface.co/soniqo/Kokoro-82M-ONNX) | 640 MB | 8(en、fr、es、it、pt、hi、ja、zh) | | [Pocket TTS 100M](https://huggingface.co/soniqo/Pocket-TTS-100M-ONNX-INT8) | 流式文本转语音(可选,固定 Alba 音色) | ~126 MB | 尚未测量 | 英语 | @@ -39,9 +40,25 @@ 模型在首次启动时通过 `ModelManager.ensureModels()` 自动下载。 -`SpeechConfig()` 默认使用 `SttModel.PARAKEET_EOU` 和 `TtsModel.KOKORO_SHORT_TURN`,让 SDK 集成和系统识别服务走低内存 Android 路径。演示应用会选用 `SttModel.PARAKEET`,使回声和听写界面使用更大的 114 语言 TDT 模型。 +`SpeechConfig()` 默认使用 `SttModel.PARAKEET_EOU` 和 `TtsModel.KOKORO_SHORT_TURN`,让 SDK 集成和系统识别服务走低内存 Android 路径。演示应用会选用 `SttModel.PARAKEET`,使回声和听写界面使用覆盖 25 种欧洲语言的较大型 TDT 模型。 -需要聚焦特定语言时,使用 `SpeechConfig(sttModel = SttModel.PARAKEET, languageHints = listOf("en", "fr"))`。如果只想固定单一语言,设置 `language = "en"`。 +两个 Parakeet 模型始终自动检测语言:它们都不接受 `language` 或 `languageHints`,也都不支持中文。要固定一种语言(包括普通话),请使用提示词条件 Nemotron 后端: + +```kotlin +val sttModel = SttModel.NEMOTRON_MULTILINGUAL +val sttBackend = SttBackend.LITERT +val modelDir = ModelManager.ensureModels( + context, + sttModel = sttModel, + sttBackend = sttBackend, +) +val config = SpeechConfig( + modelDir = modelDir, + sttModel = sttModel, + sttBackend = sttBackend, + language = "zh-CN", // 或 "zh-TW" +) +``` **Supertonic-3** 是可选启用的更高质量多语言 TTS — 通过 `SpeechConfig(ttsModel = TtsModel.SUPERTONIC)` 选用(需要 LiteRT 后端)。宿主在设备端以 44.1 kHz 运行其四个非自回归流匹配图;前端免 G2P(NFKD + Unicode 索引 — 无音素转换器),因此全部 31 种语言走同一条路径。 @@ -100,7 +117,7 @@ cd speech-android - 回声模式:转录语音并将其合成回放(无 LLM) - 听写模式:流式部分结果 - 语音悬浮窗:悬浮麦克风按钮,可向任意应用听写 -- 回声和听写界面使用 114 语言 Parakeet TDT STT +- 回声和听写界面使用覆盖 25 种欧洲语言的 Parakeet TDT STT - `SpeechRecognizer` 测试界面 — 演练系统级语音输入路径 - 带有 STT/TTS 延迟显示的聊天气泡 UI diff --git a/control-demo/README.md b/control-demo/README.md index 3f4ae97..ea436ef 100644 --- a/control-demo/README.md +++ b/control-demo/README.md @@ -33,6 +33,9 @@ Grant microphone access in the app, tap the orb, and try commands such as "set volume to three" or "what can you do?" Contact and media commands also need their corresponding Android permissions. +Call commands speak their confirmation before opening Android's system dialer; +the demo never places a call directly. + The footer shows STT, LLM, Pocket first-audio, round-trip, and memory metrics. The same data is available for device benchmarks with: diff --git a/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/KokoroLatencyTest.kt b/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/KokoroLatencyTest.kt index afa20c2..61552e1 100644 --- a/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/KokoroLatencyTest.kt +++ b/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/KokoroLatencyTest.kt @@ -100,7 +100,6 @@ class KokoroLatencyTest { modelDir = modelDir.absolutePath, useNnapi = useNnapi, pipelineMode = PipelineMode.TRANSCRIBE_ONLY, - language = "en", ), ).use { pipeline -> val start = SystemClock.elapsedRealtimeNanos() @@ -148,7 +147,6 @@ class KokoroLatencyTest { modelDir = modelDir.absolutePath, useNnapi = useNnapi, pipelineMode = PipelineMode.TRANSCRIBE_ONLY, - language = "en", ), ).use { pipeline -> val pieces = SpeechChunks.split(ControlTools.CAPABILITIES_SUMMARY) diff --git a/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/PocketTtsLatencyTest.kt b/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/PocketTtsLatencyTest.kt index 8ed2c53..3ddb003 100644 --- a/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/PocketTtsLatencyTest.kt +++ b/control-demo/src/androidTest/kotlin/audio/soniqo/speech/control/PocketTtsLatencyTest.kt @@ -45,7 +45,6 @@ class PocketTtsLatencyTest { useNnapi = false, ttsModel = TtsModel.POCKET, pipelineMode = PipelineMode.TRANSCRIBE_ONLY, - language = "en", ), ).use { pipeline -> val loadMs = elapsedMs(loadStart) @@ -96,7 +95,6 @@ class PocketTtsLatencyTest { useNnapi = false, ttsModel = TtsModel.POCKET, pipelineMode = PipelineMode.TRANSCRIBE_ONLY, - language = "en", ), ).use { pipeline -> synthesizeAndMeasure(pipeline, "Ready when you are.") diff --git a/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlAgentActivity.kt b/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlAgentActivity.kt index d401b2d..ba2f85d 100644 --- a/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlAgentActivity.kt +++ b/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlAgentActivity.kt @@ -46,6 +46,7 @@ import java.util.concurrent.atomic.AtomicBoolean import java.util.concurrent.atomic.AtomicLong import kotlin.math.abs import kotlin.math.roundToInt +import kotlinx.coroutines.CancellationException import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.channels.Channel import kotlinx.coroutines.delay @@ -113,12 +114,6 @@ class ControlAgentActivity : ComponentActivity() { private fun activeStt(): SttModel = sttOverride?.let { runCatching { SttModel.valueOf(it) }.getOrNull() } ?: DEMO_STT - // Drives the TTS language (and Nemotron's prompt slot if ever selected). - // Parakeet STT autodetects regardless — like every other Parakeet - // runtime, there is no language forcing, so accented speech can - // occasionally decode into a non-Latin script on the multilingual TDT. - private fun activeLanguage(): String = "en" - // Contextual-biasing phrases for the STT beam search: the fixed command // grammar and the brand, plus the track titles/artists currently on the // device. The command core is always present; without media permission @@ -164,7 +159,7 @@ class ControlAgentActivity : ComponentActivity() { // STT model for the demo. PARAKEET_EOU is the low-memory default // (25 languages, ~232 MB) — fast, ~2 s round trip, ~1.3 GB total. - // PARAKEET is Parakeet-TDT v3 (114 languages, auto-detect) but + // PARAKEET is Parakeet-TDT v3 (25 European languages, auto-detect) but // ~891 MB download and ~2 GB total, which is ~5× slower under // emulation and needs a real device with an NPU to feel good. private val DEMO_STT = SttModel.PARAKEET_EOU @@ -392,7 +387,6 @@ class ControlAgentActivity : ComponentActivity() { ttsModel = DEMO_TTS, pipelineMode = PipelineMode.TRANSCRIBE_ONLY, emitPartialTranscriptions = true, - language = activeLanguage(), // 0.5 s default cut people off mid-command on-device — // dictated digits and thinking pauses exceed it easily. endOfSpeechSilenceSec = 0.8f, @@ -515,8 +509,21 @@ class ControlAgentActivity : ComponentActivity() { } val turnId = store.beginTurn(text) lifecycleScope.launch(Dispatchers.Default) { - try { runAgentTurn(turnId, text, sttMs, voiceAnchored) } - finally { turnInFlight.set(false) } + try { + runAgentTurn(turnId, text, sttMs, voiceAnchored) + } catch (e: CancellationException) { + throw e + } catch (e: Exception) { + Log.e(TAG, "agent turn failed", e) + store.updateTurn(turnId) { + it.copy(toolLabel = "turn error: ${e.message}", failed = true) + } + store.addNote("turn error: ${e.message}") + micPaused = false + returnToRest() + } finally { + turnInFlight.set(false) + } } } @@ -706,7 +713,10 @@ class ControlAgentActivity : ComponentActivity() { } catch (e: Exception) { Log.e(TAG, "TTS failed", e) store.addNote("tts error: ${e.message}") - micPaused = false; returnToRest(); return + micPaused = false + returnToRest() + executeDeferredAction(outcome, turnId) + return } val presentedMs = firstPresentationMs.get().takeIf { it > 0 } ?: playbackStartMs val perf = if (playbackPerformanceMode == android.media.AudioTrack.PERFORMANCE_MODE_LOW_LATENCY) { @@ -721,6 +731,21 @@ class ControlAgentActivity : ComponentActivity() { "round ${System.currentTimeMillis() - turnAnchorMs}ms total") micPaused = false returnToRest() + executeDeferredAction(outcome, turnId) + } + + /** Launch actions that intentionally leave this Activity only after TTS. */ + private suspend fun executeDeferredAction(outcome: ToolOutcome?, turnId: Long) { + if (outcome?.deferredAction == null) return + try { + withContext(Dispatchers.Main.immediate) { + ControlTools.executeDeferredAction(outcome, device) + } + } catch (e: Exception) { + Log.e(TAG, "deferred tool execution failed", e) + store.updateTurn(turnId) { it.copy(failed = true) } + store.addNote("action error: ${e.message}") + } } // ----------------------------------------------------------------------- diff --git a/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlTools.kt b/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlTools.kt index 771c5d1..2366760 100644 --- a/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlTools.kt +++ b/control-demo/src/main/kotlin/audio/soniqo/speech/control/ControlTools.kt @@ -189,18 +189,21 @@ object ControlTools { ) } else { device.stopMusic() // a call takes over audio; stop the music - device.dial(contact.number) ToolOutcome( say ?: "Calling ${contact.name}.", label(call.name, "name" to name), + DeferredDeviceAction.Dial(contact.number), ) } } "dial_number" -> { val number = call.string("number") ?: return null device.stopMusic() // a call takes over audio; stop the music - device.dial(number) - ToolOutcome(say ?: "Calling $number.", label(call.name, "number" to number)) + ToolOutcome( + say ?: "Calling $number.", + label(call.name, "number" to number), + DeferredDeviceAction.Dial(number), + ) } "find_contact" -> { val name = call.string("name") ?: return null @@ -261,6 +264,18 @@ object ControlTools { } } + /** + * Run an external action after its spoken confirmation. Launching another + * Activity sooner triggers [android.app.Activity.onStop], which tears down + * the demo's AudioTrack while Pocket TTS is still writing to it. + */ + fun executeDeferredAction(outcome: ToolOutcome, device: DeviceActions) { + when (val action = outcome.deferredAction) { + is DeferredDeviceAction.Dial -> device.dial(action.number) + null -> Unit + } + } + private fun label(name: String, vararg args: Pair): String = "$name {${args.joinToString(", ") { "${it.first}: ${it.second}" }}}" } diff --git a/control-demo/src/main/kotlin/audio/soniqo/speech/control/DeviceActions.kt b/control-demo/src/main/kotlin/audio/soniqo/speech/control/DeviceActions.kt index 1131484..de4a22d 100644 --- a/control-demo/src/main/kotlin/audio/soniqo/speech/control/DeviceActions.kt +++ b/control-demo/src/main/kotlin/audio/soniqo/speech/control/DeviceActions.kt @@ -6,12 +6,20 @@ data class Contact(val name: String, val number: String) /** A playable local audio track. */ data class Track(val title: String, val artist: String?) +/** An Android action that must run after the spoken reply has drained. */ +sealed interface DeferredDeviceAction { + /** Opening the dialer backgrounds the demo Activity, so it cannot overlap TTS playback. */ + data class Dial(val number: String) : DeferredDeviceAction +} + /** Result of executing one tool call. */ data class ToolOutcome( /** What the agent says — model-authored `say` or a result template. */ val spoken: String, /** Compact log label, e.g. `call_contact {name: anna}`. */ val label: String, + /** Optional external action to run once [spoken] has finished playing. */ + val deferredAction: DeferredDeviceAction? = null, ) /** diff --git a/control-demo/src/test/kotlin/audio/soniqo/speech/control/ControlToolsTest.kt b/control-demo/src/test/kotlin/audio/soniqo/speech/control/ControlToolsTest.kt index a46cdb2..9239bf0 100644 --- a/control-demo/src/test/kotlin/audio/soniqo/speech/control/ControlToolsTest.kt +++ b/control-demo/src/test/kotlin/audio/soniqo/speech/control/ControlToolsTest.kt @@ -69,16 +69,20 @@ class ControlToolsTest { // ------------------------------------------------------------------ @Test - fun callContact_found_dialsAndSpeaksModelSay() { + fun callContact_found_defersDialUntilAfterModelSay() { val outcome = ControlTools.execute(call( "call_contact", "name" to ArgumentValue.Str("anna"), "say" to ArgumentValue.Str("Calling Anna now."), ), device)!! - assertEquals("+4917612345678", device.dialed) + assertNull(device.dialed) + assertEquals(DeferredDeviceAction.Dial("+4917612345678"), outcome.deferredAction) assertEquals("Calling Anna now.", outcome.spoken) assertEquals("call_contact {name: anna}", outcome.label) + + ControlTools.executeDeferredAction(outcome, device) + assertEquals("+4917612345678", device.dialed) } @Test @@ -201,15 +205,19 @@ class ControlToolsTest { // ------------------------------------------------------------------ @Test - fun dialNumber_dialsAndHonorsSay() { + fun dialNumber_defersDialAndHonorsSay() { val outcome = ControlTools.execute(call( "dial_number", "number" to ArgumentValue.Str("+493012345"), "say" to ArgumentValue.Str("Dialing."), ), device)!! - assertEquals("+493012345", device.dialed) + assertNull(device.dialed) + assertEquals(DeferredDeviceAction.Dial("+493012345"), outcome.deferredAction) assertEquals("Dialing.", outcome.spoken) + + ControlTools.executeDeferredAction(outcome, device) + assertEquals("+493012345", device.dialed) } @Test @@ -286,9 +294,12 @@ class ControlToolsTest { @Test fun callingStopsMusicFirst() { device.musicPlaying = true - ControlTools.execute(call( + val outcome = ControlTools.execute(call( "call_contact", "name" to ArgumentValue.Str("anna")), device)!! assertTrue("a call must stop any playing music", device.stopped) + assertNull("the dialer must wait until speech completes", device.dialed) + + ControlTools.executeDeferredAction(outcome, device) assertEquals("+4917612345678", device.dialed) device.stopped = false diff --git a/sdk/src/androidTest/kotlin/audio/soniqo/speech/ParakeetSttTest.kt b/sdk/src/androidTest/kotlin/audio/soniqo/speech/ParakeetSttTest.kt index 3d797cb..55aed25 100644 --- a/sdk/src/androidTest/kotlin/audio/soniqo/speech/ParakeetSttTest.kt +++ b/sdk/src/androidTest/kotlin/audio/soniqo/speech/ParakeetSttTest.kt @@ -47,8 +47,8 @@ class ParakeetSttTest { } @Test - fun sttTranscribesSynthesizedEnglishWithLanguageHints() = runBlocking { - val config = parakeetConfig(languageHints = listOf("en-US", "fr")) + fun sttAutoDetectsSynthesizedEnglish() = runBlocking { + val config = parakeetConfig() val pipeline = SpeechPipeline(config) try { pipeline.start() @@ -197,22 +197,6 @@ class ParakeetSttTest { pipeline.close() } - @Test - fun fixedLanguagePipelineStarts() { - val config = parakeetConfig(language = "en-US") - val pipeline = SpeechPipeline(config) - - assertEquals(PipelineState.Idle, pipeline.state) - pipeline.start() - assertTrue( - pipeline.state == PipelineState.Idle || - pipeline.state == PipelineState.Listening - ) - - pipeline.stop() - pipeline.close() - } - private fun pcm16ToFloat16k(pcm16: ByteArray, sourceSampleRate: Int): FloatArray { val shorts = ShortArray(pcm16.size / 2) ByteBuffer.wrap(pcm16) @@ -232,14 +216,9 @@ class ParakeetSttTest { } } - private fun parakeetConfig( - language: String = "auto", - languageHints: List = emptyList(), - ): SpeechConfig = SpeechConfig( + private fun parakeetConfig(): SpeechConfig = SpeechConfig( modelDir = modelDir, useNnapi = false, sttModel = SttModel.PARAKEET, - language = language, - languageHints = languageHints, ) } diff --git a/sdk/src/main/cpp/jni_bridge.cpp b/sdk/src/main/cpp/jni_bridge.cpp index 7274514..b6a6f08 100644 --- a/sdk/src/main/cpp/jni_bridge.cpp +++ b/sdk/src/main/cpp/jni_bridge.cpp @@ -276,7 +276,16 @@ Java_audio_soniqo_speech_NativeBridge_nativeCreate( dir + "/nemotron-multilingual-decoder.tflite", dir + "/nemotron-multilingual-joint.tflite", dir + "/vocab.json", dir + "/languages.json", nnapi); - if (lang != "auto" && !lang.empty()) m->set_language(lang); + // The core setter returns false for the special "auto" value + // after selecting autoSlot; false means an error only for a + // concrete locale. + const bool language_found = lang.empty() || m->set_language(lang); + if (lang != "auto" && !language_found) { + throw std::invalid_argument( + "Nemotron has no language prompt for '" + lang + + "'; use a locale from languages.json " + "(Chinese: zh-CN or zh-TW)"); + } h->stt = std::move(m); #else throw std::runtime_error("LiteRT STT backend not built into this SDK"); @@ -285,7 +294,16 @@ Java_audio_soniqo_speech_NativeBridge_nativeCreate( auto m = std::make_unique( dir + "/encoder.onnx", dir + "/decoder.onnx", dir + "/joint.onnx", dir + "/vocab.json", dir + "/languages.json", nnapi); - if (lang != "auto" && !lang.empty()) m->set_language(lang); + // The core setter returns false for the special "auto" value + // after selecting autoSlot; false means an error only for a + // concrete locale. + const bool language_found = lang.empty() || m->set_language(lang); + if (lang != "auto" && !language_found) { + throw std::invalid_argument( + "Nemotron has no language prompt for '" + lang + + "'; use a locale from languages.json " + "(Chinese: zh-CN or zh-TW)"); + } h->stt = std::move(m); } } else if (sttModel == STT_CANARY) { diff --git a/sdk/src/main/kotlin/audio/soniqo/speech/SpeechConfig.kt b/sdk/src/main/kotlin/audio/soniqo/speech/SpeechConfig.kt index efd32ef..5a34c27 100644 --- a/sdk/src/main/kotlin/audio/soniqo/speech/SpeechConfig.kt +++ b/sdk/src/main/kotlin/audio/soniqo/speech/SpeechConfig.kt @@ -62,10 +62,10 @@ data class SpeechConfig( /** Pipeline behavior after transcription. See [PipelineMode]. */ val pipelineMode: PipelineMode = PipelineMode.ECHO, - /** Language/locale for prompt-conditioned STT models (Nemotron) and TTS - * voice selection: e.g. "en-US", "fr", "ja-JP". "auto" lets the model - * decide. Parakeet TDT always autodetects — the transducer has no - * forcing mechanism, matching every other Parakeet runtime. */ + /** Language/locale for prompt-conditioned STT models: e.g. "en-US", "fr", + * "zh-CN". "auto" lets the model decide. Parakeet TDT and Parakeet-EOU + * always autodetect and reject a concrete value because neither model has + * a language-prompt input. */ val language: String = "auto", /** Enable noise cancellation (DeepFilterNet3). */ @@ -80,9 +80,10 @@ data class SpeechConfig( /** Interval between partial transcriptions in seconds. */ val partialTranscriptionInterval: Float = 0.5f, - /** Reserved language shortlist for future prompt-conditioned backends. - * No current STT backend consumes it: Nemotron takes a single - * [language], and Parakeet TDT always autodetects. */ + /** Reserved for a future backend that can constrain automatic detection. + * No current STT backend consumes a shortlist, so a non-empty value is + * rejected instead of being silently ignored. Nemotron takes one + * [language]; both Parakeet models always autodetect. */ val languageHints: List = emptyList(), /** RNN-T beam width for the Parakeet-EOU streaming STT. `<= 1` keeps the @@ -97,3 +98,29 @@ data class SpeechConfig( * sentence — and get cut off early. */ val endOfSpeechSilenceSec: Float = 0.5f, ) + +/** Validate language settings before JNI loads models or starts native work. */ +internal fun SpeechConfig.requireValidLanguageConfiguration() { + val requested = language.trim() + val base = requested.substringBefore('-').substringBefore('_').lowercase() + + require(base != "cn") { + "SpeechConfig.language='$language' uses the country code 'cn', not a " + + "Chinese language tag. Use 'zh-CN' for Simplified Chinese or " + + "'zh-TW' for Traditional Chinese." + } + require(languageHints.isEmpty()) { + "SpeechConfig.languageHints is not supported by any current STT backend. " + + "Parakeet models always auto-detect; for one fixed language select " + + "SttModel.NEMOTRON_MULTILINGUAL and set SpeechConfig.language." + } + + val automatic = requested.isEmpty() || requested == "auto" + if (sttModel == SttModel.PARAKEET || sttModel == SttModel.PARAKEET_EOU) { + require(automatic) { + "${sttModel.name} always auto-detects language and cannot honor " + + "SpeechConfig.language='$language'. For a fixed language select " + + "SttModel.NEMOTRON_MULTILINGUAL (for Chinese, use 'zh-CN' or 'zh-TW')." + } + } +} diff --git a/sdk/src/main/kotlin/audio/soniqo/speech/SpeechPipeline.kt b/sdk/src/main/kotlin/audio/soniqo/speech/SpeechPipeline.kt index 89d7417..7511739 100644 --- a/sdk/src/main/kotlin/audio/soniqo/speech/SpeechPipeline.kt +++ b/sdk/src/main/kotlin/audio/soniqo/speech/SpeechPipeline.kt @@ -99,7 +99,10 @@ interface SpeechPipeline : AutoCloseable { fun cancelSynthesis() {} companion object { - operator fun invoke(config: SpeechConfig): SpeechPipeline = SpeechPipelineImpl(config) + operator fun invoke(config: SpeechConfig): SpeechPipeline { + config.requireValidLanguageConfiguration() + return SpeechPipelineImpl(config) + } } } diff --git a/sdk/src/main/kotlin/audio/soniqo/speech/service/SpeechRecognitionService.kt b/sdk/src/main/kotlin/audio/soniqo/speech/service/SpeechRecognitionService.kt index cd4ccf5..07cf42b 100644 --- a/sdk/src/main/kotlin/audio/soniqo/speech/service/SpeechRecognitionService.kt +++ b/sdk/src/main/kotlin/audio/soniqo/speech/service/SpeechRecognitionService.kt @@ -90,7 +90,6 @@ open class SpeechRecognitionService : RecognitionService() { val wantPartial = recognizerIntent ?.getBooleanExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, false) ?: false val requestedLang = recognizerIntent?.let(::requestedLanguageTag) - val languageHints = recognizerIntent?.let(::languageHintTags).orEmpty() if (requestedLang != null) { Log.i(TAG, "EXTRA_LANGUAGE=$requestedLang") } @@ -111,7 +110,7 @@ open class SpeechRecognitionService : RecognitionService() { scope.launch { try { - startSession(listener, wantPartial, requestedLang, languageHints) + startSession(listener, wantPartial) } finally { starting.set(false) } @@ -121,8 +120,6 @@ open class SpeechRecognitionService : RecognitionService() { private suspend fun startSession( listener: Callback, wantPartial: Boolean, - requestedLang: String?, - languageHints: List, ) { val pipeline: SpeechPipeline val record: AudioRecord @@ -131,8 +128,10 @@ open class SpeechRecognitionService : RecognitionService() { pipeline = createPipeline( SpeechConfig( modelDir = modelDir, - language = requestedLang?.let(::languageBaseTag) ?: "auto", - languageHints = languageHints, + // Parakeet-EOU accepts no language prompt. The framework + // request is still checked against its supported language + // set above, then the model performs automatic detection. + language = "auto", emitPartialTranscriptions = wantPartial, ) ) diff --git a/sdk/src/test/kotlin/audio/soniqo/speech/SpeechConfigTest.kt b/sdk/src/test/kotlin/audio/soniqo/speech/SpeechConfigTest.kt index 72403da..f7443c1 100644 --- a/sdk/src/test/kotlin/audio/soniqo/speech/SpeechConfigTest.kt +++ b/sdk/src/test/kotlin/audio/soniqo/speech/SpeechConfigTest.kt @@ -1,6 +1,8 @@ package audio.soniqo.speech import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue import org.junit.Test class SpeechConfigTest { @@ -9,4 +11,68 @@ class SpeechConfigTest { fun `end-of-speech silence defaults to snappy command value`() { assertEquals(0.5f, SpeechConfig().endOfSpeechSilenceSec, 0f) } + + @Test + fun `cn country code is rejected with Chinese language tags`() { + val error = assertThrows(IllegalArgumentException::class.java) { + SpeechConfig( + sttModel = SttModel.NEMOTRON_MULTILINGUAL, + language = "cn", + ).requireValidLanguageConfiguration() + } + + assertTrue(error.message.orEmpty().contains("zh-CN")) + assertTrue(error.message.orEmpty().contains("zh-TW")) + } + + @Test + fun `Parakeet rejects fixed language instead of silently ignoring it`() { + val error = assertThrows(IllegalArgumentException::class.java) { + SpeechConfig( + sttModel = SttModel.PARAKEET, + language = "zh-CN", + ).requireValidLanguageConfiguration() + } + + assertTrue(error.message.orEmpty().contains("always auto-detects")) + assertTrue(error.message.orEmpty().contains("NEMOTRON_MULTILINGUAL")) + } + + @Test + fun `Parakeet EOU rejects fixed language`() { + assertThrows(IllegalArgumentException::class.java) { + SpeechConfig( + sttModel = SttModel.PARAKEET_EOU, + language = "en", + ).requireValidLanguageConfiguration() + } + } + + @Test + fun `language shortlist is rejected instead of silently ignored`() { + val error = assertThrows(IllegalArgumentException::class.java) { + SpeechConfig( + sttModel = SttModel.PARAKEET, + languageHints = listOf("en", "zh-CN"), + ).requireValidLanguageConfiguration() + } + + assertTrue(error.message.orEmpty().contains("languageHints")) + } + + @Test + fun `Nemotron accepts fixed Chinese locale`() { + SpeechConfig( + sttModel = SttModel.NEMOTRON_MULTILINGUAL, + language = "zh-CN", + ).requireValidLanguageConfiguration() + } + + @Test + fun `automatic language remains valid for both Parakeet models`() { + SpeechConfig(sttModel = SttModel.PARAKEET) + .requireValidLanguageConfiguration() + SpeechConfig(sttModel = SttModel.PARAKEET_EOU) + .requireValidLanguageConfiguration() + } } diff --git a/sdk/src/test/kotlin/audio/soniqo/speech/service/SpeechRecognitionServiceTest.kt b/sdk/src/test/kotlin/audio/soniqo/speech/service/SpeechRecognitionServiceTest.kt index 2bd6530..5072dd4 100644 --- a/sdk/src/test/kotlin/audio/soniqo/speech/service/SpeechRecognitionServiceTest.kt +++ b/sdk/src/test/kotlin/audio/soniqo/speech/service/SpeechRecognitionServiceTest.kt @@ -97,27 +97,27 @@ class SpeechRecognitionServiceTest { } @Test - fun `requested regional language is accepted`() { + fun `requested regional language is accepted with Parakeet auto-detection`() { val intent = Intent().putExtra(RecognizerIntent.EXTRA_LANGUAGE, "fr-FR") service.startListening(intent, listener) verify(timeout = 1500) { listener.readyForSpeech(any()) } - assertEquals("fr", service.lastConfig?.language) + assertEquals("auto", service.lastConfig?.language) } @Test - fun `requested language preference is accepted`() { + fun `requested language preference is accepted with Parakeet auto-detection`() { val intent = Intent().putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, "en-US") service.startListening(intent, listener) verify(timeout = 1500) { listener.readyForSpeech(any()) } - assertEquals("en", service.lastConfig?.language) + assertEquals("auto", service.lastConfig?.language) } @Test - fun `allowed languages are passed as hints`() { + fun `allowed languages do not become unsupported Parakeet hints`() { val intent = Intent().putStringArrayListExtra( RecognizerIntent.EXTRA_LANGUAGE_DETECTION_ALLOWED_LANGUAGES, arrayListOf("fr-FR", "de-DE", "ja-JP"), @@ -127,11 +127,11 @@ class SpeechRecognitionServiceTest { verify(timeout = 1500) { listener.readyForSpeech(any()) } assertEquals("auto", service.lastConfig?.language) - assertEquals(listOf("fr", "de"), service.lastConfig?.languageHints) + assertTrue(service.lastConfig?.languageHints.orEmpty().isEmpty()) } @Test - fun `requested language takes priority over allowed languages`() { + fun `requested and allowed languages leave Parakeet in auto-detect mode`() { val intent = Intent() .putExtra(RecognizerIntent.EXTRA_LANGUAGE, "en-US") .putStringArrayListExtra( @@ -142,8 +142,8 @@ class SpeechRecognitionServiceTest { service.startListening(intent, listener) verify(timeout = 1500) { listener.readyForSpeech(any()) } - assertEquals("en", service.lastConfig?.language) - assertEquals(listOf("en", "fr", "de"), service.lastConfig?.languageHints) + assertEquals("auto", service.lastConfig?.language) + assertTrue(service.lastConfig?.languageHints.orEmpty().isEmpty()) } @Test