FYI: Run models from piper with the Next-gen Kaldi subproject sherpa-onnx

[csukuangfj]

FYI: We have supported piper models in
https://github.com/k2-fsa/sherpa-onnx

Note that it does not depend on https://github.com/rhasspy/piper-phonemize

sherpa-onnx supports a variety of platforms, such as

• Windows (x86, x64)
• Linux (x64, arm, arm64), i.e., rapsberry Pi
• macOS (x64, arm64)

It also provides various programming language APIs, e.g., C/C++/Python/Kotlin/Swift/C#/Go. We also have android APKs for TTS.

You can find the installation doc at https://k2-fsa.github.io/sherpa/onnx/install/index.html

You can find the usage of piper models with sherpa-onnx at
https://k2-fsa.github.io/sherpa/onnx/tts/pretrained_models/vits.html#lessac-blizzard2013-medium-english-single-speaker

We also have a huggingface space for you to try piper models with sherpa-onnx.
Please visit
https://huggingface.co/spaces/k2-fsa/text-to-speech

---

You can find the PR supporting piper in sherpa-onnx at k2-fsa/sherpa-onnx#390

[mush42]

@csukuangfj where to find the Android APKs?

[beqabeqa473]

@csukuangfj Yes, it would be good to know about android tts as well. Could you please tell where to get it?

[csukuangfj]

I'm sorry for not getting back to you sooner.

I have been working on converting more models from piper.

Now all models of the following languages have been converted to sherpa-onnx:

• English (both US and GB)
• French
• German
• Spanish (both ES and MX)

You can find the Android APKs on the following page.
https://k2-fsa.github.io/sherpa/onnx/tts/apk.html

[beqabeqa473]

Are there using standard android text-to-speech api or not?

…

On 10/29/23, Fangjun Kuang ***@***.***> wrote: I'm sorry for not getting back to you sooner. I have been working on converting more models from piper. Now all models of the following languages have been converted to sherpa-onnx: - English (both US and GB) - French - German - Spanish (both ES and MX) You can find the Android APKs on the following page. https://k2-fsa.github.io/sherpa/onnx/tts/apk.html <img width="901" alt="Screenshot 2023-10-29 at 17 48 27" src="https://github.com/rhasspy/piper/assets/5284924/c36b2eb7-ca4a-411d-8a03-48851a8d2c09"> -- Reply to this email directly or view it on GitHub: #251 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

-- with best regards Beqa Gozalishvili Tell: +995593454005 Email: ***@***.*** Web: https://gozaltech.org Skype: beqabeqa473 Telegram: https://t.me/gozaltech facebook: https://facebook.com/gozaltech twitter: https://twitter.com/beqabeqa473 Instagram: https://instagram.com/beqa.gozalishvili

[csukuangfj]

Are there using standard android text-to-speech api or not?

@beqabeqa473

No, it uses sherpa-onnx with vits pre-trained models for tts.

Everything is open-sourced. You can find the source code for the android project at
https://github.com/k2-fsa/sherpa-onnx/tree/master/android/SherpaOnnxTts

The underlying C++ code can be found at https://github.com/k2-fsa/sherpa-onnx

The JNI C++ binding code can be found at
https://github.com/k2-fsa/sherpa-onnx/tree/master/sherpa-onnx/jni

You can find kotlin API examples at
https://github.com/k2-fsa/sherpa-onnx/tree/master/kotlin-api-examples

[beqabeqa473]

Aah, ok, i ment standard tts-engine api bindings. I may try to do it in some future to use this tts as a standard andtoid tts engine for example with screenreaders.

…

On 10/29/23, Fangjun Kuang ***@***.***> wrote: > Are there using standard android text-to-speech api or not? @beqabeqa473 No, it uses sherpa-onnx with vits pre-trained models for tts. Everything is open-sourced. You can find the source code for the android project at https://github.com/k2-fsa/sherpa-onnx/tree/master/android/SherpaOnnxTts The underlying C++ code can be found at https://github.com/k2-fsa/sherpa-onnx The JNI C++ binding code can be found at https://github.com/k2-fsa/sherpa-onnx/tree/master/sherpa-onnx/jni You can find kotlin API examples at https://github.com/k2-fsa/sherpa-onnx/tree/master/kotlin-api-examples -- Reply to this email directly or view it on GitHub: #251 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- with best regards Beqa Gozalishvili Tell: +995593454005 Email: ***@***.*** Web: https://gozaltech.org Skype: beqabeqa473 Telegram: https://t.me/gozaltech facebook: https://facebook.com/gozaltech twitter: https://twitter.com/beqabeqa473 Instagram: https://instagram.com/beqa.gozalishvili

[synesthesiam]

Thanks for doing this @csukuangfj! I'd looked into sherpa-onnx at one point, but wasn't sure how to proceed. I'd like to link to your work when you think it's stable enough; I do want to make sure people understand that pronunciations may be slightly different due to the pre-computed lexicon.

Speaking of the lexicon, could it be extended dynamically at runtime with your approach?

[csukuangfj]

@synesthesiam

but wasn't sure how to proceed.

We have detailed documentation at
https://k2-fsa.github.io/sherpa/onnx/

Could you tell us what you want to do? We can clarify the doc if you think it is not clear.

---

I do want to make sure people understand that pronunciations may be slightly different due to the pre-computed lexicon.

The lexicon.txt is generated by following the colab notebook from this repo
https://github.com/rhasspy/piper/blob/master/notebooks/piper_inference_(ONNX).ipynb

The exact code can be found at
https://github.com/csukuangfj/models/tree/master/.github/scripts

Could you explain where the difference comes from?

Speaking of the lexicon, could it be extended dynamically at runtime with your approach?

No, it cannot. If there is an OOV at runtime, it is simply ignored, though a message is printed to tell the user
that an OOV has been ignored.

I'd like to link to your work when you think it's stable enough;

Thank you! I think the support for offline VITS models is stable now. (The APIs for the VITS model are quite simple and
there should be no big changes to the APIs in the near future)

[synesthesiam]

Could you tell us what you want to do? We can clarify the doc if you think it is not clear.

I meant more "big picture" in how I should proceed. I wasn't sure if it was worth investigating porting Piper to sherpa-onnx. I'd be curious if you've noticed any speed difference.

[csukuangfj]

Thanks for doing this @csukuangfj! I'd looked into sherpa-onnx at one point, but wasn't sure how to proceed. I'd like to link to your work when you think it's stable enough; I do want to make sure people understand that pronunciations may be slightly different due to the pre-computed lexicon.

Speaking of the lexicon, could it be extended dynamically at runtime with your approach?

@synesthesiam
I am integrating piper-phonemize so that we can discard lexicon.txt in sherpa-onnx.

Could you have a look at the following two PRs?

• support building static libraries piper-phonemize#21
• Support building shared library of ucd espeak-ng#1

[csukuangfj]

https://huggingface.co/csukuangfj/vits-piper-pt_PT-tugao-medium/tree/main

I have converted all of the models from piper to sherpa-onnx.
No lexicon.txt is required any more. I am using piper-phonemize.

(No that you can all run the models on Android/iOS/Raspberry Pi, etc).

[anita-smith1]

@csukuangfj
"No lexicon.txt is required any more. I am using piper-phonemize."

does this apply to piper models only? is lexicon required for coqui tts models? I'm following up on [#257]
(#257)

I couldn't use my coqui tts converted sherpa onyx model because I had to manually add words to lexicon and there was poor pronunciation for single words.

[csukuangfj]

is lexicon required for coqui tts models?

No, it is also not required for coqui tts models

All vits models for coqui don't use lexicon.txt for sherpa-onnx.

---

I couldn't use my coqui tts converted sherpa onyx model because I had to manually add words to lexicon and there was poor pronunciation for single words.

Please look at just one coqui model at
https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models

For instance, you can look at
https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-coqui-en-ljspeech.tar.bz2

Download it, unzip it, and you will find the code for exporting models from coqui to sherpa-onnx.

[anita-smith1]

@csukuangfj meaning your notebook doesn't work anymore ? https://colab.research.google.com/drive/1cI9VzlimS51uAw4uCR-OBeSXRPBc4KoK?usp=sharing

[csukuangfj]

@csukuangfj meaning your notebook doesn't work anymore ? https://colab.research.google.com/drive/1cI9VzlimS51uAw4uCR-OBeSXRPBc4KoK?usp=sharing

I just updated the colab notebook. Please reload it.

@anita-smith1

The updated colab notebook is much much simpler than before.