Is it normal for it to be this slow?

[cochese9000]

Hi there, I'm really enjoying testing your work, it's very good indeed!

I'm running it on windows 11 with flash attn 2 and triton installed.

Python 3.12.12
Build cuda_12.6.r12.6/compiler.34431801_0

I'm trying to use the clis/moss_tts_app.py Gradio UI.

But I'm finding it very slow to use on a 13900k with a 3090rtx. I'm getting around 1 to 5 it/s and it slows down on longer generations. Even a single sentence takes around 1 minute to generate. When I add a paragraph of text, it is several minutes (2-4 minutes depending on paragraph size).

The results are very good, with emotion being realistic and not nonotonous. But the latency is surprisingly high. Is this normal?

The real-time test is much quicker, but the quality is highly variable. Using moss_tts_realtime/app.y I can generate in a few seconds, but the quality is not useable in my case.

Just for reference, here is a generation using moss_tts_app.py with a single sentence.

Generating bs1 ...:  17%|████████████████████▋                   | 177/1024 [00:30<02:24,  5.86it/s]

^^ (shortened the line so it fits in git text window)

This will slow down as I add more sentences. I have a 13700k with RTX3090 and 64GB of RAM. The VRAM does get close to full when inference is running.

Thanks!

[YWMditto]

This looks very strange. Our speed test results on the 4090 are close to the following issue. Could you please provide all the logs after running moss_tts_app.py for our reference and analysis?

#7 (comment)

[cochese9000]

Thanks, here are the logs. I did one generation of a single short sentence.

FYI, I was unable to download the models with the built-in script (it formed broken paths in the models folder, e.g. 'OpenMOSS_hyphen_Team' and the model never actually downloaded), so I manually downloaded them with the following script:

try:
    user_info = whoami()
    print(f"Logged in as: {user_info['name']}")
except:
    print("Not logged in")

model_id = "OpenMOSS-Team/MOSS-TTS"
print(f"Downloading {model_id}...")

# This downloads to HF cache but returns the local path
MODEL_PATH = snapshot_download(
    repo_id=model_id,
    local_files_only=False
)

print(f"Downloaded to: {MODEL_PATH}")

Here is the log:

Downloading OpenMOSS-Team/MOSS-TTS...
Fetching 21 files: 100%|██████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 21006.53it/s]
Download complete: : 0.00B [00:00, ?B/s]              
Downloaded to: C:\Users\me\.cache\huggingface\hub\models--OpenMOSS-Team--MOSS-TTS\snapshots\0c8df9988ab61071cdb06fe40b2bdc3132ac3b7e
[INFO] Using attn_implementation=flash_attention_2
[Startup] Preloading backend: model=C:\Users\me\.cache\huggingface\hub\models--OpenMOSS-Team--MOSS-TTS\snapshots\0c8df9988ab61071cdb06fe40b2bdc3132ac3b7e, device=cuda:0, attn=flash_attention_2
Download complete: : 0.00B [00:00, ?B/s]
Loading weights: 100%|█| 1600/1600 [00:00<00:00, 3339.66it/s, Materializing param=quantizer.quantizers.31.out_proj.parametrizations.weight.origin 
`torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|███████████████████████████████████████████████| 463/463 [00:00<00:00, 2669.93it/s, Materializing param=lm_heads.32.weight] 
[Startup] Backend preload finished in 24.12s
O:\voc\moss\clis\moss_tts_app.py:434: UserWarning: The parameters have been moved from the Blocks constructor to the launch() method in Gradio 6.0: css. Please pass these parameters to launch() instead.
  with gr.Blocks(title="MOSS-TTS Demo", css=custom_css) as demo:
C:\Users\me\anaconda3\envs\moss\Lib\site-packages\gradio\components\dataframe.py:192: UserWarning: The `col_count` parameter is deprecated and will be removed. Please use `column_count` instead.
  warnings.warn(
* Running on local URL:  http://127.0.0.1:8084
* To create a public link, set `share=True` in `launch()`.
generating...
Generating bs1 ...:   2%|█▊          | 84/4096 [00:14<11:37,  5.75it/s]
finished generating.
C:\Users\me\anaconda3\envs\moss\Lib\site-packages\gradio\processing_utils.py:689: UserWarning: Trying to convert audio automatically from float32 to 16-bit int format.
  warnings.warn(warning.format(data.dtype))

If it is off interest, the failed downloaded were going to this folder:

C:\Users\me.cache\huggingface\modules\transformers_modules\OpenMOSS_hyphen_Team (and similar, but the 'hyphen' bit seemed to be breaking it).

[Quantumatum]

I also had the same issue with the script failing to download the model:

python moss_tts_app.py
[INFO] Using attn_implementation=flash_attention_2
[Startup] Preloading backend: model=OpenMOSS-Team/MOSS-TTS, device=cuda:0, attn=flash_attention_2
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Traceback (most recent call last):
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\utils\hub.py", line 419, in cached_files
    hf_hub_download(
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\huggingface_hub\utils\_validators.py", line 85, in _inner_fn
    validate_repo_id(arg_value)
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\huggingface_hub\utils\_validators.py", line 135, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars, '-', '_' or '.'. The name cannot start or end with '-' or '.' and the maximum length is 96: 'OpenMOSS-Team\MOSS-TTS'.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\QF\Documents\moss-tss-conda\MOSS-TTS\clis\moss_tts_app.py", line 622, in <module>
    main()
  File "C:\Users\QF\Documents\moss-tss-conda\MOSS-TTS\clis\moss_tts_app.py", line 603, in main
    load_backend(
  File "C:\Users\QF\Documents\moss-tss-conda\MOSS-TTS\clis\moss_tts_app.py", line 91, in load_backend
    processor = AutoProcessor.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\models\auto\processing_auto.py", line 392, in from_pretrained
    return processor_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\QF\.cache\huggingface\modules\transformers_modules\OpenMOSS_hyphen_Team\MOSS_hyphen_TTS\0c8df9988ab61071cdb06fe40b2bdc3132ac3b7e\processing_moss_tts.py", line 205, in from_pretrained
    AutoConfig.from_pretrained(
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1356, in from_pretrained
    config_dict, unused_kwargs = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\configuration_utils.py", line 569, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\configuration_utils.py", line 624, in _get_config_dict
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\utils\hub.py", line 276, in cached_file
    file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\transformers\utils\hub.py", line 468, in cached_files
    raise OSError(f"{e}") from e
OSError: Repo id must use alphanumeric chars, '-', '_' or '.'. The name cannot start or end with '-' or '.' and the maximum length is 96: 'OpenMOSS-Team\MOSS-TTS'.

[Quantumatum]

And mine is even slower, 5x slower in fact (3080 10GB VRAM):
Generating bs1 ...: 1%|███▏ | 60/4096 [01:25<1:13:31, 1.09s/it]
Task manager shows my GPU being utilised to 100%, yet its EXTREMELY slow. I'm also running clis/moss_tts_app.py, with this text and a voice .wav attached:

Alice said to herself, “to be going messages for a rabbit! I suppose Dinah’ll be sending me on messages next!” And she began fancying the sort of thing that would happen: “ ‘Miss Alice! Come here directly, and get ready for your walk!’ ‘Coming in a minute, nurse! But I’ve got to see that the mouse doesn’t get out.’ Only I don’t think,” Alice went on, “that they’d let Dinah stop in the house if it began ordering people about like that!”

log:

(moss-tts) PS C:\Users\QF\Documents\moss-tss-conda\MOSS-TTS\clis> python moss_tts_app.py --model_path C:\Users\QF\Documents\MOSS-TTS
[INFO] Using attn_implementation=flash_attention_2
[Startup] Preloading backend: model=C:\Users\QF\Documents\MOSS-TTS, device=cuda:0, attn=flash_attention_2
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1600/1600 [00:01<00:00, 1442.47it/s, Materializing param=quantizer.quantizers.31.out_proj.parametrizations.weight.original1]
`torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 463/463 [00:00<00:00, 1358.49it/s, Materializing param=lm_heads.32.weight]
[Startup] Backend preload finished in 28.74s
C:\Users\QF\Documents\moss-tss-conda\MOSS-TTS\clis\moss_tts_app.py:436: UserWarning: The parameters have been moved from the Blocks constructor to the launch() method in Gradio 6.0: css. Please pass these parameters to launch() instead.
  with gr.Blocks(title="MOSS-TTS Demo", css=custom_css) as demo:
C:\Users\QF\miniconda3\envs\moss-tts\Lib\site-packages\gradio\components\dataframe.py:192: UserWarning: The `col_count` parameter is deprecated and will be removed. Please use `column_count` instead.
  warnings.warn(
* Running on local URL:  http://0.0.0.0:7860
* To create a public link, set `share=True` in `launch()`.
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(None)>
Traceback (most recent call last):
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\QF\miniconda3\envs\moss-tts\Lib\asyncio\proactor_events.py", line 165, in _call_connection_lost
    self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Generating bs1 ...:   0%|▊                                                                                                                                                                                                                         | 15/4096 [00:24<1:15:46,  1.11s/it]

Edit: it actually completed way before the 100% mark.. it seems that progress bar is bugged and isnt accurate

[n3r3d100ka]

I also get at rvq16 - 1.7b local mode with 3090 around 4it/s

[cochese9000]

I have tried the GGUF models and finally got it working with a fork of this repo. I used Gemini 3.1Pro and so I can't offer support, but it's working REALLY well now and I'm getting single sentence generations in ~3s now, instead of ~40s previously. it's also running in 19GB of VRAM instead of 23.6GB. I'm using the Q6k version.

It might be useful for the creators, or anyone else on windows with very slow inference. FWIW, I'm on a 3090 (24GB).

https://github.com/cochese9000/MOSS-TTS