Model trimming, Faster-whisper & more

[EliseWindbloom]

Thank you for creating this wonderful windows port, and i apologize if i'm creating this pull request incorrectly (i'm a newbie to doing pull requests).

I've made a number of changes in hopes of enhancing things and making things faster:

Training Resuming

• Edits to melo/configs/config.json template:
  • I've set skip_optimizer=false by default, which seems critical for resuming training later (if this is set to true, the model will not include the extra data required to resume training, if my understanding is correct).
  • Set train eval_interval=200 by default to make it save the model more often by default
  • Added the internal keep_ckpts=5 to the config.json, this allows you to set how many models it will keep. It will delete models past this set value, starting from the oldest model.

Model Trimming

I've discovered you can trim the model (about 66% smaller filesize on my tests). These created trimmed models should load faster i think too. Note the created files are for inference-only (as it removes the unneeded overhead from the model which seems required for training). Also it's possible this only makes the loading of the model faster and not the actual text-to-speech each time it's used.

• Created trim_models.py and trim_models.bat. Edit the bat file with the path to your models and run the bat file, or run inside conda like this (outputted models with have the word "_trimmed" at the end of them):

python trim_models.py "Path\To\Models\"

• This will ignore .pth files starting with "D_" or "DUR_" so you can just give it your training folder if you want to convert all model files in it.
• Note: Filesize of the trimmed models may vary by kilobyes for each one you create, but this is completely normal (according to Claude AI). The small differences (few KB) between trimmed checkpoints can occur due to, PyTorch's serialization process not being completely deterministic, System-level file compression varying slightly, Different memory alignments during saving, and Slight variations in how Python's pickle protocol handles the data

Audio files to wav (44100Hz sample rate)

• I've added ConvertAudiotoWav.bat which allows you to convert a very wide range of common audio file formats to wav with 44100Hz sample rate (the sample rate recommended in MeloTTS training readme). create a folder named audio, place the audio it and run the bat file to make a folder called wavs with the converted audio.

Faster-Whisper

For far faster transcribing, i've added faster-whisper

• Some background: I've discovered awhile ago that faster-whisper is far faster than the original whisper (and no notiable quality difference). However i was always plagued with cuda/cublas errors that completely made it fail on windows. I figured out the fix more recently though, which is to simply put the cuda dll files into the same folder as the script. Another user has shared them in their project here: https://github.com/Purfview/whisper-standalone-win/releases/download/libs/cuBLAS.and.cuDNN_CUDA11_win_v2.7z
• i've added transcript_fast.py and transcript_fast.bat, to use run transcript_fast.bat which will use the default directories. you can also add directories like this or run it in the conda prompt:

python transcript_fast.py --wavs_path "C:\MeloTTS-Windows\melo\data\temp\wavs" --metadata_path "C:\MeloTTS-Windows\melo\data\temp\metadata.list"

• i've changed it from "base" to "medium" Whisper model since medium tend to be much better at getting the text correct on from my tests. transcript.py and transcript_fast.py both set to medium model. The faster-whisper model is using a nice hybrid int8_float16 compute type, which is a great balance between speed and quality from my tests.

try.py Added

• I've added a simple script that lets you try a melotts voice in a text to speech loop (both build in and custom voices). In the conda environment run python try.py to use it. it will automatically find any voices pth files placed in folder named "custom" (include a config json file with matching name for each one).

Updated Readme

• I've updated the readme to note some of these changes

Extra Notes

• You may want to test out faster-whisper on your pc just in case, but i think everything else should be fine without issue.

• Some info when I tested training, in case it helps anyone (as i have no idea what the right amount of training is). For reference, my pc is runs Windows 10 and has a NVIDIA GTX 1650 with 4GB VRAM and 32GB RAM.

  • Training 5 wav files was quite fast, though i didn't take note of the exact speed.
  • Training 10 wav files (32 seconds total duration), it completed 1 epoch every 14 seconds.
  • Training 22 wav files (1min1seconds total duration), it completed about 1 epoch every minute.
  • As far as I can tell, the number on the models (like G_1000.pth, which would has the number "1000") is the "steps", which may be the same as the epoch count depending on settings or might be different.
  • According to Claude AI, the formula to get MeloTTS epochs from a checkpoint number: Epochs = checkpoint_step_number / (total_training_samples / batch_size). For example, if you have G_2000.pth, 100 wav files, and batch_size=12, then Epochs = 2000 / (100 / 12) = ~241.
  • I am thinking that around 2000 steps might be a good spot to train to (based on from my test), though honestly i'm not sure. I'm planning to do more tests.
  • As you've said in your video, you need way, way more audio than 5 wav files. I'm going to test with 2 minutes duration of audio data to see how it performs. 5 minutes audio duration might be the minimum to get you a usable quality, but would take days to reach 2000 epochs on my pc.
  • There is also a google colab for MeloTTS training that might help if you want to see how good a model you can make, but i haven't tired it.

Also, installing faster-whisper seems to cause some kind of versions conflict, but running this command after installing faster-whisper fixed the errors(i also added this to the readme):

pip install transformers==4.30.2 huggingface_hub==0.16.4