Model trimming, Faster-whisper & more

README.md

@@ -26,25 +26,46 @@ If you have trouble doing the download with the `python -m unidic download` you
 pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
 ```
 
+3. Prepare faster-whisper (optional for fast transcribing of audio files):
+   - Download cuda/cublas here [https://github.com/Purfview/whisper-standalone-win/releases/download/libs/cuBLAS.and.cuDNN_CUDA11_win_v2.7z](https://github.com/Purfview/whisper-standalone-win/releases/download/libs/cuBLAS.and.cuDNN_CUDA11_win_v2.7z), extract and place the 5 dll files directly into the `MeloTTS-Windows/melo/` folder
+   - To install faster-whisper (and prevent conflicts with it) run this from the conda window:
+```
+pip install faster-whisper==0.9.0
+pip install transformers==4.30.2 huggingface_hub==0.16.4
+```
+
 4. Run using:
 ```
 melo-ui
 ```
 
 # Local Training on Windows
+## Preparing Dataset
 1. In the `melo/data/example` folder, delete the example `metadata.list` file.
-2. If you need to convert mp3 to wav, create a folder called `mp3s` in the example folder and copy all your mp3 files into the `mp3s` folder
-3. With a conda window activated with the enviroment open in the `melo` folder, run `ConvertMp3toWav.bat` from the conda prompt. This will create a folder `data/example/wavs` with all of the converted wav files.
-4. Create a transcript file by running `python transcript.py` which will create a `data/example/metadata.list` file.
-5. Run `python preprocess_text.py --metadata data/example/metadata.list` to create the `train.list`, `config.json`, among other files in the `data/example` folder.
-6. Modify `config.json` to change the batch size, epochs, learning rate, etc.
-7. From the conda prompt run `train.bat` to start the training.
-8. File will be created within the `data/example/config` folder with the checkpoints and other logging information.
-9. To test out a checkpoint, run: `python infer.py --text "this is a test" -m "C:\ai\MeloTTS-Windows\melo\data\example\config\G_0.pth" -o output` changing the G_0 to the checkpoint you want to test with G_1000, G2000, etc.
-10. When you want to use a checkpoint from the UI, create a `melo/custom` folder and copy the .pth and `config.json` file over from the `data/example/config`, rename the .pth to a user-friendly name, and launch the UI to see it in the custom voice dropdown.
-11. To see the tensorboard, install `pip install tensorflow`
-12. Run `tensorboard --logdir=data\example\config`
-13. This will give you the local URL to view the tensorboard.
+2. MeloTTS expects wav audio files (with a sample rate of 44100Hz). If you need to convert audio to wav format (with 44100Hz sample rate), create a folder called `audio` in the example folder and copy all your audio files into the `audio` folder
+4. With a conda window activated with the enviroment open in the `melo` folder, run `ConvertAudiotoWav.bat` from the conda prompt. This will create a folder `data/example/wavs` with all of the converted wav files.
+5. Create a transcript file by running `transcript_fast.bat` which will create a `data/example/metadata.list` file using faster-whisper. Alternately, you can run `python transcript.py` to use the original whisper.
+6. Run `python preprocess_text.py --metadata data/example/metadata.list` to create the `train.list`, `config.json`, among other files in the `data/example` folder.
+7. Modify `config.json` to change the batch size, epochs, learning rate, etc.
+  - ⚠️ **Important, If you plan to Resume Training Later:**
+    - The `eval_interval` setting determines how frequently your model is saved during training
+    - For example, if `eval_interval=1000`, the model saves only once every 1000 steps
+    - If you stop training between save points, any progress since the last save will be lost
+    - For safer training sessions that you may need to resume later, use a smaller `eval_interval` value
+    - You can also adjust `n_ckpts_to_keep` to limit the max models kept (if `n_ckpts_to_keep=5`, it will delete the oldest models when their are more than 5 saved models)
+## Start Training
+1. From the conda prompt run `train.bat` to start the training.
+2. File will be created within the `data/example/config` folder with the checkpoints and other logging information.
+3. To test out a checkpoint, run: `python infer.py --text "this is a test" -m "C:\ai\MeloTTS-Windows\melo\data\example\config\G_0.pth" -o output` changing the G_0 to the checkpoint you want to test with G_1000, G2000, etc.
+4. When you want to use a checkpoint from the UI, create a `melo/custom` folder and copy the .pth and `config.json` file over from the `data/example/config`, rename the .pth to a user-friendly name, and launch the UI to see it in the custom voice dropdown.
+5. To see the tensorboard, install `pip install tensorflow`
+6. Run `tensorboard --logdir=data\example\config`
+7. This will give you the local URL to view the tensorboard.
+## Resuming Training
+1. From the conda prompt run `train.bat` again to resume the training. The training will resume from the newest G_XXXX.pth file.
+## Trimming Model
+You can trim your model to make it a way smaller filesize (which will make it load faster during the model loading process). When testing, this made the model filesize about 66% smaller. Note the created trimmed model is for inference-only(using the model just to generate audio from text) and you won't be able to train it further.
+1. Open `trim_models.bat` file in a text editor to change the directory to your G_XXXX.pth files and the save location, save the changes, then run `trim_models.bat` to create a trimmed model for inference only.
 
 # Original Readme:
 <div align="center">

melo/ConvertAudiotoWav.bat

@@ -0,0 +1,46 @@
+@echo off
+setlocal enabledelayedexpansion
+
+rem Specify input and output folders here
+set "INPUT_FOLDER=data\example\audio"
+set "OUTPUT_FOLDER=data\example\wavs"
+
+rem Create output folder if it doesn't exist
+if not exist "%OUTPUT_FOLDER%" mkdir "%OUTPUT_FOLDER%"
+
+rem Initialize counter for sequential naming
+set /a counter=1
+
+rem Loop through common audio formats that ffmpeg supports
+for %%F in (
+    "%INPUT_FOLDER%\*.mp3" 
+    "%INPUT_FOLDER%\*.m4a" 
+    "%INPUT_FOLDER%\*.wav" 
+    "%INPUT_FOLDER%\*.ogg" 
+    "%INPUT_FOLDER%\*.flac" 
+    "%INPUT_FOLDER%\*.aac" 
+    "%INPUT_FOLDER%\*.wma"
+    "%INPUT_FOLDER%\*.aiff"
+    "%INPUT_FOLDER%\*.aifc"
+    "%INPUT_FOLDER%\*.opus"
+    "%INPUT_FOLDER%\*.ape"
+    "%INPUT_FOLDER%\*.wv"
+    "%INPUT_FOLDER%\*.m4b"
+    "%INPUT_FOLDER%\*.mp2"
+    "%INPUT_FOLDER%\*.mp4"
+    "%INPUT_FOLDER%\*.mpc"
+    "%INPUT_FOLDER%\*.mka"
+    "%INPUT_FOLDER%\*.ac3"
+    "%INPUT_FOLDER%\*.dts"
+    "%INPUT_FOLDER%\*.amr"
+    "%INPUT_FOLDER%\*.au"
+    "%INPUT_FOLDER%\*.mid"
+) do (
+    rem Convert the file using ffmpeg
+    ffmpeg -i "%%F" -acodec pcm_s16le -ar 44100 "%OUTPUT_FOLDER%\!counter!.wav"
+
+    rem Increment the counter
+    set /a counter+=1
+)
+
+echo Conversion complete. Check the '%OUTPUT_FOLDER%' folder for the converted files.

melo/configs/config.json

@@ -1,7 +1,7 @@
 {
   "train": {
     "log_interval": 200,
-    "eval_interval": 1000,
+    "eval_interval": 200,
     "seed": 52,
     "epochs": 10000,
     "learning_rate": 0.0003,
@@ -10,15 +10,16 @@
       0.99
     ],
     "eps": 1e-09,
-    "batch_size": 6,
+    "batch_size": 12,
     "fp16_run": false,
     "lr_decay": 0.999875,
     "segment_size": 16384,
     "init_lr_ratio": 1,
     "warmup_epochs": 0,
     "c_mel": 45,
     "c_kl": 1.0,
-    "skip_optimizer": true
+    "skip_optimizer": false,
+    "keep_ckpts": 5
   },
   "data": {
     "training_files": "",

melo/preprocess_text.py

@@ -155,4 +155,4 @@ def main(
     logger.info("Preprocessing completed successfully")
 
 if __name__ == "__main__":
-    main()
+    main()

melo/test_gpu_cpu_switching.py

@@ -0,0 +1,187 @@
+import torch
+import gc
+from melo.api import TTS
+import os
+import melo.text.english_bert as bert
+import psutil
+import GPUtil
+
+def get_memory_usage():
+    """Get current RAM and VRAM usage"""
+    ram = psutil.virtual_memory()
+    ram_usage = ram.used / (1024 ** 3)  # Convert to GB
+
+    gpu_memory = None
+    if torch.cuda.is_available():
+        try:
+            gpus = GPUtil.getGPUs()
+            if gpus:
+                gpu = gpus[0]  # Get first GPU
+                gpu_memory = {
+                    'used': gpu.memoryUsed,  # MB
+                    'total': gpu.memoryTotal  # MB
+                }
+        except Exception as e:
+            print(f"Error getting GPU stats: {e}")
+
+    return ram_usage, gpu_memory
+
+def clear_gpu_memory():
+    """Clear both GPU and RAM memory more aggressively"""
+    # First clear CUDA memory
+    if torch.cuda.is_available():
+        for obj in gc.get_objects():
+            try:
+                if torch.is_tensor(obj):
+                    if obj.is_cuda:
+                        del obj
+            except Exception:
+                pass
+        torch.cuda.synchronize()
+        torch.cuda.empty_cache()
+
+    # Force Python garbage collection
+    gc.collect()
+
+    # Optional: Force more aggressive garbage collection
+    for _ in range(2):
+        gc.collect()
+
+    # Print memory stats to verify clearing
+    ram = psutil.virtual_memory()
+    print(f"\nAfter clearing - RAM Usage: {ram.used / (1024 ** 3):.2f} GB")
+    if torch.cuda.is_available():
+        gpus = GPUtil.getGPUs()
+        if gpus:
+            gpu = gpus[0]
+            print(f"After clearing - VRAM Usage: {gpu.memoryUsed:.2f} MB / {gpu.memoryTotal:.2f} MB")
+
+def reset_bert_model(device):
+    """Reset the global BERT model in english_bert.py only when switching devices"""
+    current_device = None
+
+    # Print memory usage before reset
+    ram_before, vram_before = get_memory_usage()
+    print(f"\nMemory before BERT reset:")
+    print(f"RAM Usage: {ram_before:.2f} GB")
+    if vram_before:
+        print(f"VRAM Usage: {vram_before['used']:.2f} MB / {vram_before['total']:.2f} MB")
+
+    # Check current device of BERT model if it exists
+    if bert.model is not None:
+        current_device = next(bert.model.parameters()).device.type
+
+        # If the model is already on the correct device, return early
+        if (current_device == 'cuda' and device == 'cuda') or \
+           (current_device == 'cpu' and device == 'cpu'):
+            return
+
+        # Explicitly move model to CPU before deletion if it's on CUDA
+        if current_device == 'cuda':
+            bert.model.cpu()
+
+        # Delete the model and clear memory
+        del bert.model
+        clear_gpu_memory()
+        bert.model = None
+
+    # Create proper dummy input for BERT initialization
+    dummy_text = "Hello world"
+    tokens = bert.tokenizer(dummy_text, return_tensors="pt")
+    word2ph = [1] * tokens["input_ids"].shape[1]
+    _ = bert.get_bert_feature(dummy_text, word2ph, device)
+
+    # Print memory usage after reset
+    ram_after, vram_after = get_memory_usage()
+    print(f"\nMemory after BERT reset:")
+    print(f"RAM Usage: {ram_after:.2f} GB")
+    if vram_after:
+        print(f"VRAM Usage: {vram_after['used']:.2f} MB / {vram_after['total']:.2f} MB")
+
+    print(f"BERT model reloaded: {current_device} -> {device}")
+
+import time  # Add this to your imports
+
+def generate_speech(model, text, spk_id, device, output_path):
+    """Generate speech with specified device"""
+    try:
+        # Print memory usage before model movement
+        ram_before, vram_before = get_memory_usage()
+        print(f"\nMemory before moving model to {device}:")
+        print(f"RAM Usage: {ram_before:.2f} GB")
+        if vram_before:
+            print(f"VRAM Usage: {vram_before['used']:.2f} MB / {vram_before['total']:.2f} MB")
+
+        # If moving from CUDA to CPU, first move model to CPU then clear CUDA memory
+        if device == "cpu" and next(model.parameters()).is_cuda:
+            model.cpu()
+            clear_gpu_memory()
+        else:
+            model = model.to(device)
+
+        reset_bert_model(device)
+        spk_id = torch.tensor([spk_id], device=device)
+
+        # Print memory usage after model movement
+        ram_after, vram_after = get_memory_usage()
+        print(f"\nMemory after moving model to {device}:")
+        print(f"RAM Usage: {ram_after:.2f} GB")
+        if vram_after:
+            print(f"VRAM Usage: {vram_after['used']:.2f} MB / {vram_after['total']:.2f} MB")
+
+        # Add timing measurement
+        start_time = time.time()
+        model.tts_to_file(text, spk_id, output_path)
+        end_time = time.time()
+        conversion_time = end_time - start_time
+        print(f"\n----Text-to-speech conversion time on {device}: {conversion_time:.2f} seconds")
+
+    finally:
+        clear_gpu_memory()
+
+
+def alternate_gpu_cpu_inference(ckpt_path, text, language="EN", output_dir="outputs"):
+    """Alternate between GPU and CPU inference"""
+    os.makedirs(output_dir, exist_ok=True)
+
+    # Initialize model
+    config_path = os.path.join(os.path.dirname(ckpt_path), 'config.json')
+
+    # Sequence of devices to test
+    devices = ["cuda", "cpu", "cuda", "cpu"]
+    current_model = None
+    current_device = None
+
+    for i, device in enumerate(devices):
+        # Skip if CUDA not available for GPU inference
+        if device == "cuda" and not torch.cuda.is_available():
+            print(f"CUDA not available, skipping GPU inference {i+1}")
+            continue
+
+        print(f"\nGenerating speech using {device.upper()} - Round {i+1}")
+
+        # Only create new model instance if switching devices or first run
+        if current_model is None or current_device != device:
+            if current_model is not None:
+                del current_model
+                clear_gpu_memory()
+
+            current_model = TTS(language=language, config_path=config_path, 
+                              ckpt_path=ckpt_path, device=device)
+            current_device = device
+
+        # Get first speaker ID
+        spk_id = list(current_model.hps.data.spk2id.values())[0]
+
+        output_path = os.path.join(output_dir, f"output_{i+1}_{device}.wav")
+
+        # Generate speech
+        generate_speech(current_model, text, spk_id, device, output_path)
+        print(f"Generated: {output_path}")
+
+if __name__ == "__main__":
+    # Example usage
+    ckpt_path = "C:/Users/lyria/Documents/Scripts/python/melotts/melotts/MeloTTS-Windows/melo/data/example_fused3voices_7wavs/output_3voices_7wavs/G_2000.pth"
+    text = "This is a test of alternating between GPU and CPU inference."
+
+    alternate_gpu_cpu_inference(ckpt_path, text)