Skip to content

SETUP.md

Latest commit

 

History

History
166 lines (116 loc) · 4.69 KB

SETUP.md

File metadata and controls

166 lines (116 loc) · 4.69 KB
Raw

MuseTalk Local Setup Guide (Windows)

System Requirements

Component Minimum Tested On
GPU NVIDIA 4GB VRAM RTX 2000 Ada 8GB
CUDA Driver 11.7+ 12.8
RAM 8GB+ 16GB
Disk Space ~20GB G: drive
OS Windows 10/11 Windows 11

Step 1 — Prerequisites

  • Anaconda installed (used from G:\anaconda)
  • Git installed
  • Python 3.10 — do NOT use 3.11+ (MMlab packages require 3.10)

Step 2 — Create Conda Environment

G:\anaconda\Scripts\conda.exe create -n musetalk python=3.10 -y

Step 3 — Clone the Repository

cd G:\
git clone https://github.com/TMElyralab/MuseTalk.git
cd MuseTalk

Step 4 — Install PyTorch (CUDA 11.8 build)

Use the CUDA 11.8 build — it works fine with CUDA 12.x drivers.

G:\anaconda\envs\musetalk\Scripts\pip.exe install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

Step 5 — Install FFmpeg

Download from BtbN GitHub Releases:

  • Download ffmpeg-master-latest-win64-gpl.zip
  • Extract to G:\ffmpeg
  • Add G:\ffmpeg\bin to your system PATH

Step 6 — Install Dependencies

G:\anaconda\envs\musetalk\Scripts\pip.exe install -r requirements.txt

Step 7 — Install MMlab Packages

G:\anaconda\envs\musetalk\Scripts\pip.exe install mmengine

G:\anaconda\envs\musetalk\Scripts\pip.exe install mmcv==2.0.1 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html

G:\anaconda\envs\musetalk\Scripts\pip.exe install chumpy --no-build-isolation

G:\anaconda\envs\musetalk\Scripts\pip.exe install mmdet==3.1.0 mmpose==1.1.0

Step 8 — Download Model Weights

# Upgrade huggingface_hub temporarily for download
G:\anaconda\envs\musetalk\Scripts\pip.exe install -U "huggingface_hub[hf_xet]"

# Set mirror for faster downloads (optional)
set HF_ENDPOINT=https://hf-mirror.com

# Download all weights
G:\anaconda\envs\musetalk\Scripts\hf.exe download TMElyralab/MuseTalk --local-dir models
G:\anaconda\envs\musetalk\Scripts\hf.exe download stabilityai/sd-vae-ft-mse config.json diffusion_pytorch_model.bin --local-dir models/sd-vae
G:\anaconda\envs\musetalk\Scripts\hf.exe download openai/whisper-tiny --local-dir models/whisper
G:\anaconda\envs\musetalk\Scripts\hf.exe download yzd-v/DWPose dw-ll_ucoco_384.pth --local-dir models/dwpose
G:\anaconda\envs\musetalk\Scripts\hf.exe download ByteDance/LatentSync latentsync_syncnet.pt --local-dir models/syncnet
G:\anaconda\envs\musetalk\Scripts\hf.exe download ManyOtherFunctions/face-parse-bisent 79999_iter.pth resnet18-5c106cde.pth --local-dir models/face-parse-bisent

# Downgrade huggingface_hub back (required by transformers==4.39.2)
G:\anaconda\envs\musetalk\Scripts\pip.exe install huggingface_hub==0.30.2

Step 9 — Fix Missing Files & Windows Bugs

9a — Create missing config.json

copy models\musetalk\musetalk.json models\musetalk\config.json

9b — Fix s3fd face detector (adrianbulat.com is very slow — use HuggingFace instead)

# Run this once in Python
from huggingface_hub import hf_hub_download
import shutil
path = hf_hub_download(repo_id='camenduru/facexlib', filename='s3fd-619a316812.pth', cache_dir='G:/tmp_hf_cache')
shutil.copy(path, r'G:\MuseTalk\musetalk\utils\face_detection\detection\sfd\s3fd.pth')

9c — Fix Windows character encoding error in preprocessing.py

In musetalk/utils/preprocessing.py lines 80 and 135, replace Japanese brackets:

# Change this:
f"Total frame:「{len(frames)}」 ..."
# To this:
f"Total frame:[{len(frames)}] ..."

Step 10 — Run Inference (CLI)

cd G:\MuseTalk
G:\anaconda\envs\musetalk\python.exe -m scripts.inference --inference_config configs\inference\test.yaml --ffmpeg_path "G:\\ffmpeg\\bin"

Edit configs/inference/test.yaml to set your video and audio paths:

task_0:
  video_path: "data/video/your_video.mp4"
  audio_path: "data/audio/your_audio.wav"

Step 11 — Run Web Interface (Gradio)

cd G:\MuseTalk
G:\anaconda\envs\musetalk\python.exe app.py --ffmpeg_path "G:\\ffmpeg\\bin" --use_float16

Open browser at: http://127.0.0.1:7860

--use_float16 is recommended — reduces VRAM usage and speeds up inference.

Notes

  • Audio must be .wav — convert with: ffmpeg -i input.mp3 output.wav
  • Short input videos work fine — MuseTalk loops them to match audio length
  • First run is slow — face landmark extraction is cached after the first run on a video
  • bbox_shift — adjust if lip sync looks off (negative = move mouth box up, positive = down)