SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis

Xiangyue Zhang*, Jianfang Li*, Jiaxu Zhang, Ziqiang Dang, Jianqiang Ren, Liefeng Bo, Zhigang Tu†

✨ICCV 2025✨

Official implementation and checkpoints for semantic-aware holistic co-speech gesture generation.

_{Keywords: co-speech gesture generation, co-speech motion generation, speech-driven motion generation, semantic gestures, BEAT2, and SMPL-X.}

📣 Updates

• [2026.07.27] 🔥 Added the 25-speaker BEAT2 release: all five motion representation models, Base and Sparse checkpoints, paired inference NPZ files, metrics, and a separate training workflow.
• [2025.09.11] 🔥 Release: - [✔] Inference code - [✔] Training code

💖 Inference Data

The two releases use different training protocols and are provided separately:

• Speaker 2 (paper protocol): download the original test .npz files from Hugging Face or Google Drive.
• All Speakers (25 English BEAT2 speakers): download the paired test .npz files from Hugging Face or Google Drive.

📊 Results

Training protocol	Speakers	FGD ↓ (×10-1)	BC ↑ (×10-1)	DIV ↑	MSE ↓ (×10-8)	LVD ↓ (×10-5)
Speaker 2 (paper)	1	4.278	7.770	12.910	6.153	6.938
All Speakers (released checkpoint)	25	3.556	5.097	8.409	4.439	1.435

The Speaker 2 row is copied from Table 1 of the paper. The All-Speaker row uses the lowest-FGD Sparse checkpoint selected by a coarse 10-epoch search followed by a local 1-epoch search. Raw metrics and checkpoint metadata are in results/all_speakers. The two rows use different training protocols and should not be interpreted as a controlled single-speaker versus multi-speaker ablation.

⚡ Quick Start

The shortest path to a runnable setup is:

conda create -n semtalk python=3.8 -y
conda activate semtalk
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt
pip install -U whisperx

If your data and weights are already ready, inference becomes:

python train.py --config configs/semtalk_sparse.yaml \
  --inference \
  --audio_infer_path demo/2_scott_0_1_1_test.wav

⚒️ Environment

Build Environment

We Recommend a python version =3.8 and cuda version >=12.1. Then build environment as follows:

# [Optional] Create a virtual env
conda create -n SemTalk python=3.8
conda activate SemTalk
# pytorch, torchvison
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
# Install with pip:
pip install -r requirements.txt
pip install -U whisperx
sudo apt-get update && sudo apt-get install -y ffmpeg  # if you don't have ffmpeg

Download Data

please refer to EMAGE and download datasets from BEAT2 for datasets and place the dataset inside the SemTalk folder, i.e. path-to-SemTalk/BEAT2

Download Weights

Download hubert-large-ls960-ft(used for extracting audio features), faster-whisper-large-v3(used for inference) weights and place itside in the SemTalk folder, i.e. path-to-SemTalk/facebook. If you are in China, you can use hf-mirror for faster and more reliable downloads. The process may take some time, so please be patient.

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download facebook/hubert-large-ls960-ft --local-dir facebook/hubert-large-ls960-ft

huggingface-cli download --resume-download Systran/faster-whisper-large-v3 --local-dir Systran/faster-whisper-large-v3

Choose the weights that match the intended protocol:

• Speaker 2 (paper protocol): download the original pretrained models and weights from Hugging Face (Google Drive mirror).
• All Speakers (25 English BEAT2 speakers): download the all-speaker representation models and weights from Hugging Face (Google Drive mirror). This archive contains all five representation models and the selected Base and Sparse checkpoints.

The Hugging Face archives exclude SMPLX_NEUTRAL_2020.npz. Download SMPL-X from the official website under its own license and place it in the path expected by the repository.

Keep the original Speaker 2 archive in its documented locations. Install the All-Speaker archive without overwriting those files:

unzip SemTalk_all_speakers_weights_25spk.zip
cp -R SemTalk_all_speakers_weights/weights/all_speakers ./weights/

The All-Speaker files use a separate subdirectory, so the original Speaker 2 paths and default configs remain unchanged.

Finally, these SemTalk folder should be orgnized as follows:

.
├── BEAT2
│   └── beat_english_v2.0.0
├── configs
├── dataloaders
├── datasets   (this folder will appear after you generate the datasets)
│   ├── beat2_cache2
│   ├── beat2_semtalk_train
│   └── semtalk_dataloader.py
├── facebook
│   └── hubert-large-ls960-ft
├── models
├── optimizers
├── src
├── Systran
│   └── faster-whisper-large-v3
├── utils
├── weights
│   ├── pretrained_vq
│   ├── smplx_models
│   ├── best_semtalk_base.bin
│   ├── best_semtalk_sparse.bin
│   └── all_speakers
│       ├── pretrained_vq
│       ├── best_semtalk_base.bin
│       └── best_semtalk_sparse.bin
├── ae_trainer.py
├── aelower_trainer.py
├── aelowerfoot_trainer.py
├── requirements.txt
├── semtalk_base_trainer.py
├── semtalk_sparse_trainer.py
└── train.py

Generate Dataset

Notice: Please make sure you are in the root directory, i.e. path-to-SemTalk.

1. To generate the training dataset, run:

python dataloaders/save_train_dataset.py

This process may take some time, so please be patient.

2. To generate the test dataset, run:

python dataloaders/save_test_dataset.py

🚀Training, Testing, and Inference

Training of SemTalk

The commands below preserve the original Speaker 2 protocol. For the separate 25-speaker preprocessing, representation-model training, Base training, Sparse training, and evaluation commands, see scripts/all_speakers/README.md.

Train RVQ-VAE

The commands below reproduce the original Speaker 2 RVQ-VAE protocol. You can either train the models yourself and place the weights under path-to-SemTalk/weights, or download the pretrained weights that match your training protocol:

• Speaker 2 (paper protocol): use the original pretrained weights.
• All Speakers (25 English BEAT2 speakers): use the all-speaker weights. The archive contains the five all-speaker representation models under weights/all_speakers/pretrained_vq, together with the selected Base and Sparse checkpoints. Follow scripts/all_speakers/README.md when training or evaluating this protocol.

python train.py --config configs/cnn_vqvae_face_30.yaml --train_rvq # face

python train.py --config configs/cnn_vqvae_hands_30.yaml --train_rvq # hands

python train.py --config configs/cnn_vqvae_upper_30.yaml --train_rvq # upper body

python train.py --config configs/cnn_vqvae_lower_foot_30.yaml --train_rvq # lower foot

python train.py --config configs/cnn_vqvae_lower_30.yaml --train_rvq # lower body

Stage1: Base Motion Generation

python train.py --config configs/semtalk_base.yaml

Notice: Once you have obtained the optimal base motion generation weights, please update the path field to base_ckpt in configs/semtalk_sparse.yaml.

Stage2: Sparse Motion Generation

python train.py --config configs/semtalk_sparse.yaml

Testing of SemTalk

Notice: Before running the test code, make sure the load_ckpt and base_ckpt paths in configs/semtalk_sparse.yaml are set correctly.

python train.py --config configs/semtalk_sparse.yaml --test_state

Inference

you can put your inference wav format aduio on ./demo path, for example, you can run:

python train.py --config configs/semtalk_sparse.yaml --inference --audio_infer_path ./demo/2_scott_0_1_1.wav

🗂️ Repository Layout

If you are opening the codebase for the first time, these folders are the important ones:

• configs/: experiment and path configuration
• dataloaders/: dataset preprocessing and dataset loading
• models/: RVQ-VAE and SemTalk model definitions
• train.py: original training entrypoint

The original research code structure is still preserved. Common repository paths are now centralized in utils/project_paths.py, so preprocessing, training, and checkpoints use the same path conventions. This also covers the local HuBERT / faster-whisper directories and shared vocab.pkl lookup, which reduces machine-specific path edits when moving the repo.

📺 Visualize

Following EMAGE, you can download SMPLX blender addon, and install it in your blender 3.x or 4.x. Click the button Add Animation to visualize the generated smplx file (like xxx.npz).

🙏 Acknowledgments

Thanks to EMAGE, DiffSHEG, our code is partially borrowing from them. Please check these useful repos.

📖 Citation

If you find our code or paper helps, please consider citing:

@inproceedings{zhang2025semtalk,
  title={SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Dang, Ziqiang and Ren, Jianqiang and Bo, Liefeng and Tu, Zhigang},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={13761--13771},
  year={2025}
}