Implement Multiturn support for EasyMagpie by Edresson · Pull Request #15764 · NVIDIA-NeMo/Speech

Edresson · 2026-06-08T12:03:42Z

What does this PR do ?

Implements multiturn support for EasyMagpieTTS, including multiturn Lhotse training/evaluation data handling, user-audio-conditioned inference, multiturn metrics/export support, and related tests.

Collection: TTS

Changelog

Added a new EasyMagpie multiturn Lhotse config: examples/tts/conf/magpietts/easy_magpietts_lhotse_multiturn.yaml.
Added support for restoring EasyMagpie models from a pretrained checkpoint before normal initialization.
Added multiturn Lhotse TTS dataset support for duplex / multi-turn conversational samples.
Added EasyMagpie multiturn user-audio inference mode via --easy_magpie_inference_mode multiturn_user_audio.
Added support for user-audio conditioning, user silence handling, user turn end-token handling, and max evaluation turns during multiturn inference.
Added distributed / multi-GPU inference support for multiturn EasyMagpie evaluation by sharding generation across ranks and merging rank-local outputs for evaluation.
Added grouped turn-level and sample-level metrics export for multiturn evaluation, including JSON/CSV outputs.
Added phoneme prediction export fields to generated .json / .csv evaluation outputs.
Added optional predicted-phoneme inference controls, including phoneme input type, phoneme sampling method, tokenizer override, and text-input dropout.
Added additional evaluation options for ASR / EOU batch sizes, emotion cosine similarity, emotion match rate, and text-annotation stripping for metrics.
Added Lhotse dataloader speaker filtering support.
Added / updated Nemotron-H decoder support needed for prefill in EasyMagpie multiturn inference.
Added unit tests for Lhotse TTS filters, MagpieTTS Lhotse dataset behavior, and EasyMagpieTTS model behavior.

Usage

Example EasyMagpie multiturn inference with a .nemo checkpoint:

# Run from the NeMo repository root.
# Replace paths with your local model, codec, evalset config, and output directory.

python examples/tts/magpietts_inference.py \
  --model_type easy_magpie \
  --easy_magpie_inference_mode multiturn_user_audio \
  --nemo_files /path/to/easy_magpie_model.nemo \
  --datasets_json_path /path/to/evalset_config.json \
  --out_dir /path/to/output \
  --codecmodel_path /path/to/codec_model.nemo \
  --batch_size 1 \
  --max_eval_turns 6 \
  --run_evaluation

copy-pr-bot · 2026-06-08T12:03:46Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Edresson · 2026-06-09T18:05:12Z

FYI, these changes was done to support prefill in one step on nemotron_h class.

shehzeen · 2026-06-22T18:22:12Z

+    return cfg_cls(**kwargs)
+
+
+def _select_runner_cls(args):


Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: paarthneekhara <paarthneekhara@users.noreply.github.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Edresson · 2026-06-24T12:45:25Z

/ok to test dc1d48b

github-actions · 2026-06-24T15:56:14Z

[🤖]: Hi @Edresson 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Edresson · 2026-06-24T17:49:53Z

/ok to test eb6b252

github-actions · 2026-06-24T22:08:55Z

[🤖]: Hi @Edresson 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Edresson · 2026-06-25T11:54:53Z

/ok to test 3b5077f

github-actions · 2026-06-25T16:09:14Z

[🤖]: Hi @Edresson 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

github-actions Bot added TTS common labels Jun 8, 2026

github-advanced-security AI found potential problems Jun 8, 2026

View reviewed changes

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 2600fb8 to 9cb8c01 Compare June 8, 2026 19:08

github-advanced-security AI found potential problems Jun 8, 2026

View reviewed changes

Comment thread examples/tts/magpietts_inference.py Fixed

Comment thread examples/tts/magpietts_inference.py Fixed

Comment thread examples/tts/magpietts_inference.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 9cb8c01 to a73132d Compare June 8, 2026 19:44

github-advanced-security AI found potential problems Jun 9, 2026

View reviewed changes

Comment thread nemo/collections/tts/data/text_to_speech_dataset_lhotse_multiturn.py Fixed

github-actions Bot removed the common label Jun 9, 2026

Edresson commented Jun 9, 2026

View reviewed changes

github-actions Bot added the common label Jun 18, 2026

github-advanced-security AI found potential problems Jun 20, 2026

View reviewed changes

Comment thread nemo/collections/tts/metrics/emotion_encoder.py Fixed

shehzeen reviewed Jun 22, 2026

View reviewed changes

Comment thread examples/tts/magpietts_inference.py

return cfg_cls(**kwargs)

def _select_runner_cls(args):

shehzeen Jun 22, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@blisc

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 2 times, most recently from c564930 to 1a43737 Compare June 23, 2026 15:36

github-advanced-security AI found potential problems Jun 23, 2026

View reviewed changes

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 2 times, most recently from b18c9ea to ba6c288 Compare June 23, 2026 18:58

paarthneekhara and others added 11 commits June 23, 2026 12:02

wip

f7f9aae

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

WIP

6cf3e02

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

speaker encoder optional

ed01ceb

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Apply isort and black reformatting

74cd9cd

Signed-off-by: paarthneekhara <paarthneekhara@users.noreply.github.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

add option to remove text embedding and lm head

382fb95

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

cas encoder layers

3899c7e

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

use IPA as text prob added during training

3711b3a

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add multiturn dataloader

a5e00b3

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Update multiturn dataloader

ca4ca99

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add multiturn config

3b44776

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add a intermediary fix for prior

7048d3c

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix unit tests

dc1d48b

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 930147e to dc1d48b Compare June 24, 2026 12:38

copy-pr-bot Bot temporarily deployed to public June 24, 2026 12:46 Inactive

copy-pr-bot Bot temporarily deployed to test June 24, 2026 12:47 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 12:50 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 13:11 Inactive

Edresson requested a review from shehzeen June 24, 2026 17:33

Add turn level volume normalization

eb6b252

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

copy-pr-bot Bot temporarily deployed to public June 24, 2026 17:50 Inactive

copy-pr-bot Bot temporarily deployed to test June 24, 2026 17:52 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 17:54 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 17:55 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 18:14 Inactive

shehzeen and others added 2 commits June 24, 2026 15:23

remove configurable cas encoder layers since it is not needed

b4aef5e

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

Left silence pad short user audios for safety

3b5077f

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

copy-pr-bot Bot temporarily deployed to public June 25, 2026 11:55 Inactive

copy-pr-bot Bot temporarily deployed to test June 25, 2026 11:57 Inactive

copy-pr-bot Bot temporarily deployed to public June 25, 2026 11:59 Inactive

copy-pr-bot Bot temporarily deployed to public June 25, 2026 12:18 Inactive

Uh oh!

Conversation

Edresson commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

Uh oh!

copy-pr-bot Bot commented Jun 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Edresson Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shehzeen Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Edresson commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Edresson commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Edresson commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Edresson commented Jun 8, 2026 •

edited

Loading