Smart Turn Detection (whisper-tiny)

A tiny, audio-only model that decides whether a speaker has finished their turn or is just pausing — the endpointing decision at the heart of a voice agent.

▶ Live demo · 🤗 Model · 📄 Report

On the official frozen smart-turn-v3.2-test set it beats Smart Turn v3.2 on Hindi (93.9%) and holds 93.7% English, at ~38 ms/clip on CPU from a single ONNX file.

• Input: raw speech (last 8s window, 16 kHz mono). No transcription.
• Output: P(turn complete) ∈ [0, 1].
• Model: trimmed whisper-tiny encoder → attention pool → MLP head. ~7.8M params.
• Data: pipecat-ai/smart-turn-data-v3.2-train, a balanced ~32k English + Hindi only subset (the target languages), filler-aware.

See REPORT.md for the full write-up (data prep, experiments, results).

Architecture

audio (last 8s, 16k) ─► log-mel [80×800]
   ─► whisper-tiny ENCODER (decoder discarded; pos-emb trimmed 1500→400 = 8s)
   ─► frame embeddings [400×384]
   ─► attention pool ─► [384]
   ─► LayerNorm → 384→128 → GELU → drop → 128→1 ─► sigmoid ─► P(complete)

Only the encoder is kept — this is classification, not ASR — and its positional embeddings are trimmed to an 8s window so it runs ~4× cheaper than the full 30s.

Setup

uv sync                     # installs torch, transformers, datasets, gradio, ...

The audio/mel cache is large (~15 GB) and is written to an external SSD via the data/ symlink (override with SMART_TURN_SSD=/path).

Run the pipeline

# 1. Stream + balance ~30k clips into an SSD memmap (~20-40 min, downloads a subset)
uv run python -m smart_turn.data

# 2. Precompute log-mel once (fast training thereafter)
uv run python -m smart_turn.featurize

# 3. Train (two experiments — see REPORT.md)
uv run python -c "from smart_turn.train import train; train(tag='frozen', freeze_encoder=True, unfreeze_top=0, epochs=4)"
uv run python -c "from smart_turn.train import train; train(tag='unfreeze2', freeze_encoder=True, unfreeze_top=2, epochs=4)"

# 4. Evaluate with per-language / filler / real-vs-synthetic slices
uv run python -m smart_turn.eval runs/unfreeze2.pt

Demo

uv run python app.py        # Gradio: record/upload speech → turn verdict

ONNX (single-file deploy)

SMART_TURN_WINDOW_S=8 uv run python -m smart_turn.export_onnx   # -> runs/smart_turn_8s.onnx (shipped)
SMART_TURN_WINDOW_S=4 uv run python -m smart_turn.export_onnx   # -> runs/smart_turn_4s.onnx (fast variant)

The graph takes a raw 16k waveform and returns P(complete) — mel is baked in, so inference needs only onnxruntime + numpy (8s ~38 ms/clip, 4s ~17 ms, CPU 1 thread):

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("runs/smart_turn_8s.onnx", providers=["CPUExecutionProvider"])
x = wav_16k[-128000:]                      # last 8s (float32 mono @ 16k)
buf = np.zeros((1, 128000), np.float32); buf[0, 128000 - len(x):] = x   # right-align
p = float(sess.run(None, {"waveform": buf})[0][0])

Layout

File	Role
smart_turn/config.py	paths, window/model constants
smart_turn/data.py	streaming download, balancing, SSD cache
smart_turn/featurize.py	precompute log-mel memmap
smart_turn/model.py	trimmed whisper encoder + attention pool + head
smart_turn/train.py	training loop (MPS/CPU), two param groups
smart_turn/eval.py	sliced metrics
smart_turn/infer.py	single-clip inference (used by the demo)
app.py	Gradio demo
notebooks/01_explore_train.ipynb	exploration + experiment narrative