Skip to content

feat(speak): Flux TTS (Speak v2 WebSocket streaming)#86

Open
GregHolmes wants to merge 3 commits into
mainfrom
gh/flux-tts-speak
Open

feat(speak): Flux TTS (Speak v2 WebSocket streaming)#86
GregHolmes wants to merge 3 commits into
mainfrom
gh/flux-tts-speak

Conversation

@GregHolmes

Copy link
Copy Markdown

Flux TTS in dg speak (Speak v2, WebSocket streaming)

Adds Flux TTS to the speak command, websocket-first per the Flux TTS CLI ticket.

What's in it

  • flux-* models → Speak v2 WebSocket streaming (speak.v2.connect); aura-* stay on Speak v1 batch REST. Raw linear16 output is wrapped in WAV so it's directly playable. New client.speak_text_stream() helper + an --encoding guard (Flux streaming is linear16/mulaw/alaw only).
  • Bug fixes (surfaced by non-prod base URLs):
    • _create_client environment now derives https (REST) + wss (WebSocket) from the host and includes agent_rest — the old builder crashed on any custom base URL.
    • verify_api_key validates against the configured base URL instead of hardcoded production, so staging/non-prod keys pass the auth guard.
  • New global options: --base-url and --api-key (the explicit-credential path already existed; nothing fed it).
  • Docs: Flux (v1 vs v2) documented in speak --help/examples/agent_help, the skills Developer Guide (also migrated off the removed pre-Fern SpeakOptions API), and the README.

Verification

  • pytest: 967 passing (incl. new Flux speak tests).
  • ruff clean on changed files.
  • Live-tested against staging: dg speak -m flux-alexis-en → valid WAV; plus aura speak, listen, read, models, and manage commands.

⚠️ Restrictions — do NOT merge/release yet

  1. Depends on an unreleased SDK. This pins deepgram-sdk>=7.5.0, which is not yet published — it's the pending Python SDK release-please PR chore(main): release 7.5.0 deepgram-python-sdk#743 (chore(main): release 7.5.0), which contains the Speak v2 streaming client this uses. This PR must not merge/release until deepgram-sdk 7.5.0 is live on PyPI.
  2. uv sync will fail until 7.5.0 ships (can't resolve >=7.5.0). The lockfile needs regenerating once 7.5.0 is published. Local testing so far used an editable install of the SDK checkout (not part of this PR).
  3. EA scope — WebSocket only for Flux. No Flux REST/batch in the CLI (per the ticket).
  4. No Flux voice listing. dg models does not list Flux voices — the models service doesn't expose them yet (GA/API-side), so it's intentionally omitted.

Route flux-* models through the Speak v2 WebSocket streaming client
(speak.v2.connect); aura-* stay on Speak v1 batch REST. Raw linear16
output is wrapped in WAV so it is directly playable. Adds a
client.speak_text_stream() helper and an --encoding guard (Flux
streaming is linear16/mulaw/alaw only).

Also:
- Fix _create_client environment: derive https (REST) + wss (WebSocket)
  from the configured host and include agent_rest (required by the SDK).
  The old builder crashed on any custom base URL.
- Fix verify_api_key to validate against the configured base URL instead
  of hardcoded production, so non-prod (staging) keys pass the auth guard.
- Add global --base-url and --api-key options (the explicit-credential
  path already existed but nothing fed it).
- Document Flux (v1 vs v2) in speak --help/examples/agent_help, the
  skills Developer Guide (migrated off the removed pre-Fern SpeakOptions
  API), and the README.
- Bump deepgram-sdk to >=7.5.0 (adds the Speak v2 streaming client).

Tests: add Flux speak coverage; update the custom base-url client test
for the new environment derivation.
@GregHolmes

Copy link
Copy Markdown
Author

@dg-coreylweathers all these tests fail because there is no 7.5.0 for Python SDK yet.

… pipe

Address PR review (B2/S1/S2) and complete the "streaming by default"
acceptance criterion for Flux TTS.

- B2: guard empty Flux streams — raise instead of writing a header-only
  WAV and reporting success (file and pipe paths). Confirmed the SDK
  iterator yields audio as raw bytes, so the isinstance filter is correct.
- S1: Flux error paths (encoding guard, streaming failure, no-audio) now
  raise click.ClickException, so they exit non-zero and are visible in
  every output format instead of returning an error result and exiting 0.
- S2: preserve the base-url transport via _split_base_url (http->ws,
  else https/wss) in client.py and auth.py, so a plaintext local/staging
  endpoint isn't forced onto TLS. agent_rest confirmed required on the SDK
  environment.
- Streaming: the pipe path now writes each chunk to stdout as it arrives
  (streaming WAV header on the first frame, flush per chunk) so playback
  starts on the first frame; the file path stays buffered because a WAV
  needs its data length up front. The streaming header uses 0x7FFFFFFF so
  ffmpeg reads it without the "may be invalid" warning while macOS
  CoreAudio still estimates duration for redirect-to-file.
- Progress: scrolling stderr output (time-to-first-audio + throttled
  "received N" lines + final summary), TTY-only so agent/CI stays clean.
- Docs/examples show the `ffplay -loglevel error` clean-demo form.

Tests: add empty-audio, streaming-failure, stdout-streaming, progress, and
http-scheme coverage.
@GregHolmes

Copy link
Copy Markdown
Author

@dg-coreylweathers thanks for the detailed review — all three findings are addressed in 32c2272, and I verified the two SDK-coupled questions against the local 7.5.0 checkout so we're not guessing.

Your questions

  • B2 — what does speak.v2.connect(...) yield for an audio frame? Raw bytes. The sync iterator (speak/v2/socket_client.py V2SocketClient.__iter__) yields message directly when isinstance(message, bytes) and otherwise a parsed V2SocketClientResponse (control messages). So the CLI's isinstance(message, bytes) filter is correct and the "empty on 100% of calls" scenario doesn't apply. I still added the empty-audio guard as a real safety net.
  • S2 — is agent_rest real/required on DeepgramClientEnvironment in 7.5.0? Yes — environment.py __init__(self, *, base, production, agent, agent_rest), all four required, and the built-in env sets all four. Also confirmed speak.v2.connect builds its WS URL as get_environment().production + "/v2/speak", which is why the scheme handling matters.

What changed

  • B2 — empty Flux stream now raises instead of writing a header-only WAV and reporting success (file and pipe paths).
  • S1 — the Flux error paths (encoding guard, streaming failure, no-audio) now raise click.ClickException, so they exit non-zero and are visible in every output format instead of returning an error result and exiting 0. The encoding guard is no longer silent in default mode.
  • S2_split_base_url preserves the input transport (httpws, else https/wss) in both client.py and auth.py, so a plaintext local/staging endpoint isn't forced onto TLS. Verified live against staging.

Plus the "streaming by default" acceptance criterion

The pipe path now writes each chunk to stdout as it arrives (streaming WAV header on the first frame, flush per chunk), so | ffplay - starts playing on the first frame instead of after the whole utterance. The file path stays buffered because a WAV needs its data length up front. There's also a scrolling stderr progress readout (time-to-first-audio + received N lines), TTY-only so agent/CI output stays clean.

One note on the pipe: an unknown-length streaming WAV makes ffmpeg print a cosmetic Packet corrupt at EOF (verified zero audio lost). I used a 0x7FFFFFFF length marker to drop the scarier "file may be invalid" line while keeping redirect-to-file correct on macOS CoreAudio, and the docs/examples show the ffplay -loglevel error clean-demo form.

Still do-not-merge until deepgram-sdk 7.5.0 is live on PyPI — the tests only pass right now against a local editable SDK checkout.

…udio

In agentic/CI mode (CI, CLAUDECODE, --agent-friendly, --non-interactive) or
with --output json, setup_output switches the default format to json and
output_result serializes the returned result via the stdout console. On the
pipe paths that stdout already carries the raw audio, so the JSON summary was
appended to the audio stream — e.g. `CLAUDECODE=1 dg speak ... > out.wav`
ended with trailing JSON, breaking the redirect-to-file guarantee.

Return None from both the Flux and Aura stdout branches so the framework
skips output_result (it guards on `result is not None`); the one-line summary
already goes to the stderr console. The -o file paths keep returning
SpeakResult since their stdout is free.

Tests updated to assert the pipe paths return None.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant