feat: Add bring-your-own ElevenLabs voice agent support #258
+1,261
−142
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Create apiVoice.ts client to fetch conversation tokens from server - Update RealtimeVoiceSession.tsx and .web.tsx to use conversationToken instead of hardcoded agentId for self-hosted deployments - Enables users to configure their own ElevenLabs agent 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add settings fields for elevenLabsUseCustomAgent, elevenLabsAgentId, elevenLabsApiKey - Create UI with toggle to enable custom agent and input fields for credentials - Update apiVoice.ts to pass custom credentials to server when enabled - Add translations for all supported languages (en, es, pl, ru, zh-Hans, ca, pt) - Update settings tests with new fields 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add Find Agent and Create/Update Agent buttons to voice settings - Add show/hide toggle for API key field - Add Save Credentials button - Add ElevenLabs agent management API functions (findHappyAgent, createOrUpdateHappyAgent) - System prompt is now embedded in code as single source of truth - Add translations for all new UI elements 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add help link that opens ElevenLabs API keys page - Update description to include required permissions: - "ElevenLabs Agents" (Write) for agent management - "Text to Speech" (Access) for TTS - Add getApiKey translation to all language files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Logs are now enabled when PUBLIC_EXPO_DANGEROUSLY_LOG_TO_SERVER_FOR_AI_AUTO_DEBUGGING is set. All voice session callbacks now log with [Voice] prefix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Allows muting the microphone during voice sessions without ending the session. When muted, audio is not sent to the LLM but the session stays active. Note: Mute functionality only works on web (native SDK limitation). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Author
|
(Update: ran some limited PR tests via the happy-devbox repo that simulates a monorepo linking the commits in these PRs together.) E2E Tests This PR includes working E2E tests using Playwright:
CI validation runs ./scripts/validate.sh which includes:
Green CI Run: https://github.com/rrnewton/happy-devbox/actions/runs/19894554194/job/57021496616 |
|
can maintainers please merge this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for users to configure their own ElevenLabs API key and voice agent instead of using the default Happy voice agent. This enables self-hosted deployments and users who want to customize their voice
assistant experience.
Changes
apiVoice.tsmodule for fetching voice tokens with custom credentials and direct ElevenLabs API callsEXPO_PUBLIC_DEBUG_VOICEenv varHow it works
Testing
Manual Test plan