Migrate to Gemini Live API

[mubashirsidiki]

Replaces OpenAI Realtime API with Google's Gemini Live API for the voice assistant backend.

API swap

• WebSocket endpoint changed from OpenAI (wss://api.openai.com/v1/realtime) to Gemini Live (wss://generativelanguage.googleapis.com/.../BidiGenerateContent)
• Auth moved from Authorization header to ?key= query param
• All message formats updated:
  • session.update → setup
  • input_audio_buffer.append → realtimeInput
  • response.output_audio.delta → serverContent.modelTurn

Audio pipeline rewritten

OpenAI used PCMU natively with no transcoding. Gemini uses raw PCM, so the server now handles:

Input

PCMU 8kHz → audioop.ulaw2lin → PCM 8kHz → audioop.ratecv upsample to 16kHz → Gemini

Output

PCM 24kHz → struct averaging 24→8kHz → audioop.lin2ulaw → PCMU → Twilio

The output converter is stateful with buffer alignment to avoid chunk-boundary artifacts.

Interrupt handling simplified

• Removed:
  • handle_speech_started_event (~30 lines)
  • timing math
  • conversation.item.truncate messages
  • item_id tracking
• Gemini handles barge-in server-side via serverContent.interrupted
• Server only needs to send clear to Twilio on interrupt

Config changes

• OPENAI_API_KEY → GEMINI_API_KEY
• Added MODEL constant
• Voice changed from alloy to Aoede
• Removed:
  • TEMPERATURE
  • LOG_EVENT_TYPES
  • SHOW_TIMING_MATH
• Added load_dotenv(override=True) to override stale system environment variables

Other

• Gemini speaks first via realtimeInput text prompt
• TwiML greeting shortened to "Connecting you now."
• Transcript logging:
  • Buffers Gemini chunks
  • Prints full response on turnComplete
  • Logs user speech via inputTranscription
• .env.example and Readme.md updated