Next Gen-Inspired AI Banking IVR POC

Production-style hackathon POC for a premium, multilingual, conversational AI banking IVR using autonomous agents.

Stack

• Backend: Node.js, Fastify, TypeScript
• Frontend: Next.js, TailwindCSS, Framer Motion
• AI/LLM: CrewAI-ready orchestration + OpenRouter
• STT/TTS: Faster-Whisper (live service) + ElevenLabs (integration point)
• Telephony: Twilio Voice + Media Streams WebSocket
• Data: MongoDB + ChromaDB
• Realtime: WebSockets

What Is Live Now

• Twilio Voice webhook returns TwiML with <Connect><Stream .../></Connect>
• Twilio media WebSocket endpoint receives audio chunks
• Mu-law 8k audio decoded to PCM16 in backend
• PCM chunks streamed to Faster-Whisper service (apps/stt)
• Live transcript fed into orchestrator and dashboard in realtime
• AI response audio sent back over Twilio bidirectional media stream

Quick Start (Local Laptop)

1. Copy env:

cp .env.example .env

2. Start dependencies locally:

• MongoDB on localhost:27017
• ChromaDB on localhost:8000 (example: chroma run --host 0.0.0.0 --port 8000)

3. One-command startup:

./scripts/start-local.sh

4. Seed data (first run or reset):

npm run seed

Services:

• Backend: http://localhost:8080
• Dashboard: http://localhost:3000
• STT: http://localhost:9000

Twilio Setup

• Voice webhook URL: POST https://<public-url>/twilio/voice
• Media stream URL is embedded by backend from TWILIO_WEBHOOK_BASE_URL as:
  • wss://<public-url>/twilio/media-stream-ws

API Endpoints

• POST /api/calls/start { callId, phone }
• POST /api/calls/utterance { callId, text }
• POST /api/calls/verify-otp { callId, otp }
• POST /twilio/voice
• GET /twilio/media-stream-ws (WebSocket)
• GET /api/health
• WS /realtime

Project Structure

• apps/backend/src/agents
• apps/backend/src/tools
• apps/backend/src/rag
• apps/backend/src/telephony
• apps/backend/src/websocket
• apps/backend/src/services
• apps/backend/src/memory
• apps/backend/src/auth
• apps/backend/src/database
• apps/backend/src/embeddings
• apps/backend/src/logs
• apps/dashboard
• apps/stt

Browser Softphone Setup (No PSTN Needed)

1. In Twilio Console, create a TwiML App:
   • Voice Request URL: https://<public-url>/twilio/client/voice
   • Method: POST
2. Create Twilio API Key/Secret (for Voice SDK token).
3. Set in .env:
   • TWILIO_ACCOUNT_SID
   • TWILIO_API_KEY_SID
   • TWILIO_API_KEY_SECRET
   • TWILIO_TWIML_APP_SID
   • TWILIO_WEBHOOK_BASE_URL
4. Restart app and open http://localhost:3000/softphone.
5. Click Connect Call and allow microphone permission.

Maintainer

This repository is owned and maintained by Disha Singh.