VOX — Real-Time Multilingual Meeting Translator (MVP)

Several people join a shared Vox room from their own laptops. One speaks Vietnamese, one English, one Chinese. Each participant sees a live shared transcript translated into their own chosen language, with who-said-what labels, plus an AI summary + action items at the end.

3 containers, no GPU: frontend (React) · backend (Go) · postgres.

Browser (React) ──mic 16k PCM (WS binary)──► Go backend ──► Soniox real-time (STT + langID + diarization)
 live captions  ◄──transcript/translation (WS)── room hub ──► LLM translate (per participant language)
                                                          └──► Postgres (persist) + end-of-meeting LLM summary

---

✨ Runs with zero API keys

The whole pipeline is demonstrable with no credentials. With keys blank, the backend uses:

• a Mock speech provider — watches your mic for voice activity and emits a scripted VN/EN/ZH standup conversation (partials → finals), and
• a Mock translator + heuristic summary — the scripted lines have baked-in, correct VN/EN/ZH translations.

So you can docker compose up, open two browser tabs in different languages, talk into the mic, and watch the full UX immediately. Add keys to make it real.

Capability	No keys (default)	With keys
STT + language ID + speaker separation	Mock (scripted)	Soniox real-time
Translation (per participant language)	Mock (baked script)	LLM (OpenAI-compatible)
Summary + action items	Heuristic	LLM

---

Quick start (Docker — the demo path)

cd vox
cp .env.example .env        # optional: add SONIOX_API_KEY / OPENAI_API_KEY for the real pipeline
docker compose up --build   # or: make up

• Frontend: http://localhost:3000
• Backend health: http://localhost:8080/healthz

Demo it (2 tabs, 2 languages)

1. Tab 1: open http://localhost:3000 → name Linh, read in Tiếng Việt → Create meeting.
2. Click 🔗 Invite to copy the room URL.
3. Tab 2: paste the URL → name Bob, read in English → Join meeting.
4. In each tab click 🎙️ Start mic and talk. Each tab shows the shared transcript in its own language, with speaker labels and live captions.
5. Click 📋 End & summarize, then ⬇ Export to download Markdown.

🔊 VOX shows captions, not audio — participants don't hear each other through Vox (use Zoom/an in-room mic for audio). Captions are the point.

---

Local dev (hot reload, no Docker for app code)

# 1) Postgres only
docker compose up -d postgres

# 2) Backend (terminal A)
make dev-backend        # go run ./cmd/server on :8080

# 3) Frontend (terminal B)
make dev-frontend       # vite on :5173, proxies /api + /ws to :8080

Open http://localhost:5173.

---

Configuration

All via env (see .env.example). Sensible defaults; everything optional.

Var	Default	Meaning
SPEECH_PROVIDER	auto	auto = Soniox if key present else mock; or force soniox/mock
SONIOX_API_KEY	(empty)	Enables real Soniox STT
SONIOX_MODEL	stt-rt-v5	Soniox real-time model
OPENAI_BASE_URL	https://api.openai.com/v1	Any OpenAI-compatible endpoint (Groq/Together/local)
OPENAI_API_KEY	(empty)	Enables real LLM translate + summary
OPENAI_MODEL	gpt-4o-mini	Chat model for translate + summary
DATABASE_URL	compose value	Postgres DSN
ALLOWED_ORIGINS	*	CORS

The backend logs its wiring on boot, e.g. VOX wiring: speech=mock translate=mock summary=heuristic.

---

Deploying behind central-nginx (9bricks.com)

This repo is live at https://vox.9bricks.com behind the shared central-nginx reverse proxy (Cloudflare → origin :443 → L4 SNI router → internal :8443).

Domain	→	Upstream
vox.9bricks.com	frontend (UI; self-proxies /api + /ws)	vox-frontend:80
api-vox.9bricks.com	backend (REST + WS, direct)	vox-backend:8080
admin-vox.9bricks.com	parked placeholder (no admin UI in MVP)	—

What wires it together:

1. /home/ubuntu/nginx-central/conf.d/vox.conf — three server { listen 8443 ssl; } blocks (the *.9bricks.com SNI is routed to :8443 by the stream block in nginx.conf), using the shared 9bricks.com cert.
2. docker-compose.yml joins the external 9bricks-cluster network so central-nginx can reach the containers. ⚠️ Use unique aliases (vox-frontend, vox-backend, vox-postgres) — generic names like postgres/backend/frontend collide with other stacks already on that shared network and silently resolve to the wrong container.
3. .env sets ALLOWED_ORIGINS=https://vox.9bricks.com,https://api-vox.9bricks.com, which switches the backend into hardened mode: CORS is restricted and the WebSocket enforces an Origin allowlist (verified: correct origin → 101, foreign origin → 403).

Apply / reload:

docker compose up -d --build
docker exec central-nginx nginx -t && docker exec central-nginx nginx -s reload

Cloudflare must be in Full SSL mode with WebSockets enabled (default on proxied records).

---

Architecture & key design decisions

Provider adapter (the GPU swap is a config change, not a rewrite)

All feature code talks to speech.SpeechProvider / SpeechSession / SpeechEvent (backend/internal/speech/provider.go). The MVP ships SonioxProvider and MockProvider. A future SelfHostedProvider (Whisper + pyannote + NLLB on a GPU) implements the same interface with zero changes to the room hub or UI — see Path to self-hosted / GPU below.

Why translation is a separate layer (important)

The build prompt suggested using Soniox's one-WebSocket "STT + translate" combo. We verified the docs: a single Soniox session translates into exactly one target language (one_way to one target_language, or two_way between two). A VOX room needs every participant's language at once, and that set changes as people join/leave.

So VOX keeps Soniox for the hard real-time part (**STT + automatic language ID

• speaker separation**) and does translation one layer up, per final utterance, into exactly the languages currently in the room (backend/internal/translate). Benefits:

• Handles N dynamic target languages (new joiner's language is picked up on the next utterance; backlog is backfilled on join).
• Cheaper than opening one Soniox session per (stream × language).
• Still hides behind the interface — SpeechEvent.Translations is honored if a future provider does translate natively.

One speech session per mic = free speaker identity

Each browser streams only its own mic over its own WebSocket, so the speaker is known with zero diarization effort — the room labels every utterance with that connection's participant name. Soniox's diarization is a bonus, not a dependency.

Audio path

AudioWorklet (frontend/public/pcm-worklet.js) captures mic audio → main thread resamples to 16 kHz mono and converts to Int16LE PCM → sent as binary WebSocket frames. Control messages are JSON text frames.

---

API & WebSocket contract

REST

• POST /api/sessions {title} → {session_id, room}
• GET /api/sessions/:id → session + participants + transcript + summaries
• POST /api/sessions/:id/end → finalize + generate summary, returns {summary, actions}
• GET /api/sessions/:id/export → Markdown download

WebSocket /ws/:room — JSON envelope {type, payload}; audio as binary frames.

• Client → server: join {name, lang} · leave · (binary) PCM audio
• Server → client: joined · participants {list} · transcript {speaker, source_lang, text, final, ...} · translation {utterance_id, target_lang, text} · summary {kind, content} · error {message}

---

Data model

sessions · participants · utterances · translations · summaries (minimal, no vectors). Schema: backend/internal/store/migrations/0001_init.sql, applied idempotently on every boot.

---

Repo layout

vox/
├── docker-compose.yml         # frontend + backend + postgres
├── backend/                   # Go (Echo + coder/websocket)
│   ├── cmd/server/main.go
│   └── internal/
│       ├── config/            # env config
│       ├── store/             # pgx + migrations
│       ├── speech/            # SpeechProvider interface + Soniox + Mock
│       ├── translate/         # Translator interface + LLM + Mock
│       ├── summary/           # end-of-meeting summary (LLM + heuristic)
│       ├── demo/              # scripted multilingual conversation (no-key demo)
│       ├── hub/               # room/connection management + broadcast
│       └── server/            # Echo routes, REST handlers, WS orchestration
└── frontend/                  # React + Vite + TS
    ├── public/pcm-worklet.js  # AudioWorklet PCM capture
    └── src/                    # audio, ws, transcript reducer, components

---

Build order & per-milestone test checklists

Built against Section 8 of the spec. Each milestone's manual checks:

M1 — Compose up. frontend + backend + postgres boot.

• docker compose up --build → 3 containers running (make ps).
• curl localhost:8080/healthz → {"status":"ok"}.
• curl -X POST localhost:8080/api/sessions -d '{"title":"t"}' → {session_id, room} (schema migrated).

M2 — Mic → backend. AudioWorklet streams 16 kHz PCM binary over WS.

• Join a room, click Start mic, allow permission.
• Backend logs show no send audio errors while you talk.
• (mock) after ~1s of speech a transcript line appears.

M3 — Speech API. Live transcript of one speaker with detected language.

• With SONIOX_API_KEY, speak VN/EN/ZH → words stream in as partials then finalize.
• The language badge matches what you spoke (auto-detected).
• Speaker label = your name.

M4 — Real-time translation. Each client reads in its own language.

• Two tabs in different languages; one person talks.
• Each tab shows the utterance translated into its language; original shown muted below.
• A late joiner sees prior lines backfilled into their language.

M5 — Multi-participant room. Shared, time-ordered transcript + labels.

• 2+ tabs joined; participants panel lists everyone with their language.
• Both speakers' utterances appear in one shared, time-ordered transcript for all tabs.
• Each line is labeled with who said it.

M6 — Summary + export.

• Click End & summarize → summary + action items appear in all tabs.
• GET /api/sessions/:id/export downloads a Markdown file with transcript + summary + actions.
• Action items include the scripted/spoken commitments (e.g. "deploy to staging before Friday").

Stretch (M7 TTS, M8 glossary) are intentionally not built — see spec §8.

---

Path to self-hosted / GPU (future, for on-prem privacy)

Trigger only when audio/text must never leave the customer's infra, or at higher scale. Nothing in the room hub, UI, or schema changes — only the provider impl and infra:

• Implement SelfHostedProvider (same SpeechProvider interface): faster-whisper (ASR, GPU) + pyannote (acoustic diarization for one-shared-mic rooms) + NLLB / local LLM (translation) in a Python service. Or license an on-prem engine (Soniox/Speechmatics on-prem).
• Add pgvector + a speaker_profiles table for cross-session voice memory.
• Add LiveKit only if Vox must host the actual audio call (per-participant tracks → still no acoustic diarization needed).
• Hardware: a single GPU box (e.g. RTX 4090 / L4).

---

Cost (demo ballpark)

Soniox real-time STT+translation ≈ $0.12/hr/stream; a 1-hour, 3-person demo ≈ $0.36. LLM translate/summary ≈ a few cents. Verify current pricing on the providers' pages.

Sources

• Soniox real-time WebSocket API — https://soniox.com/docs/api-reference/stt/websocket-api
• Soniox real-time translation — https://soniox.com/docs/translation/stt-translation/rt-translation

vox