Inclusa AI — Backend Monorepo

Inclusa AI is a real-time accessibility platform that converts live speech into synchronized transcription, AI sign-language gestures, and an AI avatar for physical, virtual, and hybrid presentation audiences.

This repository is the backend monorepo: NestJS services (TypeScript) for the API/sync/recording/notification layers, and FastAPI services (Python) for the Speech, AI, and Analytics ML pipelines.

---

---

Table of contents

• Architecture
• Pipelines
• Repository layout
• Services, ports & protocols
• Inter-service contracts
• Quick start
• Local development
• End-to-end demo path
• Environment variables
• Tooling
• Key technology decisions
• Roadmap

---

Architecture

                                 ┌─────────────────────────────────────────────┐
   Presenter mic / audio         │                  CLIENTS                     │
        │                        │  audience & presenter apps (PPTX.js/PDF.js   │
        │ audio frames (WS)      │  render slides beside accessibility panels)  │
        ▼                        └───────────────▲─────────────────────────────┘
┌──────────────────┐                             │ Socket.IO (per-session rooms)
│   asr-service     │                             │
│ (Speech Pipeline) │                    ┌────────┴─────────┐
│ Google STT / Whisper                   │   sync-service   │  ← Redis-backed Socket.IO adapter
└────────┬─────────┘                     │ (WS fan-out)     │     (horizontal scale)
         │ TranscriptChunk               └────────▲─────────┘
         ▼            ── publish ─────────────────┤ subscribe (wildcards)
    ╔═══════════════════════════════════════════ NATS ═══════════════════════════════════════╗
    ║  inclusa.pipeline.transcript.*   inclusa.pipeline.gesture.*   inclusa.pipeline.avatar.*  ║
    ║  inclusa.pipeline.session.*      inclusa.pipeline.engagement.*     inclusa.notifications ║
    ╚═══▲═══════════▲═══════════▲═══════════▲══════════════▲═══════════════════════▲═══════════╝
        │           │           │           │              │                       │
 ┌──────┴───┐ ┌─────┴────┐ ┌────┴─────┐ ┌───┴──────┐ ┌─────┴─────────┐    ┌────────┴──────────┐
 │nlp-service│ │translation│ │ gesture  │ │  avatar  │ │  presenter-   │    │ recording-service │
 │  (AI)     │ │  (AI)     │ │  (AI)    │ │  (AI)    │ │  insights     │    │ (Recording Pipe.) │
 │ segment   │ │ text→sign │ │MediaPipe │ │RPM+MP /  │ │ (Audience     │    │ persist timelines │
 │           │ │           │ │  Hands   │ │Audio2Face│ │  Analytics)   │    │ → GCS             │
 └───────────┘ └───────────┘ └──────────┘ └──────────┘ └───────┬───────┘    └───────────────────┘
                                                               │ EngagementSignal / nudges
                                                               ▼
                                                     ┌──────────────────────┐
   REST (sessions, orgs, auth) ───────────────────► │ notification-service │  ← NATS queue consumer
   ┌──────────────────┐                              │ (Notification Pipe.) │     (coaching, recording-ready,
   │   api-gateway     │  Auth0 JWT · TypeORM/Postgres│  email/push/webhook)│      session lifecycle)
   │ Swagger @ /docs   │                              └──────────────────────┘
   └──────────────────┘

  Infra: Postgres (sessions/orgs) · Redis (Socket.IO adapter) · NATS w/ JetStream (pipeline + queue)

Data flow (one utterance): presenter audio → asr-service emits TranscriptChunks → published to NATS → nlp/translation/gesture/avatar services consume and emit GestureFrame/AvatarFrame → sync-service subscribes to all pipeline subjects and fans out to the session's WebSocket room → recording-service persists the synchronized timelines to GCS → notification-service fires a "recording ready" alert. presenter-insights runs asynchronously off the main path and emits EngagementSignals / coaching nudges that surface through notification-service.

The five pipelines

Pipeline	Owned by	Responsibility
Speech	asr-service	Streaming speech-to-text (Google STT primary, Whisper fallback).
AI	nlp-service, translation-service, gesture-service, avatar-service	Segmentation → sign-language instruction data → hand gestures → avatar animation.
Audience Analytics	presenter-insights	Engagement/pace analytics + coaching, async off the main pipeline.
Recording	recording-service	Captures synced transcript/gesture/avatar timelines, writes to GCS.
Notification	notification-service	Session/recording/coaching alerts, queue-driven over NATS.

Repository layout

inclusa-ai/
├── apps/
│   ├── api-gateway/          # NestJS — auth, sessions, orgs, REST API, Swagger
│   ├── sync-service/         # NestJS — WebSocket fan-out (Redis adapter, NATS subscriber)
│   ├── recording-service/    # NestJS — Recording Pipeline → Cloud Storage (GCS)
│   ├── notification-service/ # NestJS — Notification Pipeline (NATS queue consumer)
│   └── ml-services/          # Python (uv), NOT part of the npm workspace
│       ├── asr-service/          # FastAPI — Speech Pipeline (Google STT / Whisper / mock)
│       ├── nlp-service/          # FastAPI — AI: segmentation
│       ├── translation-service/  # FastAPI — AI: text → sign instruction data
│       ├── gesture-service/      # FastAPI — AI: MediaPipe Hands gestures
│       ├── avatar-service/       # FastAPI — AI: RPM + MediaPipe (Audio2Face flag)
│       ├── presenter-insights/   # FastAPI — Audience Analytics
│       └── shared/               # Shared pydantic schemas mirroring shared-types
├── packages/
│   ├── shared-types/         # TS pipeline contracts, WS events, DTOs (source of truth)
│   └── config/               # Shared eslint/prettier/tsconfig
├── infra/
│   ├── docker/               # Dockerfiles (nestjs.Dockerfile is parameterized by APP)
│   ├── docker-compose.yml    # Local dev: Postgres, Redis, NATS + all services
│   └── k8s/                  # Placeholder manifests (not wired to local dev)
├── .github/workflows/ci.yml  # Lint/test/build affected apps + Python lint
└── README.md

Note on packages/proto: the original scaffold reserved this for gRPC .proto files. We chose NATS over gRPC for inter-service transport (see decisions), so there is no proto package. The equivalent contract lives in packages/shared-types (TS) and apps/ml-services/shared (Python).

Services, ports & protocols

Service	Port	Protocol(s)	Talks to
api-gateway	3000	HTTP REST (+ Swagger /docs)	Postgres, Auth0
sync-service	3001	WebSocket (Socket.IO), subscribes NATS	Redis, NATS, clients
recording-service	3002	HTTP REST (POST /recordings), subscribes NATS	NATS, GCS
notification-service	3003	HTTP REST (POST /notifications/test), NATS queue consumer	NATS
asr-service	8001	HTTP + WebSocket (/transcribe/stream)	(publishes NATS in full system)
nlp-service	8002	HTTP REST (POST /segment)	—
translation-service	8003	HTTP REST (POST /translate)	—
gesture-service	8004	HTTP REST (POST /gestures)	—
avatar-service	8005	HTTP REST (POST /avatar/frames)	—
presenter-insights	8006	HTTP REST (POST /insights)	—
Postgres	5432	TCP	api-gateway
Redis	6379	TCP	sync-service (Socket.IO adapter)
NATS	4222 / 8222	NATS / monitoring	pipeline + notifications

Inter-service contracts

The canonical event contract is defined once in packages/shared-types and mirrored field-for-field in apps/ml-services/shared (Python). Core types:

• TranscriptChunk — Speech Pipeline output (interim + final).
• GestureFrame — MediaPipe-style 21-point hand landmarks.
• AvatarFrame — engine-agnostic blendshapes/bone rotations.
• SessionState — per-session source of truth broadcast by sync-service.
• EngagementSignal — Audience Analytics output.

Transport: NATS (with JetStream) carries all pipeline-stage events and the notification queue — one broker for streaming fan-out, stage retries/backpressure, and durable notifications. REST is used only for request/response operations (session/org CRUD in api-gateway, the ML services' mock endpoints). WebSocket (Socket.IO) is the client-facing transport out of sync-service. NATS subjects and WS event names are defined in shared-types/src/events.ts.

Quick start

Prerequisites: Docker + Docker Compose. (For running services outside Docker: Node 20+, Python 3.11+, and uv.)

git clone <this-repo> && cd inclusa-backend
cp .env.example .env
docker compose -f infra/docker-compose.yml up --build

This starts Postgres, Redis, NATS, all four NestJS apps, and all six Python ML services with hot-reload. Then:

• API Gateway Swagger: http://localhost:3000/docs
• Health checks: :3000/health, :3001/health, :3002/health, :3003/health, :8001/health …
• NATS monitoring: http://localhost:8222

Local development

Install once at the repo root (npm workspaces link @inclusa/* packages):

npm install
npm run build -w @inclusa/shared-types   # build the shared contract first

Run a single NestJS app with hot reload (needs Postgres/Redis/NATS — start them via docker compose ... up postgres redis nats):

npm run dev:api-gateway     # or dev:sync / dev:recording / dev:notification

Run a Python service:

cd apps/ml-services/asr-service
uv run uvicorn main:app --reload --port 8001

End-to-end demo path

example

cd backend/api && npm run start:dev cd backend/realtime && npm run start:dev

## Environment variables

Copy `.env.example` to `.env` and fill the values. Typical entries (adjust to match your implementation):

APP_ENV=development PORT=8080 DATABASE_URL=postgres://user:pass@db:5432/inclusa REDIS_URL=redis://redis:6379 JWT_SECRET=supersecret ASR_PROVIDER=google|whisper GOOGLE_ASR_API_KEY=... WHISPER_MODEL_PATH=... AVATAR_SERVICE_URL=https://avatar.example.com AVATAR_API_KEY=... STORAGE_BUCKET=inclusa-recordings S3_ACCESS_KEY_ID=... S3_SECRET_ACCESS_KEY=... FIREBASE_PROJECT_ID=... AUTH0_DOMAIN=... AUTH0_CLIENT_ID=...

## Running tests

Run unit and integration tests using the project's test scripts. Example:

```bash
# Backend (NestJS)
cd backend
npm test

# Frontend
cd frontend
npm test

If integration tests require infra, start docker compose before running tests.

Deployment

Options:

• Docker images + Kubernetes (recommended for production)
• Docker Compose for staging/smaller deployments
• Managed container platforms for selective components

The scaffold is wired so the full request path can be demonstrated with mock data (no real models or cloud creds):

1. Speech in — open a WS to ws://localhost:8001/transcribe/stream?sessionId=demo (or POST /transcribe/mock). The mock ASR provider streams TranscriptChunks.
2. Pipeline — the AI stub services (/segment, /translate, /gestures, /avatar/frames) return well-formed mock frames.
3. WS broadcast — publish any pipeline frame to NATS (inclusa.pipeline.*) and sync-service fans it out to the session:demo room. Send echo over the socket to confirm wiring without the pipeline.
4. Recording write — POST http://localhost:3002/recordings {"sessionId":"demo"} writes a placeholder object (local .storage/ in mock mode, else GCS).
5. Notification fire — POST http://localhost:3003/notifications/test publishes a NotificationEvent; the queue consumer logs a mock alert.

Environment variables

• REST endpoints and WebSocket message formats are documented in docs/API.md.
• Realtime audio ingestion uses WebRTC; control and event delivery use WebSockets.

Data, privacy & security

• Analytics are aggregate-only and do not identify individuals.
• If recordings are stored, obtain consent and define retention policies.
• Encrypt data at rest (S3) and in transit (TLS).
• Minimize storing raw audio where not needed; prefer ephemeral processing.

Contributing

Every variable across all services is documented in .env.example. Copy it to .env for docker-compose, or copy each app's own .env.example for standalone runs. Safe defaults are set for local dev: ASR_PROVIDER=mock, STORAGE_MOCK=true, AUTH_DEV_BYPASS=true, FEATURE_AUDIO2FACE=false.

Tooling

• TypeScript: ESLint + Prettier via the shared packages/config.
• Python: Ruff + Black (configured per service pyproject.toml).
• Pre-commit: Husky + lint-staged (prettier --write on staged TS/JS/JSON/MD).
• CI: .github/workflows/ci.yml lints/builds/tests only the affected NestJS workspaces on a PR, plus a Python lint job.

Key technology decisions

These were the open decisions in the original brief; the choices made here (and how to swap them):

• Inter-service transport — NATS (over gRPC). One broker serves both the low-latency pipeline fan-out and the notification queue, and gives retries/backpressure via JetStream. Swap point: pipeline stages could move to gRPC for per-hop latency, but you'd then need a separate queue (e.g. BullMQ) for notifications.
• ORM — TypeORM (over Prisma). Traditional NestJS fit with decorator entities and DI. Entities live beside their modules (SessionEntity).
• JS tooling — plain npm workspaces (over pnpm/Turborepo) to keep the MVP simple. CI approximates Turbo's --filter with change detection.
• Auth — Auth0 JWT (RS256 + JWKS) for enterprise/education/government SSO. AUTH_DEV_BYPASS=true lets you develop without a tenant. Swap point: Firebase Auth — replace the issuer/JWKS/audience in jwt.strategy.ts.
• ASR — Google STT primary, Whisper fallback, both behind one provider interface (core/providers); default mock. Swap with ASR_PROVIDER.
• Storage — GCS (aligns with Google STT). STORAGE_MOCK=true writes locally.
• Avatar — Ready Player Me + MediaPipe, with NVIDIA Audio2Face behind the FEATURE_AUDIO2FACE flag (needs GPU infra), not a hard dependency.

Roadmap

• Wire real Google STT / Whisper streaming in asr-service.
• Implement real MediaPipe Hands + TensorFlow gesture generation.
• Real capture/muxing of synchronized timelines in recording-service.
• Fan notification-service out to email/push/webhook providers.
• Flesh out infra/k8s overlays (staging/production) with GPU node pools. Planned improvements:
• Support additional sign languages and regional variants
• Improve avatar realism and gesture fidelity
• Reduce ASR latency and improve accuracy
• Integrations with presentation platforms (PowerPoint, Google Slides)
• Offline/low-bandwidth modes

Troubleshooting

• Captions lagging: check ASR response time, message bus latency, and networking issues
• Avatar desync: ensure clocks are synchronized between services and verify message ordering
• Missing gestures: check translator logs and fallback behavior when translation fails

License

This project does not include a license file yet. Add a LICENSE file (e.g., MIT) to declare terms.

Contact

Project maintainers:

• SynapGridTech