AnimalAI Voice-Flow Application

A modular, voice-driven state machine application built with XState v5 + React + TypeScript.

This project demonstrates a production-ready architecture for voice-controlled applications, using the state machine as the application "spine" with clean separation of concerns.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                          UI Layer                               │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐               │
│  │ VoiceConsole│ │ StatePanel  │ │ LogsPanel   │ ...           │
│  └──────┬──────┘ └──────┬──────┘ └──────┬──────┘               │
│         │               │               │                       │
│         └───────────────┼───────────────┘                       │
│                         ▼                                       │
│                    ┌─────────┐                                  │
│                    │ AppShell│  (thin client, no business logic)│
└────────────────────┴────┬────┴──────────────────────────────────┘
                          │
                          ▼ events
┌─────────────────────────────────────────────────────────────────┐
│                      Voice Layer                                │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐               │
│  │intentRouter │ │responseBank │ │   aliases   │               │
│  └──────┬──────┘ └─────────────┘ └─────────────┘               │
│         │                                                       │
│         ▼ machine events                                        │
└─────────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────┐
│                    State Machine                                │
│  ┌─────────────────────────────────────────────────┐           │
│  │              animalMachine.ts                    │           │
│  │  (pure state machine - the application spine)    │           │
│  └──────────────────────┬──────────────────────────┘           │
│                         │                                       │
│  ┌──────────┐ ┌─────────┴─────────┐ ┌──────────┐               │
│  │ types.ts │ │    guards.ts      │ │actions.ts│               │
│  └──────────┘ └───────────────────┘ └──────────┘               │
└─────────────────────────────────────────────────────────────────┘
                          │
                          ▼ invokes
┌─────────────────────────────────────────────────────────────────┐
│                       Actors                                    │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐               │
│  │playbackActor│ │grooveSearch │ │narrationActor│              │
│  │  (timers)   │ │  (search)   │ │ (LLM/phi3)  │              │
│  └─────────────┘ └─────────────┘ └─────────────┘               │
└─────────────────────────────────────────────────────────────────┘

Folder Structure

src/
├── machine/                 # Core state machine (pure, deterministic)
│   ├── animalMachine.ts     # The spine - machine definition
│   ├── types.ts             # Context, events, and type definitions
│   ├── guards.ts            # Pure guard functions
│   └── actions.ts           # Action implementations
│
├── actors/                  # Side effects, async, cancellable
│   ├── playbackActor.ts     # Timer-based playback simulation
│   ├── grooveSearchActor.ts # Deterministic groove search
│   └── narrationActor.ts    # LLM-powered narration (phi3:mini)
│
├── voice/                   # Voice/utterance handling
│   ├── intentRouter.ts      # Utterance → intent → events
│   ├── responseBank.ts      # Fallback phrase banks
│   └── aliases.ts           # Selection phrase aliases (A/B/C)
│
├── runtime/                 # Machine runtime utilities
│   ├── interpret.ts         # Actor creation and wiring
│   └── logger.ts            # Structured logging
│
├── ui/                      # React UI (thin client)
│   ├── AppShell.tsx         # Main application shell
│   └── panels/              # UI panels
│       ├── StatePanel.tsx
│       ├── ContextPanel.tsx
│       ├── LogsPanel.tsx
│       ├── PlaybackPanel.tsx
│       ├── VoiceConsole.tsx
│       └── AnimalSaysPanel.tsx
│
├── util/                    # Shared utilities
│   ├── time.ts              # Time formatting, delays
│   └── random.ts            # Random selection helpers
│
├── App.tsx                  # Entry point
└── App.css                  # Styles

State Machine Hierarchy

idleArmed (powered on, not listening)
    ↓ WAKE
idleListening (awaiting command)
    ↓ NEW_SESSION
sessionActive (parent state)
    ├── askingVibe (prompts user)
    ├── waitingForRequest (expects groove description)
    ├── lookingUpGrooves (async search)
    ├── announceAudition (brief pause)
    ├── auditioning (A→B→C, 4 bars each)
    ├── unsureSelection (retry/fallback)
    └── jamming (loop selected groove)
error (recoverable)

Quick Start

# Install dependencies
npm install

# Start development server
npm run dev

# Build for production
npm run build

Open http://localhost:5173

Voice Implementation (Phased)

Phase 1: Voice Line Buttons (Current)

• Buttons simulate spoken phrases
• VoiceConsole sends VOICE_UTTERANCE({ text })
• intentRouter maps utterances to machine events
• No real speech recognition required

Phase 2: Browser SpeechRecognition (Future)

• Add src/adapters/speechRecognitionAdapter.ts
• Use Web Speech API
• Same intent routing, just different input source

Phase 3: TTS (Future)

• Add src/adapters/ttsAdapter.ts
• Use SpeechSynthesis or macOS say
• Speak context.animalLine on change

NarrationActor (LLM via Ollama)

The only LLM usage is in narrationActor.ts for generating Animal's spoken lines.

Setup

# Install Ollama (macOS)
brew install ollama

# Start Ollama server
ollama serve

# Pull the model
ollama pull phi3:mini

Enable

Create a .env file:

VITE_OLLAMA_BASE_URL=http://localhost:11434
VITE_NARRATION_ENABLED=true

Features

• Strict timeout: 600ms, then fallback to response bank
• JSON output: Forces { "line": "..." } format
• Validation: Invalid responses trigger fallback
• Caching: Repeated prompts return cached responses
• Graceful degradation: Works without Ollama

Use Cases

• Wake greeting
• Ask vibe line
• Audition intro
• Unsure prompt
• Jamming confirmation
• Optional candidate callouts

---

Try This Script

Follow this exact click sequence to test the full workflow:

1. Wake → New Session → Groove Request → Audition

1. Click "animal" (wake word)

   • State: idleArmed → idleListening
   • Animal greets you

2. Click "start a new session"

   • State: idleListening → sessionActive.waitingForRequest
   • Animal asks for vibe

3. Click "play a chill rock beat at 112 BPM..."

   • State: waitingForRequest → lookingUpGrooves → auditioning
   • Watch grooves A, B, C play

2. Interrupt and Select

4. While auditioning, click "first" (or "A")
   • State: auditioning → jamming
   • Selected groove loops

3. Barge In

5. Click BARGE_IN
   • State: jamming → waitingForRequest
   • Loop stops, session preserved

4. Fallback Path

6. Start a new audition
7. Click "gibberish / unclear" twice
   • Fallback buttons appear
8. Click a fallback button to select

---

Adding New Voice Commands

1. Add phrases to intentRouter.ts

const MY_COMMAND_PHRASES = [
  'do something',
  'make it happen',
];

2. Add intent type

export type IntentType = 
  | ...existing...
  | 'my_command';

3. Add state validity

const INTENT_VALID_STATES = {
  ...existing...,
  my_command: ['sessionActive.waitingForRequest'],
};

4. Handle in AppShell

case 'my_command':
  send({ type: 'MY_COMMAND' });
  break;

5. Add event to machine

Add the event type and transitions in animalMachine.ts.

---

Edge Cases Covered

Scenario	Behavior
"new session" before wake	Implicitly wakes and starts session
Groove request in wrong state	Logged as blocked
"A"/"first" during audition	Interrupts and starts jamming
2 unclear utterances	Enables fallback buttons
STOP from any state	Resets to idleArmed
BARGE_IN during jamming	Pauses without losing session
200 bars in jamming	Auto-stops (safety limit)

---

Tech Stack

• Vite - Build tool
• React 18 - UI
• TypeScript - Type safety
• XState v5 - State machine library
• @xstate/react - React bindings
• Ollama + phi3:mini - LLM for narration (optional)

---

Design Principles

1. Machine as Spine: The state machine is the source of truth
2. Pure Core: Machine logic is deterministic and testable
3. Actors for Effects: All side effects live in actors
4. Thin UI: No business logic in components
5. One LLM Actor: Narration only; everything else deterministic
6. Graceful Fallbacks: Works without external dependencies

---

License

MIT