JARVIS

A fully local, voice-driven AI assistant for Windows — a futuristic HUD that listens, talks back, controls the computer, and builds software for you. Iron Man's Jarvis, not a chatbot in a box.

Everything runs free: local speech, local LLM brain, and Claude Code (on the existing Max plan) for heavy building. No paid APIs.

What it does

• Wake word — only responds when you say "Jarvis", then acts on what follows.
• Voice in / voice out — offline speech recognition (Vosk) + a natural neural voice (edge-tts).
• Computer control — "Jarvis, open Spotify and play a song", "Jarvis, open Word", media controls.
• Conversation — anything that isn't a direct command goes to a local LLM (Ollama) for a real reply.
• Builds projects — "Jarvis, build me a snake game" → Claude Code (Opus 4.8 @ xhigh, on the Max plan) creates the project in a new folder on the Desktop, hands-free.
• Futuristic HUD — fullscreen dark interface with a live, voice-reactive 3D orb (WebGL shader); minimizes to a floating orb in the corner.

Architecture (5 stages)

Stage	Tech
Ears (speech-to-text)	Vosk (offline, Indian-English model) — hud/stt_vosk.py
Brain (intent + chat)	Keyword command engine (hud/commands.js) + local LLM via Ollama (gemma3:4b)
Hands (do things)	App launch / media keys (hud/main.js) + Claude Code headless for building (jarvis-build.ps1)
Mouth (text-to-speech)	edge-tts neural voice (en-US-BrianNeural)
Face (the HUD)	Electron + a custom WebGL orb shader (hud/index.html, renderer.js, styles.css)

Requirements

• Windows 10/11, Node.js, Python 3
• Ollama with a model: ollama pull gemma3:4b
• Claude Code signed in (Max plan) — used by the build engine, no API cost
• Python packages: pip install edge-tts vosk sounddevice
• A Vosk model in hud/models/ (not committed — see Setup)

Setup

# 1. install JS deps
cd hud
npm install

# 2. install Python voice deps
pip install edge-tts vosk sounddevice

# 3. download the offline speech model (Indian English)
#    -> unzip into hud/models/vosk-model-small-en-in-0.4
#    https://alphacephei.com/vosk/models/vosk-model-small-en-in-0.4.zip

# 4. pull the local LLM brain
ollama pull gemma3:4b

Run

# from the project root
./start-jarvis.ps1

Or run install-jarvis-app.ps1 once to put a JARVIS icon on your Desktop + Start Menu (then right-click it → Pin to taskbar).

Controls

• Say "Jarvis, …" to command it.
• Esc or the – button → shrink to corner orb · click the orb / say "maximize" → expand
• Alt+F4, the ✕ button, or say "shutdown Jarvis" → close

Scripts

File	Purpose
start-jarvis.ps1	Launches the HUD (and Ollama if needed)
jarvis-build.ps1	The headless build engine (Claude Code, Opus 4.8 @ xhigh)
install-jarvis-app.ps1	Generates the icon + Desktop/Start-Menu shortcuts
hud/	The Electron HUD (UI, orb, voice, commands, brain)

Personal single-user project. Paths are currently hardcoded to one machine; adjust PROJECT_ROOT in hud/main.js and the root in jarvis-build.ps1 for your own setup.