ESP32 Pocket LLM — Voice-Controlled Story Builder

A pocket-sized voice assistant that listens to voice input, transcribes speech, and generates responses using a local LLM running entirely on the ESP32. No cloud LLM APIs — every prompt is processed on-device and shown on a small OLED display.

ESP32 Pocket Module — compact device running the voice + LLM pipeline

---

What’s Included

The project turns a tiny on-device LLM into a voice-controlled, pocket-friendly device:

• Voice input — Button-triggered recording; speech is sent to AssemblyAI for speech-to-text.
• Local LLM — Transcribed text is fed into a small LLM (260K parameters, tinyllamas) that runs on the ESP32 — no ChatGPT or other cloud LLM calls.
• OLED output — Status (recording, uploading, transcription) and generated story/text are shown on an I2C OLED with animations and auto-scroll.
• Battery-friendly form factor — The system (ESP32, mic, display, optional battery) fits a portable, handheld use case.

Flow: button press → record → transcribe (AssemblyAI) → local LLM → text on screen, in a single device.

---

Credit & Inspiration

This project builds on DaveBben/esp32-llm (running a LLM on the ESP32), which runs a small LLM on ESP32-S3 with PSRAM and optimizations (dual-core, ESP-DSP, etc.).

Additions on top of that base:

• Screen — I2C OLED for status and LLM output.
• Microphone — I2S (e.g. INMP441) for voice capture.
• Speech-to-text — AssemblyAI API to turn recorded audio into text.
• Pocket-sized, voice-first use case — Users can talk to the LLM via voice and read the result on the device.

Core LLM-on-ESP32 idea and optimizations come from DaveBben’s work; the voice pipeline, display, and pocket-assistant experience are added in this project.

---

Hardware

• ESP32-S3 with enough PSRAM and flash (e.g. N16R8 or similar — 16MB flash, 8MB PSRAM) for the model and buffers.
• 0.96" I2C OLED (e.g. SSD1306) — SDA → GPIO 8, SCL → GPIO 9.
• INMP441 (or compatible) I2S microphone — WS → GPIO 38, SCK → GPIO 47, SD → GPIO 12.
• Button — One side to GPIO 21, other to GND (press/hold to record).
• USB for flashing and power (or battery if you add that).

Wiring diagram — OLED (I2C), INMP441 (I2S), and button

---

How It Works

1. Press and hold the button → device enters recording mode (e.g. recording indicator on OLED).
2. Speak → Audio is captured via I2S mic (e.g. 5 seconds or as configured).
3. Upload → Audio is sent to AssemblyAI for transcription (upload/processing animation on OLED).
4. Transcribe → Text is returned from AssemblyAI.
5. Local LLM → That text is used as the prompt for the on-device LLM (stories260K / tinyllamas).
6. Display → Generated tokens are streamed to the OLED with auto-scroll.

All inference runs on the ESP32; only speech-to-text uses the cloud (AssemblyAI).

---

Software Requirements

• ESP-IDF (e.g. v5.5.x) — Install guide.
• WiFi — For uploading audio to AssemblyAI.
• AssemblyAI — Sign up and add the API key in the code (e.g. in main/main.cpp or project config).

---

Setup & Build

1. Clone the repo and open the project in an ESP-IDF environment (terminal/IDE with idf.py).
2. Set WiFi and AssemblyAI in code:
   • WIFI_SSID / WIFI_PASSWORD (e.g. phone hotspot).
   • ASSEMBLYAI_API_KEY — AssemblyAI API key.
3. Ensure the LLM data files are in place (e.g. data/stories260K.bin, data/tok512.bin — or as required by your build).
4. Build and flash:

idf.py build
idf.py -p COM3 flash monitor

(Replace COM3 with your port, e.g. /dev/ttyUSB0 on Linux.)

---

Project Structure (High Level)

• main/ — Main app: WiFi, HTTP client for AssemblyAI, I2S mic, I2C OLED, button, and integration with the LLM (e.g. llm.c / llama.h).
• data/ — Pre-compiled/quantized LLM files (e.g. stories260K.bin, tok512.bin).
• images/ — Screenshots and diagrams (e.g. ESP32_Pocket_Module.gif, wiring_diagram.jpg).

The LLM core is adapted from llama2.c with ESP32-oriented optimizations (as in DaveBben’s repo).

---

Notes

• The on-device model is very small (260K parameters). It’s good for short, story-style outputs and experiments; expect limits in coherence and length.
• PSRAM and flash headroom (e.g. “around 20% usage” on a typical build) leaves room for trying other small models or extra features.
• For a permanent, compact build, the same circuit can be soldered onto a small PCB instead of a breadboard.

---

Resources

• DaveBben/esp32-llm — Original “running a LLM on the ESP32” project.
• ESP32-S3 Datasheet
• ESP-IDF Documentation
• AssemblyAI API
• ESP-IDF I2S