Reflections

A proactivity system with live perception+transcription and memory for the MentraOS smart glasses.

What this project does

Reflections streams live audio and video from MentraOS glasses, runs perception locally, and builds up a durable memory of the people around you and the conversations you have with them.

While you wear the glasses, the system:

• Transcribes speech in real time via Soniox, attributing each utterance to the correct speaker.
• Detects who is actively speaking using the LR-ASD model over video + audio, drawing green boxes around people who are talking and red boxes around those who aren't.
• Recognizes faces across sessions — the first time someone is seen speaking, their face is embedded and labeled "Person 1"; the second time, the embedding is matched and they keep the same label. On exit, Claude reads the transcript and tries to resolve those labels to real names ("Person 1" → "Alice"), which get baked into a persistent face gallery on disk.
• Overlays live captions directly on the video stream, color-coded by speaker, with under 50 µs of added per-frame latency.
• Distills conversations into a long-term memory file — press s and the agent sends everything new since the last snapshot to Claude, which updates a memory.md with durable facts, relationships, tasks, and context.
• Proactively answers questions directed at the wearer — every transcript update is screened by a local Qwen 3 1.7B + LoRA classifier (~200 ms). If it looks actionable, Claude Haiku 4.5 reads memory + recent transcript and decides what to say (or to stay silent). Approved replies are spoken through the glasses' speaker via MentraOS TTS — typically 1.5-2 s from finalized turn → answer.

The glasses are the sensor. MediaMTX is the media server. A Bun app server bridges MentraOS cloud events to your laptop. A Python viewer is the brain that runs perception, draws overlays, and manages memory.

---

How to use it

Once everything is running (see Setup below), launch the app on the glasses from the MentraOS phone app. A window titled "Mentra Live" will open on your laptop showing the glasses' camera feed with bounding boxes and live captions.

Keyboard controls

Keys are captured by the "Mentra Live" video window — click it first to give it focus.

Key	What it does
s	Snapshot to memory. Sends everything new in the transcript since the last snapshot to Claude, which updates memory.md with any new facts, people, tasks, or context. Each press only sends what's new — you can press it as often or as rarely as you like.
p	Speak a phrase. Tells the glasses to speak the default phrase (set in proactivity/speak.py) via MentraOS TTS. Useful for testing the speaker.
m	Mute / unmute the proactivity loop. Toggles whether the agent is allowed to speak unprompted. The classifier and Claude calls keep running and decisions still log either way; only the actual TTS call is suppressed when muted.
q	Quit. Stops the viewer cleanly. On quit, Claude reads the full session transcript and tries to resolve any remaining "Person N" labels to real names; matched labels are renamed in the face gallery before it's flushed to disk.

Pressing Ctrl+C in the terminal also works for quitting but is less graceful — the face gallery may not get saved.

What you'll see on screen

• Green box around a face → that person is currently speaking (LR-ASD score above threshold).
• Red box around a face → face detected but not speaking.
• Faint white box → face just detected, still waiting for the first speaker-detection score.
• Label above each box → resolved name + speaking score (e.g. Bob +0.68), or just the score if no identity has been assigned yet.
• Caption bar at the bottom → the last few finalized transcript lines, with each speaker's label in their own color. Your own speech is labeled with USER_NAME from .env.

Files produced during a session

File	Written when	Contents
transcript_updates.log	Real-time, as transcript events fire	Cumulative transcript snapshots at each sentence/speaker boundary
memory.md	On every s press	Claude-distilled durable memory (people, tasks, preferences, context)
proactivity_decisions.log	Real-time, as the proactivity agent acts	One line per turn the agent considered: classifier P, Claude verdict, what was spoken (or why it stayed silent)
models/face_gallery.npz	On quit	Face embeddings, one array per named identity
models/face_gallery.json	On quit	Gallery metadata (names, embedding counts, timestamps)

The gallery and memory.md persist across sessions. The transcript log appends — delete it manually if you want a clean slate.

Resetting

rm models/face_gallery.npz models/face_gallery.json   # forget all faces
rm memory.md                                          # wipe memory
rm transcript_updates.log                             # clear transcript history

---

Setup

Prerequisites

• Bun
• Python 3.10+
• MediaMTX (binary drop-in, no install needed)
• ngrok (free account fine)
• MentraOS glasses on the same WiFi network as your laptop

1. Register your app

1. Go to console.mentra.glass and sign in.
2. Create an app and set a unique package name (e.g. com.yourname.reflections).
3. Add the microphone and camera permissions.
4. Set the Public URL to your ngrok tunnel URL (from step 4 below).

2. Configure environment variables

cp .env.example .env

Edit .env:

Variable	Description
PACKAGE_NAME	Must match what you registered in the console
MENTRAOS_API_KEY	From console.mentra.glass
PORT	Local server port (default 3000)
WHIP_URL	http://<YOUR_LAN_IP>:8889/live/glasses/whip — use ipconfig/ifconfig to find your LAN IP
SONIOX_API_KEY	From soniox.com — required for transcription
ANTHROPIC_API_KEY	From console.anthropic.com — required for s snapshots and name resolution
USER_NAME	Your name, used to label your speech in transcripts and captions

3. Download model weights

Three model files are needed. The LR-ASD weights ship with the repo under third_party/lr_asd/weights/. You still need to download two ONNX models into models/:

• YuNet (face detection): face_detection_yunet_2023mar.onnx
• MobileFaceNet (face recognition): w600k_mbf.onnx from the buffalo_s.zip pack

4. Expose your server with ngrok

ngrok http 3000

Copy the https://...ngrok-free.app URL it prints and paste it as the Public URL in the MentraOS console.

5. Install dependencies

bun install
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

---

Running

Four terminals (one each):

# Terminal 1: ngrok tunnel
ngrok http 3000

# Terminal 2: MediaMTX
./mediamtx/mediamtx ./mediamtx/mediamtx.yml

# Terminal 3: Bun app server
bun run dev

# Terminal 4: Python viewer
source venv/bin/activate
python view_stream.py

Launch the app on the glasses from the MentraOS phone app. Within ~1 second you should see Camera session started in the Bun terminal and the "Mentra Live" window will open showing the live feed.

---

Documentation

docs.mentra.glass