EchoAssist is an iOS app that turns nearby speech into live, speaker-labeled captions — entirely on device. Stop a session and it's saved automatically as a searchable, editable transcript with an AI-generated summary. No audio or text ever leaves the phone.
Nearly 1 in 7 Americans — about 50 million people — are deaf or hard of hearing, and more than 25 million Americans don't speak English fluently. Whether the barrier is hearing loss or a language difference, the result is the same: everyday conversations aren't equally accessible. A study by the Royal National Institute for Deaf People found that 54% of deaf individuals feel excluded from conversations. Several of us have grandparents or family members who live this daily.
You can approximate what EchoAssist does by juggling Notes, Voice Memos, and a translator app — but nobody trying to follow a live conversation has time for that. EchoAssist puts live captions, speaker labels, saved transcripts, summaries, and translation in one deliberately simple app: open it, press Start, read.
- Live captions with speaker diarization — words appear on screen ~half a second after they're spoken, and each line is attributed to a speaker ("Speaker 1", "Speaker 2", …) as the diarizer tells voices apart. Freshly heard words show up instantly in a lighter shade and "solidify" once their speaker is decided.
- Haptic speaker-change cues — the phone taps when the captions switch to a different speaker, so turn-taking can be felt without watching the screen. Toggleable in Settings.
- Automatic saving — stopping a recording saves the transcript to Past Recordings with a timestamp title. No extra step to lose a conversation over.
- On-device AI summaries — each saved recording gets a short summary generated by the Apple Intelligence on-device model (Foundation Models framework). Devices without Apple Intelligence fall back to a snippet of the transcript.
- Find in transcript — browser-style ⌘F search inside a recording: every match is highlighted, the current one in orange, with an "N of M" bar to step through matches (wrapping at either end) while the view scrolls to each hit. Works on translated transcripts too.
- Translation — translate a saved transcript into Spanish, French, German, Mandarin, Japanese, or Korean using Apple's on-device Translation framework. Speaker names stay untouched; results are cached so switching languages back and forth is instant.
- Recording management — search the Past Recordings list, and open any recording to rename it, rename its speakers (e.g. "Speaker 1" → "Mom"), export the whole thing as text, or delete it.
- Guided onboarding — a swipeable walkthrough on first launch shows how to caption, revisit, translate, and manage recordings, with looping video demos. Replayable anytime from Settings → View Onboarding.
- Adjustable text size — every screen follows the system Text Size setting up to the largest accessibility sizes, or flip a toggle in Settings to set an app-specific size with a slider instead. A live preview shows the effect either way.
- Private by design — transcription, diarization, summarization, and translation all run on device. Audio and text never leave the phone, which is also why the models are downloaded up front.
- Model download center — the two speech models (~800 MB) are downloaded once, from a dedicated page in Settings (or the first-run prompt), so hitting Start never triggers a surprise download. Models can be removed anytime to reclaim space.
EchoAssist started as wireframes in Figma, aimed at one goal: someone should be able to open the app and start recording immediately, without navigating a complicated menu. Once the design was settled, we built it in Xcode, continuously checking the in-progress app against those wireframes and refining both the functionality and the UX through rounds of testing and feedback.
EchoAssist is also deliberately one app instead of several. You could approximate live transcription, notes, and translation by juggling Voice Memos, Notes, and a separate Translate app — but switching between apps mid-conversation breaks the flow and risks missing what's said next. Centralizing transcription, speaker identification, saved recordings, and translation into a single app keeps the experience fast and seamless, which matters most when someone needs live captions in the middle of a real conversation.
The pipeline (in CaptionEngine.swift) runs two streaming CoreML models over the same 16 kHz mic feed, via FluidAudio:
| Model | Job |
|---|---|
| Parakeet Unified 0.6B (streaming ASR) | What was said — punctuated, capitalized words with timestamps |
| NVIDIA Sortformer v2.1 (streaming diarizer) | Who said it — "speaker S spoke from A to B" segments |
Because both models consume the same audio clock, a TranscriptAssembler joins them purely by time: each word is matched to the speaker segment it overlaps, and words accumulate into one growing block per speaker turn. The ASR runs ~0.6 s behind real time and the diarizer ~1–2 s behind, so the transcript grows in place instead of jumping around.
EchoAssist/
├── App/ Entry point + tab bar (Settings · Recording · Past Recordings)
├── Screens/ One file per screen (live recording, past list, individual
│ recording, settings, model downloads, onboarding)
├── Support/ CaptionEngine (speech pipeline), RecordingStore (JSON
│ persistence), TranscriptSummarizer (Apple Intelligence),
│ ModelDownloadCenter, Haptics, shared theme
├── Widgets/ Reusable views (recording card, translation widget)
└── OnboardingAssets/ Looping demo clips shown in the onboarding walkthrough
Recordings persist as recordings.json in the app's Documents directory via RecordingStore.swift.
- iOS 26+, Xcode 26
- A real device with a microphone (the simulator has no usable mic input; haptics also require a device)
- ~800 MB free space for the one-time speech model download
- Apple Intelligence-capable device (iPhone 15 Pro or later) for AI summaries — older devices still get everything else
- Translation requires the iOS language packs for the chosen languages (downloaded by the system on first use)
Open EchoAssist.xcodeproj in Xcode and run. The FluidAudio dependency resolves automatically via Swift Package Manager. On first launch, the onboarding walkthrough appears; after it, tap Download Models on the Recording tab (or Settings → Speech Models) before starting a session.
Built by Rachel, Katelyn, and Ethan.
EchoAssist is released under the MIT License.
The license covers this app's own source. FluidAudio, which powers the speech pipeline, is Apache-2.0, and the speech models it downloads at runtime carry their own terms — NVIDIA's Open Model License for Sortformer, and the license listed in FluidAudio's model catalog for Parakeet. No model weights are redistributed in this repository.