AI-Powered Transcription & Read-Along Mode

[KeinNiemand]

One killer feature for me would be the ability to see the text of an audiobook along with the audio (with a highlight of the current word or sentence) so I can read and listen simultaneously, along with the ability to click on a word or phrase and jump there.

To enable this, the idea is to use a local speech-to-text model (maybe Whisper, or a newer model like IBM Granite 4.0 1B Speech which benchmarks surprisingly well at a smaller size) to convert an audiobook to text with timestamps, and use that data to display and highlight text in sync with playback. Being fully local means no data ever leaves the device.

This is something I plan on implementing eventually and eventually opening a PR for, assuming this is something you're interested in supporting, as it would massively expand the scope and size of the app (that model is going to be several gigabytes if it ends up bundled with the app).

Mostly creating this issue to keep track of this as a feature request and to discuss whether it's wanted.

[rstewa]

I like this idea. I've had a few people ask for something like this. I wonder if instead of bundling the model with the app, we could allow users to install it themselves separately. Either way, I say go for it. I'm sure we could find a clean way of integrating this in the app.

[m13v]

great feature idea. we built something similar for real-time transcription with timestamp alignment. the Whisper approach works well but the key challenge is getting accurate word-level timestamps, not just segment-level. Whisper's default output gives you start/end times per segment (usually a sentence), but for word-by-word highlighting you need the token-level timestamps. whisper.cpp exposes these but the alignment drifts over long recordings if you process the whole file at once. processing in overlapping chunks (30 second windows with 5 second overlap) and stitching results together gives much better timestamp accuracy for a 10+ hour audiobook. for model size, Granite is interesting but whisper-small gives a good accuracy/size tradeoff and runs fast on Apple Silicon via CoreML.

[m13v]

our transcription service that handles the overlapping chunk approach, word-level timestamp alignment, and CoreML model selection for Apple Silicon: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

[KeinNiemand]

Sentence level timestamps would be good enough for me personally, if it's too much effort to get world level accuracy, I'll just stick to sentence level.