Phase 4: Transcription + App-Aware Shortcuts

[sonnes]

Goal

Enable two-way communication: others speak, September transcribes. The keyboard becomes context-aware, showing shortcuts specific to the focused application.

Design Reference

Settings — Transcription

App Shortcuts (rightmost keyboard section)

The keyboard assembly shows the App Shortcuts panel on the far right, displaying the focused app name (e.g., "VS Code") and context-aware shortcuts.

Deliverables

Transcription Engines

• Apple Speech — SFSpeechRecognizer for on-device transcription. Zero configuration, private
• Whisper API — OpenAI Whisper via URLSession. Requires API key. Higher accuracy
• whisper.cpp — local inference via SPM wrapper (e.g., whisper-kit or swift-whisper). Offline, fast. Bundle a small model (base or small)

Talk Mode

• Transcription display — live text appearing as others speak, shown in the predictions area or a dedicated transcript view
• Continuous listening — microphone stays active between pauses (togglable)
• Auto-punctuation — automatically adds periods, commas, question marks (togglable)
• Context pipeline — transcribed text feeds into the AI prediction engine as conversation context, improving sentence predictions

Settings — Transcription

• Engine selection — 3 cards:
  • Apple Speech — icon (microphone), subtitle "On-device, private"
  • Whisper — icon (sparkles), subtitle "OpenAI, accurate"
  • Whisper.cpp — icon (chip), subtitle "Local, fast"
• Language dropdown — "English (US)" default, populated from engine capabilities
• Auto-Punctuation toggle — green when ON, gray when OFF, with subtitle "Automatically add periods, commas, and question marks"
• Continuous Listening toggle — with subtitle "Keep microphone active between pauses"

App Observer

• NSWorkspace observer — listen to NSWorkspace.didActivateApplicationNotification to detect frontmost app changes
• Bundle ID mapping — map app bundle identifiers to shortcut sets

App Shortcuts Section

• Dynamic shortcut loading — when the focused app changes, update the App Shortcuts section (rightmost 200pt panel)
• Focused app header — show app icon + name (e.g., "VS Code" with green dot)
• Built-in shortcut sets for common apps:
  • VS Code — Toggle Terminal, Command Palette, Go to File, Split Editor, Toggle Sidebar, Quick Fix
  • Safari — New Tab, Close Tab, Reload, Back, Forward, Show Bookmarks
  • Notes — New Note, Bold, Italic, Checklist, Table, Find
  • Mail — New Message, Reply, Forward, Archive, Flag
  • Finder — New Window, New Folder, Get Info, Quick Look
• Shortcut execution — tapping a shortcut injects the key combination into the focused app via CGEvent
• Uses ShortcutButton and ShortcutFull components from Phase 1

Acceptance Criteria

• Apple Speech transcribes spoken audio in real-time with visible text output
• Transcribed text improves AI prediction relevance
• Auto-punctuation adds periods at sentence boundaries
• Continuous listening keeps microphone active across pauses
• Switching to VS Code updates the App Shortcuts panel with VS Code shortcuts
• Switching to Safari shows Safari shortcuts
• Tapping an app shortcut executes it in the focused app
• Settings toggles persist via SwiftData

Dependencies

• Phase 1 (keyboard layout, App Shortcuts section placeholder, CGEvent injection)
• Phase 2 (AI prediction engine — transcriptions feed into prediction context)

[m13v]

I integrated whisper.cpp (via WhisperKit specifically) into a native macOS app and have some notes on the engine choices.

for the whisper.cpp option, WhisperKit is probably your best bet on Apple Silicon since it uses CoreML under the hood. the base.en model is around 150MB and transcribes faster than realtime on M1+. one gotcha - first inference after model load takes 2-3x longer than subsequent ones because of CoreML compilation, so do a warmup pass with silence at startup or the user's first words get delayed.

for CGEvent shortcut injection, watch out for apps that use a custom event tap or secure input mode. Terminal.app and password fields enable SecureEventInput which blocks CGEvent posting entirely. you can check if it's active with CGSIsSecureEventInputSet() (private API but it's just a read) and show the user a hint that shortcuts won't work in that context.

the NSWorkspace observer approach is solid. one addition - some apps change their shortcut context depending on which window is focused (like VS Code with terminal vs editor), so you might want to also watch for NSAccessibilityFocusedUIElementChangedNotification to detect sub-app context switches. it's noisier but lets you show editor shortcuts vs terminal shortcuts accurately.

[m13v]

our WhisperKit integration (model loading, warmup, streaming audio buffer management): https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

and for the CGEvent shortcut injection side, we handle that in the macOS automation server: https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift