Data race in AudioProcessor: audioEnergy/audioSamples accessed from multiple threads without synchronization

[shawnzhu]

Summary

AudioProcessor has data races on audioEnergy and audioSamples — they are written from the audio tap callback thread and read from arbitrary threads (e.g., main thread) without any synchronization.

Details

In AudioProcessor.swift, processBuffer() is called from the audio engine tap's callback thread:

// ~line 907-917, called from audio tap callback thread
func processBuffer(_ buffer: [Float]) {
    audioSamples.append(contentsOf: buffer)          // write
    // ...
    self.audioEnergy.append(newEnergy)                // write
}

Meanwhile, relativeEnergy reads these from any thread:

// ~line 210, read from any thread
public var relativeEnergy: [Float] {
    return self.audioEnergy.map { $0.rel }
}

And audioSamples is also read externally:

public var audioSamples: ContiguousArray<Float> = []  // no synchronization

This is a Swift Concurrency data race — concurrent read/write on non-isolated, non-sendable mutable state.

Impact

• When polling relativeEnergy from the main thread (e.g., for VAD-based end-of-speech detection), values can be stale, skipped, or inconsistent.
• In our case, this caused AudioProcessor.isVoiceDetected() to not work reliably when called from a @MainActor-isolated context while recording.
• Could potentially crash in optimized builds due to unsynchronized array mutations.

Reproduction

1. Start recording with audioProcessor.startRecordingLive()
2. Poll audioProcessor.relativeEnergy from the main thread in a timer
3. Observe that values are inconsistent or empty

Suggested Fix

Either:

• Make AudioProcessor an actor
• Or protect audioEnergy / audioSamples with a lock (e.g., NSLock or os_unfair_lock)

Environment

• WhisperKit 0.17.0
• Xcode 26.3
• iOS 26.3 / macOS 15

[shawnzhu]

I found this problem when fixing the end-of-speech detection, tried different options and audio energy based approach perform the best with my patch

[m13v]

hit this exact race condition running WhisperKit for live transcription in a macOS app. the audio tap callback fires on a real-time thread and any contention there causes dropouts.

what worked for us was chunking audio samples into fixed-size segments and passing them through a serial dispatch queue before they hit the transcription pipeline. the audioEnergy array gets rebuilt per-chunk rather than appended to continuously, which eliminates the shared mutable state entirely.

the other thing worth watching is that os_unfair_lock is technically correct here but can cause priority inversion if the main thread holds it while the audio thread is waiting. we ended up using a lock-free ring buffer for the audio samples path and only using a lock for the energy values which are less latency-sensitive.

[m13v]

our transcription service that handles the chunked audio pipeline: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

[shawnzhu]

@m13v Thanks for the feedback!

a lock-free approach on the audio path would be a robust solution for continuous live transcription. Although #443 doesn't meet the strict "no locks on the real-time audio thread" rule, the NSLock` is technically safe since it supports priority inheritance.

[m13v]

yeah the tradeoff between lock-free and simplicity makes sense for a library. if most users are doing batch processing rather than real-time streaming, the simpler approach is probably fine. for the live transcription case you could offer both paths behind a config flag so users who need the lock-free path can opt in without complicating the default.