fix: expire stale barge-in signal so a dead mic listener cannot hold the gate forever - #137
Merged
Merged
Conversation
…hold the gate forever _bargein_user_recording() and the legacy file watcher's interrupt branch both read /tmp/mod3-barge-in.json with no staleness check: whatever event the file last recorded was treated as still-true, forever. On 2026-07-07 the mic/VAD writer in inbound.py died mid-utterance without ever writing user_speaking_end, leaving the file pinned at user_speaking_start for roughly three days -- every speak() request in that window was silently held with "User is currently speaking," and the operator heard no voice output until noticing and deleting the file by hand. Both readers now treat a user_speaking_start event older than MOD3_BARGEIN_SIGNAL_TTL_SEC (default 120s) as idle, parsing the signal's own timestamp field (ISO-8601, tz-aware) and falling back to file mtime when that field is missing or unparseable. Missing/corrupt-file-is-idle behavior is unchanged. 120s rather than something closer to inbound.py's ~2s VAD-refresh cadence: verified VADStage.process() re-dispatches a fresh user_speaking_start on every ~2s chunk for as long as real speech continues (no onset-only guard), so a short TTL would be safe for that writer alone. But the SuperWhisper-family producers (bargein/providers/superwhisper.py and the standalone integrations/bargein-producer.py) write the start event exactly once per utterance and lean on their own internal 150s staleness self-check, since real dictation there legitimately runs 60s+. This file is provider-agnostic at the read side, so the TTL has to be safe for the slowest/most conservative writer, not tuned to the fastest one.
…tamp/corrupt) Adds TestBargeinSignalStaleness targeting _bargein_user_recording() directly: fresh start-event blocks, stale start-event (older than TTL) does not, missing timestamp falls back to mtime (both fresh and stale mtime cases), an unparseable timestamp string falls back to mtime rather than raising, corrupt JSON stays idle, and user_speaking_end is never gated by age. One end-to-end case runs the same stale signal through check_bargein_gate() to prove the fix reaches speak()/`/v1/speak`. _write_signal()'s timestamp parameter now defaults to "now" instead of a hardcoded past date, since the new TTL check would otherwise have started treating every pre-existing "signal currently blocks" test's fixture data as stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Incident (2026-07-07 → 2026-07-10): the mic/VAD writer in
inbound.pydied mid-utterance without ever writinguser_speaking_end./tmp/mod3-barge-in.jsonstayed pinned atuser_speaking_startfor roughly three days. Neither reader of that file —_bargein_user_recording()(which gatesspeak()and/v1/speakviacheck_bargein_gate()) nor the legacy file watcher's interrupt branch — checked whether the signal was stale, so every voice-output request in that window was silently held with "User is currently speaking." The operator heard no mod3 output for three days and only found the cause by inspecting the file by hand.Fix
MOD3_BARGEIN_SIGNAL_TTL_SEC(default 120s) + shared_bargein_signal_is_stale()helper inserver.py._bargein_user_recording()now treats auser_speaking_startevent older than the TTL as idle. Parses the signal's owntimestampfield (ISO-8601, tz-aware; handles a trailingZ), falling back to file mtime when that field is missing or unparseable._bargein_watcher()'s interrupt branch gets the same check —_bargein_last_mtimeresets to0.0on every process restart, so a signal file left over from a previous life of the process reads as "brand new" on the first watcher tick after restart; without the staleness check this could trigger a spurious interrupt of freshly-started playback based on days-old data.user_speaking_endsemantics are unchanged — idle either way.TTL choice: 120s, not ~15s
I verified the "does the live VAD refresh the signal continuously" assumption against
inbound.pybefore picking a default, and it's mixed:inbound.py'sVADStage.process()does refresh continuously. It re-dispatches a freshuser_speaking_startevent on every audio chunk (chunk_duration_sec=2.0default) for as long as VAD keeps detecting speech — there's no onset-only guard. For this writer alone, a short TTL (~15s) would be safe with a comfortable margin over the ~2s refresh cadence.bargein/providers/superwhisper.py(in-process, opt-in viaMOD3_BARGEIN_PROVIDERS=superwhisper) and the standaloneintegrations/bargein-producer.pyscript writeuser_speaking_startonce at recording start and hold it untiluser_speaking_end(or their own internal staleness check — 150s, since "recordings can legitimately run 60s+" per their own comments).Since
_BARGEIN_SIGNALis read the same way regardless of which producer wrote it, the TTL has to be safe for the slowest writer, not tuned to the fastest one. A 15s TTL would falsely mark a legitimate 20+ second SuperWhisper dictation as idle. 120s comfortably covers a long SuperWhisper utterance (and stays under that producer's own 150s self-heal window) while still turning a dead-writer outage into, at worst, two minutes instead of three days.Tests
New
TestBargeinSignalStalenessintests/test_gate_escalation.py:user_speaking_start→ blockeduser_speaking_start(older than TTL) → not blocked (the core regression case)check_bargein_gate()proving the fix reachesspeak()//v1/speaktimestampfield falls back to mtime (both a fresh-mtime and a stale-mtime case)timestampstring falls back to mtime rather than raisinguser_speaking_endis never gated by ageAlso fixed
_write_signal()'s hardcodedtimestamp: "2026-07-03T00:00:00Z"default, which the wall clock has now passed — every pre-existing "signal currently blocks" test would otherwise have started failing the moment TTL-checking landed, since that fixture data would read as stale. It now defaults to "now."Full suite:
1009 passed, 1 skipped(2 pre-existing, unrelatedtorchaudio-not-installed failures intest_dependency_sanity.py, confirmed present on unmodifiedmainin the same environment).ruff check/ruff format --checkclean on both touched files.Test plan
pytest tests/test_gate_escalation.py -v— 22/22 passedpytest tests/test_bargein_provider_registry.py tests/test_gate_escalation.py -v— 45/45 passedpytest tests/(full suite) — 1009 passed, 1 skipped, 2 pre-existing unrelated failuresruff check/ruff format --checkonserver.pyandtests/test_gate_escalation.py