Skip to content

fix: expire stale barge-in signal so a dead mic listener cannot hold the gate forever - #137

Merged
chazmaniandinkle merged 2 commits into
mainfrom
fix/bargein-signal-ttl
Jul 10, 2026
Merged

fix: expire stale barge-in signal so a dead mic listener cannot hold the gate forever#137
chazmaniandinkle merged 2 commits into
mainfrom
fix/bargein-signal-ttl

Conversation

@chazmaniandinkle

Copy link
Copy Markdown
Contributor

Summary

Incident (2026-07-07 → 2026-07-10): the mic/VAD writer in inbound.py died mid-utterance without ever writing user_speaking_end. /tmp/mod3-barge-in.json stayed pinned at user_speaking_start for roughly three days. Neither reader of that file — _bargein_user_recording() (which gates speak() and /v1/speak via check_bargein_gate()) nor the legacy file watcher's interrupt branch — checked whether the signal was stale, so every voice-output request in that window was silently held with "User is currently speaking." The operator heard no mod3 output for three days and only found the cause by inspecting the file by hand.

Fix

  • New MOD3_BARGEIN_SIGNAL_TTL_SEC (default 120s) + shared _bargein_signal_is_stale() helper in server.py.
  • _bargein_user_recording() now treats a user_speaking_start event older than the TTL as idle. Parses the signal's own timestamp field (ISO-8601, tz-aware; handles a trailing Z), falling back to file mtime when that field is missing or unparseable.
  • The legacy _bargein_watcher()'s interrupt branch gets the same check — _bargein_last_mtime resets to 0.0 on every process restart, so a signal file left over from a previous life of the process reads as "brand new" on the first watcher tick after restart; without the staleness check this could trigger a spurious interrupt of freshly-started playback based on days-old data.
  • Missing-file / corrupt-JSON / user_speaking_end semantics are unchanged — idle either way.

TTL choice: 120s, not ~15s

I verified the "does the live VAD refresh the signal continuously" assumption against inbound.py before picking a default, and it's mixed:

  • inbound.py's VADStage.process() does refresh continuously. It re-dispatches a fresh user_speaking_start event on every audio chunk (chunk_duration_sec=2.0 default) for as long as VAD keeps detecting speech — there's no onset-only guard. For this writer alone, a short TTL (~15s) would be safe with a comfortable margin over the ~2s refresh cadence.
  • The SuperWhisper-family producers do not. Both bargein/providers/superwhisper.py (in-process, opt-in via MOD3_BARGEIN_PROVIDERS=superwhisper) and the standalone integrations/bargein-producer.py script write user_speaking_start once at recording start and hold it until user_speaking_end (or their own internal staleness check — 150s, since "recordings can legitimately run 60s+" per their own comments).

Since _BARGEIN_SIGNAL is read the same way regardless of which producer wrote it, the TTL has to be safe for the slowest writer, not tuned to the fastest one. A 15s TTL would falsely mark a legitimate 20+ second SuperWhisper dictation as idle. 120s comfortably covers a long SuperWhisper utterance (and stays under that producer's own 150s self-heal window) while still turning a dead-writer outage into, at worst, two minutes instead of three days.

Tests

New TestBargeinSignalStaleness in tests/test_gate_escalation.py:

  • fresh user_speaking_start → blocked
  • stale user_speaking_start (older than TTL) → not blocked (the core regression case)
  • one end-to-end case through check_bargein_gate() proving the fix reaches speak() / /v1/speak
  • missing timestamp field falls back to mtime (both a fresh-mtime and a stale-mtime case)
  • unparseable timestamp string falls back to mtime rather than raising
  • corrupt JSON stays idle (unchanged pre-existing behavior)
  • user_speaking_end is never gated by age

Also fixed _write_signal()'s hardcoded timestamp: "2026-07-03T00:00:00Z" default, which the wall clock has now passed — every pre-existing "signal currently blocks" test would otherwise have started failing the moment TTL-checking landed, since that fixture data would read as stale. It now defaults to "now."

Full suite: 1009 passed, 1 skipped (2 pre-existing, unrelated torchaudio-not-installed failures in test_dependency_sanity.py, confirmed present on unmodified main in the same environment). ruff check / ruff format --check clean on both touched files.

Test plan

  • pytest tests/test_gate_escalation.py -v — 22/22 passed
  • pytest tests/test_bargein_provider_registry.py tests/test_gate_escalation.py -v — 45/45 passed
  • pytest tests/ (full suite) — 1009 passed, 1 skipped, 2 pre-existing unrelated failures
  • ruff check / ruff format --check on server.py and tests/test_gate_escalation.py

…hold the gate forever

_bargein_user_recording() and the legacy file watcher's interrupt branch
both read /tmp/mod3-barge-in.json with no staleness check: whatever event
the file last recorded was treated as still-true, forever. On 2026-07-07
the mic/VAD writer in inbound.py died mid-utterance without ever writing
user_speaking_end, leaving the file pinned at user_speaking_start for
roughly three days -- every speak() request in that window was silently
held with "User is currently speaking," and the operator heard no voice
output until noticing and deleting the file by hand.

Both readers now treat a user_speaking_start event older than
MOD3_BARGEIN_SIGNAL_TTL_SEC (default 120s) as idle, parsing the signal's
own timestamp field (ISO-8601, tz-aware) and falling back to file mtime
when that field is missing or unparseable. Missing/corrupt-file-is-idle
behavior is unchanged.

120s rather than something closer to inbound.py's ~2s VAD-refresh cadence:
verified VADStage.process() re-dispatches a fresh user_speaking_start on
every ~2s chunk for as long as real speech continues (no onset-only
guard), so a short TTL would be safe for that writer alone. But the
SuperWhisper-family producers (bargein/providers/superwhisper.py and the
standalone integrations/bargein-producer.py) write the start event exactly
once per utterance and lean on their own internal 150s staleness
self-check, since real dictation there legitimately runs 60s+. This file
is provider-agnostic at the read side, so the TTL has to be safe for the
slowest/most conservative writer, not tuned to the fastest one.
…tamp/corrupt)

Adds TestBargeinSignalStaleness targeting _bargein_user_recording()
directly: fresh start-event blocks, stale start-event (older than TTL)
does not, missing timestamp falls back to mtime (both fresh and stale
mtime cases), an unparseable timestamp string falls back to mtime rather
than raising, corrupt JSON stays idle, and user_speaking_end is never
gated by age. One end-to-end case runs the same stale signal through
check_bargein_gate() to prove the fix reaches speak()/`/v1/speak`.

_write_signal()'s timestamp parameter now defaults to "now" instead of a
hardcoded past date, since the new TTL check would otherwise have started
treating every pre-existing "signal currently blocks" test's fixture data
as stale.
@chazmaniandinkle
chazmaniandinkle merged commit ce42990 into main Jul 10, 2026
7 checks passed
@chazmaniandinkle
chazmaniandinkle deleted the fix/bargein-signal-ttl branch July 10, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant