Release v0.105.0 — an announcement whose audio never played is sent again - #102
Merged
Merged
Conversation
Third occurrence of one shape: 2026-09-15T01:42:12Z, 8.4 s after the add-on
restarted for a deploy, an announcement was queued to the cordless before that
handset had re-registered. Asterisk logged `Could not create dialog to invalid
URI '19'`, no channel existed, no audio played, and the only thing recorded was
`announce-unsettled` three minutes later. The handset's contact came back 37.9 s
after the send that failed. v0.104.0 shipped the record half; the owner's
decision after it was that such an announcement must be RETRIED automatically.
The retry lives in wakeup/scheduler.py as a third pass in the main loop, in its
own try block: that loop already ticks every 20 s, already reads the ledger with
not_before=_STARTED, already owns the settling window, and — decisively — it
SURVIVES the restart that causes this defect, which a retry held in the webui's
own process would not.
Never replaying audio that played is the hard constraint, and six locks hold it:
1. the clip join that already exists. callqos stamps the same --sound onto the
row it writes from the hangup extension, so any ANNOUNCE_TERMINAL row for a
clip excludes it permanently, at any age, in any order — including the
FILTER-mangled spelling, via clip_key(). ANNOUNCE_TERMINAL is now one tuple
shared with the reconciler instead of an inline set, so the two cannot drift.
2. two CONSECUTIVE idle observations a poll apart, so a replay cannot land on
audio in progress.
3. the budget lives in the ledger: announce-retry-attempted rows are counted from
disk, written BEFORE the Originate, and the Originate is skipped when record()
returns False. A restart cannot refill it.
4. ext freshness: a newer announcement delivered to the same room retires the
clip as ext-superseded.
5. positive state only. The green light is ami.device_idle — the single state
"not in use". Empty, unavailable, ringing, in use and any unrecognised
spelling all defer. The fail-OPEN pre-flight is what caused the incident and
is deliberately not reused for a replay.
6. a final narrow veto read AFTER the state read, so a delivered row landing
mid-decision can only appear, never be missed.
Bounded at 2 attempts and never started later than 150 s after the original
send. 150, not the 120 s settling window: at 120 the second attempt is
structurally unreachable for the ring-out and slow-re-registration shapes (both
driven in the new tests), and the cap is instead derived from the clip's own
300 s life — 150 + 30 ring + 90 clip = 270. Three invariants are pinned by test,
including reading _cleanup_announce_dir's default out of app.py's signature, and
the startup line warns "announce retry is INERT" if an override makes the first
attempt unreachable.
Also here:
* webui/announce_clip.py owns the announce directory and what a legal clip name
is, because the retry is handed a name out of a ledger in group-writable
/share: strict ann-<ext>-<32 hex> with the ext bound to the row's own ext,
realpath containment, a regular file (never a link), non-zero size, and a
re-measured duration. It returns the extensionless FULL path — the basename
would produce a call that answers and plays nothing, and no delivered row.
* the reconciler's horizon now runs from the newest attempt, so a replay in
flight is not judged undelivered, and its verdict rows carry `retries`.
* api_announce no longer arms the duplicate-suppression window when the
reachability guard could not judge. That night it was armed by an announcement
nobody heard, so an identical re-send would have been answered
{"ok": true, "skipped": "duplicate"} — the v0.85.0 lockout in a new dress.
* announce-retry-skipped (too-old | budget-exhausted | ext-superseded |
clip-gone) is written exactly once per clip and is NOT terminal: the
announcement still never arrived, so the reconciler still speaks. Transient
deferrals are logged and never recorded.
* one ledger read and one clock per tick, shared by both passes; no AMI traffic
at all unless the pure ledger scan already found a candidate.
No new notifications, no config option (ANNOUNCE_RETRY_MAX_ATTEMPTS=0 in the
environment is a complete kill switch). 63 new tests, all driving the real
callers; the suite goes 802 -> 865. 27 mutants entered, 27 killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven findings from the adversarial review of the announce retry, fixed. The two
that mattered are both about the same thing: a feature that places phone calls in
this house must not be able to place two at once, and the owner must be able to
turn it off.
1. NO TWO REPLAYS TO ONE HANDSET (major). The pass read the endpoint states ONCE
per tick and then looped over every candidate, so two unresolved announcements
to one extension fired milliseconds apart off one stale read — neither
Originate could see the other's channel. Driven through the real caller, three
queued clips for ext 19 produced THREE Originates in one tick. app.py's own
busy-guard names the consequence: "a second INVITE cannot auto-answer and would
RING the handset as call waiting", and api_announce would then report a
genuinely fresh announcement as skipped-busy behind our stale one. The
population is not hypothetical — the duplicate-window graft in this same branch
is what lets identical re-sends through as separate queued rows.
Fixed in two places, because the cost of being wrong is unsolicited ringing:
- delivery.retryable_announcements now returns at most ONE live candidate per
extension, the NEWEST announcement queued to that room; the others are
retired with the ext-superseded row that already existed for exactly this
meaning. Ties (record() stamps whole seconds) break on ledger order, which is
the producer's own order inside that second.
- _retry_announcements keeps a `fired` set: at most one Originate per extension
per tick, and the clip held back gives its clean observations BACK rather
than banking them — they were earned against the state read that this tick's
own call has just invalidated.
2. THE OFF SWITCH NOW REACHES THE ADD-ON (major). DOCS.md told the owner the
bounds could be changed in the environment and that ANNOUNCE_RETRY_MAX_ATTEMPTS=0
switched the feature off. Nothing exported it: an option is INERT until a run
script exports it, as that script's own comment says, so the only documented way
to stop the retry needed an image rebuild. Bridged properly, all four edits —
`announce_retry_attempts` (default 2, range 0-3) in config.yaml options and
schema, translations/en.yaml, and `export ANNOUNCE_RETRY_MAX_ATTEMPTS="${ARA:-2}"`
in wakeup-scheduler/run — with a test that pins the option, the range that
includes 0, the exported NAME against the one delivery.py reads, and that any
ANNOUNCE_RETRY_* the manual mentions is exported by that script. The two timings
stay code-only and DOCS.md now says so.
3. The startup bounds line is pinned through main() (minor). Deleting the
_log_retry_bounds() call left the suite green: its four references all called it
directly. That line is the only production signal separating "live and idle"
from "OFF because an import failed", "OFF at 0 attempts" and "INERT at this
poll".
4. DOCS.md says what an owner will actually notice (minor): the gate is the AUDIO,
not the reason it was missing, so an announcement nobody answers is retried too
— up to three rings for one message on the FXS phones, which have no
auto-answer. The manual had motivated the whole feature with the restart
incident, which a reader would fairly take as its only trigger.
5. The freshness lock reads QUEUE times, not delivery times (nit). A 60 s clip
answered before this announcement was even queued writes its audio-delivered
row a minute after it, which read as "the room has been spoken to since" while
the truth was the reverse — and retired the one message the room had not heard.
6. Two keys on the returned records that nothing read (`_attempt_ts`,
`last_attempt_ts`) are gone; a private field with no reader invites the next
reader to believe it is load-bearing.
7. ActiveChannels stays as it is: fail-closed, and nothing in this repo
establishes what it holds during a playback. Live stage 2d must record the real
value and delete the condition if it is empty — the code comment, the risk
register and the live plan all already say so.
870 passed (865 before; 5 new tests plus assertions on two existing ones), and
the standalone runners for config, translations, run gates, webui, app and the
announce dialplan are green. Mutants: 14 new ones for these fixes and 19 from the
original battery re-run over the two files this touches, all 33 killed, with a
control mutant that must SURVIVE — the first run of the harness reported 14 kills
from a python that had no pytest installed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gain Owner decision (2026-09-16): retry automatically when the audio never played. - wakeup/scheduler.py gains a third pass, _retry_announcements(), in its own try/except: it replays an announcement whose clip has no audio-delivered row, bounded at 2 attempts and 150 s from queueing, and only once the target endpoint reports itself registered and idle on two reads one poll apart. - The duplicate-safety join is the clip name, unchanged: a retry re-originates the SAME sound, so an audio-delivered row for that clip retires it forever. Only the newest announcement queued to a room is live; older ones retire with ext-superseded, so two INVITEs can never race to one handset. - New option `announce_retry_attempts` (default 2, range 0-3; 0 is the off switch) — config.yaml options + schema, translations, and the export in the wakeup-scheduler run script, with a test pinning all four. - app.py no longer arms the 300 s duplicate-suppression window when the pre-flight guard could not judge the endpoint: that window was being armed by an announcement nobody heard. Found in the 2026-09-15/16 review (third occurrence of the shape). 870 tests; 33 mutants killed plus a control that survived as required. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Owner decision: an announcement whose audio never played must be retried automatically. This is the third occurrence of the shape — an announcement queued seconds after a restart, before the cordless had re-registered, that died with no channel and no audio (2026-09-15T01:42:12Z).
Where it lives. A third pass in the wake-up scheduler's tick, in its own try/except so a retry that raises can neither stop verdicts being filed nor stop alarm clocks ringing. The scheduler was chosen over the webui because it survives the restart that causes the defect, already reads the ledger and endpoint states, and already originates.
Never replaying audio that played. The retry re-originates the same clip, so the existing clip-name join retires it permanently once an
audio-deliveredrow exists — the retry must not mint a new name, which would break that join. Only the newest announcement queued to a room is a live candidate; older ones retire asext-superseded, and a per-tick lock means two INVITEs can never race to one handset.Bounds, from this system's own numbers. 2 attempts, 20 s minimum age, 150 s maximum age, and a positive endpoint state observed twice a poll apart. Every occurrence of this shape was post-restart re-registration, and every endpoint was back within 46 s; two attempts a tick apart straddle that. Attempts are counted from the ledger, so a restart cannot refill the budget. 150 s is pinned against the clip's own 300 s lifetime and is deliberately not the 120 s settling window — at 120 s the second attempt was structurally unreachable.
Off switch that works.
announce_retry_attempts(default 2, range 0–3,0disables) is wired through config.yaml options and schema, translations, and the run script's export, with a test pinning all four plus the exported name against the one the code reads. The review caught that the original env-only switch could never be set from inside the add-on.Also:
app.pystops arming the 300 s duplicate-suppression window when the pre-flight guard could not judge the endpoint — that window was being armed by an announcement nobody heard.Testing
tick(),_reconcile_announcements,api_announce.Known trade-offs, accepted under the owner's decision
Live verification after deploy
Staged so nothing plays audio in the house: the new option and startup line; then the full candidate path and every refusal branch against ext 20, which has never registered; then the supersession rule. The green-light path on a real handset stays unverified until it happens naturally, and is recorded as such.
🤖 Generated with Claude Code