Skip to content

Release v0.104.0 — a link fault on an announcement raises a card; lost deliveries leave a record - #101

Merged
tesseractAZ merged 4 commits into
mainfrom
release/0.104.0
Sep 16, 2026
Merged

tesseractAZ merged 4 commits into
mainfrom
release/0.104.0

Conversation

@tesseractAZ

Copy link
Copy Markdown
Owner

Summary

From the 2026-09-15/16 review of the first day running v0.101.0–v0.103.1.

A link fault on an announcement or page raised no alert (the main fix). switchboard-callqos masked the quality branch by PLAYBACK_TAGS. That mask was written for truncated clips, but truncation exits earlier at the undelivered branch where ALARM_TAG decides, so the mask only ever silenced link conditions — MES, loss and RTT.

  • Live cost over this house's 50-day ledger: 5 legs met alert thresholds and raised nothing.
  • The worst, 2026-09-15T00:26:35Z: an announcement that played in full (stage complete, 1114 packets, 0 % loss both ways) while the handset's own RTCP reported MES 68.4 (MOS 3.42), RTT mean 287 ms, peak 883 ms. The 5-minute reachability poller read 9/9 throughout.
  • Measured noise: 1.78 → 4.75 cards per 30 days. No extra tightening.
  • The one-way-audio shape exemption is untouched; a truncated clip still stays quiet; a silent wake-up still pages.

Two deliveries that vanished now leave a record.

  • An announce queued seconds after a restart died silently. The pre-flight device_unreachable guard fails open when AMI cannot answer, which is deliberate and kept — but it now records announce-guard-unjudged so "the handset looked fine" and "nobody could tell" are distinguishable. An originate that raises or is refused records announce-originate-failed with the clip.
    • Correction found while implementing: the originate-failed row would not have fired for the live incident, because AMI accepts Async: true before Asterisk fails. announce-guard-unjudged is the row that closes that shape.
  • A wake-up that replaces a pending one now names the displaced time on the same set row — one row, so the snooze reader still matches it.

Two smaller fixes.

  • callqos records ts_iso beside the epoch ts (schema v5). The two spellings in one folder made a reviewer report "no calls" for a window holding 18. ts keeps its type: devhealth floats it and post_alert keys the card id on it.
  • A misdial (zero duration or unreachable, no transmit figure) can no longer be the nearest leg that judges cordless audio. The predicate is deliberately narrow: a call that was scored 0.0 on real audio is still matched, because an empty measurement is not the same fact as a call that never happened.

Corrected at source: the "append-only and unbounded" comments (the ledger is capped at 300 rows and rolling), SECURITY.md's /data counts (re-dated), and DOCS.md's claim that playback legs never raise a quality alert.

Testing

  • 802 tests pass on the merged tree.
  • Mutation batteries: 11/11 and 20/20 killed, no survivors. One first-pass survivor was fixed by adding fixtures that exercise each clause of the misdial predicate separately.
  • New tests use the real recorded legs as fixtures, with identifiers masked.
  • test_playback_legs_are_recorded_but_never_alert was rewritten and renamed rather than deleted: its one-way-audio block is unchanged and its silence block inverts.
  • Leak scan clean.

Live verification after deploy

  • Next announcement or page over a degraded link: a quality card appears.
  • Next add-on restart with an announcement queued inside the settling window: an announce-guard-unjudged row.
  • Next wake-up replacement: the displaced time on the set row.
  • Next call: ts_iso present and matching ts.

🤖 Generated with Claude Code

tesseractAZ and others added 4 commits September 15, 2026 18:42
…ts an ISO twin; a misdial cannot judge the cordless

FIX 1 — a link fault on a playback leg raised nothing.

build_record's `else:` branch ended `_notify = notify and (a.tag or "") not in
PLAYBACK_TAGS`. That mask was written for TRUNCATED clips — "a page cut short is
not an emergency" — and truncation cannot reach the branch: delivery_failures()
is evaluated above and every cut-short scripted leg takes the `if undelivered:`
path, where ALARM_TAG already decides. So the only thing the mask could ever
silence was a LINK condition — MES, loss or RTT — on the same wire the next
conversation uses.

Cost, measured over this house's 50-day ledger: 5 legs met the alert thresholds
and raised nothing. The worst, 2026-09-15T00:26:35Z, was an announcement that
played in FULL (stage complete, 1114 packets, 0 % loss both ways) to the WiFi
cordless while the handset's own RTCP reported MES 68.4 (MOS 3.42), RTT mean
287 ms and peak 883 ms. Nothing else saw it: the 5-minute reachability poller
read 9/9 throughout.

The line is now `_notify = notify`. No other tightening: noise goes from 1.78 to
4.75 cards per 30 days on the same real data. The other use of PLAYBACK_TAGS —
the one-way-audio detector — is untouched; that mask is about a leg's shape and
it is correct.

FIX 2 — two timestamp types in one /share folder read as "no calls".

`ts` here is epoch seconds; every sibling ledger in /share/switchboard writes an
ISO-8601 string. A reviewer filtering the folder by ISO prefix matched nothing on
this one and reported "no calls" for a window that held 18, silently — a string
compared to an int is just False. `ts` keeps its type, because devhealth's
load_callqos_legs() floats it on every line and post_alert() keys the card id on
it. `ts_iso` is added beside it, derived from the SAME integer so the two cannot
drift across a second boundary. Record schema v4 -> v5, documented in the version
block.

FIX 3 — a misdial could be chosen as the leg that judges cordless audio.

devhealth's matcher took the nearest ledger leg, and a zero-duration misdial
(quality "unreachable", hcause 34, billsec 0, mes_tx null — live: 2026-09-15
12:50:54Z, 13:11:49Z, 13:11:52Z, all ext 19 dialling an invalid room) qualified.
When one won, there was nothing in it to agree or disagree with, the verdict read
"unmeasured", and a real degrade was suppressed by a call that never happened.
All three were dialled BY the cordless, so the own-extension preference would
have handed them the verdict even where distance did not — the filter therefore
runs before the nearest is chosen, not after.

leg_carries_no_measurement() is deliberately narrow: a leg must both be unable to
have carried audio (zero duration, or "unreachable") AND have no transmit figure.
An answered call Asterisk simply could not score — the 13:00:31Z operator leg,
mes_tx 0.0 on 18 s of audio — is still matched and still reads "unmeasured".
Playback legs are never filtered, so the playback-skip gate cannot be reopened.

Tests: three new (the real 2026-09-15 announce leg with its measured metrics plus
truncation and silent-wake-up guards; the ts/ts_iso twin read back through both
ledger writers and through devhealth's own consumer; the misdial matcher with a
real-shaped leg 10 s from a genuine one). test_playback_legs_are_recorded_but_
never_alert is rewritten and renamed — its one-way-audio block is unchanged, its
MES-59 block inverts to notify True, and the name no longer asserts a silence the
code does not keep.

Suite: 778 passed. Mutation battery on the changed guards: 11 entered, 11 killed,
0 survivors; both source files byte-identical to their backups afterwards, green
baseline before and after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e claims

An announcement that fails to originate, and a wake-up that replaces another,
both used to leave the delivery ledger looking exactly like a system that had
never been asked. Observability only — no retry, no new escalation, no push.

ANNOUNCE (2026-09-15T01:42:12Z, third occurrence of this shape). 8.4 s after an
add-on restart the webui queued an announcement to the cordless before ext 19
had re-registered; Asterisk logged `Could not create dialog to invalid URI '19'`
and the clip never played. app.py HAS a pre-flight guard written for that, and it
passed: ami.get_device_state() returns "" on any failure to read, AMI was not
answering seconds after the restart, and an empty state is deliberately not
"unreachable" so that a state-read hiccup can never silence an alarm. The
fail-open is kept. What is fixed is that it was SILENT.

  * ami.device_state_unjudged() — a third predicate over the same normalised
    string (now written once, in _norm_device_state): true when the read was
    empty or a spelling nothing classifies. It changes no behaviour; it gives
    the caller something true to record.
  * app.py writes `announce-guard-unjudged` before the Originate, carrying the
    clip, and proceeds exactly as before. Kept OUT of the reconciler's resolved
    set on purpose: treating it as a verdict would silently retire the announce
    reconciler for every announcement whose state read hiccuped.
  * `announce-originate-failed` replaces the two bare literals the announce path
    wrote on failure. They carried no clip and spelled the same two names the
    wake-up path writes for its own originates, so an announce failure could be
    joined to nothing. One row per attempt, the cause on the row as `reason`
    (ami-error | refused). The wake-up rows are untouched.

WAKE-UP (2026-09-15). A 06:20 set at 13:10:35Z was silently replaced at
13:15:09Z by a 04:00 one, 4 m 51 s before it was due — the store keeps one entry
per extension, so the ledger showed `set 06:20` and then no ring and no cancel,
which is also what a lost wake-up looks like. store.set_wakeup() now hands back
the entry it displaced (on the RETURNED dict only; a `replaced` key in the file
would chain), and record_wakeup_change() names both times on the `set` row it
already writes. One row, not two — a separate outcome would fall outside
room_changed_wakeup()'s match, so a replacement dialled on the ringing room's own
phone, which is what a snooze is, would stop counting as a change.

STALE CLAIMS, corrected at the source:
  * devhealth/poller.py called the call ledger "append-only and unbounded". It is
    neither: switchboard-callqos caps it at MAX_RECORDS = 300 and rewrites
    through os.replace, so the inode changes under any reader holding it open.
    The live file was at exactly 300 rows, oldest 2026-07-27. The header comment
    in switchboard-callqos said "append-only" too.
  * SECURITY.md's private-log figures are re-dated rather than rewritten: 2,043
    and 16 when first counted, 2,385 and 18 on 2026-09-15.
  * switchboard-callqos append_record() now states why the /data ledger goes by
    name while append_outcome() uses the O_NOFOLLOW helper for /share, rather
    than making them consistent: the threat is the host-writable /share, and the
    MAX_RECORDS cap is enforced by rewrite + os.replace(), which acts on the NAME
    — a planted link is replaced, not written through. The remaining exposure is
    the read, and it is named.

Tests: 799 pass (was 775). The announce rows are driven through the real
api_announce handler with the state guards unstubbed; the replacement is driven
through all three real setters (dial-42 AGI, dashboard POST, operator console).
Mutation battery on the changed guards: 20 entered, 20 killed, 0 survivors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t deliveries leave a record

- switchboard-callqos: the quality branch no longer masks by PLAYBACK_TAGS. A
  truncated clip cannot reach that branch (it exits at the undelivered branch,
  where ALARM_TAG decides), so the mask only ever silenced MES/loss/RTT — five
  legs in 50 days, worst an announce that played in full while the handset
  reported MES 68.4, RTT mean 287 ms, peak 883 ms. Measured noise: 1.78 -> 4.75
  cards/30 d. The one-way-audio shape exemption at :502 is unchanged.
- callqos records ts_iso beside the epoch ts (schema v5); ts keeps its type
  because devhealth floats it and post_alert keys the card id on it.
- devhealth: a leg that never connected (zero duration or 'unreachable' with no
  transmit figure) can no longer be the nearest leg that judges the cordless.
- webui: an announce whose pre-flight device-state read is unjudgeable now
  records announce-guard-unjudged and still proceeds; an originate that raises
  or is refused records announce-originate-failed with the clip.
- wakeup/store: a set that replaces a pending wake-up names the displaced time
  on the same set row (one row, so the snooze reader still matches it).
- Corrected at source: the "append-only and unbounded" ledger comments, the
  SECURITY.md /data counts (re-dated), and DOCS.md's claim that playback legs
  never raise a quality alert.

Found in the 2026-09-15/16 live log review. 802 tests; 31 mutants killed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tesseractAZ
tesseractAZ merged commit 24ab74f into main Sep 16, 2026
10 checks passed
@tesseractAZ
tesseractAZ deleted the release/0.104.0 branch September 16, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant