feat(lane-alarm): a lane that breaks a third of the time is broken, however short each break is - #11495
Merged
Conversation
…owever short each break is The escalation rule is a RUN: the unbroken tail of one verdict, measured against a one-hour minimum. A lane that recovers between episodes never builds one, however often it breaks. Measured 2026-08-19: `chain-detail-staleness` sat 851s behind a 300s threshold for 12.5 minutes, recorded `stale` in lane_health -- verdict, age_ms and all -- and opened nothing, because no single episode lasted the 45 minutes three consecutive quarter-hourly ticks would need. `chain_detail`'s own write-latency tail says that is not rare: p95 718s, p99 969s, and metagraphed-infra#492 has it at p99 1244s a week earlier. So a lane can be past its threshold for minutes, several times a day, forever, and the issue tracker stays empty. A second rule now runs beside the first: faulty on at least a third of a lane's ticks over 24 hours. Reported as its own kind rather than folded into `stale`, because it is a different finding -- the lane is not down, it is unreliable, and an issue that says "stale on every tick" about a lane that is fine most of the time would be false. Three gates keep it from becoming the noise this file exists to prevent: - TWELVE SAMPLES MINIMUM. A ratio over three ticks is an anecdote: a daily lane that logged twice, once badly, would read as 50% faulty. - A THIRD. This file already documents the noise floor -- `chain-detail` flicked stale 64 times in a day while healthy -- and a third sits above a deploy blip and below the case this was written for. - NOT ALREADY IN A RUN. A lane in a long unbroken stale run is also 100% faulty by this measure; the run rule describes it better, and two issues for one fault is exactly what this file's own comments call the failure that gets an alerting system muted. It counts BOTH fault verdicts, because a lane alternating `stale` and `unknown` is no more trustworthy than one doing either consistently -- and the run rule, keyed on a single verdict, sees two short runs and alarms on neither. `faultRates` is optional on the plan input, so every existing caller keeps exactly its previous behaviour, and the exhaustive `LaneAlarmKind` switch made the compiler demand the new message rather than letting it default. The test fixture needed a fix to be honest: the rate SQL also selects `MIN(checked_at) AS since`, so a substring match on that alone handed run rows to the rate reader and would have made every flapping test a silent no-op. Closes #11488
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-wss-lb | 2c1c08e | Aug 19 2026, 12:03 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 2c1c08e | Aug 19 2026, 12:03 PM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 2c1c08e | Aug 19 2026, 12:03 PM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11495 +/- ##
=======================================
Coverage 95.65% 95.65%
=======================================
Files 788 788
Lines 46926 46959 +33
Branches 17236 17251 +15
=======================================
+ Hits 44886 44919 +33
Misses 527 527
Partials 1513 1513
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The escalation rule is a run — the unbroken tail of one verdict, measured against a one-hour minimum. A lane that recovers between episodes never builds one, however often it breaks.
The case
Measured 2026-08-19,
chain-detail-staleness:851s against a 300s threshold — 2.8x over — recorded faithfully, escalated nowhere, because no single episode lasted the 45 minutes three consecutive quarter-hourly ticks would need.
That is not a rare shape.
chain_detail's own write-latency tail is p95 718s / p99 969s, and metagraphed-infra#492 measured p99 1244s a week earlier. So a lane can sit past its threshold for minutes, several times a day, forever, and the tracker stays empty.The second rule
Faulty on at least a third of a lane's ticks over 24 hours. Its own kind, not folded into
stale, because it is a different finding — the lane is not down, it is unreliable, and an issue saying "stale on every tick" about a lane that is fine most of the time would be false.Three gates against noise
chain-detailflicked stale 64 times in a day while healthy — so the threshold has to sit above a deploy blip and below the case this was written for.It counts both fault verdicts: a lane alternating
staleandunknownis no more trustworthy than one doing either consistently, and the run rule — keyed on a single verdict — sees two short runs and alarms on neither.Compatibility
faultRatesis optional on the plan input, so every existing caller keeps exactly its previous behaviour; a test pins that. The exhaustiveLaneAlarmKindswitch made the compiler demand the new message rather than letting it fall through — which is the property its comment already claimed and now has been exercised.A fixture bug this surfaced
The rate SQL also selects
MIN(checked_at) AS since, which the test fixture matched on to serve run rows. Left alone it would have handed run rows to the rate reader and made every flapping test a silent no-op. The fixture now matches the rate query first, on a substring unique to it.Verification
Full suite 960 files / 22,124 passed, build, lint, format, all 72 CI validators, patch coverage 100% (30/30 lines, 31/31 branches). 18 new tests.
Closes #11488