Skip to content

fix: end a DNS health episode when the trouble stops, not when the dwell does - #32

Merged
thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-health-recovery-rate
Aug 19, 2026
Merged

thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-health-recovery-rate

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

Closes #28. Stacked on #31 (both touch src/dns.rs).

What was wrong

#22 gave the resolution verdict a time-based dwell and episodes became
observable. The clearing condition either side of it was still counted in
queries, and it turned out to set the shape of everything the check reported.

Recovery needed success_threshold consecutive successes — about a second of
clean operation on a gateway whose health path sees ~2 events/s (cache hits
return before admission and never reach it). During a brownout where even a
third of queries fail, a run of two successes turns up almost immediately, so
the dwell expiring is what ended every episode. 76 episodes over 4.6h of
soak:

min 60.0s   median 64.5s   p90 73.0s   max 80.5s      (dwell = 60s)

Every one within 20.5s of the constant. That distribution describes
DEGRADED_DWELL, not the trouble.

Re-degrading then needed only 3 consecutive failures (~1.6s), so a continuous
problem was published as ~60s degraded followed by ~2.5 minutes healthy, on
repeat — 16.4 episodes/hour. The gateway read healthy through most of a
brownout.

What changed

Clearing is now a ratio: at most one failure in eight across the last 128
outcomes, and never read off fewer than 32 of them. While failures keep arriving
at a rate a working resolver would not produce, the episode continues — so its
duration measures how long the trouble lasted.

The window is a u128 bitmask, so this is a shift and a count_ones() per
query with no allocation and no timer.

Unchanged deliberately:

  • Detection stays count-based. It is fast and that half was working.
  • The dwell stays as a floor on episode length; it is what made an episode
    observable in the first place.
  • success_threshold still applies, so the documented knob keeps its meaning.

Why the ratio tolerates failures at all

The measured gateway carries ~3% failures with a healthy upstream — the provider
resolver has a constant background, roughly one exhausted query every 18s. A
rule demanding a clean window would leave the check degraded forever. One in
eight sits well above that background and well below a brownout.

Operational impact

Degraded duration becomes meaningful, so "degraded for longer than N minutes"
becomes a usable alert where before it could never fire. Expect fewer, longer
episodes: a sustained brownout will now show as one long episode instead of a
train of 60s ones. Readiness is unaffected — Degraded still reads ready.

Rollback: revert the commit; clearing returns to consecutive successes.

Validation

cargo fmt --all --check, cargo clippy --all-targets -- -D warnings,
cargo test --all-targets — clean, 204 tests. Four new/rewritten tests: a
sustained failure rate holding the verdict degraded well past the dwell and
clearing only when the trouble stops, a healthy gateway's background still
recovering, a handful of outcomes not clearing a verdict, and the original dwell
test kept intact.

…ell does

#22 gave the resolution verdict a time-based dwell, and an episode became
observable instead of cancelling inside the same timestamp. The clearing
condition either side of it was still counted in queries, and it turned
out to set the shape of everything the check reported.

Recovery needed `success_threshold` consecutive successes, which is about
a second of clean operation on a gateway whose health path sees roughly
two events a second. During a brownout where even a third of queries
fail, a run of two successes turns up almost immediately, so the dwell
expiring is what ended every episode. Measured over 4.6h of soak, 76
episodes: median 64.5s, min 60.0s, max 80.5s, against a 60s dwell. The
distribution describes the constant, not the trouble. Re-degrading then
needed three consecutive failures, about 1.6s, so a continuous problem
was published as 60s degraded followed by 2.5 minutes healthy, over and
over, and the gateway read healthy through most of a brownout.

Clear on a ratio instead: at most one failure in eight across the last
128 outcomes, and never off fewer than 32 of them. While failures keep
arriving at a rate a working resolver would not produce, the episode
continues, so its duration measures how long the trouble lasted. The
dwell stays as a floor on episode length — it is what made an episode
observable — and detection stays count-based, because that half was
working and fast.

The ratio has to tolerate a background: the observed gateway runs at
about 3% failures with a healthy upstream, and a rule demanding none
would leave it degraded forever.

Refs #28
@thedancingdeveloper
thedancingdeveloper force-pushed the fix/dns-health-recovery-rate branch from 0872de6 to 7b644ff Compare August 19, 2026 02:56
@thedancingdeveloper
thedancingdeveloper merged commit df72df4 into main Aug 19, 2026
6 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the fix/dns-health-recovery-rate branch August 19, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DNS health recovers on two consecutive successes, so sustained upstream trouble is published as a train of 60-second episodes

1 participant