fix: end a DNS health episode when the trouble stops, not when the dwell does - #32
Merged
Merged
Conversation
…ell does #22 gave the resolution verdict a time-based dwell, and an episode became observable instead of cancelling inside the same timestamp. The clearing condition either side of it was still counted in queries, and it turned out to set the shape of everything the check reported. Recovery needed `success_threshold` consecutive successes, which is about a second of clean operation on a gateway whose health path sees roughly two events a second. During a brownout where even a third of queries fail, a run of two successes turns up almost immediately, so the dwell expiring is what ended every episode. Measured over 4.6h of soak, 76 episodes: median 64.5s, min 60.0s, max 80.5s, against a 60s dwell. The distribution describes the constant, not the trouble. Re-degrading then needed three consecutive failures, about 1.6s, so a continuous problem was published as 60s degraded followed by 2.5 minutes healthy, over and over, and the gateway read healthy through most of a brownout. Clear on a ratio instead: at most one failure in eight across the last 128 outcomes, and never off fewer than 32 of them. While failures keep arriving at a rate a working resolver would not produce, the episode continues, so its duration measures how long the trouble lasted. The dwell stays as a floor on episode length — it is what made an episode observable — and detection stays count-based, because that half was working and fast. The ratio has to tolerate a background: the observed gateway runs at about 3% failures with a healthy upstream, and a rule demanding none would leave it degraded forever. Refs #28
thedancingdeveloper
force-pushed
the
fix/dns-health-recovery-rate
branch
from
August 19, 2026 02:56
0872de6 to
7b644ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #28. Stacked on #31 (both touch
src/dns.rs).What was wrong
#22 gave the resolution verdict a time-based dwell and episodes became
observable. The clearing condition either side of it was still counted in
queries, and it turned out to set the shape of everything the check reported.
Recovery needed
success_thresholdconsecutive successes — about a second ofclean operation on a gateway whose health path sees ~2 events/s (cache hits
return before admission and never reach it). During a brownout where even a
third of queries fail, a run of two successes turns up almost immediately, so
the dwell expiring is what ended every episode. 76 episodes over 4.6h of
soak:
Every one within 20.5s of the constant. That distribution describes
DEGRADED_DWELL, not the trouble.Re-degrading then needed only 3 consecutive failures (~1.6s), so a continuous
problem was published as ~60s degraded followed by ~2.5 minutes healthy, on
repeat — 16.4 episodes/hour. The gateway read healthy through most of a
brownout.
What changed
Clearing is now a ratio: at most one failure in eight across the last 128
outcomes, and never read off fewer than 32 of them. While failures keep arriving
at a rate a working resolver would not produce, the episode continues — so its
duration measures how long the trouble lasted.
The window is a
u128bitmask, so this is a shift and acount_ones()perquery with no allocation and no timer.
Unchanged deliberately:
observable in the first place.
success_thresholdstill applies, so the documented knob keeps its meaning.Why the ratio tolerates failures at all
The measured gateway carries ~3% failures with a healthy upstream — the provider
resolver has a constant background, roughly one exhausted query every 18s. A
rule demanding a clean window would leave the check degraded forever. One in
eight sits well above that background and well below a brownout.
Operational impact
Degraded duration becomes meaningful, so "degraded for longer than N minutes"
becomes a usable alert where before it could never fire. Expect fewer, longer
episodes: a sustained brownout will now show as one long episode instead of a
train of 60s ones. Readiness is unaffected —
Degradedstill reads ready.Rollback: revert the commit; clearing returns to consecutive successes.
Validation
cargo fmt --all --check,cargo clippy --all-targets -- -D warnings,cargo test --all-targets— clean, 204 tests. Four new/rewritten tests: asustained failure rate holding the verdict degraded well past the dwell and
clearing only when the trouble stops, a healthy gateway's background still
recovering, a handful of outcomes not clearing a verdict, and the original dwell
test kept intact.