fix: debounce DNS health over time, and let refusals reach the verdict - #26
Merged
Merged
Conversation
`failure_threshold` and `success_threshold` are counts of queries, not durations. At the ~31 qps the reported gateway served, three failures is about 100ms and two successes about 65ms, so the debounce converted a burst of upstream failures into a state change and back before anything could observe it: 23 degraded episodes, every one back to healthy in the same second and 15 of them within the same timestamp. A gauge read at scrape time never sampled one, so `egressy_dns_resolution_healthy` read 1 throughout an incident, and the self-cancelling pairs evicted 46 of the 200 bounded transition slots, pushing the first 40 minutes of uptime out of the window. Hold a degraded verdict for a minute before a recovery may clear it, so an episode outlives a scrape interval and the history records state that meant something. Entering degraded is unchanged: detection stays fast, recovery becomes deliberate. Refusals now reach the verdict too. `observe_udp_health` was only called from the worker, after admission, so a query refused by the concurrency limit contributed nothing — the largest failure population on that gateway, 14,343 refusals against 229 forwarding failures, was invisible to the check by construction. A client whose queries are all refused has no working DNS, and every signal the gateway published said healthy. The cause is fixed for the length of an episode and reported as `dns.queries_refused` or `dns.upstream_udp_failures`, so the reason code does not alternate while the check stays degraded. An unchanged verdict is now re-published at most every 30 seconds rather than on every query, which at 31 qps was a snapshot clone and a broadcast per query, all of them saying the same thing. Refs #22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22. Last of the four fixes from the same incident (#19, #20, #21).
What was wrong
The debounce was counted in queries, not time.
failure_threshold3 andsuccess_threshold2 are counts. At the ~31 qps the reported gateway servedthat is about 100ms to go degraded and 65ms to come back, so a burst of
upstream failures became a state change and back before anything could observe
it — 23 degraded episodes, every one returning to healthy in the same second
and 15 of them within the same timestamp:
egressy_dns_resolution_healthyis a gauge read at scrape time, so a statethat exists for under a second between scrapes 15-60s apart is never sampled:
it read 1 throughout the incident. The self-cancelling pairs also occupied 46
of the 200 bounded transition slots, pushing the first 40 minutes of uptime out
of the window.
The dominant failure never reached the health model at all.
observe_udp_healthwas only called from the worker, after admission, so aquery refused by the concurrency limit returned before any worker was spawned
and contributed nothing. That was 14,343 refusals against 229 forwarding
failures — the largest failure population by two orders of magnitude, invisible
to the check by construction. A client whose queries were all being refused had
no working DNS, and every signal the gateway published said healthy.
What changed
episode outlives a scrape interval and the transition history records state
that meant something. Entering degraded is unchanged — detection stays fast,
recovery becomes deliberate. Sub-second cancelling pairs can no longer be
recorded, which is the third bullet of the issue handled by construction.
fixed for the length of an episode and reported as
dns.queries_refusedordns.upstream_udp_failures, so the reason code cannot alternate while thecheck stays degraded and churn the history a second way.
query. At 31 qps the old path was a snapshot clone and a broadcast per query,
all saying the same thing.
Operational impact
egressy_dns_resolution_healthyand/api/v2/statusnow actually reportresolver trouble, which is what the gauge added in #12 was for. Expect the
check to spend real time degraded on a gateway that is genuinely struggling —
that is the point — and alerts keyed on it to fire where they previously could
not.
Readiness is unaffected:
readinesstreatsDegradedas ready and onlyStarting/Unavailable/Failedblock, so a held degraded verdict cannot markthe container unhealthy or change protection. Aggregate
availabilitywillread
Degradedduring an episode, which is the intended visibility.Rollback: revert the commit; the verdict returns to per-query debouncing.
No config or contract change — the check id
dns.upstream_udpand the gaugeare unchanged, and
dns.queries_refusedis a new reason-code value on anexisting free-form field.
Validation
cargo fmt --all --check,cargo clippy --all-targets -- -D warnings,cargo test --all-targets— all clean. The single count-based health test isreplaced by five: consecutive failures still required before degrading, a
degraded verdict outliving the burst that caused it, refusals degrading the
verdict on their own, the cause staying fixed across an episode, and an
unchanged verdict not being republished per query.