Skip to content

fix: debounce DNS health over time, and let refusals reach the verdict - #26

Merged
thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-health-debounce
Aug 18, 2026
Merged

thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-health-debounce

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

Closes #22. Last of the four fixes from the same incident (#19, #20, #21).

What was wrong

The debounce was counted in queries, not time. failure_threshold 3 and
success_threshold 2 are counts. At the ~31 qps the reported gateway served
that is about 100ms to go degraded and 65ms to come back, so a burst of
upstream failures became a state change and back before anything could observe
it — 23 degraded episodes, every one returning to healthy in the same second
and 15 of them within the same timestamp:

02:31:08 healthy -> degraded  dns.upstream_udp_failures
02:31:08 degraded -> healthy  dns.udp_healthy

egressy_dns_resolution_healthy is a gauge read at scrape time, so a state
that exists for under a second between scrapes 15-60s apart is never sampled:
it read 1 throughout the incident. The self-cancelling pairs also occupied 46
of the 200 bounded transition slots, pushing the first 40 minutes of uptime out
of the window.

The dominant failure never reached the health model at all.
observe_udp_health was only called from the worker, after admission, so a
query refused by the concurrency limit returned before any worker was spawned
and contributed nothing. That was 14,343 refusals against 229 forwarding
failures — the largest failure population by two orders of magnitude, invisible
to the check by construction. A client whose queries were all being refused had
no working DNS, and every signal the gateway published said healthy.

What changed

  • A degraded verdict is held for 60s before a recovery may clear it, so an
    episode outlives a scrape interval and the transition history records state
    that meant something. Entering degraded is unchanged — detection stays fast,
    recovery becomes deliberate. Sub-second cancelling pairs can no longer be
    recorded, which is the third bullet of the issue handled by construction.
  • Admission refusals feed the verdict alongside upstream failures. The cause is
    fixed for the length of an episode and reported as dns.queries_refused or
    dns.upstream_udp_failures, so the reason code cannot alternate while the
    check stays degraded and churn the history a second way.
  • An unchanged verdict is re-published at most every 30s rather than on every
    query. At 31 qps the old path was a snapshot clone and a broadcast per query,
    all saying the same thing.

Operational impact

egressy_dns_resolution_healthy and /api/v2/status now actually report
resolver trouble, which is what the gauge added in #12 was for. Expect the
check to spend real time degraded on a gateway that is genuinely struggling —
that is the point — and alerts keyed on it to fire where they previously could
not.

Readiness is unaffected: readiness treats Degraded as ready and only
Starting/Unavailable/Failed block, so a held degraded verdict cannot mark
the container unhealthy or change protection. Aggregate availability will
read Degraded during an episode, which is the intended visibility.

Rollback: revert the commit; the verdict returns to per-query debouncing.
No config or contract change — the check id dns.upstream_udp and the gauge
are unchanged, and dns.queries_refused is a new reason-code value on an
existing free-form field.

Validation

cargo fmt --all --check, cargo clippy --all-targets -- -D warnings,
cargo test --all-targets — all clean. The single count-based health test is
replaced by five: consecutive failures still required before degrading, a
degraded verdict outliving the burst that caused it, refusals degrading the
verdict on their own, the cause staying fixed across an episode, and an
unchanged verdict not being republished per query.

`failure_threshold` and `success_threshold` are counts of queries, not
durations. At the ~31 qps the reported gateway served, three failures is
about 100ms and two successes about 65ms, so the debounce converted a
burst of upstream failures into a state change and back before anything
could observe it: 23 degraded episodes, every one back to healthy in the
same second and 15 of them within the same timestamp. A gauge read at
scrape time never sampled one, so `egressy_dns_resolution_healthy` read 1
throughout an incident, and the self-cancelling pairs evicted 46 of the
200 bounded transition slots, pushing the first 40 minutes of uptime out
of the window.

Hold a degraded verdict for a minute before a recovery may clear it, so
an episode outlives a scrape interval and the history records state that
meant something. Entering degraded is unchanged: detection stays fast,
recovery becomes deliberate.

Refusals now reach the verdict too. `observe_udp_health` was only called
from the worker, after admission, so a query refused by the concurrency
limit contributed nothing — the largest failure population on that
gateway, 14,343 refusals against 229 forwarding failures, was invisible
to the check by construction. A client whose queries are all refused has
no working DNS, and every signal the gateway published said healthy. The
cause is fixed for the length of an episode and reported as
`dns.queries_refused` or `dns.upstream_udp_failures`, so the reason code
does not alternate while the check stays degraded.

An unchanged verdict is now re-published at most every 30 seconds rather
than on every query, which at 31 qps was a snapshot clone and a broadcast
per query, all of them saying the same thing.

Refs #22
@thedancingdeveloper
thedancingdeveloper merged commit 1121dfd into main Aug 18, 2026
6 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the fix/dns-health-debounce branch August 18, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DNS health debounce is counted in queries, not time, so resolver failures flap sub-second and are never observable

1 participant