Skip to content

fix: answer refused and failed DNS queries instead of dropping them - #24

Merged
thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-answer-refusals
Aug 18, 2026
Merged

thedancingdeveloper merged 1 commit into
mainfrom
fix/dns-answer-refusals

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

Closes #20.

What was wrong

Both failure paths in the forwarder logged and moved on without sending the
client anything. From the client's side, silence from a refusal and silence
from a gateway that has gone away are indistinguishable, so it waits out its
own resolver timeout — glibc's default is 5s with two attempts per nameserver —
and then retries. Under load those retries arrive as a second wave against the
capacity that refused the first.

On the reported gateway that was 14,343 refusals and 229 forwarding failures
over 2h39m, every one of them silent, arriving in bursts of hundreds inside the
same millisecond. The worst bucket was the first 90 seconds of process uptime:
1,053 refusals, because the client arrived with a backlog of unanswered retries
queued from before the restart. A gateway that answered REFUSED would have shed
that backlog in one round trip.

What changed

  • REFUSED when admission control declined to try — a distinct condition
    from failure, and what RCODE 5 is for.
  • SERVFAIL when the gateway tried and could not get an answer, including
    when no upstream is configured yet.

Both responses are built from the query: same transaction ID, same question,
QR and RA set, and the OPT record echoed back when the client sent one, as RFC
6891 requires. A message that cannot be parsed is still dropped — there is no
question to echo and no transaction to answer, so there is nothing truthful to
send.

TCP gets the same treatment on both paths. A refusal there costs a read and a
write rather than a permit, on the grounds that a client which has already
opened a connection is already waiting.

Operational impact

Clients fail fast and apply their own backoff instead of blocking for seconds
on silence, which removes the retry amplification that turned an upstream
hiccup into sustained refusal. The failure also becomes visible from inside the
enrolled container: an operator sees an error to attribute slow resolution to,
where before the gateway said nothing.

No config change and no new metrics — refusals are already counted per limit in
egressy_dns_queries_refused_total.

Rollback: revert the commit. Behaviour returns to dropping; nothing
persists.

Validation

cargo fmt --all --check, cargo clippy --all-targets -- -D warnings,
cargo test --all-targets — all clean. Four new tests cover the response
shape (id, question, QR/RA, and the forwarder's own validate_response
accepting it), the two rcodes staying distinguishable, unanswerable messages
being dropped rather than guessed at, and a refused query actually arriving at
a client socket.

Both failure paths in the forwarder logged and moved on without sending
the client anything. From the client's side silence from a refusal and
silence from a gateway that has gone away are the same event, so it waits
out its own resolver timeout — glibc's default is 5s with two attempts
per nameserver — and then retries. Under load those retries arrive as a
second wave against the capacity that refused the first, which is how an
upstream hiccup turned into sustained refusal: 14,343 refusals and 229
forwarding failures on the reported gateway, every one of them silent,
arriving in bursts of hundreds within the same millisecond.

Answer instead:

* REFUSED when admission control declined to try, which is the condition
  RCODE 5 exists for;
* SERVFAIL when the gateway tried and could not get an answer, including
  when no upstream is configured yet.

Both are built from the query itself — same transaction, same question,
QR and RA set, and the OPT record echoed when the client sent one, as RFC
6891 requires. A message that cannot be parsed is still dropped: there is
no question to echo and no transaction to answer, so there is nothing
truthful to send.

TCP gets the same treatment on both paths. A refusal there costs a read
and a write rather than a permit, because a client that has already
opened a connection is already waiting.

Refs #20
@thedancingdeveloper
thedancingdeveloper merged commit 9574e9e into main Aug 18, 2026
6 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the fix/dns-answer-refusals branch August 18, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DNS forwarder drops refused and failed queries silently instead of answering, provoking client retry storms

1 participant