fix: answer refused and failed DNS queries instead of dropping them - #24
Merged
Merged
Conversation
Both failure paths in the forwarder logged and moved on without sending the client anything. From the client's side silence from a refusal and silence from a gateway that has gone away are the same event, so it waits out its own resolver timeout — glibc's default is 5s with two attempts per nameserver — and then retries. Under load those retries arrive as a second wave against the capacity that refused the first, which is how an upstream hiccup turned into sustained refusal: 14,343 refusals and 229 forwarding failures on the reported gateway, every one of them silent, arriving in bursts of hundreds within the same millisecond. Answer instead: * REFUSED when admission control declined to try, which is the condition RCODE 5 exists for; * SERVFAIL when the gateway tried and could not get an answer, including when no upstream is configured yet. Both are built from the query itself — same transaction, same question, QR and RA set, and the OPT record echoed when the client sent one, as RFC 6891 requires. A message that cannot be parsed is still dropped: there is no question to echo and no transaction to answer, so there is nothing truthful to send. TCP gets the same treatment on both paths. A refusal there costs a read and a write rather than a permit, because a client that has already opened a connection is already waiting. Refs #20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #20.
What was wrong
Both failure paths in the forwarder logged and moved on without sending the
client anything. From the client's side, silence from a refusal and silence
from a gateway that has gone away are indistinguishable, so it waits out its
own resolver timeout — glibc's default is 5s with two attempts per nameserver —
and then retries. Under load those retries arrive as a second wave against the
capacity that refused the first.
On the reported gateway that was 14,343 refusals and 229 forwarding failures
over 2h39m, every one of them silent, arriving in bursts of hundreds inside the
same millisecond. The worst bucket was the first 90 seconds of process uptime:
1,053 refusals, because the client arrived with a backlog of unanswered retries
queued from before the restart. A gateway that answered REFUSED would have shed
that backlog in one round trip.
What changed
from failure, and what RCODE 5 is for.
when no upstream is configured yet.
Both responses are built from the query: same transaction ID, same question,
QR and RA set, and the OPT record echoed back when the client sent one, as RFC
6891 requires. A message that cannot be parsed is still dropped — there is no
question to echo and no transaction to answer, so there is nothing truthful to
send.
TCP gets the same treatment on both paths. A refusal there costs a read and a
write rather than a permit, on the grounds that a client which has already
opened a connection is already waiting.
Operational impact
Clients fail fast and apply their own backoff instead of blocking for seconds
on silence, which removes the retry amplification that turned an upstream
hiccup into sustained refusal. The failure also becomes visible from inside the
enrolled container: an operator sees an error to attribute slow resolution to,
where before the gateway said nothing.
No config change and no new metrics — refusals are already counted per limit in
egressy_dns_queries_refused_total.Rollback: revert the commit. Behaviour returns to dropping; nothing
persists.
Validation
cargo fmt --all --check,cargo clippy --all-targets -- -D warnings,cargo test --all-targets— all clean. Four new tests cover the responseshape (id, question, QR/RA, and the forwarder's own
validate_responseaccepting it), the two rcodes staying distinguishable, unanswerable messages
being dropped rather than guessed at, and a refused query actually arriving at
a client socket.