open_path() with explicit local_ip: PATH_RESPONSE never reaches on_path_response_received, path stuck ValidationFailed

[cuzic]

Summary

When a client opens a new multipath Connection::open_path() with an explicit local_ip (FourTuple::new(remote, Some(ip))), while another path to the same remote address is already validated with local_ip: None, the new path's PATH_CHALLENGE is sent and retransmitted correctly (per RFC 9002 PTO), but PathState::on_path_response_received is never invoked for that path — even though raw UDP data is confirmed arriving at the correct local socket. The path eventually gets abandoned with ValidationFailed.

This reproduces reliably on a real Android device with real network conditions (WiFi + cellular), independent of:

• whether the new path targets the same remote address as the already-validated path, or a completely different one (tested IPv4 vs IPv6)
• whether the new path is opened concurrently with other path-opening attempts, or fully serialized after everything else has settled

Versions

• noq / noq-proto / noq-udp: 1.0.1 (also reproduced by instrumenting the noq-proto-v1.0.1 tag directly)
• Client: Android (aarch64), runtime-tokio, custom AsyncUdpSocket/UdpSender implementation (see "Setup" below)
• Server: Linux x86_64, noq::Endpoint::server() with a plain UDP socket

Setup

• One QUIC connection, multipath negotiated, client side.
• Path 0 (PathId::ZERO): established via Endpoint::connect(remote_a, ...).
• Path "Secondary": opened via Connection::open_path(FourTuple::new(remote_a, None), PathStatus::Available) — i.e. same remote as path 0, no explicit local IP. This one validates successfully.
• Path "Physical": opened via Connection::open_path(FourTuple::new(remote_a_or_b, Some(specific_local_ip)), PathStatus::Available), where the socket used to send/receive on this path is a separate UdpSocket explicitly bound to specific_local_ip (via Android's Network.bindSocket(), fd handed to Rust via UdpSocket::from_raw_fd). This one never validates.

The client-side AsyncUdpSocket implementation is a thin multiplexer over a "default" socket (bound 0.0.0.0:0, used when Transmit::src_ip is None) and N "named" sockets (used when Transmit::src_ip matches their bound local IP). poll_recv tries the default socket first, then each named socket, setting RecvMeta::dst_ip accordingly (None for default, Some(local_ip) for named). This mirrors the pattern shown in the crate's own multipath examples.

What we tried to rule out

1. Same remote address for both paths. Changed the "Physical" candidate's remote to a completely different address (moved from IPv4 to a distinct globally-routable IPv6 address on the same server, verified independently reachable via raw UDP send/recv). Still failed identically.
2. Concurrent path opens racing for CIDs / anti-amplification budget. Serialized path opening so that "Physical" only starts after "Secondary" has fully validated and no other open_path() call is in flight. Still failed identically, including for a single physical candidate with nothing else concurrently opening.
3. MTU / fragmentation. Instrumented the AsyncUdpSocket/UdpSender implementation directly: confirmed 1200-byte PATH_CHALLENGE-sized datagrams are sent from the named socket and 1200-byte responses are received back on the same named socket with the correct dst_ip, repeatedly, for several seconds — i.e. raw UDP round-trips are working.

Diagnostic evidence (client-side, from a temporarily patched noq-proto-v1.0.1)

We added logging to ensure_path, record_path_challenge_sent, and on_path_response_received (writing directly to a file since eprintln!/tracing weren't visible in our Android test harness). One real-device trace (direct_host scenario, path "Secondary" plus one physical candidate "PhysicalWifi" retried 3 times before giving up):

ensure_path validated=true  path_id=PathId(1) network_path=(local: <unspecified>, remote: 204.12.203.210:45823)
record_path_challenge_sent  token=8310967745856842139  network_path=(local: <unspecified>, remote: 204.12.203.210:45823) self.network_path=(local: <unspecified>, remote: 204.12.203.210:45823)
on_path_response_received   token=8310967745856842139  sent_network_path=(local: <unspecified>, ...) current_network_path=(local: <unspecified>, ...) is_probably_same_path=true

ensure_path validated=false path_id=PathId(2) network_path=(local: 192.168.10.80, remote: 204.12.203.210:45823)
record_path_challenge_sent  token=11718703244394908866 network_path=(local: 192.168.10.80, ...) self.network_path=(local: 192.168.10.80, ...)
record_path_challenge_sent  token=4528551501049309925  network_path=(local: 192.168.10.80, ...) self.network_path=(local: 192.168.10.80, ...)
record_path_challenge_sent  token=13872093659869132038 network_path=(local: 192.168.10.80, ...) self.network_path=(local: 192.168.10.80, ...)
[... PathId(3), PathId(4) — two more retries, same pattern, three record_path_challenge_sent calls each ...]

Across the whole trace: record_path_challenge_sent is called 19 times across PathId(2), PathId(3), PathId(4) (the three retry attempts for the physical candidate), but on_path_response_received is called exactly twice, both for PathId(1) (the local_ip: None path) and never once for any of the local_ip: Some(192.168.10.80) paths.

This is a lower-level symptom than a is_probably_same_path mismatch (PathState::network_path field comparison) — the response-handling function is never reached at all for these paths' tokens, despite:

• the challenges genuinely being sent (repeated retransmission on PTO expiry is the expected behavior when no response arrives, which is consistent with what we see)
• raw UDP responses genuinely arriving at the correct socket (per the earlier AsyncUdpSocket-level instrumentation)

This points to something between "bytes are handed to AsyncUdpSocket::poll_recv's caller" and "the PATH_RESPONSE frame reaches PathState::on_path_response_received" — e.g., a decrypt/CID-demux step silently dropping or misrouting datagrams for paths whose network_path.local_ip is Some(..), or an issue in how packets for a given PathId get matched when three sequential retries (PathId(2), (3), (4)) share largely the same underlying local/remote addressing but different CIDs.

Reproduction notes

We don't yet have a minimal, self-contained reproduction using the crate's own test harness (ConnPair/ManyToManyRouting etc.) — our reproduction is on a real Android device over real WiFi/cellular NAT paths, which we can't easily hand you directly. We're happy to help build a minimal repro if a maintainer can point us to the most likely suspect (packet decode / CID lookup path for a freshly-opened PathId with an explicit local_ip), or if there's a known limitation here we should be aware of (e.g. explicit-local_ip paths needing something we're not providing, like a prior add_nat_traversal_address/NAT-traversal negotiation, which we are not using).

Happy to share the (temporary, throwaway) diagnostic patch if useful.

[flub]

A reproducer in a -proto test would be nice. From the description I think the ConnPair with the default BasicRouting should be able to produce this.

I also think that on trace-level logging no packets are completely silently discarded, so you should be able to rule this out.

Do you see the received path response frame logged at all? On verbose logging it should probably log a PathResponse and maybe also as PATH_RESPONSE.

[divagant-martian]

Could you provide TRACE level logs for this? I suspect we might get more clarity about this from them. I suspect this relates to mapped addresses. From looking at the code this is a bug we do need to fix as dual stack instances might report back ipv6 addresses even if they ultimately are ipv4 ones. But i'd like to confirm this issue does relate to my findings