feat: Netlink ARP/NDP monitor — proper MAC resolution for provider/ECMP deployments

[redlemonbe]

Context

Current XDP TX path uses naive MAC inversion (swap src/dst from incoming frame). This works for 95% of use cases (direct LAN, single-hop deployments) but fails in provider environments with:

• Intermediate routers / load balancers between client and server
• ECMP (Equal-Cost Multi-Path) routing — response path ≠ request path
• Asymmetric routing — packet arrives via Router A, should leave via Router B

Problem

Naive MAC inversion in an ECMP/provider setup sends responses to a black hole.

Solution — Netlink RTMGRP_NEIGH monitor + ArcSwap

Architecture

Kernel ARP/NDP table
        │
        │ NETLINK_ROUTE (RTMGRP_NEIGH)
        ▼
Background Netlink monitor thread
        │
        │ ArcSwap atomic update
        ▼
MAC lookup table (HashMap<IpAddr, MacAddr>)
        │
        │ lock-free read
        ▼
XDP TX fast path — no syscall, no ARP request

How it works

1. On startup: open NETLINK_ROUTE socket in monitoring mode (RTMGRP_NEIGH)
2. Background thread receives kernel events on every ARP/NDP create/update/delete
3. Updates HashMap<IpAddr, MacAddr> via ArcSwap — zero contention with fast path
4. XDP TX path does mac_table.get(next_hop_ip) — single lock-free lookup, no syscall

Next-hop resolution for ECMP

For provider deployments with asymmetric routing:

• Query kernel via ip route get <client_ip> (Netlink RTM_GETROUTE) to find next-hop
• Store next-hop MAC in response cache at ArcSwap insert time
• On route change (Netlink RTM_NEWROUTE/RTM_DELROUTE) → invalidate affected entries

Passive MAC learning (complementary)

Extract (src_ip, src_mac) from every incoming XDP frame to pre-populate the table before the kernel ARP entry exists. Zero overhead — data is already in the UMEM buffer.

Fallback

If MAC not found in table → XDP_PASS to kernel for normal UDP response path. Silent, no packet drop.

Current behaviour

Naive src/dst MAC swap — works on direct L2 segments, fails with intermediate routing equipment.

Target deployments unlocked

• ISP/provider recursive resolvers behind routers
• Anycast nodes behind ECMP load balancers (feat: anycast-ready deployment support — health thresholds, graceful drain, node identity, PROXY protocol #21)
• Multi-PoP deployments with asymmetric routing

Related

• feat: anycast-ready deployment support — health thresholds, graceful drain, node identity, PROXY protocol #21 — anycast deployment (ECMP is standard in anycast infrastructure)
• feat: zero-alloc DNS fast path — XDP hot path bypass with Hickory slow path fallback #26 — zero-alloc fast path (Netlink table lookup must be zero-alloc on hot path)

[redlemonbe]

Implementation pattern — async Netlink observer, never block the fast path

The anti-pattern to avoid

Request → MAC cache miss → Netlink call → wait → respond

This destroys latency. Never call Netlink synchronously from the XDP worker.

The correct pattern

Dedicated background thread — listens exclusively on the Netlink socket, receives neighbor table updates (RTM_NEWNEIGH / RTM_DELNEIGH), pushes to ArcSwap<HashMap<IpAddr, MacAddr>>.

Read-only fast path — XDP worker only reads the cache. On miss → XDP_PASS immediately, no syscall, no wait.

Progressive warm-up (Day 1 → Day 3)

State	Behavior
Cache cold (Day 1)	Everything via XDP_PASS — correct, kernel handles it
Cache warming (Day 2)	Netlink populates table in background, hit rate grows
Cache warm (Day 3)	Fast path handles majority of traffic zero-copy

Resilience guarantee

If the Netlink thread crashes or the cache is incomplete — the network keeps working via XDP_PASS fallback. No interruption, no packet loss. The fast path is an acceleration layer, not a critical dependency.

This transforms the MAC resolution from a hard dependency into a progressive optimization.