accurately detect default gateways

[aojea]

Address several critical edge cases in default interface detection and introduces a robust, rootless networking test framework.

1. Point-to-Point Interfaces: Removed the r.Gw == nil check. Previously, this caused the agent to completely ignore active VPNs and tunnels (like Wireguard or tun/tap) because P2P links route directly out of the device without a Gateway IP.
2. Route Metrics: The kernel relies on metrics (Priority) to determine the active route in a multi-WAN setup. The old code ignored this, returning all interfaces. It now correctly isolates the lowest metric for IPv4 and IPv6 independently, while preserving ECMP support.
3. Multipath Link Lookup: Fixed a bug where multipath routes were queried using the parent route's link index (which is often 0) rather than the nexthop's link index.

[aojea]

/assign @gauravkghildiyal @dkennetzoracle @tamilmani1989

[k8s-ci-robot]

@aojea: GitHub didn't allow me to assign the following users: dkennetzoracle, tamilmani1989.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @gauravkghildiyal @dkennetzoracle @tamilmani1989

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dkennetzoracle]

Some nits / questions, but I think it's a great add and a correctness fix. I'm not 100% sure it fixes the NonUplinkChecker situation on #138 on OKE, I'd need to double check. RA injects IPv6 defaults for us. If they share a metric (which I think they do) I still need something like NonUplinkChecker. If they don't I can drop NonUplinkChecker.

This definitely addresses one of the sub-issues in that PR, and is correct. LGTM!

[gauravkghildiyal]

I'm not 100% sure it fixes the NonUplinkChecker situation on #138 on OKE, I'd need to double check.

@dkennetzoracle I'm very much interested in hearing about this after you've had the opportunity to try this out

[dkennetzoracle]

@gauravkghildiyal - I will let you know! It's very hard for me to get access to GB200+ so I need to make the most of my time on them haha. I'm pretty sure each NIC gets a default route for ipv6 and these use the kernel's RA priority (so 1024) so we wouldn't be able to differentiate the 8 RDMA NICs from each other by metric. They all look like peer's from he kernel's pov.

So they'd all get excluded still here, haha. But it should work for the Azure case

[anson627]

I just verified this PR on AWS p4d.24large, the primary NiC with default gateway/route (e.g. ens32) is excluded, and 4 EFA/ENA devices are properly included in the resource slice:

Device	Type	Interface	PCI Address	NUMA	IP
pci-0000-10-01-0	ENA	ens33	0000:10:01.0	0	192.168.157.168/19
pci-0000-10-1b-0	EFA	—	0000:10:1b.0	0	rdmap16s27
pci-0000-20-01-0	ENA	ens65	0000:20:01.0	0	192.168.155.210/19
pci-0000-20-1b-0	EFA	—	0000:20:1b.0	0	rdmap32s27
pci-0000-90-01-0	ENA	ens129	0000:90:01.0	1	192.168.145.129/19
pci-0000-90-1b-0	EFA	—	0000:90:1b.0	1	rdmap144s27
pci-0000-a0-01-0	ENA	ens161	0000:a0:01.0	1	192.168.132.181/19
pci-0000-a0-1b-0	EFA	—	0000:a0:1b.0	1	rdmap160s27

[anson627]

verified this PR on Azure GB300, the primary NiC (eth0) with default gateway is excluded, while Azure accelerated networking VF (e.g. enP8051s1) together with other mlx5_* NiCs are included

#	Name	RDMA Device	PCI Address	NUMA	Vendor	RDMA
1	pci-0101-00-00-0	mlx5_0	0101:00:00.0	0	Mellanox	true
2	pci-0102-00-00-0	mlx5_1	0102:00:00.0	0	Mellanox	true
3	pci-0103-00-00-0	mlx5_2	0103:00:00.0	1	Mellanox	true
4	pci-0104-00-00-0	mlx5_3	0104:00:00.0	1	Mellanox	true
5	pci-1f73-00-02-0	—	1f73:00:02.0	0	Mellanox	true

/lgtm

[gauravkghildiyal]

Holding to have time to go through this and allow resolution of a few comments.

/hold

[aojea]

RA injects IPv6 defaults for us. If they share a metric (which I think they do) I still need something like NonUplinkChecker. If they don't I can drop NonUplinkChecker.

From a kubernetes architectural and security perspective, we like to have certain guardrails so users can not shoot themselves on the feet. Any logic that allows an interface with a default route to be unmounted or modified based on an opportunistic RA injection is very risky.:

• In the Kubernetes networking model, there is typically no difference between the control plane and the dataplane traffic. If an interface intended for the dataplane receives a default route via RA, it will be 'hijacking' control plane traffic, so the kubelet and pods trying to connect to the apiserver will fail (unless this secondary interface has access too, but this looks a very convoluted setup with routing loops). This will effectively disconnect the entire node from the cluster.

• We already were hit by security issues "abusing" RA, so right now is RA processing disabled by default in all container interfaces, the same attack can be performed at the node level

A vulnerability was found in all versions of containernetworking/plugins before version 0.8.6, that allows malicious containers in Kubernetes clusters to perform man-in-the-middle (MitM) attacks. A malicious container can exploit this flaw by sending rogue IPv6 router advertisements to the host or other containers, to redirect traffic to the malicious container. (CVE-2020-10749)

I still think we need a better mechanism for improving filtering, so I encourage you to check @gauravkghildiyal proposal in #152 , that I think will be able to accomodate the NonUplinkChecker functionality perfectly

[dkennetzoracle]

@aojea aligned with @gauravkghildiyal 's issue, it seems like a good fix and a generally better way to approach the problem!

[tamilmani1989]

changes lgtm. left a comment, i will leave to your discretion

[netlify]

✅ Deploy Preview for dranet canceled.

Name	Link
🔨 Latest commit	52aa19a
🔍 Latest deploy log	https://app.netlify.com/projects/dranet/deploys/69e2958e6384bc0008a17d96

[aojea]

pushed addressing latest comments, great reviews, caught some flaws:

• clarify function scope explicitly indicating it queries for the unspecified routes 0.0.0.0/0, ::/0 or nil that are the kernel default networks
• adding new unit tests

[dkennetzoracle]

@aojea @gauravkghildiyal I can confirm this fixed our issues (besides NIC orphaning) on OCI. I pulled in our OKE changes to this branch locally and deployed, and didn't need any additional changes. We made a few additional changes on our side, and fixed our dual ipv6 stack deployment. With these changes, the only thing we are missing is the cri upstream changes.

[aojea]

/hold cancel

based on the feedback from OKE and Azure, @gauravkghildiyal for final review

[gauravkghildiyal]

Perfect, thanks for validating @anson627 and @dkennetzoracle

Thanks for the change @aojea

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anson627, aojea, dkennetzoracle, gauravkghildiyal, tamilmani1989

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [aojea,gauravkghildiyal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment