Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions vppbld/patches/0015-vxlan-encap-inner-aware-flow-hash.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Aaron Bernardino <aaronber@microsoft.com>
Date: Fri, 10 Jul 2026 00:00:00 +0000
Subject: [PATCH] vxlan: inner-aware flow hash in the encap node

The VXLAN encap node derives the outer UDP source port (and the buffer
flow hash used for underlay ECMP) from vnet_l2_compute_flow_hash(). That
helper reads the inner ethertype at current_data + l2.l2_len. For an
L3-routed VXLAN tunnel the inner packet arrives already routed (the
ip4/ip6-rewrite adjacency prepends the inner Ethernet header) and
l2.l2_len is 0/unset, so the helper misparses the frame and returns a
value that does not depend on the inner L3/L4 headers. Every inner flow
to a given tunnel endpoint then maps to the same outer UDP source port
and collapses onto a single underlay ECMP path / bond member.

This is the exact case that patch 0011 (inner-aware flow hash) left out of
scope on the assumption that "VXLAN ... their outer UDP source port is the
standard-mandated entropy carrier (RFC 7348 section 4.2), so the existing
outer-5-tuple hash already distributes inner flows." That assumption only
holds if the encap node actually populates the outer UDP source port from
the inner flow -- which it does not for L3-routed tunnels. This patch
fixes the entropy carrier at its source rather than adding the peek-inner
machinery of 0011 to the underlay.

Read the inner Ethernet ethertype directly at the buffer current pointer
(the inner Ethernet header is present for both L2 and L3 tunnels at the
encap call sites) and hash the inner IPv4/IPv6 5-tuple. Tagged inner
frames are handled by skipping up to two 802.1Q/802.1ad VLAN tags before
reading the inner ethertype; this cannot be delegated to
vnet_l2_compute_flow_hash() because that helper depends on l2.l2_len,
which is not valid on the routed path (it shares storage with
ip.flow_hash in the buffer opaque union). Anything that is not plain
inner IPv4/IPv6 still falls back to the generic vnet_l2_compute_flow_hash().

Signed-off-by: Aaron Bernardino <aaronber@microsoft.com>
---
diff --git a/src/plugins/vxlan/encap.c b/src/plugins/vxlan/encap.c
index 9d304b7..475239d 100644
--- a/src/plugins/vxlan/encap.c
+++ b/src/plugins/vxlan/encap.c
@@ -57,6 +57,44 @@ format_vxlan_encap_trace (u8 * s, va_list * args)
}
#endif

+static_always_inline u32
+vxlan_encap_inner_flow_hash (vlib_buffer_t * b)
+{
+ /* For L3 VXLAN tunnels the inner packet is routed (not bridged) into this
+ * encap node, so vnet_buffer(b)->l2.l2_len is not set. The generic
+ * vnet_l2_compute_flow_hash() then misparses the frame and returns a value
+ * that does not depend on the inner L3/L4 headers, collapsing every inner
+ * flow for a given tunnel endpoint onto a single outer UDP source port and
+ * a single underlay ECMP path. The inner Ethernet header is present at the
+ * current data pointer, so read its ethertype directly and hash the inner
+ * IP 5-tuple. Skip up to two VLAN tags so tagged inner frames are hashed on
+ * their real inner headers too. Fall back to the generic L2 hash for
+ * anything that is not plain inner IPv4/IPv6. */
+ ethernet_header_t *eh = vlib_buffer_get_current (b);
+ u16 ethertype = clib_net_to_host_u16 (eh->type);
+ u8 *l3h = (u8 *) (eh + 1);
+
+ if (ethertype == ETHERNET_TYPE_VLAN || ethertype == ETHERNET_TYPE_DOT1AD)
+ {
+ ethernet_vlan_header_t *vlan = (ethernet_vlan_header_t *) l3h;
+ ethertype = clib_net_to_host_u16 (vlan->type);
+ l3h = (u8 *) (vlan + 1);
+ if (ethertype == ETHERNET_TYPE_VLAN)
+ {
+ vlan = (ethernet_vlan_header_t *) l3h;
+ ethertype = clib_net_to_host_u16 (vlan->type);
+ l3h = (u8 *) (vlan + 1);
+ }
+ }
+
+ if (ethertype == ETHERNET_TYPE_IP4)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we support tagged routed packets? today it will fall back to vnet_l2_compute_flow_hash, which doesn't have proper l2.l2_len set. So the problem still exists for such flows. Also, should we check l2.l2_len first? If it is properly set, we can use vnet_l2_compute_flow_hash, which makes uses properly parsed l2 headers. If it is not set, we fall back to this path to parse the l2 header.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yue-fred-gao

You are right that tagged routed frames still fall through today, I will fix that.
On why I would not check l2.l2_len first: on a routed frame it is not reliable. The packet reaches the encap node through ip4/ip6-rewrite, and l2.l2_len sits on the same memory as ip.flow_hash in the buffer opaque union. The IP lookup always writes that field before we get here, so l2_len ends up as 0 or a hash byte, not a real header length, and there is no valid bit to tell the difference. Trusting it would sometimes send routed frames back into the same misparse we are fixing, which is why 0015 reads the inner header directly.
So instead of gating on l2_len, I will make the direct parse skip VLAN tags and hash the real inner IP:
if (ethertype == ETHERNET_TYPE_VLAN || ethertype == ETHERNET_TYPE_DOT1AD)
{
ethernet_vlan_header_t *vlan = (ethernet_vlan_header_t *) l3h;
ethertype = clib_net_to_host_u16 (vlan->type);
l3h = (u8 ) (vlan + 1); / handles single and QinQ tags */
}
The bridged path stays identical, tagged routed frames now hash correctly, and anything unusual still falls back to vnet_l2_compute_flow_hash. I will fold this into 0015 and revalidate on the testbed before pushing.

Co-authored with CoPilot

+ return ip4_compute_flow_hash ((ip4_header_t *) l3h, IP_FLOW_HASH_DEFAULT);
+ else if (ethertype == ETHERNET_TYPE_IP6)
+ return ip6_compute_flow_hash ((ip6_header_t *) l3h, IP_FLOW_HASH_DEFAULT);
+
+ return vnet_l2_compute_flow_hash (b);
+}
+
always_inline uword
vxlan_encap_inline (vlib_main_t *vm, vlib_node_runtime_t *node,
vlib_frame_t *from_frame, u8 is_ip4)
@@ -125,8 +163,8 @@ vxlan_encap_inline (vlib_main_t *vm, vlib_node_runtime_t *node,
vlib_buffer_t *b1 = b[1];
b += 2;

- u32 flow_hash0 = vnet_l2_compute_flow_hash (b0);
- u32 flow_hash1 = vnet_l2_compute_flow_hash (b1);
+ u32 flow_hash0 = vxlan_encap_inner_flow_hash (b0);
+ u32 flow_hash1 = vxlan_encap_inner_flow_hash (b1);

/* Get next node index and adj index from tunnel next_dpo */
if (sw_if_index0 != vnet_buffer (b0)->sw_if_index[VLIB_TX])
@@ -351,7 +389,7 @@ vxlan_encap_inline (vlib_main_t *vm, vlib_node_runtime_t *node,
vlib_buffer_t *b0 = b[0];
b += 1;

- u32 flow_hash0 = vnet_l2_compute_flow_hash (b0);
+ u32 flow_hash0 = vxlan_encap_inner_flow_hash (b0);

/* Get next node index and adj index from tunnel next_dpo */
if (sw_if_index0 != vnet_buffer (b0)->sw_if_index[VLIB_TX])
--
2.34.1
Loading