Skip to content

fix(srv6): implement DelConnectivity with NLRI-key teardown and candidate-path failover#1025

Open
ryskn wants to merge 2 commits into
projectcalico:masterfrom
ryskn:srv6-delconnectivity
Open

fix(srv6): implement DelConnectivity with NLRI-key teardown and candidate-path failover#1025
ryskn wants to merge 2 commits into
projectcalico:masterfrom
ryskn:srv6-delconnectivity

Conversation

@ryskn

@ryskn ryskn commented May 21, 2026

Copy link
Copy Markdown
Contributor

Description

SRv6Provider.DelConnectivity was a FIXME no-op. This implements SR Policy withdraw/teardown and the candidate-path failover that follows it (RFC 9012 NLRI key, RFC 9256 candidate paths).

  • A withdraw is handled per NLRI key <Distinguisher, Color, Endpoint>, tearing down only the candidate path it identifies — never collaterally removing other policies on the same endpoint.
  • After teardown the orphaned prefix is re-steered onto the next-best surviving candidate of the same behavior on the same endpoint; the survivor is installed on demand (it was masked by the higher-priority winner).
  • The SR Policy NLRI cache is keyed by the NLRI key and upserted in place; on a BSID change the prior BSID is torn down after the steering is re-pointed, and a pendingBsidCleanup queue retries cleanups VPP still resolves through so a superseded BSID never leaks.
  • NO_SUCH_INNER_FIB / UNSPECIFIED from VPP are treated as idempotent "already gone" on delete (debug, not warn).

common.SRv6Tunnel now carries Color and Distinguisher; getSRPolicy populates them so the SRv6PolicyDeleted dispatch is NLRI-key targeted.

How Has This Been Tested?

Unit tests for the teardown / failover / cache paths are included (connectivity/srv6_test.go, fake VPP, no dataplane needed): NLRI-key precision, candidate-path failover (orphan collection + on-demand install + re-steer with dedupe), idempotent delete, BSID-change cleanup ordering, the re-point-failed guard, and the pending-cleanup retry.

Also verified end-to-end on a 3-node IPv6 single-stack cluster with an external gobgpd controller. From a worker, a remote-node pod prefix (fd20::b91:ca7f:1e60:6840/122) is steered; SR Policy A is installed then withdrawn.

Prior behavior — a withdraw double-deletes and re-steers twice:

level=warning DelSRv6Steering bsid=cafe::aaaa prefix=fd20::b91:ca7f:1e60:6840/122: VPPApiError: No such inner FIB / VRF (-4)
level=warning DelSRv6Policy  bsid=cafe::aaaa: VPPApiError: Unspecified Error (-1)
level=info    re-steered prefix=fd20::b91:ca7f:1e60:6840/122 onto surviving bsid=cafe::bbbb behavior=8
level=info    re-steered prefix=fd20::b91:ca7f:1e60:6840/122 onto surviving bsid=cafe::bbbb behavior=8   # duplicate

With this change — clean withdraw, single failover, no warn/error:

level=info SRv6Provider DelConnectivity ...
level=info re-steered prefix=fd20::b91:ca7f:1e60:6840/122 onto surviving bsid=cafe::307 behavior=8

BSID change (re-advertise the same NLRI key with a new BSID) — the prior BSID is removed from VPP and the steering re-points:

after A:            show sr policies => [..., cafe::aaaa]   steer: fd20::b91:.../122 -> cafe::aaaa
after B (new bsid): show sr policies => [..., cafe::cccc]   steer: fd20::b91:.../122 -> cafe::cccc   # old cafe::aaaa removed

…date-path failover

SRv6Provider.DelConnectivity was a FIXME no-op. Implement SR Policy
withdraw/teardown and the candidate-path failover that follows it
(RFC 9012 NLRI key, RFC 9256 candidate paths):

- A withdraw is handled per NLRI key <Distinguisher, Color, Endpoint>, so
  it tears down only the candidate path it identifies and never
  collaterally removes other policies on the same endpoint.
- After teardown the orphaned prefix is re-steered onto the next-best
  surviving candidate of the same behavior on the same endpoint; the
  survivor is installed on demand (it was masked by the higher-priority
  winner and never reached CreateSRv6Tunnel).
- The SR Policy NLRI cache is keyed by the NLRI key and upserted in place;
  on a BSID change the prior BSID is torn down after the steering has been
  re-pointed, and a pendingBsidCleanup queue retries cleanups that VPP
  still resolves through so a superseded BSID never leaks.
- NO_SUCH_INNER_FIB and UNSPECIFIED from VPP are treated as idempotent
  "already gone" in a delete context (debug, not warn); other errors warn.
- getPolicyNode skips candidates with a nil SrPolicy (nil-deref guard) and
  uses a strict priority comparison so a tie deterministically keeps the
  first candidate.

common.SRv6Tunnel carries Color and Distinguisher; getSRPolicy populates
them on success, on the withdraw short-circuit, and on the mixed-behavior
reject so the SRv6PolicyDeleted dispatch is NLRI-key targeted.

Signed-off-by: Ryosuke Nakayama <ryosuke.nakayama@ryskn.com>
@ryskn ryskn force-pushed the srv6-delconnectivity branch from 547dfb5 to 1f464dd Compare May 21, 2026 16:36
@aritrbas

Copy link
Copy Markdown
Collaborator

/codebuild_run(1f464dd)

@ryskn

ryskn commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Because MakePathSRv6Tunnel sets Color = trafficType (4/6), every DT6 SR Policy on a given endpoint collapses to the same <Color=6, Endpoint> identity, so a node's own self-delivery policy and an externally-injected policy targeting the same endpoint are indistinguishable as candidate paths. As a result, if a prefix is steered toward a node B that doesn't host that pod and the path is later withdrawn, failover re-points the prefix onto B's self-delivery policy and decap-blackholes it (no fallback to the native owner).

Given the Color = trafficType model this is RFC 9256-correct behavior, not a defect in this PR — it only surfaces with cross-endpoint SR-TE and is harmless in plain node-to-node operation. A follow-up PR will add a guard that skips failover candidates whose endpoint isn't the prefix's native owner, leaving it unsteered to fall back to native routing.

@ryskn

ryskn commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

delSRPolicy already tears down the node-IP /128 steering that steerNodeIPViaSID installs for host-network-backed ClusterIPs. That steering is keyed on the node's End.DT6 BSID, so it is collected and deleted by the same BSID match as the pod-prefix steerings, and re-steered by the failover path when a candidate survives — no dedicated teardown is needed.

One fix was required for that to be correct: the node-IP /128 steering lives in PodVRFIndex, not the main table, but resteerOrphan rebuilt the steering without a FibTable, so a candidate-path failover would have re-pointed it into table 0 and silently broken host-network-backed ClusterIPs. The orphan set now carries the full SrSteer (including FibTable) and the re-steer preserves it, so pod-prefix (main table) and node-IP /128 (PodVRFIndex) steerings each fail over into their original table.

Unit tests (fake VPP, no dataplane):

  • TestDelSRPolicy_FailoverPreservesNodeIPSteeringFibTable — the node-IP /128 stays in PodVRFIndex after failover.
  • TestDelSRPolicy_TearsDownNodeIPSteeringWhenNoSurvivor — when the last candidate is withdrawn, the node-IP /128 steering is removed and not re-steered (the host plane falls back to native routing).

Verified on a Linux node (fake-VPP unit tests, no real dataplane): the teardown/failover tests and the existing failover/NLRI tests are green.

IPv4 node IPs / End.DT4 remain out of scope (steerNodeIPViaSID is IPv6-only); dual-stack is tracked as a follow-up.

resteerOrphan rebuilt the steering without a FibTable, so a candidate-path
failover re-pointed every orphaned prefix into the main table. Pod-prefix
steerings live there so they were unaffected, but the node-IP /128 steering
(steerNodeIPViaSID, for host-network-backed ClusterIPs) lives in PodVRFIndex and
would have been silently moved to table 0 on failover, breaking the host plane.

Carry the full SrSteer (including FibTable) in the orphan set and preserve it on
re-steer, so pod-prefix (main table) and node-IP /128 (PodVRFIndex) steerings
each fail over into their original table. The same delSRPolicy BSID match also
tears the node-IP /128 steering down when no candidate survives.

Tests: TestDelSRPolicy_FailoverPreservesNodeIPSteeringFibTable,
TestDelSRPolicy_TearsDownNodeIPSteeringWhenNoSurvivor.

Signed-off-by: Ryosuke Nakayama <ryosuke.nakayama@ryskn.com>
@ryskn ryskn force-pushed the srv6-delconnectivity branch from 0a36f50 to f0f66ad Compare June 3, 2026 10:30
@sknat

sknat commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

/codebuild_run(f0f66ad)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants