Skip to content

Fix IPv6 CIDR flapping in k8s.ovn.org/host-cidrs annotation#3167

Draft
smulje wants to merge 1 commit intoopenshift:release-4.21from
smulje:fix/ipv6-host-cidrs-flapping-4.21
Draft

Fix IPv6 CIDR flapping in k8s.ovn.org/host-cidrs annotation#3167
smulje wants to merge 1 commit intoopenshift:release-4.21from
smulje:fix/ipv6-host-cidrs-flapping-4.21

Conversation

@smulje
Copy link
Copy Markdown

@smulje smulje commented Apr 29, 2026

Summary

Fixes a race condition where the k8s.ovn.org/host-cidrs annotation continuously flaps between dual-stack and IPv4-only states on nodes with IPv6 enabled (particularly with SLAAC).

Problem

On nodes with IPv6 enabled, the k8s.ovn.org/host-cidrs annotation exhibits unstable behavior, alternating between:

  • ["10.46.xx.62/21","2620:52:0:2ef8:7058:xxx:3bd9:e3dc/64"] (dual-stack)
  • ["10.46.xx.62/21"] (IPv4-only)

This flapping occurs every 30-60 seconds and impacts workloads that depend on stable host network information.

Root Cause

The addressManager.sync() function has a race condition with IPv6 Duplicate Address Detection (DAD):

  1. Periodic sync scans network interfaces to rebuild the address map
  2. IPv6 DAD timing: IPv6 addresses may still be in "tentative" state during the scan
  3. Result: sync() misses the IPv6 address and updates annotation to IPv4-only
  4. Netlink watcher later discovers the IPv6 address via kernel events
  5. Result: Annotation updated back to dual-stack
  6. Repeat: Next sync cycle triggers the same race

Solution

Perform two interface scans with a 100ms delay and take the union of both results. This ensures IPv6 addresses that are completing DAD during the first scan are captured in the second scan.

Changes

  1. New helper function scanInterfaceAddresses(): Extracts interface scanning logic for reuse and returns (sets.Set[string], error) to propagate failures

  2. Modified sync() function: Performs double-scan with union and includes error handling with fallback logic:

    • Both scans succeed: use union (catches IPv6 during DAD)
    • One scan fails: use the successful scan result
    • Both scans fail: abort early to preserve existing annotation
  3. Better error handling: Continues on per-interface errors instead of aborting entire sync, and prevents transient netlink.LinkList() failures from incorrectly clearing the annotation

The 100ms delay accommodates typical IPv6 DAD completion times (most complete within 100-200ms) without significantly impacting sync performance (runs every 30s).

Testing

Reproduction Environment

  • Cluster: OpenShift 4.20.1, OVN-Kubernetes
  • Network: Dual-stack (IPv4 + IPv6 via SLAAC)
  • Affected nodes: All nodes with IPv6 enabled
  • Kernel: RHEL CoreOS 9.6

Reproduction Steps

  1. Enable IPv6 on cluster nodes (via SLAAC or static)
  2. Monitor annotation: oc get node <node> -o jsonpath='{.metadata.annotations.k8s\.ovn\.org/host-cidrs}'
  3. Observe flapping between dual-stack and IPv4-only every 30-70 seconds

Verification

Before fix: Flapping detected every 40-70 seconds
After fix: Annotation remains stable with dual-stack

Impact

  • Positive: Eliminates annotation instability on IPv6-enabled clusters
  • Performance: Negligible (adds 100ms to sync interval, which runs every 30s)
  • Compatibility: No breaking changes, backward compatible
  • Side effects: None identified

Related Issues

  • Affects OpenShift 4.15+ (when k8s.ovn.org/host-cidrs annotation was introduced)
  • Impacts egress IP feature on dual-stack clusters
  • May affect any controller watching node annotations for network changes

Fixes a race condition where the k8s.ovn.org/host-cidrs annotation
continuously flaps between dual-stack and IPv4-only states on nodes
with IPv6 enabled (particularly with SLAAC).

Root cause: The addressManager.sync() function scans network interfaces
every 30s to rebuild the address map. When IPv6 addresses are still in
"tentative" state during Duplicate Address Detection (DAD), sync()
misses them and updates the annotation to IPv4-only. The netlink watcher
later discovers the IPv6 address via kernel events and updates back to
dual-stack, creating a continuous flapping cycle.

Solution: Perform two interface scans with a 100ms delay and take the
union of both results. This ensures IPv6 addresses completing DAD during
the first scan are captured in the second scan.

Changes:
1. New scanInterfaceAddresses() helper function that extracts interface
   scanning logic for reuse and returns (sets.Set[string], error) to
   propagate failures

2. Modified sync() function performs double-scan with union and includes
   error handling with fallback logic:
   - Both scans succeed: use union (catches IPv6 during DAD)
   - One scan fails: use the successful scan result
   - Both scans fail: abort early to preserve existing annotation

3. Better error handling: continues on per-interface errors instead of
   aborting entire sync, and prevents transient netlink.LinkList()
   failures from incorrectly clearing the annotation

The 100ms delay accommodates typical IPv6 DAD completion times (most
complete within 100-200ms) without significantly impacting sync
performance (runs every 30s).

Tested on OpenShift 4.20.1 with dual-stack networking. Before fix:
annotation flapped every 40-70s. After fix: annotation remains stable.
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 29, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Enterprise

Run ID: 8c909b1f-ae94-46e8-9528-a2d7830f665a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 29, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 29, 2026

Hi @smulje. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 29, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: smulje
Once this PR has been reviewed and has the lgtm label, please assign martinkennelly for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant