[Dualtor][Prefix-route]: Fix NHG member stale OID causing remove failure on dualtor prefix-route mode by rajkumar1-arista · Pull Request #4886 · sonic-net/sonic-swss

rajkumar1-arista · 2026-09-14T07:45:53Z

Why I did it

On dualtor topologies with neighbor_mode: prefix-route, deleting an ECMP
static route after all its nexthop mux cables have transitioned to standby
causes a persistent orchagent failure. The SAI_API_NEXT_HOP_GROUP remove
operation fails with SAI_STATUS_NOT_EXECUTED.

When mux cables go to standby, MuxPrefixBasedNbrHandler::disable() calls
invalidnexthopinNextHopGroup() to remove NHG members from SAI. Unlike the
host-route handler which always pairs invalidate with validate to swap members,
the prefix-route handler intentionally does not add tunnel replacements —
route-to-tunnel redirection is handled separately via setBulkRouteNH().

However, invalidnexthopinNextHopGroup() does not update the in-memory
nhopgroup_members map after removing members from SAI. The stale OIDs remain
in the map. When the route is later deleted, removeNextHopGroup() reads these
stale OIDs and submits them for bulk removal. SAI meta validation fails with
SAI_STATUS_ITEM_NOT_FOUND, and SAI_BULK_OP_ERROR_MODE_STOP_ON_ERROR causes
the remaining members to get SAI_STATUS_NOT_EXECUTED.

The issue does not occur with neighbor_mode: host-route because that handler
always calls validnexthopinNextHopGroup() after invalidation, which overwrites
the stale OID with a fresh one.

Work item tracking

Microsoft ADO (number only):

How I did it

In invalidnexthopinNextHopGroup(), set the member OID to
SAI_NULL_OBJECT_ID after successful SAI removal. This preserves seq_id
for ordered ECMP re-add via validnexthopinNextHopGroup() while clearly
marking the member as already removed from SAI.
In removeNextHopGroup(), skip members whose OID is SAI_NULL_OBJECT_ID
since they have already been removed from SAI.
Add a mock unit test (RemoveNhgAfterAllMembersInvalidatedWithoutReplacement)
that exercises the exact sequence: create ECMP route, invalidate all members
without replacement, delete route and verifies no SAI failures occur.
Add a virtual switch test (test_multi_nexthop_all_standby_then_delete_route)
that creates an ECMP route while mux is active, transitions all mux ports to
standby, deletes the route, and verifies: NHG members are cleaned up from
ASIC DB, no SAI error is logged, and orchagent remains healthy.

How to verify it

On an active-standby DualToR device with neighbor_mode: prefix-route:

# Step 1: Set 2 mux ports to active and resolve neighbors
sudo config mux mode active Ethernet4
sudo config mux mode active Ethernet8
sudo ip neigh replace 192.168.0.3 lladdr aa:bb:cc:dd:ee:01 dev Vlan1000
sudo ip neigh replace 192.168.0.4 lladdr aa:bb:cc:dd:ee:02 dev Vlan1000

# Step 2: Create ECMP static route
sudo config route add prefix 9.9.9.0/24 nexthop 192.168.0.3,192.168.0.4

# Step 3: Set mux ports to standby
sudo config mux mode standby Ethernet4
sudo config mux mode standby Ethernet8

# Step 4: Delete the route
sudo config route del prefix 9.9.9.0/24

# Step 5: Verify no SAI errors
sudo grep "SAI_STATUS_NOT_EXECUTED" /var/log/syslog | tail -5
# Expected: no output (no errors logged)

Run the new mock UT: RemoveNhgAfterAllMembersInvalidatedWithoutReplacement
in tests/mock_tests/routeorch_ut.cpp.
Run the new VS test: test_multi_nexthop_all_standby_then_delete_route in
tests/test_mux_prefixroute.py.

Which release branch to backport (provide reason below if selected)

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO): #4885

Failure type: day-one issue

Reason for backport: the issue is reproducible on 202605 active-standby
DualToR setups with neighbor_mode: prefix-route and causes sonic-mgmt tests like route/test_static_route.py to fail due to presence of syslog error that is being caught by logAnalyzer

Tested branch

Test result

202605: SONiC.branch.202605-ars.f113c325-buildimage.ccd774c0f0bab05d35518f9fab1ac40a401b5ca1-review.801796.5-2026.09.13.15.46 -
Passing_test_static_route.xml
NHG_bug_fixed.log

Description for the changelog

Fix NHG member stale OID causing SAI remove failure on dualtor with prefix-route neighbor mode.

Link to config_db schema for YANG module changes

N/A — no YANG or config_db schema changes.

A picture of a cute animal (not mandatory but encouraged)


 /\     /\
{  `---'  }
{  O   O  }
~~>  V  <~~
 \  \|/  /
  `-----'__
  /     \  `^\_
 {       }\ |\_\_   W
 |  \_/  |/ /  \_\_( )
  \__/  /(_E     \__/
    (  /

On dualtor topologies with neighbor_mode=prefix-route, when mux cables transition to standby, MuxPrefixBasedNbrHandler::disable() calls invalidnexthopinNextHopGroup() to remove NHG members from SAI. Unlike MuxNbrHandler (host-route mode) which always pairs invalidnexthop with validnexthop to swap members, the prefix-route handler intentionally does not add tunnel replacements. route-to-tunnel redirection is handled separately via setBulkRouteNH() at the per-prefix level. However, invalidnexthopinNextHopGroup() does not update the in-memory nhopgroup_members map after removing members from SAI. The stale SAI OIDs persist in the map. When the route is later deleted, removeNextHopGroup() reads these stale OIDs and submits them to the ObjectBulker for bulk removal. SAI meta validation fails with SAI_STATUS_ITEM_NOT_FOUND for the first stale OID, and SAI_BULK_OP_ERROR_MODE_STOP_ON_ERROR causes remaining members to get SAI_STATUS_NOT_EXECUTED. This triggers handleSaiFailure(), setting a persistent orchagent unhealthy flag that floods syslog with ERR messages every ~1 second. Fix invalidnexthopinNextHopGroup() to set the member OID to SAI_NULL_OBJECT_ID after successful SAI removal, preserving seq_id for ordered ECMP re-add. Fix removeNextHopGroup() to skip members with SAI_NULL_OBJECT_ID since they are already removed from SAI. Add missing UTs and swss tests Signed-off-by: rajkumar1 <rajkumar1@arista.com>

azure-pipelines · 2026-09-14T07:46:02Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

mssonicbld · 2026-09-14T07:46:12Z

/azp run

azure-pipelines · 2026-09-14T07:46:23Z

Azure Pipelines: Successfully started running 1 pipeline(s).

mssonicbld · 2026-09-14T12:16:28Z

This PR has backport request for branch(es): 202605.
Added label(s) for branch(es) 202605.

_{---Powered by SONiC BuildBot}

mssonicbld · 2026-09-14T12:16:32Z

The Tested branch section has been ticked and Test result is provided for branch(es): 202605. Added label(s): Tested for 202605 Branch.

_{---Powered by SONiC BuildBot}

zjswhhh · 2026-09-15T06:04:43Z

Hi @Ndancejic - can you help review?

Ndancejic · 2026-09-15T18:19:33Z

Looks good to me! @rajkumar1-arista have you tested with both prefix and host route mode to confirm the behavior for host route mode hasn't changed?

Ndancejic · 2026-09-15T22:30:49Z

@manamand2020 for review as well

manamand2020 · 2026-09-16T00:34:34Z

LGTM. @rajkumar1-arista do you want to verify the nexthop ref count as well in the tests since invalidnexthopinNextHopGroup() only removes SAI members; NeighOrch refcount is unchanged until the NHG itself is destroyed.

rajkumar1-arista · 2026-09-17T06:02:10Z

Looks good to me! @rajkumar1-arista have you tested with both prefix and host route mode to confirm the behavior for host route mode hasn't changed?

Yes, this was verified for host-route mode.

Click to expand

admin@gd274:~$ show version --brief

SONiC Software Version: SONiC.branch.202605-ars.34c55f68-buildimage.94f39efbfba9878881126e45c3b926decb74dd48-review.801796.6-2026.09.15.21.27
SONiC OS Version: 13
Distribution: Debian 13.7
Kernel: 6.12.41+deb13-sonic-amd64
Build commit: 2c28d7a75
Build date: Tue Sep 15 22:15:20 UTC 2026
Built by: jenkins@jenkins-arsonic-k8s-arsonic-backend-k8s-117770-cqt2r-g680m

Platform: x86_64-arista_7260cx3_64
HwSKU: Arista-7260CX3-D108C8
ASIC: broadcom
ASIC Count: 1
Serial Number: SSJ18010321
Model Number: DCS-7260CX3-64
Hardware Revision: 03.00
Uptime: 05:40:51 up 22:45,  1 user,  load average: 1.50, 1.54, 1.36
Date: Thu 17 Sep 2026 05:40:51
admin@gd274:~$ sudo grep "SAI_STATUS_NOT_EXECUTED" /var/log/syslog | tail -5
admin@gd274:~$
admin@gd274:~$
admin@gd274:~$ show mux config | head -8
SWITCH_NAME    PEER_TOR
-------------  ----------
gd333          10.1.0.33
port         state    ipv4             ipv6               cable_type    soc_ipv4    soc_ipv6    prober_type    neighbor_mode
-----------  -------  ---------------  -----------------  ------------  ----------  ----------  -------------  ---------------
Ethernet0    auto     192.168.0.2/32   fc02:1000::2/128                                         software       host-route
Ethernet4    auto     192.168.0.3/32   fc02:1000::3/128                                         software       host-route
Ethernet8    auto     192.168.0.4/32   fc02:1000::4/128                                         software       host-route
admin@gd274:~$
admin@gd274:~$
admin@gd274:~$ show mux status | head -5
PORT         STATUS    SERVER_STATUS    HEALTH     HWSTATUS    LAST_SWITCHOVER_TIME
-----------  --------  ---------------  ---------  ----------  ---------------------------
Ethernet0    standby   standby          unhealthy  consistent  2026-Sep-16 13:06:18.366025
Ethernet4    standby   standby          unhealthy  consistent  2026-Sep-16 13:06:17.726109
Ethernet8    standby   standby          unhealthy  consistent  2026-Sep-17 04:49:28.457891
admin@gd274:~$
admin@gd274:~$ sudo config mux mode active Ethernet4
port       state
---------  ----------
Ethernet4  INPROGRESS
admin@gd274:~$ sudo config mux mode active Ethernet8
port       state
---------  ----------
Ethernet8  INPROGRESS
admin@gd274:~$ show mux status | head -5
PORT         STATUS    SERVER_STATUS    HEALTH     HWSTATUS    LAST_SWITCHOVER_TIME
-----------  --------  ---------------  ---------  ----------  ---------------------------
Ethernet0    standby   standby          unhealthy  consistent  2026-Sep-16 13:06:18.366025
Ethernet4    active    active           unhealthy  consistent  2026-Sep-17 05:41:34.602152
Ethernet8    active    active           unhealthy  consistent  2026-Sep-17 05:41:39.322464
admin@gd274:~$ sudo ip neigh replace 192.168.0.3 lladdr aa:bb:cc:dd:ee:01 dev Vlan1000
admin@gd274:~$ sudo ip neigh replace 192.168.0.4 lladdr aa:bb:cc:dd:ee:02 dev Vlan1000
admin@gd274:~$ sudo config route add prefix 9.9.9.0/24 nexthop 192.168.0.3,192.168.0.4
admin@gd274:~$ show ip route 9.9.9.0/24
Routing entry for 9.9.9.0/24
  Known via "static", distance 1, metric 0, best
  Last update 00:00:09 ago
  * 192.168.0.3, via Vlan1000
  * 192.168.0.4, via Vlan1000

admin@gd274:~$ sudo config mux mode standby Ethernet4
port       state
---------  ----------
Ethernet4  INPROGRESS
admin@gd274:~$ sudo config mux mode standby Ethernet8
port       state
---------  ----------
Ethernet8  INPROGRESS
admin@gd274:~$ show mux status | head -5
PORT         STATUS    SERVER_STATUS    HEALTH     HWSTATUS      LAST_SWITCHOVER_TIME
-----------  --------  ---------------  ---------  ------------  ---------------------------
Ethernet0    standby   standby          unhealthy  consistent    2026-Sep-16 13:06:18.366025
Ethernet4    standby   standby          unhealthy  consistent    2026-Sep-17 05:42:43.686706
Ethernet8    standby   standby          unhealthy  consistent    2026-Sep-17 05:42:47.686158
admin@gd274:~$ sudo config route del prefix 9.9.9.0/24
admin@gd274:~$ sudo grep "SAI_STATUS_NOT_EXECUTED" /var/log/syslog | tail -5
admin@gd274:~$ sudo grep "SAI_STATUS_NOT_EXECUTED" /var/log/syslog | wc -l
0
admin@gd274:~$

(host_route)test_static_route.tar.gz

Signed-off-by: rajkumar1 <rajkumar1@arista.com>

mssonicbld · 2026-09-17T06:10:01Z

/azp run

rajkumar1-arista · 2026-09-17T06:10:07Z

do you want to verify the nexthop ref count as well in the tests since invalidnexthopinNextHopGroup() only removes SAI members; NeighOrch refcount is unchanged until the NHG itself is destroyed.

Thanks, added in the UT

azure-pipelines · 2026-09-17T06:10:13Z

Azure Pipelines: Successfully started running 1 pipeline(s).

mssonicbld · 2026-09-18T05:15:49Z

/azp run

azure-pipelines · 2026-09-18T05:15:59Z

Azure Pipelines: Successfully started running 1 pipeline(s).

rajkumar1-arista requested a review from prsunny as a code owner September 14, 2026 07:45

mssonicbld added the Request for 202605 Branch label Sep 14, 2026

mssonicbld added the Tested for 202605 branch Tested for 202605 branch label Sep 14, 2026

mssonicbld added the Cherry Pick Conflict_202605 label Sep 14, 2026

add ref count check

819fe1c

Signed-off-by: rajkumar1 <rajkumar1@arista.com>

Ndancejic approved these changes Sep 17, 2026

View reviewed changes

Merge branch 'sonic-net:master' into NHG-bug-fix

451a47b

Conversation

rajkumar1-arista commented Sep 14, 2026

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch

Test result

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

azure-pipelines Bot commented Sep 14, 2026

Uh oh!

mssonicbld commented Sep 14, 2026

Uh oh!

azure-pipelines Bot commented Sep 14, 2026

Uh oh!

mssonicbld commented Sep 14, 2026

Uh oh!

mssonicbld commented Sep 14, 2026

Uh oh!

zjswhhh commented Sep 15, 2026

Uh oh!

Ndancejic commented Sep 15, 2026

Uh oh!

Ndancejic commented Sep 15, 2026

Uh oh!

manamand2020 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rajkumar1-arista commented Sep 17, 2026

Uh oh!

mssonicbld commented Sep 17, 2026

Uh oh!

rajkumar1-arista commented Sep 17, 2026

Uh oh!

azure-pipelines Bot commented Sep 17, 2026

Uh oh!

mssonicbld commented Sep 18, 2026

Uh oh!

azure-pipelines Bot commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

manamand2020 commented Sep 16, 2026 •

edited

Loading