Conversation
handleMuxConfigNotification() skipped switchMuxState() when the requested mode already matched the in-memory mux state. That state is bootstrapped from the first probe after a linkmgrd restart and is never reconciled against the hardware, so a matching `config mux mode active|standby` was silently dropped: the CLI reported success, the driver was never told, and the port was left reporting an inconsistent mux status. Drop the comparison so an explicit Active or Standby config is always forwarded to the driver. switchMuxState() is already called with forceSwitch on these paths, so the mismatch case is unchanged. Auto, Manual and Detached modes keep their existing behavior. Fixes sonic-net/sonic-buildimage#26391 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nikola Dancejic <ndancejic@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command. |
Collaborator
|
This PR has backport request label(s) for branch(es): 202605, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202605: <test result>) in the Test result section as well in your PR description. ---Powered by SONiC BuildBot
|
zjswhhh
approved these changes
Aug 27, 2026
Contributor
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
nazariig
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
Fixes sonic-net/sonic-buildimage#26391
ActiveActiveStateMachine::handleMuxConfigNotification()only calledswitchMuxState()when the requested mode differed from the in-memory mux statems(mCompositeState). When they already matched, control fell through to the state transition recalculation, which issues nosetMuxState()and therefore never asks the driver to toggle.The in-memory mux state is not proof that the hardware agrees with it. After a linkmgrd restart it is bootstrapped from the first state db / probe notification without being reconciled against the driver. A subsequent
config mux mode active|standbythat happens to match the bootstrapped state was then silently dropped: the CLI reported success, no toggle was sent, and the port was left reporting an inconsistent mux status indefinitely.Type of change
Approach
What is the motivation for this PR?
An explicit
config mux mode active|standbyfrom an operator or from a test is an unconditional instruction to program the hardware. Treating it as a no-op whenever linkmgrd's cached state happens to match leaves the port stuck with an inconsistent mux status and no way to recover short of restarting linkmgrd again.Work item tracking
How did you do it?
Dropped the
ms(mCompositeState) != ...comparison from theActiveandStandbybranches so an explicit config is always forwarded to the driver.switchMuxState()is already invoked withforceSwitchset on these paths, so the mismatch case is unchanged. The matching case now reuses the same path, which means it also suspends the heartbeat when enforcingStandbyto let the peer ToR toggle, and posts aSwitchingStartmetric that pairs with theSwitchingEndposted when the driver responds.This additionally reconciles a restarted linkmgrd with the hardware, because a config notification deferred during initialization is replayed once all components are up.
Auto,ManualandDetachedmodes keep their existing behavior.How did you verify/test it?
Unit tests — full
make test-targetssuite passes (194 tests, 12 suites). Five new cases added toLinkManagerStateMachineActiveActiveTest:MuxActiveConfigActiveReDrivesMuxStateconfig mux mode activewhile alreadyActivestill drives the driverMuxStandbyConfigStandbyReDrivesMuxStateStandbyMuxConfigActiveAfterRestartMatchingBootstrappedStateMuxConfigActiveBeforeInitMatchingBootstrappedStateMuxActiveConfigAutoDoesNotReDriveMuxStateautois unchanged, no redundant toggleMuxActivDefaultRouteStateMuxConfigActivewas updated for the new expectedsetMuxStatecount.Hardware — validated A/B on an active-active dual-ToR testbed, swapping only the linkmgrd binary between runs:
handleMuxConfigNotification: mux config mode: 2is logged, followed by a no-op transition(P: Active, M: Active, L: Up) -> (P: Active, M: Active, L: Up)with noswitchMuxStateline, andLAST_SWITCHOVER_TIMEdoes not advance. This matches the syslog in the linked issue.switchMuxState: <port>: Switching MUX state to 'Active', a state db update, and an advancingLAST_SWITCHOVER_TIME. The port stayedactive/healthy/consistent, i.e. no flap.dualtor_mgmt/test_grpc_periodical_sync.py::test_mux_forwarding_state_consistencypasses with the change (both parameters).Note for reviewers: that test is not a reliable gate for this bug. It only reaches the short-circuit if the in-memory state bootstraps to
Activewhile the hardware is stillstandby, which in the reported environment came from a failing gRPC probe. On a healthy setup linkmgrd probes the hardware correctly, so the follow-up config takes the mismatch path and passes either way. The syslog observable above is the reliable signal.Any platform specific information?
None. The change is in the active-active state machine and is platform independent.
Documentation
No documentation change needed; this restores the documented behavior of
config mux mode.