Skip to content

[active-active] Always drive an explicit mux config to the driver - #352

Open
Ndancejic wants to merge 1 commit into
sonic-net:masterfrom
Ndancejic:fix/26391-mux-config-redrive
Open

Ndancejic wants to merge 1 commit into
sonic-net:masterfrom
Ndancejic:fix/26391-mux-config-redrive

Conversation

@Ndancejic

Copy link
Copy Markdown

Description of PR

Summary:
Fixes sonic-net/sonic-buildimage#26391

ActiveActiveStateMachine::handleMuxConfigNotification() only called switchMuxState() when the requested mode differed from the in-memory mux state ms(mCompositeState). When they already matched, control fell through to the state transition recalculation, which issues no setMuxState() and therefore never asks the driver to toggle.

The in-memory mux state is not proof that the hardware agrees with it. After a linkmgrd restart it is bootstrapped from the first state db / probe notification without being reconciled against the driver. A subsequent config mux mode active|standby that happens to match the bootstrapped state was then silently dropped: the CLI reported success, no toggle was sent, and the port was left reporting an inconsistent mux status indefinitely.

Type of change

  • Bug fix
  • New feature
  • Doc/Design
  • Unit test

Approach

What is the motivation for this PR?

An explicit config mux mode active|standby from an operator or from a test is an unconditional instruction to program the hardware. Treating it as a no-op whenever linkmgrd's cached state happens to match leaves the port stuck with an inconsistent mux status and no way to recover short of restarting linkmgrd again.

Work item tracking

How did you do it?

Dropped the ms(mCompositeState) != ... comparison from the Active and Standby branches so an explicit config is always forwarded to the driver.

switchMuxState() is already invoked with forceSwitch set on these paths, so the mismatch case is unchanged. The matching case now reuses the same path, which means it also suspends the heartbeat when enforcing Standby to let the peer ToR toggle, and posts a SwitchingStart metric that pairs with the SwitchingEnd posted when the driver responds.

This additionally reconciles a restarted linkmgrd with the hardware, because a config notification deferred during initialization is replayed once all components are up.

Auto, Manual and Detached modes keep their existing behavior.

How did you verify/test it?

Unit tests — full make test-targets suite passes (194 tests, 12 suites). Five new cases added to LinkManagerStateMachineActiveActiveTest:

Test Covers
MuxActiveConfigActiveReDrivesMuxState config mux mode active while already Active still drives the driver
MuxStandbyConfigStandbyReDrivesMuxState same for Standby
MuxConfigActiveAfterRestartMatchingBootstrappedState config matching a bootstrapped state after restart reaches the driver
MuxConfigActiveBeforeInitMatchingBootstrappedState config deferred during init is replayed to the driver
MuxActiveConfigAutoDoesNotReDriveMuxState auto is unchanged, no redundant toggle

MuxActivDefaultRouteStateMuxConfigActive was updated for the new expected setMuxState count.

Hardware — validated A/B on an active-active dual-ToR testbed, swapping only the linkmgrd binary between runs:

  • Unpatched: handleMuxConfigNotification: mux config mode: 2 is logged, followed by a no-op transition (P: Active, M: Active, L: Up) -> (P: Active, M: Active, L: Up) with no switchMuxState line, and LAST_SWITCHOVER_TIME does not advance. This matches the syslog in the linked issue.
  • Patched: the same trigger produces switchMuxState: <port>: Switching MUX state to 'Active', a state db update, and an advancing LAST_SWITCHOVER_TIME. The port stayed active / healthy / consistent, i.e. no flap.

dualtor_mgmt/test_grpc_periodical_sync.py::test_mux_forwarding_state_consistency passes with the change (both parameters).

Note for reviewers: that test is not a reliable gate for this bug. It only reaches the short-circuit if the in-memory state bootstraps to Active while the hardware is still standby, which in the reported environment came from a failing gRPC probe. On a healthy setup linkmgrd probes the hardware correctly, so the follow-up config takes the mismatch path and passes either way. The syslog observable above is the reliable signal.

Any platform specific information?

None. The change is in the active-active state machine and is platform independent.

Documentation

No documentation change needed; this restores the documented behavior of config mux mode.

handleMuxConfigNotification() skipped switchMuxState() when the requested mode
already matched the in-memory mux state. That state is bootstrapped from the
first probe after a linkmgrd restart and is never reconciled against the
hardware, so a matching `config mux mode active|standby` was silently dropped:
the CLI reported success, the driver was never told, and the port was left
reporting an inconsistent mux status.

Drop the comparison so an explicit Active or Standby config is always forwarded
to the driver. switchMuxState() is already called with forceSwitch on these
paths, so the mismatch case is unchanged. Auto, Manual and Detached modes keep
their existing behavior.

Fixes sonic-net/sonic-buildimage#26391

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Nikola Dancejic <ndancejic@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request label(s) for branch(es): 202605, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202605: <test result>) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

@zjswhhh

zjswhhh commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@abelamit abelamit added the Tested for 202605 branch Tested for 202605 branch label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: [202511][Dual-ToR AA] | Failed to set mux status to active

5 participants