Skip to content

[route] Fix route consistency snapshot flake - #1403

Open
LinJin23 wants to merge 2 commits into
Azure:masterfrom
LinJin23:fix-route-snapshot-race
Open

LinJin23 wants to merge 2 commits into
Azure:masterfrom
LinJin23:fix-route-snapshot-race

Conversation

@LinJin23

@LinJin23 LinJin23 commented Sep 7, 2026

Copy link
Copy Markdown

Description of PR

Summary:
Serialize ASIC_DB route snapshot collection on the controller to avoid flaky failures from concurrent pytest-ansible calls.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • New Test case
    • Skipped for non-supported platforms
  • Test case improvement

Back port request

  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO): https://msazure.visualstudio.com/One/_workitems/edit/38896300/?view=edit
Failure type: other - test infrastructure race condition

Tested branch

  • master
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • N/A

Test result

Approach

What is the motivation for this PR?

get_route_prefix_snapshot_from_asicdb launches concurrent controller threads that reuse the same pytest-ansible/SonicHost dispatcher. In the observed failures, one Ansible transaction raised AnsibleConnectionFailure and did not post its completion signal, causing the main thread to report a route snapshot timeout.

How did you do it?

Removed the controller-side threading and queue logic and collected each ASIC snapshot serially.

How did you verify/test it?

Ran route/test_route_consistency.py on 202511; all 4 test cases passed.

Any platform specific information?

The failure was observed on an Arista-7280DR3AM-36 multi-ASIC HWSKU. The change is not platform-specific.

Supported testbed topology if it's a new test case?

N/A

Documentation

N/A

Signed-off-by: Lin Jin <linjin@microsoft.com>
Signed-off-by: Lin Jin <linjin@microsoft.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@LinJin23
LinJin23 marked this pull request as ready for review September 8, 2026 15:43
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@LinJin23

LinJin23 commented Sep 8, 2026

Copy link
Copy Markdown
Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 1403 in repo Azure/sonic-mgmt.msft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants