Skip to content

[action] [PR:27910] Scale BGP route-convergence timeouts by neighbor count in IDF isolation tests - #1418

Merged
bingwang-ms merged 1 commit into
Azure:202512from
mssonicbld:cherry/msft-202512/27910
Sep 16, 2026
Merged

bingwang-ms merged 1 commit into
Azure:202512from
mssonicbld:cherry/msft-202512/27910

Conversation

@mssonicbld

Copy link
Copy Markdown
Collaborator

Description of PR

Summary:
Scale BGP route-convergence wait_until timeouts in bgp/test_seq_idf_isolation.py by the number of BGP neighbors, instead of using flat/fixed timeouts. On large topologies (250+ downlink/uplink BGP neighbors), parsing and re-converging routes across all neighbors takes noticeably longer than on small topologies, causing the previous fixed timeouts (600s / 180s) to trigger false test failures even though the DUT itself converges correctly and quickly.

Type of change

  • Bug fix

Back port request

  • 202205
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202605

Approach

What is the motivation for this PR?

test_idf_isolation_no_export_with_config_reload and test_idf_isolated_withdraw_all (and the closely related test_idf_isolation_withdraw_all_with_config_reload) intermittently fail on large-scale topologies with hundreds of BGP neighbors. The failures show routes not yet converged/withdrawn on some neighbors, but manual inspection of the DUT's BGP RIB/Adj-RIB-Out and route-map configuration confirms the DUT itself is already in the correct, converged state. The actual cause is that parsing/verifying routes across all neighbors takes long enough on large topologies that the fixed wait_until timeouts don't leave enough retry attempts to observe convergence within the time budget.

How did you do it?

  • Added a scaled_route_convergence_timeout(neigh_hosts, base=180, per_neighbor=5, maximum=1800) helper in route_checker.py that computes min(maximum, base + per_neighbor * len(neigh_hosts)).
  • Changed assert_only_loopback_routes_announced_to_neighs()'s timeout parameter to default to None, auto-computing the scaled value via the new helper when not explicitly passed. This transparently benefits all existing callers (e.g. in test_traffic_shift.py, test_startup_tsa_tsb_service.py) without requiring changes to those files.
  • Updated test_idf_isolation_no_export_with_config_reload's two hardcoded wait_until(600, ...) calls to use the scaled timeout based on the test's neighbor count.

How did you verify/test it?

Ran the three affected tests live on a physical testbed with a large-scale topology (250+ downlink/uplink BGP neighbors):

  • test_idf_isolation_no_export_with_config_reload — previously failed at 600s; passed after scaling (IPv6 convergence took ~20 minutes).
  • test_idf_isolated_withdraw_all — previously failed at a flat 180s; passed after scaling (~700s total).
  • test_idf_isolation_withdraw_all_with_config_reload — shares the same pattern; verified it also benefits from the new default via the same call path used by test_idf_isolated_withdraw_all.

Confirmed via vtysh/ansible ad-hoc commands that the DUT's own BGP RIB and Adj-RIB-Out were already correctly converged well before the old timeouts expired, confirming this is a test-timing issue rather than a functional bug in the isolation feature.

Any platform specific information?

None specific to a platform; applies to any topology with a large number of BGP neighbors.

Supported testbed topology if it's a new test case?

Not a new test case; existing t2/lt2 topology tests.

Documentation

N/A

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

…on tests

### Description of PR
Summary:
Scale BGP route-convergence `wait_until` timeouts in `bgp/test_seq_idf_isolation.py` by the number of BGP neighbors, instead of using flat/fixed timeouts. On large topologies (250+ downlink/uplink BGP neighbors), parsing and re-converging routes across all neighbors takes noticeably longer than on small topologies, causing the previous fixed timeouts (600s / 180s) to trigger false test failures even though the DUT itself converges correctly and quickly.

### Type of change
- [x] Bug fix

### Back port request
- [ ] 202205
- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [x] 202511
- [x] 202605

### Approach
#### What is the motivation for this PR?
`test_idf_isolation_no_export_with_config_reload` and `test_idf_isolated_withdraw_all` (and the closely related `test_idf_isolation_withdraw_all_with_config_reload`) intermittently fail on large-scale topologies with hundreds of BGP neighbors. The failures show routes not yet converged/withdrawn on some neighbors, but manual inspection of the DUT's BGP RIB/Adj-RIB-Out and route-map configuration confirms the DUT itself is already in the correct, converged state. The actual cause is that parsing/verifying routes across all neighbors takes long enough on large topologies that the fixed `wait_until` timeouts don't leave enough retry attempts to observe convergence within the time budget.

#### How did you do it?
- Added a `scaled_route_convergence_timeout(neigh_hosts, base=180, per_neighbor=5, maximum=1800)` helper in `route_checker.py` that computes `min(maximum, base + per_neighbor * len(neigh_hosts))`.
- Changed `assert_only_loopback_routes_announced_to_neighs()`'s `timeout` parameter to default to `None`, auto-computing the scaled value via the new helper when not explicitly passed. This transparently benefits all existing callers (e.g. in `test_traffic_shift.py`, `test_startup_tsa_tsb_service.py`) without requiring changes to those files.
- Updated `test_idf_isolation_no_export_with_config_reload`'s two hardcoded `wait_until(600, ...)` calls to use the scaled timeout based on the test's neighbor count.

#### How did you verify/test it?
Ran the three affected tests live on a physical testbed with a large-scale topology (250+ downlink/uplink BGP neighbors):
- `test_idf_isolation_no_export_with_config_reload` — previously failed at 600s; passed after scaling (IPv6 convergence took ~20 minutes).
- `test_idf_isolated_withdraw_all` — previously failed at a flat 180s; passed after scaling (~700s total).
- `test_idf_isolation_withdraw_all_with_config_reload` — shares the same pattern; verified it also benefits from the new default via the same call path used by `test_idf_isolated_withdraw_all`.

Confirmed via `vtysh`/ansible ad-hoc commands that the DUT's own BGP RIB and Adj-RIB-Out were already correctly converged well before the old timeouts expired, confirming this is a test-timing issue rather than a functional bug in the isolation feature.

#### Any platform specific information?
None specific to a platform; applies to any topology with a large number of BGP neighbors.

#### Supported testbed topology if it's a new test case?
Not a new test case; existing `t2`/`lt2` topology tests.

### Documentation
N/A

Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Original PR: sonic-net/sonic-mgmt#27910

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

1 similar comment
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@bingwang-ms
bingwang-ms merged commit 1d08c54 into Azure:202512 Sep 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants