[action] [PR:27910] Scale BGP route-convergence timeouts by neighbor count in IDF isolation tests - #1418
Merged
Conversation
…on tests ### Description of PR Summary: Scale BGP route-convergence `wait_until` timeouts in `bgp/test_seq_idf_isolation.py` by the number of BGP neighbors, instead of using flat/fixed timeouts. On large topologies (250+ downlink/uplink BGP neighbors), parsing and re-converging routes across all neighbors takes noticeably longer than on small topologies, causing the previous fixed timeouts (600s / 180s) to trigger false test failures even though the DUT itself converges correctly and quickly. ### Type of change - [x] Bug fix ### Back port request - [ ] 202205 - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [x] 202511 - [x] 202605 ### Approach #### What is the motivation for this PR? `test_idf_isolation_no_export_with_config_reload` and `test_idf_isolated_withdraw_all` (and the closely related `test_idf_isolation_withdraw_all_with_config_reload`) intermittently fail on large-scale topologies with hundreds of BGP neighbors. The failures show routes not yet converged/withdrawn on some neighbors, but manual inspection of the DUT's BGP RIB/Adj-RIB-Out and route-map configuration confirms the DUT itself is already in the correct, converged state. The actual cause is that parsing/verifying routes across all neighbors takes long enough on large topologies that the fixed `wait_until` timeouts don't leave enough retry attempts to observe convergence within the time budget. #### How did you do it? - Added a `scaled_route_convergence_timeout(neigh_hosts, base=180, per_neighbor=5, maximum=1800)` helper in `route_checker.py` that computes `min(maximum, base + per_neighbor * len(neigh_hosts))`. - Changed `assert_only_loopback_routes_announced_to_neighs()`'s `timeout` parameter to default to `None`, auto-computing the scaled value via the new helper when not explicitly passed. This transparently benefits all existing callers (e.g. in `test_traffic_shift.py`, `test_startup_tsa_tsb_service.py`) without requiring changes to those files. - Updated `test_idf_isolation_no_export_with_config_reload`'s two hardcoded `wait_until(600, ...)` calls to use the scaled timeout based on the test's neighbor count. #### How did you verify/test it? Ran the three affected tests live on a physical testbed with a large-scale topology (250+ downlink/uplink BGP neighbors): - `test_idf_isolation_no_export_with_config_reload` — previously failed at 600s; passed after scaling (IPv6 convergence took ~20 minutes). - `test_idf_isolated_withdraw_all` — previously failed at a flat 180s; passed after scaling (~700s total). - `test_idf_isolation_withdraw_all_with_config_reload` — shares the same pattern; verified it also benefits from the new default via the same call path used by `test_idf_isolated_withdraw_all`. Confirmed via `vtysh`/ansible ad-hoc commands that the DUT's own BGP RIB and Adj-RIB-Out were already correctly converged well before the old timeouts expired, confirming this is a test-timing issue rather than a functional bug in the isolation feature. #### Any platform specific information? None specific to a platform; applies to any topology with a large number of BGP neighbors. #### Supported testbed topology if it's a new test case? Not a new test case; existing `t2`/`lt2` topology tests. ### Documentation N/A Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
Collaborator
Author
|
Original PR: sonic-net/sonic-mgmt#27910 |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Collaborator
Author
|
/azp run |
1 similar comment
Collaborator
Author
|
/azp run |
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. |
9 tasks
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
Scale BGP route-convergence
wait_untiltimeouts inbgp/test_seq_idf_isolation.pyby the number of BGP neighbors, instead of using flat/fixed timeouts. On large topologies (250+ downlink/uplink BGP neighbors), parsing and re-converging routes across all neighbors takes noticeably longer than on small topologies, causing the previous fixed timeouts (600s / 180s) to trigger false test failures even though the DUT itself converges correctly and quickly.Type of change
Back port request
Approach
What is the motivation for this PR?
test_idf_isolation_no_export_with_config_reloadandtest_idf_isolated_withdraw_all(and the closely relatedtest_idf_isolation_withdraw_all_with_config_reload) intermittently fail on large-scale topologies with hundreds of BGP neighbors. The failures show routes not yet converged/withdrawn on some neighbors, but manual inspection of the DUT's BGP RIB/Adj-RIB-Out and route-map configuration confirms the DUT itself is already in the correct, converged state. The actual cause is that parsing/verifying routes across all neighbors takes long enough on large topologies that the fixedwait_untiltimeouts don't leave enough retry attempts to observe convergence within the time budget.How did you do it?
scaled_route_convergence_timeout(neigh_hosts, base=180, per_neighbor=5, maximum=1800)helper inroute_checker.pythat computesmin(maximum, base + per_neighbor * len(neigh_hosts)).assert_only_loopback_routes_announced_to_neighs()'stimeoutparameter to default toNone, auto-computing the scaled value via the new helper when not explicitly passed. This transparently benefits all existing callers (e.g. intest_traffic_shift.py,test_startup_tsa_tsb_service.py) without requiring changes to those files.test_idf_isolation_no_export_with_config_reload's two hardcodedwait_until(600, ...)calls to use the scaled timeout based on the test's neighbor count.How did you verify/test it?
Ran the three affected tests live on a physical testbed with a large-scale topology (250+ downlink/uplink BGP neighbors):
test_idf_isolation_no_export_with_config_reload— previously failed at 600s; passed after scaling (IPv6 convergence took ~20 minutes).test_idf_isolated_withdraw_all— previously failed at a flat 180s; passed after scaling (~700s total).test_idf_isolation_withdraw_all_with_config_reload— shares the same pattern; verified it also benefits from the new default via the same call path used bytest_idf_isolated_withdraw_all.Confirmed via
vtysh/ansible ad-hoc commands that the DUT's own BGP RIB and Adj-RIB-Out were already correctly converged well before the old timeouts expired, confirming this is a test-timing issue rather than a functional bug in the isolation feature.Any platform specific information?
None specific to a platform; applies to any topology with a large number of BGP neighbors.
Supported testbed topology if it's a new test case?
Not a new test case; existing
t2/lt2topology tests.Documentation
N/A
Signed-off-by: Sonic Build Admin sonicbld@microsoft.com