[202405][code sync] Restore BuildVS CI

[mssonicbld]

Summary

• retain the public-to-msft 202405 sync from [code sync] Merge public 202405 into msft 202405 #3063
• retain the release-local quilt patch that replaces Python 3.10-only match/case dispatch with equivalent if/elif logic for Bullseye/Python 3.9
• complete the missing libyang1 YANG-management compatibility fixes required by the pinned utilities revision
• preserve original grouping metadata for CLI generation and correct stale BGP chassis-detection mocks without weakening expected output
• restore artifact staging independently of disabled Docker/PTF publication so downstream tests can download the built images
• pin the management-container runtime used by the seven required KVM jobs to a validated immutable image compatible with the release test framework
• request the approved registry-qualified cEOS 4.29.3M-1 name for the six cEOS jobs, preserving the SONiC-neighbor job and all image-availability guards; actual KVM parameter forwarding is currently blocked in Elastictest, as detailed below

This PR supersedes #3063. It must be merged with a merge commit (no squash or rebase) to preserve the exact public-source ancestry already recorded by commit 6dfec61.

Initial root cause

Build 1213980 fails while building sonic_utilities-1.2-py3-none-any.whl under Python 3.9:

File "/sonic/src/sonic-utilities/generic_config_updater/patch_sorter.py", line 1615
    match op:
          ^
SyntaxError: invalid syntax

The incompatible statement was introduced by 38991e8574f3, included in the 202405 submodule tip 45eb547d0f3b. Both PR 3063's base and head already use that same gitlink, so the code-sync merge did not introduce the defect; its clean VS build exposed it.

Follow-up root causes and changes

Build 1215396 successfully applied the Python 3.9 patch, then failed in the full utilities suite: 32 failed, 3205 passed, 3 skipped. The failure was in wheel unit tests, before the VS image/KVM stages.

Area	Cause	Repair
Config management	load_module_str_name() was missing from the release YANG library	Adapt the helper from upstream commit 609bbb8fd7f1d286955191813acb678d0e7f1b91, associated with sonic-net/sonic-buildimage#24414
Generic config updater	Dependency lookup assumed leaf nodes and mishandled containers, list entries and /	Adapt the libyang1 traversal and test-isolation fixes from upstream commit 098d4cde7894dc333fecf51f2b900d99af8ed38e, sonic-net/sonic-buildimage#26563; retain value matching and remove duplicate paths
YANG grouping and CLI generation	Uses compilation and CLI grouping metadata have different requirements	Backport grouping compilation fixes; add a release-local original-schema helper and utilities patch so CLI grouping labels, ordering and refinements remain intact
BGP summary tests	Tests mocked get_platform_info(), but is_chassis() now reads CONFIG_DB directly	Patch both chassis-detection bindings; retain the expectation of 20 external peers, not 23

The YANG changes are adapted backports, not verbatim cherry-picks. The original-schema CLI adapter and BGP mock adjustments are release-local integration fixes. The existing Python 3.9 patch, utilities gitlink 45eb547d0f3b129d9f59ff01f3bf5809b6c8f489, loadData(..., quiet=False) interface and code-sync merge ancestry remain unchanged.

Local validation

Initial syntax repair:

• Confirmed the unmodified pinned source fails to compile with Python 3.9 at match op.
• Confirmed the quilt patch applies through the same src/<name>.patch/series mechanism used by slave.mk.
• Confirmed the patched file compiles with Python 3.9.
• Ran TestBulkLeafListMoveGenerator and TestLowLevelMoveGenerator: 32 passed.

Follow-up repair, using Bullseye, Python 3.9.2, pytest 6.0.2, libyang 1.0.73 and real swsscommon bindings:

Coverage	Result
Affected utilities baseline, before follow-up repairs	32 failed, 452 passed, 2 existing skips
Same affected utilities after repairs	484 passed, 2 existing skips
Full utilities suite	3237 passed, 3 existing skips
YANG-management suite	66 passed
Independent dependency/leafref isolation	2 passed
Builds of both changed wheels	Passed
Installed-wheel integration	13 passed
Installed-package Python 3.9 compilation	Passed

All 32 failure identities from build 1215396 passed locally. Existing CLI golden dictionaries and BGP peer-count expectations are unchanged; repeated CLI parsing also checks that schema mutations do not leak.

The local validation container is not byte-identical to the CI slave, and full SONiC image/KVM validation is delegated to the PR pipeline.

Artifact publication follow-up

Build 1216384, on 3970e73cfb6d9c67a874d21a371ca3b3b1b770d1, passed all five image jobs, including BuildVS. Downstream Elastictest jobs then failed while downloading images.

The VS artifact upload log reports Processed 0 files. Upstream commit de4ea070236d0d985607e93116f130f2fc185ccf disabled Docker/PTF publication with an early exit 0 in the shared post-build script, inadvertently bypassing the artifact-copy commands too.

Commit b96bb96a6d4782cffdde58404c9e4624db1fc965 moves staging into its own always() step and fails on copy errors. Docker/PTF publication stays disabled; artifact names, directory layout and success/failure upload conditions are preserved.

Focused pipeline-script checks reproduced the empty output before the change (4 failed) and passed afterward (4 passed). These exercise file staging with small fixtures for single-ASIC VS, multi-ASIC VS, a hardware image and nested packages, plus failure propagation when target is missing; they do not substitute for the real image and KVM pipeline.

Build 1216662, on commit b96bb96a6d4782cffdde58404c9e4624db1fc965, passed all five image jobs. VS publication processed 1531 files, and the actual /target/sonic-vs.img.gz is downloadable (HTTP 200, 1,872,063,233 bytes, valid gzip header), confirming the empty-artifact defect is fixed.

Management-controller compatibility follow-up

The next failure was ADD_TOPOLOGY_FAILED, before VM creation or test execution. The release sonic-mgmt checkout 89377ddcf81abc72fa981ea08d5c29f645265edc was paired with the floating management image's Python 3.12.3 / Ansible 2.20.9 runtime:

• The legacy conn_graph_facts module's #!/usr/bin/env python shebang was treated as a single executable name; the local module failed to launch with exit 127.
• The legacy SSH plugin could not import imp, and the YAML callback plugin failed to load.
• The same signatures were confirmed on multi-ASIC, dual-ToR, T0 and T1-LAG workers. This was a controller/runtime compatibility failure, not a DUT-image regression.

Commit 11bde3d3405b8a3817b4515782d9776387003383 uses the existing SETUP_CONTAINER_PARAMS / setup-container.sh -i interface to select the official 20250530 management image by immutable digest:

sonicdev-microsoft.azurecr.io:443/docker-sonic-mgmt@sha256:d9ee5be397d022d72140b8ab8e7f64161bc766ebac1b4eee8165c115ad82bb26

The selected runtime is Python 3.8.10 / Ansible core 2.13.13. Isolated A/B probes against the unchanged release checkout reproduced 6 failures with the exact failing image and 6 passes with the selected image: four real Ansible module launches and the two original plugin-loading checks. YAML parsing also confirmed that all seven required KVM jobs receive the same pin and that all other pipeline settings are unchanged.

These probes preserve the actual CI callback configuration. Separately forcing ANSIBLE_STDOUT_CALLBACK=yaml still exposes a legacy full-result rendering issue; that experimental override is neither enabled nor suppressed by this change. Local module/plugin probes do not replace the complete topology and test pipeline.

DUT images, sonic-mgmt refs, PTF tags, test selections, worker counts, callback settings and shared Elastictest infrastructure remain unchanged. No tests have been disabled or expected outputs weakened. Build 1216924, on commit 11bde3d3405b8a3817b4515782d9776387003383, confirmed the selected runtime and successful minigraph-module execution and DUT VM kickstart. The previous interpreter/plugin failure signatures are absent from the inspected preparation logs.

cEOS provisioning follow-up

The cEOS-based jobs then reached the release role's explicit missing-image guard: bare ceosimage:4.29.3M-1 and ceosimage:4.29.3M were not found, the local cEOS64-lab-4.29.3M.tar archive was absent, and the example download URLs were unusable.

The approved exact-version registry source is soniccr1.azurecr.io/ceosimage:4.29.3M-1. Its manifest/configuration metadata was verified, and worker provisioning uses registry-qualified references. Commit 96167e04fefbe2286934ee68b8bd88d33e6282b5 requests that override through the existing ADD_TOPO_PARAMS / ceos_image interface for the six cEOS jobs only.

This remains a guarded configuration candidate pending end-to-end forwarding and image-availability evidence: the available logs do not establish whether each failed worker held the fully qualified cache entry. The old role does not automatically pull an absent registry image, and this change does not bypass that limitation or any image-availability guard. No archive URL, credential, neighbor-type change or cEOS-version change is introduced.

Pipeline parsing confirms the exact six cEOS bindings, the unchanged SONiC-neighbor job, the retained management-runtime pin and unchanged test selections/worker counts. Actual worker image lookup and complete test execution still must pass in CI.

Build 1217169, on latest commit 96167e04fefbe2286934ee68b8bd88d33e6282b5 (test merge cb71e21ba390e83445ae6d6ccceeee4799534b6f), completed failed on 2026-09-11 at 05:53 UTC:

Gate	Final result
All five image builds	Succeeded
t0-sonic KVM job, including test execution	Succeeded; framework final result: 67 passed, 9 skipped, 0 failures, 0 errors
Six cEOS KVM jobs: DPU, multi-ASIC, T1-LAG, T0-2VLANs, T0 and dual-ToR	Failed during preparation, before test execution

The original t0-sonic execution records retain one first-attempt warm-reboot failure in the test framework's log-extraction step (extract_log.py, comparing NoneType and datetime.datetime); the PTF portion had returned rc=0 / OK. The existing automatic retry passed. The final successful job is therefore not a claim that every first attempt passed, and this PR did not add retries or skips.

All six failed jobs' preparation logs confirm the same terminating guard: Failed, no working ceos image download URL is found. They establish an external execution-path blocker: add_topo_params is saved correctly in the test plan, but Elastictest's KVM path does not forward it into the actual add-topo command. The inspected first and retry commands omit the requested ceos_image override and fall back to the unusable example archive URLs. Consequently, this run did not perform the qualified-image cache lookup; it does not disprove that cache's availability.

Further buildimage-only retries cannot resolve that missing server-side forwarding. The remaining work requires a separately reviewed scheduler correction or approved worker provisioning. Full CI is not green, and no test or image-availability guard has been bypassed.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).