[vpp] saivpp VXLAN L3 decap: source-independent decap, inner source MAC, BD teardown fix

[aaronber0614]

Description of PR

Summary:
Enable L3 VXLAN decap on the sonic-vpp platform (saivpp / vslib/vpp) and fix
several pre-existing decap setup/teardown defects found during review.

Part of sonic-net/sonic-buildimage#25777

Depends on the VPP patch series in sonic-net/sonic-platform-vpp#262 being in the
built image first. All changes are confined to vslib/vpp.

Changes:

• Source-independent (secondary-VTEP) VNET tunnel decap (decap_any) on both the
  primary and secondary L3 VNET paths; L2 EVPN tunnels intentionally keep exact
  outer-source validation.
• Set the L3 VXLAN tunnel inner source MAC to the router MAC via a new
  sw_interface_set_mac_by_index binary-API helper, so the inner Ethernet header
  matches what HW ASICs and the VNET decap tests expect (default was VPP's
  auto-generated 02:fe:.. MAC).
• Review hardening of the secondary-VTEP decap setup/teardown: refcount the
  shared VRF0 VTEP local-receive route so tearing down one VNI does not break
  its siblings; check every VPP API return in create_vxlan_decap_term and roll
  back the BD/BVI on failure instead of returning success half-programmed; and
  install the L3 decap terms from the tunnel-create path too, so a tunnel
  created after its map entry still gets programmed.
• Detach the VXLAN tunnel interface from its bridge-domain before deleting the
  BD in remove_vpp_vxlan_decap: the tunnel was still a BD member at delete time
  (it is removed later by remove_vpp_vxlan_encap), so
  vpp_bridge_domain_add_del(is_add=0) failed with -120 (BD in use). Benign (BD
  reclaimed later) but it tripped loganalyzer during teardown.

Files: vslib/vpp/{TunnelManager.cpp, TunnelManager.h, SwitchVpp.cpp,
vppxlate/SaiVppXlate.c, vppxlate/SaiVppXlate.h}.

Type of change

• Bug fix
• New feature

Approach

What is the motivation for this PR?

Enable correct L3 VXLAN decap on sonic-vpp and remove decap setup/teardown
defects: source-dependent decap, wrong inner source MAC, and the BD-in-use
teardown error.

How did you do it?

Changes in vslib/vpp (TunnelManager, SwitchVpp, SaiVppXlate) to install
source-independent decap terms, set the tunnel inner source MAC, and fix the BD
member-detach ordering on teardown.

How did you verify/test it?

Rebuilt syncd-vpp, hot-swapped onto a t1-lag-vpp testbed, and ran the VXLAN
regression suite: test_vnet_decap 4/4, test_vxlan_multiple_tunnels 16/16, and
vxlan/test_vxlan_ecmp.py with zero -120 bridge-domain-delete errors and zero
leaked bridge-domains after teardown.

Any platform specific information?

sonic-vpp (vslib/vpp) only; no other platform is affected.

Documentation

No doc/HLD changes.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR enhances the sonic-vpp (vslib/vpp) VXLAN implementation to support correct L3 VNET decapsulation behavior (including source-independent decap) and hardens VXLAN decap setup/teardown to avoid known failure modes during tunnel lifecycle operations.

Changes:

• Add support for source-independent VXLAN decap terms (“decap_any”) by encoding a flag into decap_next_index for patched VPP VXLAN behavior.
• Add a VPP binary-API helper to set a tunnel interface MAC by sw_if_index, and use it so L3 VXLAN inner source MAC matches the router MAC.
• Add L3 secondary-VTEP decap term install/teardown logic and a late-tunnel hook to handle out-of-order tunnel vs. map-entry creation; adjust BD teardown ordering to avoid BD-in-use errors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vslib/vpp/vppxlate/SaiVppXlate.h	Adds decap-any flag encoding definitions, extends VXLAN tunnel struct, and declares sw_interface_set_mac_by_index.
vslib/vpp/vppxlate/SaiVppXlate.c	Implements sw_interface_set_mac_by_index and encodes decap_any into the wire decap_next_index.
vslib/vpp/TunnelManager.h	Introduces APIs and state for L3 secondary-VTEP decap term install/teardown and refcounting.
vslib/vpp/TunnelManager.cpp	Programs decap-any for L3 VNET, sets tunnel MAC to router MAC, adds L3 decap term install/remove logic, and fixes BD teardown ordering.
vslib/vpp/SwitchVpp.cpp	Invokes the late-tunnel hook after VXLAN tunnel creation to backfill missing L3 decap terms.

[yejianquan]

Thanks — the updates in ff8519fe all look right to me.

The IPv6 fix is better than what I suggested: forcing s6_addr[0] = 0x20 makes the invariant hold unconditionally instead of only detecting a violation after the fact, and you're right that an assert() would have been compiled out under NDEBUG anyway. The best-effort policy comment on install_l3_vxlan_decap_terms() and the explicit version-coupling sentence on the flag both answer what I was asking for.

On the third stale reference at SaiVppXlate.h:253 — good catch, and thanks for leaving it out of scope. That one is mine (06e38d8a); I'll put up a separate one-line change for it rather than widen this PR.

One thing I'd like to flag before this merges

I think the Copilot comment on the early return in install_l3_vxlan_decap_terms() is a genuine gap, not a false positive, and it happens to sit on the exact scenario this series enables. The guard is keyed on the map-entry OID:

sai_object_id_t term_oid;
sai_deserialize_object_id(map_entry_serialized_oid, term_oid);
if (m_vxlan_decap_term_map.find(term_oid) != m_vxlan_decap_term_map.end()) {
    return SAI_STATUS_SUCCESS;   // already installed
}

but the work it guards is per-tunnel — the function goes on to loop over every tunnel referencing the mapper and create a decap term for each, storing them as a vector under that one key. So:

1. Map entry E is created while tunnel A already references mapper M. A term is installed for A and m_vxlan_decap_term_map[E] becomes non-empty.
2. Tunnel B is created later against the same mapper M. handle_l3_vxlan_tunnel_create(B) walks M's entries, reaches E, and returns at the guard above.
3. B's secondary-VTEP decap term is never installed, and nothing retries it.

The pure late-tunnel case you designed the hook for is safe, because when no tunnel exists yet created stays empty and the if (!created.empty()) means nothing is recorded, so the next hook re-attempts. The same is true when the only candidate is the primary VTEP and is_local_skip fires. The gap needs at least one non-local tunnel installed at map-entry time plus a further tunnel arriving afterwards — which is plausibly what a multi-VTEP ordering in test_vxlan_multiple_tunnels.py produces, and that test is being added to the t1-lag-vpp PR set in sonic-mgmt#26220.

Making the guard per (map entry, VTEP) rather than per map entry would close it — e.g. keep the vector but skip only the tunnels whose VTEP IP already has a term recorded under that entry, so a newly-appearing tunnel still gets one. That also makes the function properly idempotent in the sense the comment claims.

Worth confirming against a run of test_vxlan_multiple_tunnels.py once the image carries these changes, since that's the case that would expose it.

On the other three Copilot comments

For what it's worth, I read the one about the handle_l2_vxlan_tunnel_map_entry reference in the handle_l3_vxlan_tunnel_create() comment as a false positive: the VNI_TO_VIRTUAL_ROUTER_ID branch genuinely does live inside handle_l2_vxlan_tunnel_map_entry, so the comment is accurate — the function name is just historically misleading. Renaming it is out of scope here.

The two about unchecked return codes (sw_interface_set_mac_by_index() in create_vpp_vxlan_encap(), and the unbridge in remove_vpp_vxlan_decap()) both seem fair to me. The MAC one especially: if it fails the tunnel silently keeps the 02:fe:.. auto MAC, which is the exact condition the change exists to prevent, and the caller still sees success — at minimum worth an error log.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

Thanks @yejianquan — you're right on all three, and I've pushed the fix as fc3bb733.

Per-(map entry, VTEP) guard

Implemented exactly as you suggested. I kept the vector under the map-entry key but replaced the whole-entry early-return with per-VTEP idempotency. The function now:

• looks up the entry's existing vector (if any) instead of returning on its presence;
• in the tunnel loop, skips only VTEPs whose source IP already has a term recorded — either under this entry from a prior hook (existing) or installed earlier in the same pass (created) — using the existing saivs::sai_ip_address_equal on TunnelVPPData.src_ip;
• appends the newly-created terms to the entry's vector (vec.insert(vec.end(), ...)) instead of overwriting it.

So the step-3 scenario you described — tunnel B arriving after entry E was processed with tunnel A already termed — now installs B's decap term and appends it, while A is skipped. That also makes the function properly idempotent in the sense the comment claims. I checked the teardown path (handle_l2_vxlan_tunnel_map_entry_removal) iterates the full vector, so appending is safe on removal.

The two unchecked return codes

Both added as (non-fatal) error logs:

• sw_interface_set_mac_by_index() in create_vpp_vxlan_encap() — logs if setting the router MAC fails, so the silent 02:fe:.. auto-MAC fallback (the exact condition the change exists to prevent) is now diagnosable instead of returning success.
• the unbridge set_sw_interface_l2_bridge_by_index(..., is_add=false, ...) in remove_vpp_vxlan_decap() — logs if the detach fails, which is what precedes the -120 (BD in use) on the subsequent BD delete.

The handle_l2_vxlan_tunnel_map_entry naming comment

Agreed — false positive, left as-is. The VNI_TO_VIRTUAL_ROUTER_ID branch does live in handle_l2_vxlan_tunnel_map_entry; the name is just historically misleading.

Verification

Built these changes into a docker-syncd-vpp image and ran test_vxlan_multiple_tunnels.py on a t1-lag-vpp testbed — i.e. the multi-VTEP ordering you flagged: 16/16 passed, BGP 24/24 established, no new cores.

And thanks for taking the third stale reference at SaiVppXlate.h:253 in a separate change — left it out of scope here.

Co-authored with CoPilot

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

vslib/vpp/TunnelManager.cpp:470

• remove_vpp_vxlan_decap() returns the bridge-domain id to dynamic_bd_id_pool before attempting to delete the bridge domain. If vpp_bridge_domain_add_del(bd_id, false) fails (e.g., BD still in use), the id can be reallocated while the old bridge-domain still exists; the current code also logs a successful delete unconditionally.

This issue also appears on line 1305 of the same file.

    m_switch_db->dynamic_bd_id_pool.free(tunnel_data.bd_id);
    refresh_interfaces_list();
    //bd is create automatically when the fist interface is add to it but requires manual deletion
    vpp_bridge_domain_add_del(tunnel_data.bd_id, false);

vslib/vpp/TunnelManager.cpp:1309

• remove_vxlan_decap_term() frees the bridge-domain id back to dynamic_bd_id_pool before deleting the bridge-domain (and ignores the delete return code). If vpp_bridge_domain_add_del() fails, the id can be reused while the old bridge-domain still exists; the current code also logs successful teardown regardless of BD delete result.

    snprintf(hw_bvi_ifname, sizeof(hw_bvi_ifname), "bvi%u", tunnel_data.bd_id);
    delete_bvi_interface(hw_bvi_ifname);
    m_switch_db->dynamic_bd_id_pool.free(tunnel_data.bd_id);
    refresh_interfaces_list();
    vpp_bridge_domain_add_del(tunnel_data.bd_id, false);

vslib/vpp/TunnelManager.cpp:1297

• remove_vxlan_decap_term() deletes the decap-only VXLAN tunnel without setting req.decap_any=true. Since vpp_vxlan_tunnel_add_del encodes the source-independent decap behavior via the high-bit flag, omitting it on delete can leave the decap-any entry/tunnel state behind in VPP.

    req.instance = ~0;
    req.vni = tunnel_data.vni;
    req.decap_next_index = ~0;
    sai_ip_address_t src_nc = tunnel_data.src_ip;
    sai_ip_address_t_to_vpp_ip_addr_t(src_nc, req.src_address);

vslib/vpp/SwitchVpp.cpp:1528

• handle_l3_vxlan_tunnel_create() is invoked unconditionally even if create_l2_vxlan_tunnel() fails and SwitchVpp::create returns a failure status. That can program L3 secondary-VTEP decap state for a tunnel create that ultimately fails, potentially leaking VPP resources.

        sai_status_t status = m_tunnel_mgr.create_l2_vxlan_tunnel(object_id, sw_if_index);
        SWSS_LOG_INFO("L2 VXLAN tunnel create for %s: status=%d sw_if_index=%u",
            serializedObjectId.c_str(), status, sw_if_index);

        // Late-tunnel hook: install any L3 secondary-VTEP decap terms whose
        // TUNNEL_MAP_ENTRY was created before this tunnel existed (M3).
        m_tunnel_mgr.handle_l3_vxlan_tunnel_create(object_id);
        return status;

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[aaronber0614]

@yejianquan quick follow up on your cross note about the map entry granularity early return in install_l3_vxlan_decap_terms().

Fixed in fc3bb733. Rather than returning as soon as the map entry has any recorded term, it now skips only the VTEPs that already have a decap term under that entry and installs one for any tunnel that appears later, appending to the entry's vector instead of overwriting it. That also covers the Copilot comment on the same early return, plus the two unchecked return codes (the router MAC in create_vpp_vxlan_encap and the unbridge in remove_vpp_vxlan_decap).

Worth recording for the merge notes though. I ran test_vxlan_multiple_tunnels.py on a t1-lag-vpp testbed with a docker-syncd-vpp image carrying the fix and it passes 16 of 16, and it also passes on the pre fix base ff8519fe. So the exact ordering that would expose the gap you described isn't actually reproduced by the test as it stands today. I'm treating the change as a defensive correctness fix for the multi VTEP decap path rather than something the current test forces, and the single VTEP path behaves exactly as before.

For the record the run stayed healthy end to end, BGP 24 of 24 and no new cores. If you think it's worth locking down I can add a dedicated multi VTEP ordering case to the sonic-mgmt set as a follow up so this path gets explicit coverage.

Co-authored with CoPilot

[yue-fred-gao]

This function is very similar to sw_interface_set_mac. Please refactor to reuse the common code

[aaronber0614]

Resolved. Please re-review @yue-fred-gao

[yue-fred-gao]

do we need to set decap_any flag for deletion?

[aaronber0614]

No. On delete VPP finds the tunnel by its key (dst, vni, encap_fib, src-port, plus src for v4), which does not include decap_next_index, then uses the flag it stored at create (t->decap_any) to also remove the wildcard entry and clear L3 promiscuous. So the remove request does not need to carry decap_any; setting it would be a no-op. See the delete path of vnet_vxlan_add_del_tunnel in patch 0017.

Co-authored with CoPilot

[yue-fred-gao]

This is only added to create. Do we need to handle tunnel being deleted when tunnel map entries are kept?

[aaronber0614]

The secondary VTEP decap terms are anchored to the map entries, not the tunnel: handle_l2_vxlan_tunnel_map_entry_removal frees them (via m_vxlan_decap_term_map) on TUNNEL_MAP_ENTRY removal. Only create needs the late tunnel hook, because a map entry can arrive before its tunnel; delete has no symmetric race since removing the entries frees the terms.
You are right about the gap: a TUNNEL deleted while its TUNNEL_MAP_ENTRYs are kept would leak, since nothing sweeps the terms then. Normal VNET teardown removes the entries first, so it does not happen in practice, but I added a symmetric handle_l3_vxlan_tunnel_removal sweep as defense in depth, pushed in 9ccc457. It runs before remove_internal (while the tunnel and its DECAP_MAPPERS links are still resolvable) and is refcount aware, since terms are deduplicated per VTEP: it frees a term only once no surviving VXLAN tunnel still references that mapper with the same VTEP source IP.

Co-authored with CoPilot

[yue-fred-gao]

I feel this is very SONiC specific behaviour. Is there stronger hint from other SAI API calls? I think I don't quite understand the sequence of SAI API calls leading to this. Can you please share the SAI calls captured from sairedis.rec? Essentially, how SONiC express primary and secondary vtep? how decap-any is derived from SAI attributes?

[aaronber0614]

It is not a heuristic. decap_any is set only on the L3 VNET path, both mapper directions. On the encap side (VIRTUAL_ROUTER_ID_TO_VNI, in tunnel_encap_nexthop_action) every tunnel built toward a remote peer is created with decap_any=true, so the local VTEP decaps VXLAN from any outer source. On the decap side (VNI_TO_VIRTUAL_ROUTER_ID, via install_l3_vxlan_decap_terms) the same flag is set, but that explicit term is only created for a VTEP source IP that is not one of our own interfaces: create_vxlan_decap_term probes with vpp_sw_interface_find_by_ip and skips the local (primary) VTEP, since it is already decapped by the encap path. The L2 EVPN path (VLAN mappers) never sets decap_any, so exact source matching is preserved there. So primary VTEP = the ENCAP_SRC_IP that matches a local interface (Loopback0); a non-local VTEP source is the only case that gets its own explicit term plus a VRF0 local-receive. Remote peer VTEPs, the tunnel destinations, are separate and only appear as tunnel-encap nexthops. The line at 1131 is just the placeholder dst for the decap only term: VPP needs a valid dst different from src, so I set a fixed high byte for v6 and 0.0.0.0 for v4. That is a VPP API constraint, not where decap-any is decided.
On a stronger SAI hint, and to answer how SONiC expresses this: I captured a sairedis.rec from a t1-lag-vpp VNET run. Per VNET, SONiC creates four tunnel maps (L2 VLAN<->VNI and L3 VirtualRouter<->VNI), one VXLAN TUNNEL with ENCAP_SRC_IP set to the local VTEP (10.1.0.32) and PEER_MODE P2MP, and a single TUNNEL_TERM_TABLE_ENTRY of type P2MP with DST_IP equal to that local VTEP and no SRC_IP. Per the SAI spec P2MP matches an exact dst with a masked (any) source, so that one term already expresses source independent (decap-any) decap for the local VTEP. Primary VTEP = ENCAP_SRC_IP (the same address as the term DST_IP); the remote peer VTEPs show up only as SAI_NEXT_HOP_TYPE_TUNNEL_ENCAP nexthops (IPs like 100.0.1.10), with no decap term of their own. So the SAI native decap-any signal is the P2MP term type, and a cleaner v2 would key VXLAN decap-any off it rather than the mapper type plus local-interface check. saivpp already reads the term type on the IPinIP path but the VXLAN term path currently just stores the object. Note this single-loopback topology never emits a non-local VTEP source, so the explicit secondary-term branch above is not exercised here; decap is entirely the primary encap path plus this one P2MP term.

Excerpt:

c|SAI_OBJECT_TYPE_TUNNEL:...6cc|TYPE=VXLAN|ENCAP_SRC_IP=10.1.0.32
|ENCAP_MAPPERS=[VLAN_ID_TO_VNI,VIRTUAL_ROUTER_ID_TO_VNI]
|DECAP_MAPPERS=[VNI_TO_VLAN_ID,VNI_TO_VIRTUAL_ROUTER_ID]|PEER_MODE=P2MP
c|SAI_OBJECT_TYPE_TUNNEL_TERM_TABLE_ENTRY:...6cd|TYPE=P2MP|VR_ID=...002
|DST_IP=10.1.0.32|TUNNEL_TYPE=VXLAN|ACTION_TUNNEL_ID=...6cc
c|SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:...6ce|TYPE=VIRTUAL_ROUTER_ID_TO_VNI|VR=...002|VNI=10000
c|SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:...6cf|TYPE=VNI_TO_VIRTUAL_ROUTER_ID|VNI=10000|VR=...002
c|SAI_OBJECT_TYPE_NEXT_HOP:...6d0|TYPE=TUNNEL_ENCAP|IP=100.0.1.10|TUNNEL_ID=...6cc
IPv6 VNET emits the same shape (term TYPE=P2MP, DST_IP=fc00:1::32).

Co-authored with CoPilot

[yue-fred-gao]

Thanks for the explanation.

[yue-fred-gao]

The vtep_ip is from encap_source_ip in tunnel object and you said "but that explicit term is only created for a VTEP source IP that is not one of our own interfaces". Does this mean the encap_source_ip can be an ip not a local interface address? is this ip advertised by bgp otherwise how remote can reach it?

[aaronber0614]

Correct. vtep_ip is the tunnel ENCAP_SRC_IP. In the standard VNET model that is always the local Loopback0 VTEP, which BGP advertises into the underlay so remote peers reach us, so it is never non local in a normal config. The primary VTEP guard therefore always fires and the secondary VTEP term branch below is a defensive path a single loopback config never hits

Co-authored with CoPilot

[yue-fred-gao]

The SAI_TUNNEL_ATTR_ENCAP_SRC_IP in above TUNNEL is 10.1.0.32, which is the normal case with loopback address. Is there a real usecase where the encap source ip is not a local ip? You said it is a defensive path so I am wondering if it is really necessary.

[aaronber0614]

Hi @yue-fred-gao , I modified the comment. It's not defensive but necessary. The encap source IP does not have to be a local IP. test_vxlan_multiple_tunnels configures :

1. the loopback0 tunnel (10.1.0.32, local) and
2. a second tunnel whose SRC is the Loopback0 IP BUT with the last octet set to 100 or 101. (e.g. 10.1.0.100). This address is not assigned to an interface. It then sends the outer packet to this special SRC and expects DUT to decap it.

The test assumes that on ASICs, (not specific to VPP) that the DUT is required to decap the special SRC even if not one of its interface address. VPP is the odd one out in how it gets there. VPP only local-receives an IP that is on an interface or has an explicit local route, so without this branch there is no local-receive for the special src, the outer packet is never punted to vxlan-input, and it is not decapped.

The change adds a VRF0 /32 or /128 VPP_NEXTHOP_LOCAL receive for exactly those non-interface VTEP IPs, while the primary Loopback0 VTEP is skipped by the local-interface guard.
In our run that is what takes the special-src subcases from failing to passing.

Co-authored with CoPilot

[yue-fred-gao]

Thanks. Honestly, I don't know how the secondly vtep works in production, especially how the second vtep's source ip is advertised. There is no integration with frr to advertise vtep's source ip via routing protocols. The PR is solid to support the feature. Just how it is intended to be used is unclear. If you can find a document to describe the use case and ways to advertise the source ip, that will be helpful to understand it.

[aaronber0614]

@yue-fred-gao . I looked at this a little bit more. So the "secondary" vtep is a sonic-mgmt test construct, and not really something that's done in production. It's needed to test decap for src that is not the box's interface addess.

[yue-fred-gao]

Thanks for the clarification. Maybe the sonic-mgmt goes too far to verify some behavior that is not required in production. It might even be some platform specific behavior. For example, some may implement vtep as an interface, which would naturally use vtep source ip as the pseduo interface ip. But sonic doesn't model vtep as an interface so the source ip is not added to the control plane and advertised by routing protocols.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[yue-fred-gao]

The vtep_ip is from encap_source_ip in tunnel object and you said "but that explicit term is only created for a VTEP source IP that is not one of our own interfaces". Does this mean the encap_source_ip can be an ip not a local interface address? is this ip advertised by bgp otherwise how remote can reach it?

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).