Bug: [dualtor][vpp] Standby mux port reports ~99% RX_DRP — ACL policy denies are charged to the interface discard counter

[lolyu]

Tracked under umbrella issue #26046.

Fix is up for review as sonic-platform-vpp#280 — [vpp][acl] do not count policy denies as interface drops.

Is it platform specific

vpp

Importance or Severity

Low–Medium. Cosmetic in the sense that forwarding is correct — but it makes a healthy standby ToR indistinguishable from a broken one on the counter an operator reaches for first, and it fails an upstream test case.

Description of the bug

On a dual-ToR VPP testbed, a standby mux port reports a large and growing RX_DRP count — around 99% of everything it receives — while forwarding perfectly correctly. Dropping that traffic is exactly what a standby ToR is supposed to do.

The traffic is denied by the mux drop ACL. In VPP the ACL deny path sends the buffer to error-drop, and interface_drop_punt() charges every buffer it sees to the ingress interface's /if/drops. saivpp exports that as SAI_PORT_STAT_IF_IN_DISCARDS, which utilities_common/portstat.py maps to the RX_DRP column:

# utilities_common/portstat.py
NStats = namedtuple("NStats", "rx_ok, rx_err, rx_drop, ...")
...
1: ['SAI_PORT_STAT_IF_IN_ERRORS'],      # -> RX_ERR
2: ['SAI_PORT_STAT_IF_IN_DISCARDS'],    # -> RX_DRP

So a deliberate policy decision is reported as a discard.

Where SONiC expects an ACL drop to be counted. Upstream sonic-mgmt is explicit. For discard_group == "ACL", tests/drop_packets/test_drop_counters.py asserts the per-ACE ACL rule counter increments, then — on any platform not listed as having combined counters — asserts that neither RX_DRP nor RX_ERR moved:

if acl_drops != pkt_number:
    pytest.fail(...)                      # per-ACE counter MUST increment
if not COMBINED_ACL_DROP_COUNTER:
    executor.submit(ensure_no_l3_and_l2_drops, duthost, packets_count=pkt_number)

VPP is already declared as such a platform. tests/drop_packets/combined_drop_counters.yml lists the platforms where ACL and L2 drop counters legitimately are combined (x86_64-mlnx, x86_64-dell.*, x86_64-arista.*, x86_64-cel_seastone.*, x86_64-nokia.*). x86_64-kvm_x86_64-r0 (VPP) appears in that file's l2_l3 list but deliberately not in acl_l2 — added by @AkeelAli in sonic-mgmt#24031. Upstream has therefore already declared that on VPP an ACL drop must not land in the L2/RX_DRP counter; today VPP contradicts that declaration.

Note: an earlier revision of this issue argued from RFC 1213. That was overstated — ifInDiscards is "packets which were chosen to be discarded even though no errors had been detected... one possible reason ... could be to free up buffer space", so buffer exhaustion is an example rather than the definition, and a strict reading leaves RX_DRP defensible. The justification is the SONiC-level convention above, not the RFC. See the PR discussion.

Note the drops themselves are correct and must stay observable. The fix keeps them fully visible as per-node and per-ACE errors in show errors and show acl-plugin acl; only the interface discard counter stops being charged.

Steps to Reproduce

1. Deploy a dual-ToR VPP KVM testbed (dualtor-aa-vpp, or any active-standby dual-ToR VPP topology).
2. Toggle a mux port so one ToR is standby for it.
3. Send upstream traffic from the server through that mux port.
4. Run show interfaces counters on the standby ToR.

Equivalently, run the upstream test case, which exists on sonic-mgmt master (added by #23577):

dualtor_io/test_normal_op.py::test_upstream_standby_rx_drop_check

Actual Behavior and Expected Behavior

Actual — nearly every received packet is counted as a discard:

Port Ethernet16 on standby ToR vlab-vpp-04 has RX_DRP increment of 5706
which exceeds threshold of 10 (before=0, after=5706)

Expected — RX_DRP stays at (or near) zero on a standby mux port, because nothing is being discarded for lack of resources. With sonic-platform-vpp#280 applied, the standby ToR received 5762 packets on the mux port with RX_DRP=0, while show errors still reported the denies against acl-plugin-in-ip4-l2, and the test above passes.

Relevant log output

$ show interfaces counters
      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP  ...
 Ethernet16        U     5706       ...        ...         0      5706  ...

Meanwhile the denies are visible where they belong:

$ vppctl show errors
   Count            Node                    Reason
    5706   acl-plugin-in-ip4-l2            ACL deny packets

Additional context — proposed fix

sonic-platform-vpp#280 adds patch 0019-acl-do-not-count-policy-denies-as-interface-drops.patch:

1. src/vnet/buffer.h — allocate VNET_BUFFER_F_POLICY_DROP on bit 19, the first free bit.
2. src/plugins/acl/dataplane_node.c — set the flag at the single ACL deny choke point in acl_fa_inner_node_fn().
3. src/vnet/interface_output.c — interface_drop_punt() discounts flagged buffers from the interface drop counter, reusing the per-buffer loop already added by patch 0008, so there is no new pass over the frame.

The flag is set only for ACL_FA_ERROR_ACL_DROP. Drops from a genuine lack of resources, such as ACL_FA_ERROR_ACL_TOO_MANY_SESSIONS, remain real discards and keep incrementing the interface counter — which is the distinction this change is about.

Once #280 merges, platform/vpp needs a submodule advance here to pick it up (VPP_VERSION moves to 2606-0.7).