bench: Tier-0 smoke working end-to-end + Tier-1 (TRex/sink) scaffolding - #1
Merged
Merged
Conversation
LeTuR
force-pushed
the
bench/tier0-working-plus-tier1-scaffold
branch
from
May 25, 2026 18:34
9c0f018 to
8f850e1
Compare
2 tasks
Brings the Tier-0 smoke test green end-to-end and lays in the TRex/sink
VM scaffolding for the future Tier-1 lab box. Per tests/benchmarks.md
§ 0.6/§ 0.8, this satisfies the security-effectiveness precondition on
a developer workstation while reserving the throughput numbers for a
substrate that can produce them honestly.
Tier-0 smoke fixes:
- Topology: host bridge IPs moved off `.1` so they no longer collide
with the SUT's own LAN/WAN addresses.
- Q35 NIC naming: cloud-init and rules.nft now reference `enp1s0`
/`enp2s0` (the Q35 PCIe-slot names libvirt actually hands out), not
the i440fx `ens3`/`ens4` they were originally written for.
- Cloud-init seed delivery: `render-cloud-init.sh` injects user-data
into `/var/lib/cloud/seed/nocloud/` on the instance qcow2 via
`virt-customize`. The well-known path always wins ds-identify;
previous attempts via cidata-labeled CD-ROMs were unreliable on
customised images.
- `bench/Makefile`: new `base-customise` target pre-bakes packages
into the cached base image via virt-customize (the bench bridges
have no NAT, so cloud-init's `packages:` step has nothing to fetch).
- `smoke.sh`: gates tests on `cloud-init status --wait`, runs the
ruleset-loaded check up front via `systemctl is-active` (race-free)
so iperf3 saturation can't briefly stall the check.
- `bench/SETUP.md`: § 3a documents the qemu user/group override
(`/etc/libvirt/qemu.conf`: `user = "$USER"`, `group = "kvm"`) that
qemu:///system needs to read disks under `/home`. `guestfs-tools`
added to the package list; troubleshooting table extended.
Tier-1 scaffolding (runs but explicitly out-of-scope on Tier 0):
- `bench/candidates/trex/` — TRex VM cloud-init: software-mode TRex
with MAC-resolved 2-NIC config (required even number; port 1 is a
dummy because RX accounting comes from the sink, not from TRex).
- `bench/candidates/sink/` — receive-only WAN endpoint at
203.0.113.200 whose `/sys/class/net/enp1s0/statistics/rx_packets`
the b00 driver reads over SSH for RX counting. A dedicated VM
avoids the kernel-loopback shortcut that nukes any TRex setup where
TX and RX endpoints share a Linux network namespace.
- `bench/scripts/b00.{sh,py}` — TRex-driven binary-search NDR/PDR
sweep across the seven RFC 2544 frame sizes. Runnable but the
header comment is explicit that on Tier 0 the result is a harness
ceiling, not a SUT measurement.
- `bench/Makefile`: `trex-fetch`, `trex-up`, `trex-down`, `sink-up`,
`sink-down`, `b-00` targets; `tear-down` covers all VMs.
Verified locally: `make tear-down && make smoke CAND=nftables` exits 0
with `result.json:.security_effectiveness_precondition.satisfied=true`.
…e result Tier-0 produces meaningful functional/correctness numbers but cannot produce trustworthy throughput numbers — the workstation virtio path bottlenecks the measurement well before the SUT does. Make that explicit in three places so future readers don't chase what was never measurable here. - bench/README.md: new "Scope decision: throughput is Tier-1 only" section with reasoning (1-vCPU SUT + virtio, § 0.6 control logic) and a "what stays in scope on Tier 0" list (smoke, functional correctness, footprint metrics B-21..B-24). The TRex/sink scaffolding stays in the tree so the Tier-1 box runs `make b-00` as-is when it arrives. Also a "Latest Tier-0 smoke run" block showing today's pass — verdict, per-probe table, host invariants, Tier-0 caveat. - bench/ROADMAP.md: Tier-0 "what it can measure" now lists footprint metrics; "what it can't measure" calls out B-00..B-13 explicitly with the substrate-bottleneck reasoning and a pointer to the scaffolding that's runnable for Tier 1.
LeTuR
force-pushed
the
bench/tier0-working-plus-tier1-scaffold
branch
from
May 25, 2026 18:53
8f850e1 to
b5e52b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
make smoke CAND=nftables) is green end-to-end — six tests pass, security-effectiveness precondition (tests/benchmarks.md§ 0.8) is satisfied, ~14 Gbps virtio-bounded iperf3 between gen and gen-WAN-netns through the SUT.make trex-up,make sink-up,make b-00 CAND=nftables). They run on the workstation but the result is the harness ceiling, not the SUT's; that's labelled in three places.bench/README.mdandbench/ROADMAP.md.What's in the diff
Commit 1 —
feat(bench): working Tier-0 smoke + Tier-1 (TRex/sink) scaffolding(16 files, +927 / -83)Tier-0 smoke fixes:
.1).enp1s0/enp2s0, notens3/ens4).virt-customizeinto the well-known/var/lib/cloud/seed/nocloud/path on the instance qcow2 — replaces the cidata-ISO approach that was unreliable on customised images.bench/Makefilenewbase-customisetarget pre-bakes packages into the cached base image (bench bridges have no NAT).smoke.shgates tests oncloud-init status --wait; ruleset check viasystemctl is-activeup front.bench/SETUP.md§ 3a: qemu user/group override +guestfs-tools+ troubleshooting table.Tier-1 scaffolding:
bench/candidates/trex/andbench/candidates/sink/— separate sink VM avoids the kernel-loopback shortcut that nukes any setup where TX and RX live in the same Linux namespace.bench/scripts/b00.{sh,py}— TRex-driven binary-search NDR/PDR across seven RFC 2544 frame sizes.trex-fetch,trex-up,trex-down,sink-up,sink-down,b-00;tear-downcovers all VMs.Commit 2 —
docs(bench): Tier-0 scope decision + latest smoke result(2 files, +69 / -4)bench/README.md: "Scope decision: throughput is Tier-1 only" + "Latest Tier-0 smoke run" with today's pass.bench/ROADMAP.md: Tier-0 "what it can't measure" calls out B-00..B-13 with reasoning.Why two commits
The diff is large but it's two coherent stories — one of code/config that makes the bench do things, one of docs that says what it won't try to do. Splitting the Makefile by hunk into three commits adds noise without help.
Test plan
cd bench && make tear-downmake smoke CAND=nftables→ exit 0,result.json:.security_effectiveness_precondition.satisfied = truemake b-00 CAND=nftablesruns the sameb00.{sh,py}against real NICs; no harness rewrite needed.Out of scope (separate PRs)