From 4940f08c48af156deb091c7d8fd70ee6c68f6b59 Mon Sep 17 00:00:00 2001 From: letur Date: Mon, 25 May 2026 20:31:30 +0200 Subject: [PATCH 1/2] feat(bench): working Tier-0 smoke + Tier-1 (TRex/sink) scaffolding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings the Tier-0 smoke test green end-to-end and lays in the TRex/sink VM scaffolding for the future Tier-1 lab box. Per tests/benchmarks.md § 0.6/§ 0.8, this satisfies the security-effectiveness precondition on a developer workstation while reserving the throughput numbers for a substrate that can produce them honestly. Tier-0 smoke fixes: - Topology: host bridge IPs moved off `.1` so they no longer collide with the SUT's own LAN/WAN addresses. - Q35 NIC naming: cloud-init and rules.nft now reference `enp1s0` /`enp2s0` (the Q35 PCIe-slot names libvirt actually hands out), not the i440fx `ens3`/`ens4` they were originally written for. - Cloud-init seed delivery: `render-cloud-init.sh` injects user-data into `/var/lib/cloud/seed/nocloud/` on the instance qcow2 via `virt-customize`. The well-known path always wins ds-identify; previous attempts via cidata-labeled CD-ROMs were unreliable on customised images. - `bench/Makefile`: new `base-customise` target pre-bakes packages into the cached base image via virt-customize (the bench bridges have no NAT, so cloud-init's `packages:` step has nothing to fetch). - `smoke.sh`: gates tests on `cloud-init status --wait`, runs the ruleset-loaded check up front via `systemctl is-active` (race-free) so iperf3 saturation can't briefly stall the check. - `bench/SETUP.md`: § 3a documents the qemu user/group override (`/etc/libvirt/qemu.conf`: `user = "$USER"`, `group = "kvm"`) that qemu:///system needs to read disks under `/home`. `guestfs-tools` added to the package list; troubleshooting table extended. Tier-1 scaffolding (runs but explicitly out-of-scope on Tier 0): - `bench/candidates/trex/` — TRex VM cloud-init: software-mode TRex with MAC-resolved 2-NIC config (required even number; port 1 is a dummy because RX accounting comes from the sink, not from TRex). - `bench/candidates/sink/` — receive-only WAN endpoint at 203.0.113.200 whose `/sys/class/net/enp1s0/statistics/rx_packets` the b00 driver reads over SSH for RX counting. A dedicated VM avoids the kernel-loopback shortcut that nukes any TRex setup where TX and RX endpoints share a Linux network namespace. - `bench/scripts/b00.{sh,py}` — TRex-driven binary-search NDR/PDR sweep across the seven RFC 2544 frame sizes. Runnable but the header comment is explicit that on Tier 0 the result is a harness ceiling, not a SUT measurement. - `bench/Makefile`: `trex-fetch`, `trex-up`, `trex-down`, `sink-up`, `sink-down`, `b-00` targets; `tear-down` covers all VMs. Verified locally: `make tear-down && make smoke CAND=nftables` exits 0 with `result.json:.security_effectiveness_precondition.satisfied=true`. --- bench/Makefile | 153 ++++++++++++++++- bench/SETUP.md | 29 +++- .../candidates/nftables/cloud-init/user-data | 14 +- bench/candidates/nftables/rules.nft | 25 +-- bench/candidates/sink/cloud-init/meta-data | 2 + bench/candidates/sink/cloud-init/user-data | 56 ++++++ bench/candidates/trafficgen/README.md | 60 +++++++ .../trafficgen/cloud-init/user-data | 96 +++++++++-- bench/candidates/trex/cloud-init/meta-data | 2 + bench/candidates/trex/cloud-init/user-data | 145 ++++++++++++++++ bench/scripts/b00.py | 159 ++++++++++++++++++ bench/scripts/b00.sh | 143 ++++++++++++++++ bench/scripts/render-cloud-init.sh | 67 +++++--- bench/scripts/smoke.sh | 47 ++++-- bench/topology/lan-thurward.xml | 6 +- bench/topology/wan-thurward.xml | 6 +- 16 files changed, 927 insertions(+), 83 deletions(-) create mode 100644 bench/candidates/sink/cloud-init/meta-data create mode 100644 bench/candidates/sink/cloud-init/user-data create mode 100644 bench/candidates/trafficgen/README.md create mode 100644 bench/candidates/trex/cloud-init/meta-data create mode 100644 bench/candidates/trex/cloud-init/user-data create mode 100644 bench/scripts/b00.py create mode 100755 bench/scripts/b00.sh diff --git a/bench/Makefile b/bench/Makefile index 5cf4152..ae0df12 100644 --- a/bench/Makefile +++ b/bench/Makefile @@ -105,6 +105,34 @@ $(BASE_IMAGE): .PHONY: base-image base-image: $(BASE_IMAGE) +# Packages pre-baked into the cached base image. The bench bridges are +# `forward mode='none'` (no external NAT), so guest VMs cannot reach +# package mirrors at cloud-init time. virt-customize installs the union +# of tools every candidate / generator needs before the image is ever +# cloned. A sentinel file inside the image records that customisation +# ran, so re-runs are idempotent and don't double-install. +BASE_PACKAGES := nftables,tcpdump,iperf3,conntrack,iputils-ping,iputils-arping,jq,iproute2,python3 +BASE_CUSTOM_MARK := /var/lib/thurward-bench.customised + +$(BASE_IMAGE).customised: $(BASE_IMAGE) + @if ! command -v virt-customize >/dev/null 2>&1; then \ + echo "ERROR: virt-customize not found (install guestfs-tools)"; exit 1; \ + fi + @if sudo virt-cat -a $(BASE_IMAGE) $(BASE_CUSTOM_MARK) >/dev/null 2>&1; then \ + echo " base image already customised"; \ + else \ + echo " customising base image with: $(BASE_PACKAGES)"; \ + sudo virt-customize -a $(BASE_IMAGE) \ + --install $(BASE_PACKAGES) \ + --write '/etc/cloud/cloud.cfg.d/99-thurward-nocloud.cfg:datasource_list: [ NoCloud, None ]' \ + --truncate /etc/machine-id \ + --run-command 'mkdir -p /var/lib && date -u +%FT%TZ > $(BASE_CUSTOM_MARK)'; \ + fi + @touch $@ + +.PHONY: base-customise base-customize +base-customise base-customize: $(BASE_IMAGE).customised + # --------------------------------------------------------------------- # Networks # --------------------------------------------------------------------- @@ -139,20 +167,19 @@ net-down: # Per-instance paths. SUT_DISK = $(IMAGES_INST_DIR)/thur-sut-$(CAND).qcow2 -SUT_SEED = $(IMAGES_INST_DIR)/thur-sut-$(CAND)-seed.iso SUT_NAME = thur-sut-$(CAND) SUT_DIR = $(BENCH_DIR)/candidates/$(CAND) SUT_USER = bench .PHONY: sub-up -sub-up: net-up $(BASE_IMAGE) $(HARNESS_KEY) +sub-up: net-up $(BASE_IMAGE).customised $(HARNESS_KEY) @if [ ! -d $(SUT_DIR) ]; then echo "ERROR: $(SUT_DIR) does not exist (CAND=$(CAND))"; exit 1; fi @mkdir -p $(IMAGES_INST_DIR) @if [ ! -f $(SUT_DISK) ]; then \ echo "Cloning base image to instance: $(SUT_DISK)"; \ qemu-img create -f qcow2 -F qcow2 -b $(BASE_IMAGE) $(SUT_DISK) 20G; \ + $(SCRIPTS_DIR)/render-cloud-init.sh $(CAND) sut $(SUT_DISK) $(HARNESS_KEY_PUB); \ fi - @$(SCRIPTS_DIR)/render-cloud-init.sh $(CAND) sut $(SUT_SEED) $(HARNESS_KEY_PUB) @if ! virsh -c $(LIBVIRT_URI) dominfo $(SUT_NAME) >/dev/null 2>&1; then \ echo "Defining $(SUT_NAME)"; \ virt-install --connect $(LIBVIRT_URI) \ @@ -161,7 +188,6 @@ sub-up: net-up $(BASE_IMAGE) $(HARNESS_KEY) --cpu host-passthrough \ --osinfo debian12 \ --disk path=$(SUT_DISK),format=qcow2,bus=virtio \ - --disk path=$(SUT_SEED),device=cdrom,bus=sata \ --network network=lan-thurward,model=virtio \ --network network=wan-thurward,model=virtio \ --graphics none --console pty,target_type=serial \ @@ -193,16 +219,15 @@ sub-shell: # --------------------------------------------------------------------- GEN_DISK := $(IMAGES_INST_DIR)/thur-gen.qcow2 -GEN_SEED := $(IMAGES_INST_DIR)/thur-gen-seed.iso GEN_NAME := thur-gen .PHONY: gen-up -gen-up: net-up $(BASE_IMAGE) $(HARNESS_KEY) +gen-up: net-up $(BASE_IMAGE).customised $(HARNESS_KEY) @mkdir -p $(IMAGES_INST_DIR) @if [ ! -f $(GEN_DISK) ]; then \ qemu-img create -f qcow2 -F qcow2 -b $(BASE_IMAGE) $(GEN_DISK) 20G; \ + $(SCRIPTS_DIR)/render-cloud-init.sh trafficgen gen $(GEN_DISK) $(HARNESS_KEY_PUB); \ fi - @$(SCRIPTS_DIR)/render-cloud-init.sh trafficgen gen $(GEN_SEED) $(HARNESS_KEY_PUB) @if ! virsh -c $(LIBVIRT_URI) dominfo $(GEN_NAME) >/dev/null 2>&1; then \ virt-install --connect $(LIBVIRT_URI) \ --name $(GEN_NAME) \ @@ -210,7 +235,6 @@ gen-up: net-up $(BASE_IMAGE) $(HARNESS_KEY) --cpu host-passthrough \ --osinfo debian12 \ --disk path=$(GEN_DISK),format=qcow2,bus=virtio \ - --disk path=$(GEN_SEED),device=cdrom,bus=sata \ --network network=lan-thurward,model=virtio \ --network network=wan-thurward,model=virtio \ --graphics none --console pty,target_type=serial \ @@ -229,6 +253,117 @@ gen-down: echo " $(GEN_NAME): down"; \ fi +# --------------------------------------------------------------------- +# WAN-side sink VM (for TRex B-NN: avoids same-VM loopback shortcut) +# --------------------------------------------------------------------- + +SINK_DISK := $(IMAGES_INST_DIR)/thur-sink.qcow2 +SINK_NAME := thur-sink +SINK_WAN_IP := 203.0.113.200 + +.PHONY: sink-up +sink-up: net-up $(BASE_IMAGE).customised $(HARNESS_KEY) + @mkdir -p $(IMAGES_INST_DIR) + @if [ ! -f $(SINK_DISK) ]; then \ + echo "Cloning base image to sink instance"; \ + qemu-img create -f qcow2 -F qcow2 -b $(BASE_IMAGE) $(SINK_DISK) 20G; \ + $(SCRIPTS_DIR)/render-cloud-init.sh sink gen $(SINK_DISK) $(HARNESS_KEY_PUB); \ + fi + @if ! virsh -c $(LIBVIRT_URI) dominfo $(SINK_NAME) >/dev/null 2>&1; then \ + virt-install --connect $(LIBVIRT_URI) \ + --name $(SINK_NAME) \ + --memory 1024 --vcpus 1 \ + --cpu host-passthrough \ + --osinfo debian12 \ + --disk path=$(SINK_DISK),format=qcow2,bus=virtio \ + --network network=wan-thurward,model=virtio \ + --graphics none --console pty,target_type=serial \ + --import --noautoconsole --noreboot; \ + fi + @if [ "$$(virsh -c $(LIBVIRT_URI) domstate $(SINK_NAME))" != "running" ]; then \ + virsh -c $(LIBVIRT_URI) start $(SINK_NAME); \ + fi + @echo " $(SINK_NAME): running (WAN $(SINK_WAN_IP))" + +.PHONY: sink-down +sink-down: + @if virsh -c $(LIBVIRT_URI) dominfo $(SINK_NAME) >/dev/null 2>&1; then \ + virsh -c $(LIBVIRT_URI) destroy $(SINK_NAME) 2>/dev/null || true; \ + virsh -c $(LIBVIRT_URI) undefine $(SINK_NAME) --nvram 2>/dev/null || true; \ + echo " $(SINK_NAME): down"; \ + fi + +# --------------------------------------------------------------------- +# TRex generator lifecycle (replaces gen-up for RFC-9411 measurements) +# --------------------------------------------------------------------- + +TREX_TARBALL := $(BENCH_DIR)/images/trex/trex-latest.tar.gz +TREX_DISK := $(IMAGES_INST_DIR)/thur-trex.qcow2 +TREX_NAME := thur-trex +TREX_LAN_IP := 10.10.0.200 + +# Fetch TRex tarball (~270 MB). Cisco's CDN doesn't send the +# intermediate cert in the TLS chain, so we use -k and verify the +# tarball is a valid gzip afterwards. +$(TREX_TARBALL): + @mkdir -p $(dir $@) + @if [ ! -f $@ ]; then \ + echo "Downloading TRex tarball (~270 MB)..."; \ + curl -fSL -k --connect-timeout 30 -o $@.tmp \ + https://trex-tgn.cisco.com/trex/release/latest; \ + file $@.tmp | grep -q 'gzip compressed' || { echo "ERROR: not a gzip"; rm $@.tmp; exit 1; }; \ + mv $@.tmp $@; \ + fi + @ls -lh $@ + +.PHONY: trex-fetch +trex-fetch: $(TREX_TARBALL) + +.PHONY: trex-up +trex-up: net-up $(BASE_IMAGE).customised $(HARNESS_KEY) $(TREX_TARBALL) + @mkdir -p $(IMAGES_INST_DIR) + @if [ ! -f $(TREX_DISK) ]; then \ + echo "Cloning base image to TRex instance"; \ + qemu-img create -f qcow2 -F qcow2 -b $(BASE_IMAGE) $(TREX_DISK) 30G; \ + $(SCRIPTS_DIR)/render-cloud-init.sh trex gen $(TREX_DISK) $(HARNESS_KEY_PUB); \ + fi + @if ! virsh -c $(LIBVIRT_URI) dominfo $(TREX_NAME) >/dev/null 2>&1; then \ + echo "Defining $(TREX_NAME)"; \ + virt-install --connect $(LIBVIRT_URI) \ + --name $(TREX_NAME) \ + --memory 4096 --vcpus 4 \ + --cpu host-passthrough \ + --osinfo debian12 \ + --disk path=$(TREX_DISK),format=qcow2,bus=virtio \ + --network network=lan-thurward,model=virtio \ + --network network=wan-thurward,model=virtio \ + --memorybacking hugepages=on \ + --graphics none --console pty,target_type=serial \ + --import --noautoconsole --noreboot; \ + fi + @if [ "$$(virsh -c $(LIBVIRT_URI) domstate $(TREX_NAME))" != "running" ]; then \ + virsh -c $(LIBVIRT_URI) start $(TREX_NAME); \ + fi + @echo " $(TREX_NAME): running (LAN $(TREX_LAN_IP))" + +.PHONY: trex-down +trex-down: + @if virsh -c $(LIBVIRT_URI) dominfo $(TREX_NAME) >/dev/null 2>&1; then \ + virsh -c $(LIBVIRT_URI) destroy $(TREX_NAME) 2>/dev/null || true; \ + virsh -c $(LIBVIRT_URI) undefine $(TREX_NAME) --nvram 2>/dev/null || true; \ + echo " $(TREX_NAME): down"; \ + fi + +# B-00: reference path, empty ruleset. SUT is booted in pass-through +# (no rules.nft loaded — see candidates/nftables/rules-empty.nft). +B00_RESULT_DIR := $(RESULTS_DIR)/b-00/$(TODAY) + +.PHONY: b-00 +b-00: trex-up sink-up sub-up + @mkdir -p $(B00_RESULT_DIR) + @$(SCRIPTS_DIR)/b00.sh $(CAND) $(B00_RESULT_DIR) $(HARNESS_KEY) $(TREX_LAN_IP) $(SINK_WAN_IP) + @echo " result: $(B00_RESULT_DIR)/result.json" + # --------------------------------------------------------------------- # Smoke test (Tier-0 reachability + iperf3 sanity) # --------------------------------------------------------------------- @@ -252,7 +387,7 @@ status: @virsh -c $(LIBVIRT_URI) list --all .PHONY: tear-down -tear-down: sub-down gen-down net-down +tear-down: sub-down gen-down trex-down sink-down net-down @echo " harness fully torn down" .PHONY: distclean diff --git a/bench/SETUP.md b/bench/SETUP.md index 751ca31..60150e9 100644 --- a/bench/SETUP.md +++ b/bench/SETUP.md @@ -25,7 +25,8 @@ sudo pacman -S --needed \ qemu-full libvirt virt-install \ dnsmasq bridge-utils iproute2 \ cloud-image-utils edk2-ovmf swtpm \ - jq libguestfs iperf3 + jq libguestfs guestfs-tools iperf3 \ + tcpdump ``` `qemu-full` pulls in everything; on slim setups `qemu-base` + @@ -38,6 +39,28 @@ sudo systemctl enable --now libvirtd.socket sudo systemctl enable --now virtlogd.socket ``` +## 3a. Run qemu as your user + +`qemu:///system` defaults to running the qemu process as the +`libvirt-qemu` system user — which can't traverse `/home/` to read +the bench instance disks stored under `bench/images/instances/`. Tell +libvirt to run qemu as your user instead: + +``` +sudo tee -a /etc/libvirt/qemu.conf >/dev/null </Repositories/.../bench/images/instances/ +# are readable by the qemu process. +user = "$USER" +group = "kvm" +EOF +sudo systemctl restart libvirtd.service +``` + +Without this, `make sub-up` fails with +`error: Cannot access storage file ... (as uid:955, gid:955): Permission denied`. + ## 4. User groups Your shell user needs to be in `libvirt` and `kvm`: @@ -118,4 +141,6 @@ If both bridges came up, you're ready. | `Could not access KVM kernel module: Permission denied` | User not in `kvm` group, or `/dev/kvm` permissions are wrong (`ls -l /dev/kvm` should be `0660 kvm`). | | Wayland + libvirt graphical viewer doesn't open | Use `virsh console ` (text) instead of `virt-viewer`. The harness doesn't need a GUI. | | `network 'lan-thurward' is not active` after host reboot | Networks are not auto-started; run `make net-up` again. | -| nftables guest can't reach package mirrors | Cloud-init runs *after* the firewall ruleset loads; the guest is intentionally isolated. Install packages into the cloud-image with `virt-customize` instead. | +| nftables guest can't reach package mirrors | Cloud-init runs *after* the firewall ruleset loads; the guest is intentionally isolated. The Makefile pre-bakes all packages via `make base-customise` — re-run that target if you change `BASE_PACKAGES`. | +| `error: Cannot access storage file ... (as uid:955, gid:955)` | Run § 3a — `qemu:///system` is still running qemu as `libvirt-qemu`. Bench disks live under `/home`, which that user can't traverse. | +| Guest boots but `localhost login:` instead of `thur-*` | Cloud-init never read the seed. The harness injects the seed into `/var/lib/cloud/seed/nocloud/` on the instance qcow2 via `virt-customize`; if this step fails (e.g. `guestfs-tools` missing) cloud-init falls back to default state. Check `make sub-up` output. | diff --git a/bench/candidates/nftables/cloud-init/user-data b/bench/candidates/nftables/cloud-init/user-data index bfe6f35..63ac282 100644 --- a/bench/candidates/nftables/cloud-init/user-data +++ b/bench/candidates/nftables/cloud-init/user-data @@ -1,7 +1,7 @@ #cloud-config # nftables SUT cloud-init for thurward bench harness. # -# Two NICs (libvirt assigns ens3 = lan-thurward, ens4 = wan-thurward). +# Two NICs (Q35: enp1s0 = lan-thurward on pci.1, enp2s0 = wan-thurward on pci.2). # Static IPs; no DHCP on either bridge (see topology/lan-thurward.xml). # IP forwarding on; nftables ruleset baked in via runcmd before the # default-deny policy locks the SUT down. @@ -31,11 +31,11 @@ write_files: network: version: 2 ethernets: - ens3: + enp1s0: dhcp4: false addresses: [10.10.0.1/24] optional: true - ens4: + enp2s0: dhcp4: false addresses: [203.0.113.1/24] optional: true @@ -63,11 +63,9 @@ write_files: [Install] WantedBy=multi-user.target -packages: - - nftables - - tcpdump - - iperf3 - - conntrack +# Packages are pre-baked into the base image by `make base-customise` +# (the bench bridges have no external NAT, so apt is unreachable at +# cloud-init time). See bench/Makefile $(BASE_PACKAGES). runcmd: - sysctl --system diff --git a/bench/candidates/nftables/rules.nft b/bench/candidates/nftables/rules.nft index 7221583..99be655 100644 --- a/bench/candidates/nftables/rules.nft +++ b/bench/candidates/nftables/rules.nft @@ -14,9 +14,9 @@ # `destination_fqdn` semantic; the sidecar-resolver + ipset feeder # pattern needed for parity is its own ADR (deferred). # -# Interfaces (per cloud-init): -# ens3 = LAN (10.10.0.0/24) -# ens4 = WAN (203.0.113.0/24) +# Interfaces (per cloud-init, Q35 machine type): +# enp1s0 = LAN (10.10.0.0/24) +# enp2s0 = WAN (203.0.113.0/24) # Firewall's own LAN IP is 10.10.0.1 to match the DNS proxy in # rules.yaml's `allow-dns-out`. @@ -45,6 +45,11 @@ table inet thurward { # intentionally elided in Tier 0 (no sidecar resolver yet). ip saddr @lan_clients tcp dport 443 ct state new,established accept + # allow-iperf3: LAN to TCP/5201 for the bench harness throughput + # test (smoke.sh § test 4). Not a real-world rule — only present + # on the bench candidate so we can measure forwarded throughput. + ip saddr @lan_clients tcp dport 5201 ct state new,established accept + # allow-established: any returning traffic. ct state established,related accept @@ -56,15 +61,15 @@ table inet thurward { type filter hook input priority 0; policy drop; # DNS to the firewall (matches `allow-dns-out` in rules.yaml). - ip saddr @lan_clients iifname "ens3" udp dport 53 accept - ip saddr @lan_clients iifname "ens3" tcp dport 53 accept + ip saddr @lan_clients iifname "enp1s0" udp dport 53 accept + ip saddr @lan_clients iifname "enp1s0" tcp dport 53 accept # SSH from LAN — harness access only; never on a real deployment. - ip saddr @lan_clients iifname "ens3" tcp dport 22 accept + ip saddr @lan_clients iifname "enp1s0" tcp dport 22 accept # ICMP echo for the smoke test. - iifname "ens3" icmp type echo-request accept - iifname "ens3" icmpv6 type echo-request accept + iifname "enp1s0" icmp type echo-request accept + iifname "enp1s0" icmpv6 type echo-request accept ct state established,related accept counter comment "default-deny-input" @@ -84,7 +89,7 @@ table ip nat { # DNAT: 203.0.113.10:443/tcp -> 10.10.0.5:443/tcp # Matches the worked DNAT in examples/rules.yaml:23-28. - iifname "ens4" tcp dport 443 ip daddr 203.0.113.10 \ + iifname "enp2s0" tcp dport 443 ip daddr 203.0.113.10 \ dnat to 10.10.0.5:443 } @@ -93,6 +98,6 @@ table ip nat { # SNAT/masquerade: 10.10.0.0/24 -> WAN interface IP. # Matches the worked SNAT in examples/rules.yaml:18-22. - oifname "ens4" ip saddr 10.10.0.0/24 masquerade + oifname "enp2s0" ip saddr 10.10.0.0/24 masquerade } } diff --git a/bench/candidates/sink/cloud-init/meta-data b/bench/candidates/sink/cloud-init/meta-data new file mode 100644 index 0000000..abe0f2f --- /dev/null +++ b/bench/candidates/sink/cloud-init/meta-data @@ -0,0 +1,2 @@ +instance-id: sink-001 +local-hostname: thur-sink diff --git a/bench/candidates/sink/cloud-init/user-data b/bench/candidates/sink/cloud-init/user-data new file mode 100644 index 0000000..9d13323 --- /dev/null +++ b/bench/candidates/sink/cloud-init/user-data @@ -0,0 +1,56 @@ +#cloud-config +# WAN-side sink VM for the TRex B-NN benchmarks. +# +# Receive-only endpoint at 203.0.113.200/24. TRex on the LAN side +# transmits packets destined for this address; the SUT forwards them +# across the firewall; the sink receives them on enp1s0. The b00.py +# driver reads /sys/class/net/enp1s0/statistics/rx_packets via SSH +# before and after each trial to count received packets. +# +# A separate VM is used (rather than a second NIC on the TRex VM) +# because Linux's local-route shortcut sends any packet destined for +# a locally-assigned IP via loopback, bypassing the wire entirely. +# With this VM, the SUT is doing real routing between two distinct +# Layer-2 endpoints — matching the topology of a production deployment. + +hostname: thur-sink +manage_etc_hosts: true + +users: + - name: bench + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + ssh_authorized_keys: + - "@@HARNESS_SSH_KEY@@" + +write_files: + - path: /etc/sysctl.d/99-thurward-sink.conf + permissions: "0644" + content: | + # No forwarding — this is a leaf endpoint. + net.ipv4.ip_forward = 0 + # Don't reply to unicast pings if the source isn't on our subnet — + # we want the SUT to be the gateway, not us responding to anything. + # (Standard behaviour; left explicit for clarity.) + net.ipv4.icmp_echo_ignore_broadcasts = 1 + net.ipv6.conf.all.disable_ipv6 = 1 + + - path: /etc/netplan/60-thurward-sink.yaml + permissions: "0600" + content: | + network: + version: 2 + ethernets: + enp1s0: + dhcp4: false + addresses: [203.0.113.200/24] + routes: + - to: default + via: 203.0.113.1 + optional: true + +# Packages pre-baked into the base image by `make base-customise`. + +runcmd: + - sysctl --system + - netplan apply || true diff --git a/bench/candidates/trafficgen/README.md b/bench/candidates/trafficgen/README.md new file mode 100644 index 0000000..ce578d3 --- /dev/null +++ b/bench/candidates/trafficgen/README.md @@ -0,0 +1,60 @@ +# `candidates/trafficgen/` — Tier-0 iperf3 + ping generator + +A single Debian 12 VM that hosts **both** endpoints of the smoke test, +one on each side of the firewall: + +``` + ┌──────────────────── thur-gen ────────────────────┐ + │ │ +LAN side │ root netns wan netns │ WAN side +10.10.0.100 │ enp1s0 ── 10.10.0.100/24 enp2s0 ── 203.0.113.100│ 203.0.113.100 + │ default via default via │ + │ 10.10.0.1 ───┐ ┌── 203.0.113.1│ + │ │ │ │ + └─────────────────────────┼─────────┼──────────────┘ + ▼ ▲ + ┌─────────────────────┐ + │ thur-sut- │ the firewall under test + │ 10.10.0.1 → fwd │ + │ 203.0.113.1 ← │ + └─────────────────────┘ +``` + +## Why a netns? + +Without the split, both `10.10.0.100` and `203.0.113.100` would be +local addresses in the same kernel namespace. An `iperf3 -c +203.0.113.100` started from the same VM would be short-circuited via +loopback by the kernel — and quietly bypass the firewall entirely. The +smoke test would then report a passing run that proved nothing. + +Putting `enp2s0` into a `wan` network namespace removes +`203.0.113.0/24` from the root netns's address table. The only route +left from root to WAN-side addresses is the default route via +`10.10.0.1` — i.e. through the SUT. That is what guarantees the +FORWARD chain is actually exercised. + +## Files + +- `cloud-init/user-data` — declares the netns helper script, the + systemd unit that runs it, and the `iperf3-server@.service` template + that execs into the netns. +- `cloud-init/meta-data` — boilerplate (instance-id, hostname). + +## What runs at boot + +1. `netplan apply` — addresses `enp1s0` only. +2. `trafficgen-wan-ns.service` — moves `enp2s0` into the `wan` netns, + assigns `203.0.113.100/24`, installs the default route via + `203.0.113.1`. +3. `iperf3-server@5201.service` — enabled at boot; the unit `ExecStart` + wraps `ip netns exec wan iperf3 -s`, so the server listens inside the + netns. + +## Useful from inside the VM + +``` +sudo /usr/local/sbin/trafficgen-wan-ns status # see wan-netns addrs/routes +sudo ip netns exec wan ss -lntp # see wan-side listeners +sudo ip netns exec wan tcpdump -ni enp2s0 # capture on the WAN side +``` diff --git a/bench/candidates/trafficgen/cloud-init/user-data b/bench/candidates/trafficgen/cloud-init/user-data index 90e9dfc..a3036ae 100644 --- a/bench/candidates/trafficgen/cloud-init/user-data +++ b/bench/candidates/trafficgen/cloud-init/user-data @@ -1,9 +1,14 @@ #cloud-config # Traffic-generator VM for thurward bench harness. # -# Two NICs, one on each bridge. Hosts both the "LAN client" and the -# "WAN server" personas — that gives us a single VM with iperf3 -# endpoints on both sides of the firewall. +# Two NICs, one on each bridge. enp1s0 stays in the root namespace as the +# LAN client. enp2s0 is moved into the `wan` network namespace where the +# iperf3 server lives. Without this split, both 10.10.0.100 and +# 203.0.113.100 would be local addresses in the same netns and the kernel +# would short-circuit any client connection to 203.0.113.100 via +# loopback — bypassing the firewall and silently invalidating the smoke +# numbers. With the split, the only route from root-netns to +# 203.0.113.0/24 is via the SUT. # # Tier-0 only: iperf3 + ping. TRex lives in its own VM in a later pass. @@ -23,6 +28,7 @@ write_files: content: | # No forwarding on the generator — it's a leaf. net.ipv4.ip_forward = 0 + net.ipv6.conf.all.disable_ipv6 = 1 - path: /etc/netplan/60-thurward-gen.yaml permissions: "0600" @@ -30,38 +36,98 @@ write_files: network: version: 2 ethernets: - ens3: + enp1s0: dhcp4: false addresses: [10.10.0.100/24] routes: - - to: 203.0.113.0/24 + - to: default via: 10.10.0.1 optional: true - ens4: + enp2s0: + # enp2s0 is handed off to the `wan` netns by trafficgen-wan-ns.service. + # Leaving it unmanaged here keeps systemd-networkd from re-claiming it. dhcp4: false - addresses: [203.0.113.100/24] + dhcp6: false optional: true + - path: /usr/local/sbin/trafficgen-wan-ns + permissions: "0755" + content: | + #!/bin/bash + # Move enp2s0 into the `wan` netns and configure it inside. + set -euo pipefail + NS=wan + DEV=enp2s0 + ADDR=203.0.113.100/24 + GW=203.0.113.1 + case "${1:-}" in + up) + ip netns add "$NS" 2>/dev/null || true + if ip -o link show "$DEV" >/dev/null 2>&1; then + ip link set "$DEV" netns "$NS" + fi + ip netns exec "$NS" ip link set lo up + ip netns exec "$NS" ip link set "$DEV" up + ip netns exec "$NS" ip addr add "$ADDR" dev "$DEV" 2>/dev/null || true + ip netns exec "$NS" ip route replace default via "$GW" + ;; + down) + if ip netns list | awk '{print $1}' | grep -qx "$NS"; then + ip netns exec "$NS" ip link set "$DEV" netns 1 2>/dev/null || true + ip netns del "$NS" 2>/dev/null || true + fi + ;; + status) + ip netns exec "$NS" ip -br addr || true + ip netns exec "$NS" ip route || true + ;; + *) + echo "usage: $0 up|down|status" >&2 + exit 2 + ;; + esac + + - path: /etc/systemd/system/trafficgen-wan-ns.service + permissions: "0644" + content: | + [Unit] + Description=Move enp2s0 into the `wan` network namespace + After=network.target systemd-networkd.service + Before=iperf3-server@.service + + [Service] + Type=oneshot + RemainAfterExit=yes + ExecStart=/usr/local/sbin/trafficgen-wan-ns up + ExecStop=/usr/local/sbin/trafficgen-wan-ns down + + [Install] + WantedBy=multi-user.target + - path: /etc/systemd/system/iperf3-server@.service permissions: "0644" content: | [Unit] - Description=iperf3 server on port %i - After=network.target + Description=iperf3 server on port %i (in wan netns) + Requires=trafficgen-wan-ns.service + After=trafficgen-wan-ns.service [Service] - ExecStart=/usr/bin/iperf3 -s -p %i -1 + Type=exec + ExecStart=/usr/sbin/ip netns exec wan /usr/bin/iperf3 --server --port %i Restart=on-failure + RestartSec=1 [Install] WantedBy=multi-user.target -packages: - - iperf3 - - iputils-ping - - tcpdump - - jq +# Packages are pre-baked into the base image by `make base-customise` +# (the bench bridges have no external NAT, so apt is unreachable at +# cloud-init time). See bench/Makefile $(BASE_PACKAGES). runcmd: - sysctl --system - netplan apply || true + - systemctl daemon-reload + - systemctl enable --now trafficgen-wan-ns.service + - systemctl enable iperf3-server@5201.service diff --git a/bench/candidates/trex/cloud-init/meta-data b/bench/candidates/trex/cloud-init/meta-data new file mode 100644 index 0000000..bcab264 --- /dev/null +++ b/bench/candidates/trex/cloud-init/meta-data @@ -0,0 +1,2 @@ +instance-id: trex-001 +local-hostname: thur-trex diff --git a/bench/candidates/trex/cloud-init/user-data b/bench/candidates/trex/cloud-init/user-data new file mode 100644 index 0000000..a01da50 --- /dev/null +++ b/bench/candidates/trex/cloud-init/user-data @@ -0,0 +1,145 @@ +#cloud-config +# TRex VM for thurward bench harness — single-NIC TX-only generator. +# +# TRex transmits on enp1s0 (LAN side) toward 203.0.113.200 (sink VM +# on the WAN bridge). The SUT forwards from LAN to WAN. RX accounting +# is done by reading /sys/class/net/enp1s0/statistics/rx_packets on +# the sink VM via SSH from the b00.py driver — not via a TRex RX port, +# because Linux would short-circuit any locally-bound destination IP +# through loopback and bypass the firewall entirely. +# +# Topology: +# +# TRex VM (10.10.0.200) ── tx ──> SUT enp1s0 (10.10.0.1) +# SUT forwards +# SUT enp2s0 (203.0.113.1) ── tx ──> sink VM (203.0.113.200) +# +# TRex runs in --software (af_packet) mode on virtio NICs. + +hostname: thur-trex +manage_etc_hosts: true + +users: + - name: bench + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + ssh_authorized_keys: + - "@@HARNESS_SSH_KEY@@" + +write_files: + - path: /etc/netplan/60-thurward-trex.yaml + permissions: "0600" + content: | + network: + version: 2 + ethernets: + enp1s0: + dhcp4: false + addresses: [10.10.0.200/24] + routes: + - to: default + via: 10.10.0.1 + optional: true + enp2s0: + # Required by TRex (port_limit must be even) but we don't + # use it for RX — packets are addressed to the sink VM at + # 203.0.113.200, which counts them via /sys statistics over + # SSH. enp2s0 stays UP but unconfigured at L3. + dhcp4: false + dhcp6: false + optional: true + + - path: /etc/sysctl.d/99-thurward-trex.conf + permissions: "0644" + content: | + net.ipv4.ip_forward = 0 + net.ipv6.conf.all.disable_ipv6 = 1 + vm.nr_hugepages = 512 + + - path: /etc/trex_cfg.yaml + permissions: "0644" + content: | + ### TRex platform config — 2 ports (TRex requires an even number). + ### Port 0 = enp1s0 (LAN, TX → SUT → sink). Port 1 = enp2s0 (no IP, + ### unused for RX measurement; sink VM counts packets instead). + - port_limit : 2 + version : 2 + interfaces : ["enp1s0", "enp2s0"] + port_info : + - dest_mac : "@@DEST_MAC_0@@" + src_mac : "@@SRC_MAC_0@@" + - dest_mac : "ff:ff:ff:ff:ff:ff" + src_mac : "@@SRC_MAC_1@@" + platform : + master_thread_id : 0 + latency_thread_id : 1 + dual_if : + - socket : 0 + threads : [2,3] + + - path: /usr/local/sbin/trex-prepare + permissions: "0755" + content: | + #!/bin/bash + # Render /etc/trex_cfg.yaml with this VM's source MAC and the + # SUT's LAN MAC as next-hop. Re-run via `systemctl restart` once + # the SUT is reachable so ARP resolves to the real MAC, not the + # broadcast fallback. + set -euo pipefail + cfg=/etc/trex_cfg.yaml + SRC0=$(ip -br link show enp1s0 | awk '{print $3}') + SRC1=$(ip -br link show enp2s0 | awk '{print $3}') + arping -I enp1s0 -c 1 -w 2 10.10.0.1 >/dev/null 2>&1 || true + DEST0=$(ip neigh show 10.10.0.1 dev enp1s0 | awk '{print $5}' | head -1) + : "${DEST0:=ff:ff:ff:ff:ff:ff}" + sed -i \ + -e "s/@@SRC_MAC_0@@/$SRC0/" -e "s/@@DEST_MAC_0@@/$DEST0/" \ + -e "s/@@SRC_MAC_1@@/$SRC1/" \ + "$cfg" + echo "trex-prepare: SRC0=$SRC0 DEST0(SUT-LAN)=$DEST0 SRC1=$SRC1" + + - path: /etc/systemd/system/trex-server.service + permissions: "0644" + content: | + [Unit] + Description=TRex stateless server (software af_packet mode) + After=network-online.target trex-prepare.service + Wants=network-online.target + Requires=trex-prepare.service + + [Service] + Type=simple + WorkingDirectory=/opt/trex/current + ExecStart=/opt/trex/current/t-rex-64 -i --software --no-scapy-server --no-watchdog + Restart=on-failure + RestartSec=2 + + [Install] + WantedBy=multi-user.target + + - path: /etc/systemd/system/trex-prepare.service + permissions: "0644" + content: | + [Unit] + Description=Render /etc/trex_cfg.yaml with live MACs + After=network-online.target + Wants=network-online.target + + [Service] + Type=oneshot + RemainAfterExit=yes + ExecStart=/usr/local/sbin/trex-prepare + + [Install] + WantedBy=multi-user.target + +# Packages pre-baked into the base image by `make base-customise`. + +runcmd: + - sysctl --system + - netplan apply || true + - mkdir -p /opt/trex + - bash -c 'tar -xzf /var/tmp/trex-latest.tar.gz -C /opt/trex && ln -sfn /opt/trex/$(tar tzf /var/tmp/trex-latest.tar.gz | head -1 | tr -d /) /opt/trex/current' + - systemctl daemon-reload + - systemctl enable --now trex-prepare.service + - systemctl enable --now trex-server.service diff --git a/bench/scripts/b00.py b/bench/scripts/b00.py new file mode 100644 index 0000000..7d80e1e --- /dev/null +++ b/bench/scripts/b00.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +# b00.py — runs INSIDE the TRex VM. Drives TRex's stateless API to +# perform a binary-search NDR/PDR sweep across the seven RFC 2544 +# frame sizes (B-00 reference path per tests/benchmarks.md § 0.6). +# +# Topology: TRex single-port TX on enp1s0, packets dest=$SINK_IP. +# RX accounting reads /sys/class/net/enp1s0/statistics/rx_packets on +# the sink VM via SSH at start and end of each trial. Loss is +# (tx - rx_delta) / tx. +# +# Output: JSON on stdout — picked up by scripts/b00.sh. +import json +import os +import subprocess +import sys +import time + +TREX_CLIENT_LIB = "/opt/trex/current/automation/trex_control_plane/interactive" +sys.path.insert(0, TREX_CLIENT_LIB) + +from trex_stl_lib.api import STLClient, STLStream, STLPktBuilder, STLTXCont, STLError # noqa: E402 +from scapy.layers.inet import IP, UDP # noqa: E402 +from scapy.layers.l2 import Ether # noqa: E402 + +RFC2544_FRAMES = [64, 128, 256, 512, 1024, 1280, 1518] +TRIAL_SEC = 10 # § 0.11 calls for 60s; tunable down for iteration speed +SEARCH_ITER = 8 +PDR_THRESHOLD = 0.005 # 0.5% drop tolerance for PDR +# Ceiling for the binary search. Set to a realistic Tier-0 upper bound +# (SW-mode TRex + 1-vCPU SUT VM with virtio caps well below 1 Mpps). +# Set too high and the search wastes all its iterations above the actual +# NDR and reports 0; set too low and we cap the result artificially. +RATE_HIGH_PPS = 500_000 + +SRC_IP = "10.10.0.200" +SINK_IP = os.environ.get("SINK_IP", "203.0.113.200") +SRC_PORT = 1024 +DST_PORT = 5000 + +# How to read the sink VM's RX packet counter. +SINK_KEY = "/home/bench/.ssh/harness" # uploaded by b00.sh +SINK_SSH_OPTS = ["-i", SINK_KEY, "-o", "StrictHostKeyChecking=no", + "-o", "UserKnownHostsFile=/dev/null", "-o", "LogLevel=ERROR", + "-o", "ConnectTimeout=5"] + + +def sink_rx_packets(): + """Read /sys/class/net/enp1s0/statistics/rx_packets from the sink VM.""" + cmd = ["ssh", *SINK_SSH_OPTS, f"bench@{SINK_IP}", + "cat /sys/class/net/enp1s0/statistics/rx_packets"] + out = subprocess.check_output(cmd, text=True, timeout=10).strip() + return int(out) + + +def make_stream(frame_size_b): + pad_len = max(0, frame_size_b - 4 - 14 - 20 - 8) # FCS(4)+ETH(14)+IP(20)+UDP(8) + base_pkt = Ether() / IP(src=SRC_IP, dst=SINK_IP) / UDP(sport=SRC_PORT, dport=DST_PORT) / ("x" * pad_len) + pkt = STLPktBuilder(pkt=base_pkt) + return STLStream(packet=pkt, mode=STLTXCont(pps=1)) # rate set via mult= at start + + +def run_trial(c, frame_size_b, rate_pps, duration_s): + c.reset(ports=[0]) + stream = make_stream(frame_size_b) + c.add_streams(stream, ports=[0]) + c.clear_stats() + rx_before = sink_rx_packets() + c.start(ports=[0], mult=f"{rate_pps}pps", duration=duration_s, force=True) + c.wait_on_traffic(ports=[0], timeout=duration_s + 30) + # Sink counter updates may lag a hair after the last frame; small sleep. + time.sleep(0.5) + rx_after = sink_rx_packets() + stats = c.get_stats() + tx = stats[0].get("opackets", 0) + rx = max(0, rx_after - rx_before) + if tx == 0: + return tx, rx, 1.0, 0.0 + loss_pct = max(0.0, min(1.0, (tx - rx) / tx)) + achieved_mpps = (tx / duration_s) / 1e6 + return tx, rx, loss_pct, achieved_mpps + + +def search_one_size(c, frame_size_b): + lo, hi = 0, RATE_HIGH_PPS + ndr_pps = 0 + pdr_pps = 0 + history = [] + + for i in range(SEARCH_ITER): + mid = (lo + hi) // 2 + if mid < 10_000: + break + tx, rx, loss, mpps = run_trial(c, frame_size_b, mid, TRIAL_SEC) + history.append({"iter": i + 1, "rate_pps": mid, "tx": tx, "rx": rx, + "loss_pct": loss, "achieved_mpps": mpps}) + if loss == 0: + ndr_pps = max(ndr_pps, mid) + lo = mid + elif loss <= PDR_THRESHOLD: + pdr_pps = max(pdr_pps, mid) + lo = mid + else: + hi = mid + + def gbps(pps, size_b): + return (pps * size_b * 8) / 1e9 + + return { + "frame_size_b": frame_size_b, + "ndr_pps": ndr_pps, + "ndr_gbps": gbps(ndr_pps, frame_size_b), + "pdr_pps": pdr_pps, + "pdr_gbps": gbps(pdr_pps, frame_size_b), + "history": history, + } + + +def main(): + c = STLClient(server="127.0.0.1") + try: + c.connect() + c.reset(ports=[0]) + c.set_port_attr(ports=[0], promiscuous=True) + + # Sanity-probe the sink RX path before kicking off the sweep. + rx0 = sink_rx_packets() + print(f"sink RX counter at start: {rx0}", file=sys.stderr, flush=True) + + results = [] + for fs in RFC2544_FRAMES: + t0 = time.time() + r = search_one_size(c, fs) + r["elapsed_s"] = round(time.time() - t0, 1) + results.append(r) + print(f" frame={fs}B NDR={r['ndr_pps']:>10} pps ({r['ndr_gbps']:.3f} Gbps) " + f"PDR={r['pdr_pps']:>10} pps ({r['pdr_gbps']:.3f} Gbps) in {r['elapsed_s']}s", + file=sys.stderr, flush=True) + + out = { + "frame_sizes_b": RFC2544_FRAMES, + "trial_duration_s": TRIAL_SEC, + "search_iterations": SEARCH_ITER, + "pdr_threshold": PDR_THRESHOLD, + "rx_source": f"sink:{SINK_IP}:enp1s0", + "per_size": results, + } + json.dump(out, sys.stdout) + except STLError as e: + print(f"TRex error: {e}", file=sys.stderr) + sys.exit(1) + finally: + try: + c.disconnect() + except Exception: + pass + + +if __name__ == "__main__": + main() diff --git a/bench/scripts/b00.sh b/bench/scripts/b00.sh new file mode 100755 index 0000000..ffa1b57 --- /dev/null +++ b/bench/scripts/b00.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# b00.sh — orchestrate B-00 (reference path with empty ruleset) on the +# nftables candidate using TRex + MLRsearch-flavoured binary NDR/PDR +# search. Per tests/benchmarks.md § 0.6, this is a per-environment +# control: the result tells us where the virtio path tops out. +# +# **NOTE — Tier-0 throughput is out of scope.** On the developer +# workstation, the virtio + 1-vCPU-SUT path bottoms out the SUT well +# before reaching the firewall's actual capability. The B-00 number +# here is the harness ceiling, not a SUT measurement. The script is +# kept runnable so the Tier-1 lab box can use it as-is. See +# bench/README.md "Scope decision" for details. +# +# Usage: b00.sh +set -euo pipefail + +CAND="${1:?candidate required}" +RESULT_DIR="${2:?result dir required}" +HARNESS_KEY="${3:?harness key path required}" +TREX_LAN_IP="${4:?trex LAN IP required}" +SINK_WAN_IP="${5:?sink WAN IP required}" + +BENCH_DIR="$(cd "$(dirname "$0")/.." && pwd)" +SUT_LAN_IP=10.10.0.1 +RAW_LOG="$RESULT_DIR/raw.log" +RESULT_JSON="$RESULT_DIR/result.json" +mkdir -p "$RESULT_DIR" +: >"$RAW_LOG" + +log() { printf '[b-00] %s\n' "$*" | tee -a "$RAW_LOG"; } +trex_ssh() { + ssh -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR -o ConnectTimeout=10 bench@"$TREX_LAN_IP" "$@" +} +sut_ssh() { + ssh -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR -o ConnectTimeout=10 bench@"$SUT_LAN_IP" "$@" +} +sink_ssh() { + ssh -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR -o ConnectTimeout=10 bench@"$SINK_WAN_IP" "$@" +} + +log "candidate=$CAND result-dir=$RESULT_DIR trex=$TREX_LAN_IP sink=$SINK_WAN_IP" + +# --- Wait for SSH on all three VMs ------------------------------------------- +log "waiting for SUT + TRex + sink SSH ..." +# Sink lives on the WAN bridge; the host can reach it via thur-wan0 (203.0.113.254). +for tgt in "$SUT_LAN_IP" "$TREX_LAN_IP" "$SINK_WAN_IP"; do + for i in {1..60}; do + if ssh -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR -o ConnectTimeout=3 bench@"$tgt" true 2>/dev/null; then + log " $tgt: reachable after ${i}s"; break + fi + sleep 2 + [ "$i" = 60 ] && { log " $tgt: NOT REACHABLE — aborting"; exit 1; } + done +done + +# --- Wait for cloud-init on all VMs ----------------------------------------- +log "waiting for cloud-init to finish on SUT + TRex + sink ..." +sut_ssh "cloud-init status --wait" >>"$RAW_LOG" 2>&1 || true +trex_ssh "cloud-init status --wait" >>"$RAW_LOG" 2>&1 || true +sink_ssh "cloud-init status --wait" >>"$RAW_LOG" 2>&1 || true + +# --- Verify SUT ruleset state ------------------------------------------------ +# B-00 wants pass-through. The nftables candidate's stock rules.nft has +# `policy drop` on forward, so for B-00 we either flush rules or rely +# on the candidate having a pass-through mode. Easiest: flush ruleset +# on the SUT just for the duration of B-00. +log "B-00: flushing SUT ruleset for the duration of the test (pass-through)" +sut_ssh "sudo nft flush ruleset && sudo sysctl -w net.ipv4.ip_forward=1" >>"$RAW_LOG" 2>&1 + +# --- Re-resolve TRex MACs after SUT is up ----------------------------------- +log "re-running trex-prepare so it learns the SUT's MACs" +trex_ssh "sudo systemctl restart trex-prepare.service && sudo systemctl restart trex-server.service" >>"$RAW_LOG" 2>&1 +sleep 5 # give TRex a moment to bind sockets + +# --- Wait for TRex RPC to come up ------------------------------------------- +log "waiting for TRex RPC on $TREX_LAN_IP:4501 ..." +for i in {1..30}; do + if trex_ssh "ss -tnlp 2>/dev/null | grep -q ':4501'" 2>/dev/null; then + log " TRex RPC: up after ${i}s"; break + fi + sleep 2 + [ "$i" = 30 ] && { log " TRex RPC: NOT UP — aborting"; trex_ssh 'sudo journalctl -u trex-server.service -n 50 --no-pager' >>"$RAW_LOG" 2>&1 || true; exit 1; } +done + +# --- Drive the benchmark --------------------------------------------------- +# Copy the Python driver + harness SSH key to TRex VM. The driver needs +# the key to SSH to the sink VM and read its RX counter each trial. +log "uploading b00.py + harness key to TRex VM" +scp -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR \ + "$BENCH_DIR/scripts/b00.py" bench@"$TREX_LAN_IP":/tmp/b00.py >>"$RAW_LOG" 2>&1 +trex_ssh "mkdir -p ~/.ssh && chmod 700 ~/.ssh" >>"$RAW_LOG" 2>&1 +scp -i "$HARNESS_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o LogLevel=ERROR \ + "$HARNESS_KEY" bench@"$TREX_LAN_IP":.ssh/harness >>"$RAW_LOG" 2>&1 +trex_ssh "chmod 600 ~/.ssh/harness" >>"$RAW_LOG" 2>&1 +# Pre-warm ARP for the sink from the TRex VM so its first ssh isn't slow. +trex_ssh "ssh -i ~/.ssh/harness -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=10 bench@$SINK_WAN_IP true" >>"$RAW_LOG" 2>&1 || true + +log "running B-00 (this will take several minutes)" +B00_JSON=$(trex_ssh "SINK_IP=$SINK_WAN_IP python3 /tmp/b00.py" 2>>"$RAW_LOG") +echo "$B00_JSON" >"$RESULT_DIR/trex-raw.json" + +# --- Wrap in our standard envelope ----------------------------------------- +HOST_KERNEL=$(uname -srv) +HOST_CPU=$(grep -m1 '^model name' /proc/cpuinfo | sed -e 's/^[^:]*: //') + +python3 - "$B00_JSON" "$CAND" "$HOST_KERNEL" "$HOST_CPU" >"$RESULT_JSON" <<'PY' +import json, sys +trex = json.loads(sys.argv[1]) +out = { + "schema_version": 1, + "result_type": "b-00", + "candidate": sys.argv[2], + "tier": "0-workstation-virtio", + "generator": "trex-software-af_packet", + "search": "binary NDR/PDR (MLRsearch-flavoured)", + "host_invariants": { + "kernel": sys.argv[3], + "cpu_model": sys.argv[4], + "numa_pinning": "none", + "hugepages": "guest-only", + "nic": "virtio-net", + "queue_count": 1, + }, + "frame_sizes_b": trex["frame_sizes_b"], + "per_size": trex["per_size"], + "notes": [ + "B-00 / reference path control (tests/benchmarks.md § 0.6).", + "Tier-0 numbers characterise the virtio path, not the SUT envelope.", + "TRex runs in --software (af_packet) mode; no DPDK pmd binding.", + "Trial duration is shorter than § 0.11 (60s/trial) to keep the harness usable on a workstation; longer trials are a tuning knob in scripts/b00.py.", + ], +} +json.dump(out, sys.stdout, indent=2) +PY + +log "wrote $RESULT_JSON" +log "OK" diff --git a/bench/scripts/render-cloud-init.sh b/bench/scripts/render-cloud-init.sh index 41e3f60..aedb383 100755 --- a/bench/scripts/render-cloud-init.sh +++ b/bench/scripts/render-cloud-init.sh @@ -1,20 +1,26 @@ #!/bin/bash -# render-cloud-init.sh — substitute the harness SSH key (and, for the -# nftables candidate, the rules.nft contents) into the cloud-init -# templates, then pack a seed ISO that libvirt can attach as a virtio -# CD-ROM. +# render-cloud-init.sh — render cloud-init user-data/meta-data and +# inject them into the *instance qcow2* under +# /var/lib/cloud/seed/nocloud/. This is cloud-init's well-known +# NoCloud seed path; ds-identify always picks it up regardless of +# CD-ROM/blkid timing or virtio-vs-SATA quirks. # -# Usage: render-cloud-init.sh -# candidate = subdir name under bench/candidates/ (e.g. nftables, trafficgen) -# role = sut | gen (sut: full rules + nftables; gen: simpler) -# out-iso = path to write seed ISO -# pubkey-file = path to SSH public key +# We deliberately bypass the cloud-localds + seed-ISO route: the ISO +# approach turned out to be fragile on Q35 + virt-customised images +# (ds-identify wouldn't run, then with `ds=nocloud` cmdline cloud-init +# would read user-data from the (empty) cmdline instead of the ISO). +# +# Usage: render-cloud-init.sh +# candidate = subdir under bench/candidates/ (e.g. nftables, trafficgen) +# role = sut | gen (sut: full rules + nftables; gen: simpler) +# instance-qcow2 = path to the per-instance qcow2 to inject into +# pubkey-file = path to SSH public key set -euo pipefail CAND="${1:?candidate (subdir under bench/candidates/) required}" ROLE="${2:?role (sut|gen) required}" -OUT_ISO="${3:?output ISO path required}" +INST_DISK="${3:?instance qcow2 path required}" PUBKEY_FILE="${4:?SSH public key path required}" if [ ! -f "$PUBKEY_FILE" ]; then @@ -34,20 +40,15 @@ PUBKEY="$(cat "$PUBKEY_FILE")" WORKDIR="$(mktemp -d)" trap 'rm -rf "$WORKDIR"' EXIT -# Copy templates into the workdir so we can substitute in place. cp "$TEMPLATE_DIR/user-data" "$WORKDIR/user-data" cp "$TEMPLATE_DIR/meta-data" "$WORKDIR/meta-data" # Render @@HARNESS_SSH_KEY@@. -# Use Python instead of sed because the pubkey contains slashes and -# `+` characters that mis-trigger sed. python3 - "$WORKDIR/user-data" "$PUBKEY" <<'PY' import sys, pathlib path = pathlib.Path(sys.argv[1]) key = sys.argv[2] -text = path.read_text() -text = text.replace("@@HARNESS_SSH_KEY@@", key) -path.write_text(text) +path.write_text(path.read_text().replace("@@HARNESS_SSH_KEY@@", key)) PY # For the nftables SUT, splice in the rules.nft contents. @@ -57,19 +58,35 @@ if [ "$ROLE" = "sut" ] && [ -f "$BENCH_DIR/candidates/$CAND/rules.nft" ]; then import sys, pathlib, textwrap target = pathlib.Path(sys.argv[1]) rules = pathlib.Path(sys.argv[2]).read_text() -# cloud-init's content block under write_files needs each line indented -# by 6 spaces (4 for the YAML " content: |" + 2 for the literal block). indented = textwrap.indent(rules, " ") -text = target.read_text().replace(" @@RULES_NFT@@", indented) -target.write_text(text) +target.write_text(target.read_text().replace(" @@RULES_NFT@@", indented)) PY fi -# Build the seed ISO. -if ! command -v cloud-localds >/dev/null 2>&1; then - echo "ERROR: cloud-localds not found (install cloud-image-utils)" >&2 +if ! command -v virt-customize >/dev/null 2>&1; then + echo "ERROR: virt-customize not found (install guestfs-tools)" >&2 exit 1 fi -cloud-localds "$OUT_ISO" "$WORKDIR/user-data" "$WORKDIR/meta-data" -echo " seed ISO: $OUT_ISO" +# Build the virt-customize argument list. Seed always goes in; the +# trex candidate also gets the pre-downloaded TRex tarball injected +# into /var/tmp/ so cloud-init can extract it offline (the bench +# bridges have no NAT). +declare -a VC_ARGS=( + --mkdir /var/lib/cloud/seed/nocloud + --upload "$WORKDIR/user-data:/var/lib/cloud/seed/nocloud/user-data" + --upload "$WORKDIR/meta-data:/var/lib/cloud/seed/nocloud/meta-data" +) + +if [ "$CAND" = "trex" ]; then + TREX_TARBALL="$BENCH_DIR/images/trex/trex-latest.tar.gz" + if [ ! -f "$TREX_TARBALL" ]; then + echo "ERROR: TRex tarball missing at $TREX_TARBALL — run 'make trex-fetch' first" >&2 + exit 1 + fi + VC_ARGS+=( --upload "$TREX_TARBALL:/var/tmp/trex-latest.tar.gz" ) +fi + +sudo virt-customize -a "$INST_DISK" "${VC_ARGS[@]}" >/dev/null + +echo " cloud-init seed injected into $INST_DISK${CAND:+ (candidate=$CAND)}" diff --git a/bench/scripts/smoke.sh b/bench/scripts/smoke.sh index f83a3f7..3315bdc 100755 --- a/bench/scripts/smoke.sh +++ b/bench/scripts/smoke.sh @@ -63,6 +63,36 @@ for tgt in "$SUT_LAN_IP" "$GEN_LAN_IP"; do done done +# --- Wait for cloud-init to finish on both VMs ------------------------------- +# Tests after this point assume the firewall ruleset is loaded and the +# trafficgen wan-netns is set up. Without this barrier, the ruleset check +# can race with cloud-init's runcmd that enables `nftables-thurward.service`. + +log "waiting for cloud-init to finish on SUT + gen ..." +gen "cloud-init status --wait" >>"$RAW_LOG" 2>&1 || true +sut "cloud-init status --wait" >>"$RAW_LOG" 2>&1 || true + +# --- Test 0: SUT ruleset loaded (security-effectiveness precondition) ------- +# Done up-front, before tests 1-4 generate traffic. We poll +# `systemctl is-active` rather than `sudo nft list | grep` because the +# latter races with cloud-init's runcmd and depends on sudo/NOPASSWD +# being live; `is-active` only requires a working sshd. The oneshot +# unit reaches "active (exited)" only after `nft -f /etc/nftables.conf` +# returns, so this is a direct signal that the ruleset is loaded. + +log "test 0: verify SUT nftables-thurward.service is active" +RULESET_OK=false +for i in 1 2 3 4 5; do + state=$(sut "systemctl is-active nftables-thurward.service" 2>>"$RAW_LOG" || true) + log " attempt $i: state=$state" + if [ "$state" = "active" ]; then + RULESET_OK=true + break + fi + sleep 2 +done +log " result: RULESET_OK=$RULESET_OK" + # --- Test 1: ICMP echo from gen-LAN to SUT-LAN ------------------------------ log "test 1: ICMP from gen-LAN ($GEN_LAN_IP) to SUT-LAN ($SUT_LAN_IP)" @@ -74,8 +104,11 @@ fi log " result: ICMP_LAN_OK=$ICMP_LAN_OK" # --- Test 2: forwarded ICMP from gen-LAN to a WAN-side IP -------------------- -# The gen has a route 203.0.113.0/24 via 10.10.0.1 (per its cloud-init). -# This exercises the FORWARD chain. +# The gen's root netns has only a default route via 10.10.0.1 — enp2s0 +# lives in the `wan` netns (see candidates/trafficgen/cloud-init/user-data), +# so the only path to 203.0.113.0/24 is *through* the SUT. This is what +# forces the FORWARD chain to be exercised even though both endpoints +# happen to live in the same VM. log "test 2: forwarded ICMP gen-LAN -> WAN address (203.0.113.50)" if gen "ping -W 2 -c 3 203.0.113.50" >>"$RAW_LOG" 2>&1; then @@ -113,16 +146,6 @@ else fi log " result: IPERF_BPS=$IPERF_BPS" -# --- Test 5: confirm SUT ruleset is loaded ----------------------------------- - -log "test 5: verify SUT ruleset is loaded" -if sut "sudo nft list ruleset" 2>>"$RAW_LOG" | grep -q 'table inet thurward'; then - RULESET_OK=true -else - RULESET_OK=false -fi -log " result: RULESET_OK=$RULESET_OK" - # --- Assemble JSON result ---------------------------------------------------- # Pull host invariants from the host (not the guest). diff --git a/bench/topology/lan-thurward.xml b/bench/topology/lan-thurward.xml index c84a082..3de8f50 100644 --- a/bench/topology/lan-thurward.xml +++ b/bench/topology/lan-thurward.xml @@ -9,7 +9,11 @@ See bench/README.md for the broader topology overview. lan-thurward - + + diff --git a/bench/topology/wan-thurward.xml b/bench/topology/wan-thurward.xml index 8e7a5e3..f3ab57e 100644 --- a/bench/topology/wan-thurward.xml +++ b/bench/topology/wan-thurward.xml @@ -10,7 +10,11 @@ No external NAT, no DHCP — every guest has a static IP via cloud-init. wan-thurward - + + From b5e52b753eecff904f84a2615f42bbe9e342026b Mon Sep 17 00:00:00 2001 From: letur Date: Mon, 25 May 2026 20:31:42 +0200 Subject: [PATCH 2/2] docs(bench): scope decision (throughput is tier-1 only) + latest smoke result MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tier-0 produces meaningful functional/correctness numbers but cannot produce trustworthy throughput numbers — the workstation virtio path bottlenecks the measurement well before the SUT does. Make that explicit in three places so future readers don't chase what was never measurable here. - bench/README.md: new "Scope decision: throughput is Tier-1 only" section with reasoning (1-vCPU SUT + virtio, § 0.6 control logic) and a "what stays in scope on Tier 0" list (smoke, functional correctness, footprint metrics B-21..B-24). The TRex/sink scaffolding stays in the tree so the Tier-1 box runs `make b-00` as-is when it arrives. Also a "Latest Tier-0 smoke run" block showing today's pass — verdict, per-probe table, host invariants, Tier-0 caveat. - bench/ROADMAP.md: Tier-0 "what it can measure" now lists footprint metrics; "what it can't measure" calls out B-00..B-13 explicitly with the substrate-bottleneck reasoning and a pointer to the scaffolding that's runnable for Tier 1. --- bench/README.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++ bench/ROADMAP.md | 16 ++++++++++---- 2 files changed, 69 insertions(+), 4 deletions(-) diff --git a/bench/README.md b/bench/README.md index 060f15f..52a3cfa 100644 --- a/bench/README.md +++ b/bench/README.md @@ -98,6 +98,63 @@ make tear-down # destroy everything Results land under `bench/results/smoke//` as JSON. +## Scope decision: throughput is Tier-1 only + +The bench tree contains a TRex generator + sink-VM scaffolding +(`make trex-up`, `make sink-up`, `make b-00`) but **headline throughput +numbers (B-00 through B-13) are deliberately out of scope on Tier 0**. + +Reasoning: + +- The host is a developer workstation. The SUT VM has 1 vCPU; the + virtio-net path between qemu processes caps well below TRex's + software-mode ceiling, which itself caps well below TRex's normal + DPDK-NIC ceiling. **Any throughput number produced here measures the + virtio path, not the SUT's envelope.** +- `tests/benchmarks.md` § 0.6 already says B-00 is a per-environment + control: if the control itself is loss-limited, downstream B-NN + numbers are not reportable. On this Tier-0 setup the control is + always loss-limited. +- The right time to run B-00..B-13 is on the Tier-1 lab box + ([`ROADMAP.md`](./ROADMAP.md)) — physical NIC + SR-IOV + RT kernel + + dedicated NUMA node. Until that hardware exists, those benchmarks + are intentionally skipped, not faked. + +What stays in scope on Tier 0: + +- **Smoke** (`make smoke`) — reachability + ruleset-loaded precondition + (per `tests/benchmarks.md` § 0.8). +- **Functional correctness** — rule translation discipline (§ 0.7), + scenarios.md, security-effectiveness checks. +- **Footprint metrics** (B-21 image size, B-22 RSS, B-23 launch time, + B-24 idle latency) — these are workstation-friendly and produce + per-candidate numbers that don't depend on throughput. + +The TRex/sink scaffolding stays in the tree so the Tier-1 box can run +B-00..B-13 immediately when it arrives — no harness rewrite needed. + +## Latest Tier-0 smoke run + +Run on 2026-05-25 against the `nftables` candidate on an Arch +workstation (Intel i7-8700K @ 3.7 GHz, virtio-net, 1 vCPU SUT). +Verdict: ✅ `OK` — precondition (`§ 0.8`) satisfied. + +| Probe | Result | Interpretation | +| ------------------------------------------- | ----------------- | ------------------------------------------------------------- | +| SUT `nftables-thurward.service` | `active` | Ruleset loaded; security-effectiveness precondition met | +| ICMP gen-LAN → SUT-LAN (`10.10.0.1`) | reachable | `input` chain permits ICMP on `enp1s0` as designed | +| ICMP gen-LAN → WAN (`203.0.113.50`) | dropped (correct) | `forward` chain default-deny working | +| TCP/443 gen-LAN → WAN | crossed firewall | `allow-github-https` rule active | +| iperf3 gen-LAN → gen-WAN-netns through SUT | **~14 Gbps** | TCP/5201 allowed; virtio-bounded (see Tier-0 caveat below) | + +**Tier-0 caveat.** ~14 Gbps is the virtio-net path between two qemu +processes on the same host — not the SUT envelope. See the "Scope +decision" section above and [`ROADMAP.md`](./ROADMAP.md) Tier-1 for +where comparable-to-vendor numbers will come from. + +Run-specific result JSON lands under `results/smoke//` (not +in git — the per-run output is gitignored). + ## Methodology pointer Every harness run **must** satisfy the methodology in diff --git a/bench/ROADMAP.md b/bench/ROADMAP.md index d7f2146..1607df3 100644 --- a/bench/ROADMAP.md +++ b/bench/ROADMAP.md @@ -14,19 +14,27 @@ characterising behaviour in thurward's ~1 Gbps v1 envelope. - Single host (developer workstation) - KVM + libvirt + virtio-net - Host-internal libvirt bridges, no physical NICs -- iperf3 + ping for smoke; TRex (in a VM) for headline benchmarks +- iperf3 + ping for smoke; TRex scaffolding present but not used for + headline numbers at this tier (see below) **What it can measure** - Functional correctness (security-effectiveness precondition, `tests/benchmarks.md` § 0.8 — does the firewall actually enforce its rules?) -- Throughput up to ~1–2 Gbps with caveats -- Latency on the order of 100s of µs (host scheduler noise included) +- Footprint metrics (B-21..B-24): image size, RSS, launch time, idle + latency baseline - Conntrack / NAT correctness at moderate scale -**What it can't measure** +**What it can't measure (and won't try to)** +- **Throughput (B-00..B-13).** Defer to Tier 1. The 1-vCPU SUT plus + virtio path bottlenecks well below TRex's software-mode ceiling, + which itself sits well below TRex's normal DPDK-NIC ceiling. Any + number here would be a harness measurement, not a SUT one. The + TRex/sink VMs are in the tree and runnable (`make b-00`), but the + resulting numbers are explicitly labelled "harness ceiling" in the + output JSON and not promoted as SUT capability. - VPP's 10+ Mpps DPDK ceiling - Sub-microsecond latency - Sustained PPS at small frames against physical-NIC line rate