diskIostat: grouped iostat averages swap/cache/member disks into diskServiceTime, masking a saturated data volume

[MagnaCapax]

Problem

scripts/lib/diskIostat.php measures disk latency over every block device matching PMSS_BLOCK_DATA_DEVICE_NAME_PATTERN (sd*|vd*|xvd*|nvme*|mmcblk*, scripts/lib/runtime.php). It runs one grouped sample:

iostat -xm 120 2 -g grp1 <all matching devices>

It then reports the grp1 row: diskAwait = group r_await, diskServiceTime = group w_await.

The group row is a request-weighted average across all of those devices. A busy, fast device that is not the data volume therefore drowns out a saturated data volume:

• Guest with a swap disk (e.g. vda = data, vdb = swap). In one sample, swap ran ~3000 r/s + ~2800 w/s at 0.3–2.3 ms, while the data disk was at 100% util, r_await 83 ms / w_await 171 ms. The group w_await from the same sample works out to ~2.9 ms. The reported service time looked healthy while the volume holding user data was saturated.
• Bare metal. The pattern takes each RAID member disk and any cache NVMe, but not the md device the data actually lives on. The average therefore blends member disks and cache devices rather than describing the data volume.

Downstream consumers treat diskServiceTime as the data-volume latency, so a saturated node can look nominal.

Candidate fixes (choose one)

1. Exclude swap devices (from /proc/swaps) and cache-only devices from the grouped sample.
2. Sample the device(s) that back the data mount (/home) and report those, falling back to the group.
3. Report the per-device max of r_await/w_await alongside the group value (new fields), so consumers can use the worst device.

Option 2 matches what the metric is used for. Option 3 is the smallest change that keeps existing fields as they are.

Notes

• diskServiceTime is w_await only, so read-heavy saturation shows up only in diskAwait. Worth stating in the field docs whichever fix lands.
• diskIostat.php awk column-shift bug on sysstat 12+ — diskAwait/diskServiceTime/throughputs read wrong columns #580 (column-shift) fixed the parser; this is about which devices are sampled, not about how the columns are parsed.

(An average is a very polite way to hide the worst disk.)

---

_{Tier-3 declared at filing: --why multiple-fix-paths — GH#631 tier gate. Declared fix size: 30 lines vs 29-line body — size gate (operator directive 2026-07-29). Owner: scripts/lib/diskIostat.php in MagnaCapax/PMSS — owner gate.}

[MagnaCapax]

Two additions from reading the code paths:

1. diskAwait is affected just as much as diskServiceTime. Both come from the same grp1 row. A consumer that uses diskAwait (group r_await) as its latency metric sees the same dilution, and for read-heavy load that is the field that matters.

2. diskQuantity has the same root cause. It counts every sampled device. A guest with one data disk and one swap disk reports 2, and bare metal counts root and cache devices alongside the data array's members. Anything that reads diskQuantity as "number of disks behind the data volume" gets the wrong number.

Both fields come from the same device list, so there is one place to fix. Option 2 above covers both:

• Resolve the block device(s) behind the /home mount, following /sys/class/block/<dev>/slaves through md, dm and bcache down to the leaf disks.
• Sample and count only those.
• If resolution fails, fall back to the current all-devices list.

[MagnaCapax]

Commit 4d3ac82 adds homeDiskAwait, homeDiskServiceTime, and homeDiskQuantity to the serialized iostat snapshot. The /home source is resolved through the shared mount helper, block-device slaves are followed to sampled leaf disks, and read/write latencies are weighted by their respective request rates. Unresolved topology or missing rows yields null for all three new fields. The existing group fields and iostat command retain their values and meaning.

Hermetic tests cover a guest data disk alongside swap, six md members alongside a sampled root disk, bcache over md, a mapper device over a partition, unresolved and unsampled leaves, missing iostat rows, zero-traffic plain means, and serialized legacy-field values. The development runner passed 3,181 tests with zero failures; PHP lint, test-php.sh, test-bash.sh, PHP 7.3 compatibility checks, docblock lint, and doctrine lint passed.

(The old ledger stays put.)

— Sampsa Pellervoinen 🌱

[MagnaCapax]

Commit 4c18893 removes the three parallel homeDisk fields and their per-device weighting. The existing snapshot fields now use an iostat group built only from resolved /home leaf disks; diskQuantity counts those sampled disks. If /home cannot be resolved, sampling still falls back to the full discovered list. The snapshot key set matches the pre-4d3ac825 set.

The development Runner and required lint and testing scripts passed. ADR 0070 now records the in-place decision. (The group now measures its intended ground.)

— Sampsa Pellervoinen 🌱