M6 Sweep, the whole matrix

[tamnd]

The whole matrix instead of one cell. Roughly ten thousand runs over about eight days on the large host, which means the interesting engineering here is not throughput, it is what happens when something goes wrong on day six.

Scope

• cache-bench doctor, checking cores, the core split, memtier and its version, the PMU, perf_event_paranoid, free memory against the profile's limit, the projected working set against that limit, and the load average. It refuses a profile the host cannot honestly run rather than warning and continuing.
• cache-bench sweep, iterating prog then threads then pipeline then perf then run, in that order, matching the original so a partial results directory stays comparable, and so all 31 runs of a cell happen together in time and a disturbance shows up as one bad cell instead of smeared across all of them.
• Restartable by file existence, and a result file that does not parse is re-run rather than accepted. Silently trusting a truncated file is the failure mode that costs a whole sweep.
• A lock file per results directory. One session at a time against one clone.
• results/logs/sweep.jsonl, one object per run, with the load average recorded before each one so a noisy window can be found afterwards rather than guessed at.
• results/failures.json as a first class output. A cell that failed is absent with a footnote, never zero, because a zero on a chart is a claim and an absence is not.
• An ETA, derived once there are enough completed runs to derive one rather than from an estimate scaled off somebody else's silicon.

Exit, and it is a gate

A smoke sweep of every engine at one pipeline depth and two thread counts completes unattended, killing it halfway and restarting it loses no completed work and repeats no completed run, and the failure file is empty or every entry in it is understood.

[tamnd]

The machine half of doctor is in, #47.

It reads what the kernel publishes about the box it is on, the kernel and the distribution, the CPU and how many of it there are, the memory and how much of it is actually available, the frequency governor, which mitigations are still open and the load average, and it does that through parsers that each take a string, so they are tested against the text an ARM box, an x86 box and a container each produce rather than only against the machine they were written on. A fact the machine does not publish comes back as missing and is printed as missing, never as a guess.

Then it measures the profile against what it found and refuses rather than warns. A profile that names more cores than the machine has, because the pins in it name cores by number and a pin naming a core that is not there is either refused by the kernel in the middle of a run or quietly narrowed so that a sweep meant for sixteen cores runs on four and produces numbers. A profile that sweeps the cycles half of the matrix on a host with no counters. A working set that would not fit in what is available, which is a sweep that measures the page cache. A load average that says somebody else is using the machine. Every one of those produces plausible numbers rather than an error, which is why each of them stops here. A warning printed at the start of a job that runs for eight days is read by nobody.

Two more things came with it. --deep starts each of the seven servers in turn, waits for it to answer over the socket, stops it and checks the group is gone, which is the check no file can make: a binary of the wrong architecture, a server built without unix socket support and a Garnet whose runtime is not installed all read as a correct config and all fail on the first run of a sweep instead. --write records what the machine is in host.json next to the results, refusing to write a fact the machine did not give rather than writing unknown, since that file is the whole of what a published results directory says about where its numbers came from.

The same core check is now shared with run, which is what the integration test failing on a four core CI runner turned up. It was measuring against the reference profile, whose pins name cores 0 to 31, and the kernel refused the mask with Invalid argument and nothing saying which argument. The test now carries a two core profile of its own, and the underlying case gets a sentence explaining what is wrong with it.

Next is the sweep itself, the three boxes about the loop and the restart rule and the lock.

[tamnd]

#48 is in. cache-bench sweep measures the whole matrix in the original's order, prog then threads then pipeline then perf then run, so all the runs of one cell happen together in time and a disturbed window shows up as one bad cell rather than smeared across the sweep.

Restarting is by file existence, and a file that will not parse is not a measured cell, so a half written run left behind by a machine that lost power gets measured again. The lock is taken once for the whole session rather than once per cell, which is what let the run command and the sweep share the same measure step instead of each having its own.

That is three more boxes ticked. The journal, the failure file and the ETA are next, and they are written and waiting on a PR of their own.

[tamnd]

#49 is in, and the scope list is now ticked all the way down.

A sweep leaves two files behind. logs/sweep.jsonl is a line per run attempted with the load average taken before each one, which is the file that answers what was happening at three in the morning on day six. failures.json is what is missing and why, counted across restarts, because a cell that failed once reads differently from a cell that has failed four times.

A failed cell no longer stops the sweep now that there is somewhere for the reason to go, and three failures in a row from one engine leaves that engine alone for the rest of the session. The next sweep tries it again. The ETA comes from the last twenty runs this sweep did and says nothing until it has three of them.

What is left in this milestone is the exit, and it is a gate: a smoke sweep of every engine on a real host, killed halfway and restarted. That needs the hosts, so it happens alongside M7.

[tamnd]

Half of the exit gate here got tested for real today, by accident, which is the best way for it to happen.

The large host tears its virtual machine down when whatever started it goes away, so a draft sweep started there was killed four separate times, each one mid cell rather than at a boundary. That is a harder test than the one this issue describes, because killing it halfway was not a controlled SIGINT, it was the whole machine going out from under a running memtier.

It came through it exactly as specified. Three runs had completed and been written. On restart the sweep announced 477 cells rather than 480, which is the arithmetic this box asks for: no completed work lost and no completed run repeated. The runs directory holds run_1, run_2 and run_3 of the first cell and nothing else, so the run that was in flight at each kill left no partial file behind to be trusted later. The failure file is an empty list, which is right, because a machine going away is not a cell that failed.

The one thing it needs a human for is the lock file, and the error message is good enough that it does not need anything more. A killed sweep leaves its lock behind, the next start refuses with the path, the holding pid and what to do about it, and removing it is the whole fix. That is the correct trade for a lock whose job is to stop two sessions writing to one results directory.

What is not shown yet is the other half of the gate, which is a smoke sweep completing unattended. That needs a host that stays up, and where that stands is on #9 rather than here.