M5 Runner, one run on real hardware

[tamnd]

One benchmark run on real hardware, start to finish, written to one JSON file. Everything before this milestone is arithmetic over data somebody else measured. This is where the project starts measuring.

The sequence is fixed and does not vary by engine. Start the server pinned to the first half of the cores, with persistence off and a memory limit large enough that nothing is ever evicted, listening on a unix socket. Wait for it to answer by connecting, not by sleeping. Run memtier once as a warmup and throw it away. Run it again, pinned to the other half of the cores, and keep that. Stop the server, wait for the process group to be gone, and check that nothing was left behind.

Scope

• All seven adapters, each with its own thread flag, persistence off, memory limit and unix socket argv.
• Readiness by probe. Sleeping instead is how a benchmark ends up measuring a server that was still growing its hash table.
• CPU pinning through sched_setaffinity in a pre_exec hook rather than by shelling out to taskset. This is the only unsafe block in the tree and it carries a safety comment.
• Process group termination and a stray process check after every run, because a server that survived its run competes with the next one for the same cores and the result looks like a real number.
• The memtier driver, its argv, and a parser for its output. The argv is validated rather than assumed, since a flag that memtier silently ignores is a benchmark that silently measures something else.
• The warmup pass, which is not a courtesy. Several of these engines allocate lazily and a first pass measures the allocator.
• The perf driver, the PMU probe, and CSV parsing, with cycles counted on the server process only and only for the measured pass.
• cache-bench run, one cell, one file.

Exit, and it is a gate

Every one of the seven servers starts, answers, completes a run and stops cleanly on a real Linux host, with and without perf attached, and no stray process is left behind by any of them.

[tamnd]

#39 landed the memtier driver, which is one box off M5.

The argv is built in one place from the profile and the pass, and there is a test that compares what comes out for the reference profile against the exact command line the original's published numbers came out of. The original spreads the same flags across a shell script and a Go program whose defaults disagree, so a flag can drift there without anything noticing.

The parser is the half worth arguing about. The original reads memtier's JSON with a path query that returns zero for a path that is not there, so a memtier that renamed a field, or a run where a third of the connections were refused, still produces a result file, and that file charts as real bars sitting on the axis. Three checks before a result is accepted here: the stats object exists, the completed operation count is within a thousandth of the count that was asked for, and all five requested percentiles came back. Each of the three is a failed run rather than a number.

The count check is the one that catches the case nobody sees otherwise. memtier does not always divide operations evenly across connections, so an exact match is not something to demand, but anything past a thousandth is connections that died mid run rather than rounding.

Spawning is not in this PR. It goes through the supervisor in cb-cache, which is next, and the parser is the half that can be tested without a machine with memtier on it.

[tamnd]

#41 is merged, which covers three of the boxes above: readiness by probe, CPU pinning in a pre_exec hook, and process group termination with a stray check.

The supervisor is the piece everything else in this milestone sits on. A server is started in its own process group, pinned in the child itself between fork and exec rather than by wrapping it in taskset, and its output goes to a file next to the run. Stopping asks first with SIGTERM, waits, and only then kills, and it does not return until the whole group is gone. If something is still there after the kill, that is reported as a stray rather than passed over, because a server that survived its run competes with the next one for the same cores and the result looks like a real number.

Readiness is a protocol round trip instead of a sleep. A connection is opened and the smallest command the protocol has is sent, PING for the six that speak RESP and version for memcached, and the server is ready when it answers correctly. Accepting a connection is not the same thing, and neither is a sleep, which cannot tell a slow server from one that died on its first flag. A server that exits while being waited for is reported as exited straight away rather than after the full deadline.

The pinning is the only unsafe in the tree. It is denied at the workspace level rather than forbidden so that these three sites can allow it individually, each with a safety comment saying why it is sound. Recorded as D20.

Next up is the perf driver and the PMU probe, then the seven adapters wired to the supervisor, the warmup pass, and cache-bench run.

[tamnd]

The perf driver landed in #42.

Counters are taken over the server process for the length of the two measured passes and nothing else. Not over memtier, which would count the load generator's work as the server's, and not over the machine, which would count whatever else the box was doing. perf is attached after the warmup and stopped with SIGINT, because that is the signal it answers by printing its counters rather than by dying with them unwritten. Its output goes to a file next to the run rather than through a pipe, so a capture survives a harness that fell over between the interrupt and the read.

The PMU probe is two checks rather than one, because either check on its own gives the wrong answer on a machine somebody will actually run this on. A cpu event source has to be listed by the kernel, and a live perf stat -e cycles has to come back with a number, because a virtual machine can have the directory and still answer <not supported> for every hardware event, which is the common case and the one that matters. perf_event_paranoid is read separately so that a host with real counters and a locked down kernel is told it has a setting to change rather than told it has no PMU. Three states, three different things to do about them.

The counters are read from -x, output rather than from the table perf prints for a person, and CPUs utilized is computed from task-clock and a measured duration rather than scraped out of a comment column. The original parses the human table with two string helpers that return an empty string when they do not find what they are looking for, so a perf that changed its column widths would produce zeros that chart as real bars. That is recorded as D21.

[tamnd]

memtier is actually run now, in #44.

Three invocations make up a run and all three go through one path. The only difference between them is the ratio in the argv and whether the caller keeps the answer. memtier is pinned to the load generator half of the machine, the half the server was kept off, because a load generator sharing cores with the server measures the two of them fighting. What it prints while it works goes to a log next to the run rather than to the terminal, which is where a refused connection says so, and it is also the difference between a two week job and gigabytes of scrollback.

The warmup is checked exactly like the two measured passes. Its numbers are thrown away, but a warmup that did not run means the measured SET pass is partly a measurement of hash table growth, which is the one thing the warmup exists to prevent, so it fails the run rather than passing quietly.

Two things that only matter when something has already gone wrong. A pass has a ceiling on how long it may take, because a pass that never finishes is memtier waiting on a server that stopped answering, and every later run in a sweep is queued behind it. And the result file is cleared before the pass that writes it runs, because a file left by an earlier pass holds real numbers from a real run, so a memtier that died before writing would otherwise produce a result that looks entirely credible and belongs to something else. That last one is the failure that cannot be spotted afterwards, which is why it is handled before it can happen rather than checked for later.

[tamnd]

The seven adapters are wired to the supervisor in #43, and every server is asked what it is before it is measured.

There is one start path and it does not vary by engine. Clear the socket, start the binary pinned to the cache half of the cores with its output going to a file next to the run, wait for it to answer, and later stop the group and confirm nothing survived. An engine given special handling here would be an engine measured differently from the rest, which is the thing the fairness rules exist to prevent, so the adapters hold flags and nothing else and the process handling is shared.

The socket is cleared before a server starts, because several of these refuse to bind a path that already exists and a run whose server was killed leaves one behind. Only a socket is removed. Anything else at that path is somebody's file, and a benchmark that deletes one would be a worse failure than the one it was avoiding.

Version detection is the original's: run the binary with --version, keep the first line, strip the colour escapes, trim. Nothing tries to parse a version number out of it, because the seven do not agree on a format and a number parsed wrong is worse than the line the server printed. That string goes verbatim into every result file the server produces, and the generated README's version table is written from those strings rather than typed, so a build swapped halfway through a sweep shows up in the results instead of in a footnote nobody wrote.

[tamnd]

#45 is merged, which is the last box on this list.

cache-bench run <cache> --threads N --pipeline N --perf yes|no --run N measures one cell once and writes one file. Ask the binary its version, clear any leftover socket, start the server pinned to the cache half of the cores, wait for it to answer a real command, warm up, attach perf if the cell wants counters, run the set pass and the get pass, detach, stop the group, confirm nothing survived, write. A run that fails anywhere in that writes nothing, and a cell already on disk is skipped rather than measured again, which is the whole of how a sweep that takes days is restartable.

Five checks in there are not in the original, and each one is a failure that produces numbers rather than an error. A stray on the socket stops the run instead of being pkilled along with somebody else's processes. The results directory is locked for the length of the run. The result file and its directory entry are flushed before the run is called finished. A perf cell is refused on a machine whose counters do not answer. A thread count larger than the pinned cores is refused. One more went in while fixing CI: a profile written for a bigger machine than the one running it is refused up front, because a pin naming some cores that exist and some that do not is narrowed by the kernel rather than refused, and the run still produces numbers.

There is also one whole run in the test suite now, against a cache server and a load generator that are not real. Every piece of the runner had tests around it and none of them could check the sequence: that the socket in the argv is the socket the readiness probe connects to, that the file memtier is told to write is the file that gets read back, and that what comes out is a result file of the shape the rest of the harness reads. It takes about a second and needs nothing installed.

That is all eight boxes. The exit condition on this milestone is a gate rather than a checklist, and it has not been run: seven servers each starting, answering, completing a run and stopping cleanly with and without perf needs a Linux host with all seven built on it. Everything up to that point is verified, on three operating systems in CI and against fakes end to end, but the gate itself is hardware work and it stays open until it runs on hardware. v0.5.0 will say the same thing rather than imply otherwise.