M8 Publish, two results directories

[tamnd]

The deliverable. Two complete, internally consistent results directories, committed, with the raw data next to them.

Two directories and not one. The large host has 32 cores and reproduces the reference topology but has no PMU, so it cannot produce the cycles charts at all. The small host has a real PMU and produces all 154. Taking throughput from one and cycles from the other and presenting them as one set gives a chart set in which no two charts are necessarily comparable, and that is a worse outcome than a missing chart. Each set is internally consistent and every chart says which host it came from.

Scope

• The publish sweep on the large host, 31 runs per cell, throughput and latency, 146 of the 154 charts.
• The publish sweep on the small host, 31 runs per cell, both perf modes, all 154 charts including cycles.
• output.json and host.json committed next to the charts, so anybody can redraw every chart from committed data without trusting us.
• The raw run files archived as a release asset rather than committed. Seventeen thousand small files in git history is a tax on every clone forever, and they are still the evidence, so they ship. The mechanism is proven on the draft directory: results-wsl32coarse-20260909 carries 4636 files, the tag deliberately does not start with v so the crate release workflow does not fire on it, and cache-bench archive --check reads it back against its own manifest. This box stays open until the two publish sweeps have theirs.
• Coefficient of variation reviewed per cell before anything is published, and any cell that was clearly disturbed re-run.
• The README carrying the narrow claim these numbers support and the sentence saying what they do not, wherever a number appears.
• Every crate published to crates.io on a tag, so cargo install cache-bench is how somebody gets this onto a box they are about to measure.
• No chart from this project presented beside a chart from the original. Different architecture, different absolute numbers, and engines that are close together can genuinely reorder.

Exit, and it is a gate

Both directories are committed and complete, every chart is stamped with its host, redrawing every chart from the committed output.json reproduces the committed PNGs byte for byte, and the results README says plainly what the numbers may and may not be used for.

[tamnd]

#50 is in. Every crate in the workspace now goes to crates.io on a tag, which is the last piece of getting this onto a machine that is about to be measured: cargo install cache-bench rather than a git checkout and a build.

Eight crates go up, in dependency order, after the GitHub release exists. That order is deliberate. A GitHub release can be deleted and a version on crates.io cannot, so the irreversible half runs last and only after every gate has passed.

The first release is the slow one. crates.io lets you create five new crate names in a burst and then one every ten minutes, and there are eight new names here, so the job will sit and wait for the last three. It has a two hour timeout and it only sleeps when the failure actually looks like a rate limit. Anything else fails immediately, because waiting ten minutes to find out a build is broken is a waste of everybody's afternoon. The job also skips any version already on crates.io, so a release that dies halfway is finished by dispatching the workflow again rather than by hand.

Two things moved to make this possible. A cargo package only carries files inside its own directory, so the golden files and the fonts had to come in from the top of the tree. The fixtures now live at crates/cb-core/golden and are reachable as cb_core::golden, and the fonts live at crates/cb-chart/assets/fonts. There are no ../../../ includes left anywhere. The binary crate was also renamed from cb-cli to cache-bench, so the thing you install and the thing you type are the same word.

New CI job: package runs cargo package --workspace --locked, which builds every crate from its own tarball. That is the only check that catches an include reaching out of a crate root, and it is worth having on every push rather than on the day a tag is cut, because by then the version is published and cannot be replaced.

[tamnd]

The coefficient of variation review is now a command rather than a morning. #65 landed cache-bench spread.

Every chosen file has carried spread.sets.opsec_cv and spread.gets.opsec_cv since the statistics milestone, and nothing read them back. The scope line above says the spread is reviewed per cell before anything is published, which until now meant somebody sitting down with output.json at the end of a week long sweep, which is the worst moment to be asking a person to be careful. It has been done exactly once, by hand, and the result is the results/wsl32coarse/NOTES.md paragraph saying not to quote yo's and rugo's bars at eight and sixteen threads.

Run against that same directory the command reproduces that paragraph:

spread    96 cells in results/wsl32coarse, against a coefficient of variation of 0.05

engine        worst  over  where
dragonfly      0.08     1  threads 1
garnet         0.31     6  threads 1, 8 and 16
memcache       0.26    10  threads 1, 8 and 16
pogocache      0.06     1  threads 16
redis          0.01     0
rugo           0.62     9  threads 1, 8 and 16
valkey         0.02     0
yo             0.85    10  threads 1, 8 and 16

37 of 96 cells are above 0.05, so a number from one of them is a picture of which run won rather than of how fast the engine is

--check fails instead of only saying so, and --over moves the line, which defaults to the 0.05 those notes were already written in. A cell is judged on the worse of its two passes, because a GET chart drawn from a cell whose SET pass was a coin toss is still drawn from a disturbed cell. It throws nothing away and re-runs nothing, since what to do about a noisy cell is a decision with a machine and a deadline in it.

The scope line stays unticked, because the tool is not the review. The review happens against the two publish sweeps and neither of them exists yet.

Where it gets used first is the epyc8coarse draft queued on the eight core host, and there is a reason to be worried about it. That box turned out to be a KVM guest on shared hardware. A yo microbenchmark on it came back with every cell between 0.14 and 0.74 even with the box's own CI runners stopped and the load average down at 3 on 8 cores, which is a machine that cannot be made quiet from inside. A cache-bench sweep is a different shape of measurement, thirty second runs with thirty one of them a cell rather than four second runs with five, and averaging is exactly what bursty interference is vulnerable to, so it may well survive where the microbenchmark did not. The way to find out is to run spread on that directory and look at Redis and Valkey. They came out at 0.01 and 0.02 on the 32 core box. If they are not near that on the eight core box then that host cannot publish either, and M8 has one results directory in it and not two.

The 32 core host is not helping this week. It was rebooted a few hours ago, which killed the wsl32coarse re-sweep against yo v0.3.27 at run 225 of 480, and it now has somebody else's bazel build and a benchmark of their own on it. The sweep restarts off the files already on disk, so nothing measured is lost, and it is queued behind a quiet gate rather than started into that.

[tamnd]

#66 landed cache-bench archive, which is the tool behind the raw data scope line. The box stays unticked, because a tool that can make a release asset is not a release asset, and there is no cache-bench release yet for one to hang off.

What it does. archive --dir D --out P writes a gzipped tar of runs, logs and the small files beside them, with a MANIFEST in it holding the SHA-256 and the byte count of every member. --check P recomputes all of them, and --check P --dir D also asks whether every chosen file the committed output.json was built from is in the archive, which is what ties an asset to the charts a reader is looking at.

Why it went in now rather than at publish time. The .gitignore has said since the first commit that the raw data ships as a tarball rather than in the tree, and nothing made that tarball, so the evidence behind the published draft sweep existed on the 32 core desktop box and nowhere else. That box has rebooted three times in two days and somebody else uses it. The 480 run draft is now off it and packed: 4708 files, 11 MB, and all 384 chosen files behind results/wsl32coarse/output.json are accounted for in it.

The logs go in with the runs. A run file is derived from the pair of load generator files beside it, so logs is the layer underneath and it is the layer somebody arguing with a number ends up in. On that draft it is 3841 files against 864 runs, and it is most of the 11 MB.

Nothing else on this milestone has moved. The two publish sweeps are still waiting on hardware: the 8 core host is a third of the way through a 480 run draft with about a day left on it, and the 32 core host is being used by somebody else.

[tamnd]

#67 is merged, and the box about the narrow claim travelling with the numbers is ticked.

What was wrong is that the caveat lived in one of the three generated documents and the other two are the ones people open. A link to LINEAR.md is the link that gets shared, and a reader following it landed on 77 pictures of bars with no host, no method and no caveat anywhere above them. The README said all of it correctly and they never passed the README.

Both chart indexes now open, above the first chart, with the CPU, the logical CPU count, the distribution and the profile that was run with its own description, the line saying this is a port and these are not the original's numbers, a link back to README.md and to NOTES.md where there is one, the sentence about the x axis being the server's I/O threads and not the client count, and MAY_NOT word for word as the README carries it. Three documents, one caveat, same string, and a test asserts it is above the first chart rather than merely present.

Golden mode names no host and prints no caveat. A document generated from the spec has nothing measured behind it, and a caveat printed where there is nothing to caveat is what teaches people to skip caveats.

The charts themselves still carry no stamp in the picture, and that stays. A hostname drawn into a PNG makes its hash depend on which machine drew it, which would undo the determinism check that says a chart is a function of its data and nothing else.

That leaves five boxes here, and four of them are the two publish sweeps and the data beside them, which are the hardware gate. The release asset box needs a release for the asset to hang off, and there is none yet.

[tamnd]

Four pull requests landed since the last note here, all of them out of one finding in the draft sweep, and none of them tick a box above because every box above needs a publish sweep on a host that is not free yet. They change what a publish sweep will be allowed to say when it does happen, so they belong here.

#70 refuses a run whose load generator threads did not finish together. memtier reports Ops/sec as the completed operation count over Total duration, and Total duration runs to the first thread to finish rather than the last. Where the threads finish together those are the same number to within a hundredth of a percent. Where they do not, one pass in the draft sweep had a thread finish after 1.387 seconds while the last took 17.682, and 25.6 million operations over the first of those came out as 19.4 million a second, higher than any other engine anywhere on that box. That is D25. A pass whose last thread ran more than a quarter longer than its first is now refused.

#71 adds cache-bench recheck, which exists because that check arrived after 480 runs were already on disk. Every check the parser makes is a function of the bytes memtier wrote and those bytes are kept, all three passes of every run, so a sweep measured last month can be judged by this month's checks without measuring anything again. That is the difference between an hour and a week on a box that is busy, and on a publish sweep at 31 runs per cell it is the difference between an hour and never. It moves rather than deletes, renumbers the runs above a refusal so a cell does not stop at a hole, and records everything in failures.json exactly as the day would have. The same pull request gives choose a floor of three runs, because over two runs the median is one of them and the best and the worst are both of them, so four files would carry two numbers and show a reader a spread that is the entire sample.

#72 is the one that matters most for the box above about saying plainly what the numbers may be used for. The generated README took the shape of the sweep from the profile, so a directory that had lost cells since it was measured went on claiming 96 of them. It now says how many cells are plotted when that is fewer than the profile asked for, gives both reasons, and points at failures.json.

#73 ships the refused runs in the release archive, under runs/refused and counted separately from the runs. A run that was refused is the evidence for why it was refused, and the argument about where a threshold sits is an argument about what it kept and what it did not, so leaving them out would mean the one directory that can settle it is the one nobody has.

#69 is open with all of that applied to the existing draft directory. 83 of its 480 runs no longer pass, 18 of 96 cells have no bar left in them, and the losses are not spread evenly: yo and rugo lose their whole multithreaded half, Garnet loses two cells, and the five other engines lose nothing. That is the finding rather than a gap in the data.

The consequence for this milestone is worth stating before either publish sweep starts. A 31 run cell will refuse individual runs and keep going where a 5 run cell loses the whole cell, so the publish sweeps should come out of this in better shape than the draft did. What will not change is that a cell an engine cannot serve evenly stays empty, and the README will say so on the page rather than in a note beside it.

[tamnd]

The evidence for the draft directory is now published, in results-wsl32coarse-20260909, and #74 links it from the README and from the notes beside the charts.

4636 files: the 709 runs the charts are drawn from, the 83 a check refused after the fact, and all 3841 log files, which are the bytes memtier and the servers wrote. Before publishing it I checked it for anything that names the machine. host.json inside it is byte for byte the committed one, which already passes the anonymity test, the only address anywhere in it is 127.0.0.1, and the only hostnames are github.com and dragonflydb.io out of two engines' startup banners. There is a build directory path in the load generator output and it names a user and a directory rather than a host. Scrubbing it was the obvious next thought and is the wrong move, because the whole value of the logs is that they are the unmodified bytes the check reads, and a scrubbed log is one somebody has to trust rather than verify.

Two things this settles for the publish sweeps. The tag does not start with v, so the crate release workflow does not fire on a data release, which is the sort of thing better found now than on the day a week long sweep finishes. And the compression is good: 4636 files come to 11 MB, so seventeen thousand at 31 runs a cell is not going to be an awkward asset.

[tamnd]

The second draft directory is published, in #76, with its raw runs and logs under the tag results-epyc8coarse-20260910. None of the scope boxes here move, because all of these are about the publish sweeps at 31 runs a cell and both published directories are drafts at five. But three things about this milestone are now known that were not before, and two of them are problems.

The archive mechanism is proven on both hosts rather than one. results-epyc8coarse-20260910 is 4705 files, the tag does not start with v so the crate release workflow does not fire on it, and cache-bench archive --check reads it back against its own manifest. That box still stays open until the two publish sweeps have theirs, but there is nothing left to find out about how.

The redraw half of the exit gate holds locally on the first directory. chart --check results/wsl32coarse/graphs.sha256 reports 146 charts match, and shasum -a 256 -c of the same committed manifest gives 146 OK, so redrawing from the committed output.json reproduces the committed PNGs byte for byte on this machine. What is not proven is that a different machine gets the same bytes, and CI does not run the check at all, which is the real content of #75 and what that issue should be rewritten around.

The first problem is the small host. It has the PMU and it is the only source of the eight cycles charts, and it is also nowhere near quiet enough for a publish sweep. 91 of its 95 plotted cells vary by more than a twentieth from run to run, against 18 of 78 on the large host. The sweep already waits for a quiet load average before it measures, and that wait buys nothing: the 65 runs taken before the wait was tight enough came out a median of about 1.09 times the runs that replaced them, moving up as often as down. Steal in /proc/stat is zero, so this is neighbours on the box rather than the hypervisor. Thirty one runs a cell will narrow the error on the median, which is the point of thirty one, but it does not make a cell measured through somebody else's build into a measurement of a cache server. That has to be resolved before the small host's publish sweep is worth the eight days, and the honest options are to get the box to ourselves, to find another host with a PMU, or to publish the cycles charts with the spread stated on them.

The second is the coefficient of variation box above. It says any cell that was clearly disturbed is re-run. On the large host that is a handful of cells. On the small host, on this evidence, it is nearly all of them, and a re-run of a disturbed cell on a box that is disturbed all the time just spends another twenty minutes. That box needs a threshold and a stopping rule rather than a judgement call, and cache-bench spread --over is most of the mechanism already.

The last thing is that the draft on the small host also turned up the two generated-document bugs fixed in #76, both of which were invisible on a host with no PMU. Same pattern as M7: the drafts keep paying for themselves.

[tamnd]

#81 is merged. It ticks no box here but it takes one thing off the path to the first two.

The 32 core box could not reach the end of its matrix and nothing in the harness was going to let it. Sixteen load generator threads on that VM finish wildly apart on the deeper pipelines, the parser refuses the rate because memtier divides the whole operation count by the first thread to finish, and the five runs of one cell sit next to each other in the order the matrix goes in. So a shape that box cannot measure is three failures in a row, three in a row puts the whole engine down for the session, and a wrapper that restarts the sweep stops at the same shape every time. One cell there had been attempted fourteen times.

failures.json had counted attempts per cell since it was written and nothing read the count. Now four attempts and the cell is left alone, counted apart from the cells already on disk, and the failure the sweep exits with says how many of the missing cells have had their tries so a restarting script can tell a sweep worth running from one that would do nothing. --retry-failed throws the counts away for a machine that has changed.

It is worth saying which cells this loses, because a box that cannot measure a cell is a fact about the published table and not only about the sweep. The committed wsl32coarse failure file already refuses 40 cells of rugo, 33 of yo and 10 of garnet for exactly this reason, so the shape is known and it is not particular to one engine. The publish sweep at 31 runs a cell will have the same holes and they belong in that directory's NOTES.md rather than in a footnote here.

Next on this milestone, in order: get the two yo only sweeps that are running now off the boxes so the boxes are free, then the publish sweep on the 8 core box, which is the one that produces all 154 charts, then the 32 core box.

[tamnd]

#82 is merged and it is the other half of #81.

#81 gave a cell four attempts and then left it alone, which was the right rule and never got a chance to apply. The 32 core box was redeployed onto it and stopped at 45 cells of 60 on the first sweep. Its log says a shape of yo failed three times, the whole engine went down for the session, and the eleven cells after it were never attempted. The attempt count needs four full sweeps to reach four and the engine was gone after one.

The give up rule was written for an engine the machine cannot run at all, where attempting a thousand cells to learn that costs a day, and that case is real and stays. What tells it apart from a bad shape is whether the engine has ever produced a cell on this machine. One on the disk, or one measured in this session, and the sweep walks past the bad shape instead of abandoning the engine. It pays one run for each remaining bad cell, once, because #81's count then leaves those cells alone next time.

The box is rebuilding on that commit now. The two yo only sweeps are what stands between this milestone and the publish sweeps, and this was the last thing in the harness stopping one of them from reaching the end of its matrix.

[tamnd]

#82 worked and here is what it did on the machine it was written for.

The 32 core box was at 45 cells of 60 and stopping after one sweep that measured nothing. On the rebuilt binary its first sweep measured 2 and attempted all twelve of the cells the give up rule had been leaving, its second measured 1, and it converged at 52 cells with all eight remaining cells having had their four tries and being left alone. It stopped by itself with the right sentence rather than by a wrapper deciding it had waited long enough.

before   swept 60 cells: 0 measured here, 45 already on disk, 3 failed, 11 left alone because their engine was given up on
after    swept 60 cells: 2 measured here, 45 already on disk, 12 failed, 1 left alone after 4 tries each
then     swept 60 cells: 1 measured here, 51 already on disk, 4 failed, 4 left alone after 4 tries each
last     sweep measured nothing this time and holds 52 cells, so it has run out of cells it is willing to try

Seven cells that were unreachable before, and a terminal state that is a fact about the box rather than a place a sweep happened to stop.

Both yo only sweeps are now off the boxes, which is what this milestone was waiting for. The 8 core box finished its full 60 cells with no failures at all in three hours. Both directories have been reduced and combined. The eight cells the 32 core box cannot measure are the same shape its committed failures.json already refuses 40 cells of rugo and 10 of garnet under, so the publish sweep there will have the same holes and they belong in that directory's NOTES.md.

The 8 core box is the next thing here, and it is the one that produces all 154 charts. It has another tenant on it at the moment and a cycles measurement queued behind that, both of which are hours rather than days.