Throughput, latency and CPU cycles for eight cache servers, measured with memtier_benchmark and charted from committed data.
This is a Rust port of tidwall/cache-benchmarks, which is the original and which deserves the credit for the methodology. The port exists for three reasons. It adds yo and rugo as a seventh and an eighth subject. It runs on hardware that is not a 32 core AWS instance, which the original assumes throughout. And it draws the charts in Rust instead of shelling out to Python and matplotlib, so the chart layer can be tested and so the same input produces the same PNG on every machine.
Every stage is written. doctor, run, sweep, mem, choose, combine, spread, recheck, chart, docs, archive and verify all work: run measures one cell and writes one file, sweep is the loop that measures the other ten thousand and keeps a record of what it did, and everything downstream of those files has been working since the chart milestone. What is missing is the hardware gate that says all eight servers come up and go away again on a real Linux box, and a reference sweep. The milestones say what each stage has to land and what it is gated on.
The first results are in results/wsl32coarse, which is a draft sweep rather than the reference matrix: three thread counts instead of six and five runs a cell instead of thirty one, on a host with no hardware PMU, so there are 146 charts and none of them are cycles. It is published because a draft is what finds the layout bugs and the provenance holes that a week long sweep should not find. Read its own README before reading its charts, and read its NOTES.md before quoting anything at all. Eighteen of its ninety six cells have no bar in them, because three of the eight engines did not serve all 256 connections alike and a run whose load generator threads finish that far apart is refused rather than charted. Two of the engines lose their whole multithreaded half.
The second is in results/epyc8coarse, an 8 core host, the same draft shape and the first of the two with a hardware PMU. It loses only fourteen of its 480 runs to that check rather than eighty three, because it offers 64 connections where the other offers 256 and a split that is badly uneven at 256 is only slightly uneven at 64, so it measures the multithreaded half the other host could not. What it gives up is quiet: ninety one of its ninety five plotted cells vary by more than a twentieth from run to run, against eighteen of seventy eight on the 32 core box. Its NOTES.md says what can be read off a directory like that, which is ratios inside one engine and not rates.
Every results directory comes with the raw output.json next to it, so anyone can redraw every chart without trusting us. The individual run files and the load generator's own output for every pass of every run are too many to put in git history, so they go on a release instead: the archives for wsl32coarse and epyc8coarse hold the runs the charts come from, the runs a check refused, and every log, and cache-bench archive --check reads one back against the manifest inside it. That is what makes a check added after the measuring is done worth adding, since recheck can judge an old sweep from those logs without measuring anything again.
Eight cache servers, all with persistence off, all over a local unix socket, so no network stack is in the measurement.
| Server | Thread flag |
|---|---|
| memcached | -t |
| Redis | --io-threads |
| Valkey | --io-threads |
| Dragonfly | --proactor_threads |
| Garnet | --miniothreads, --maxiothreads, --minthreads, --maxthreads |
| Pogocache | -t |
| yo | --threads |
| rugo | --threads |
The x axis of every chart is that thread count, swept from 1 to 16. Each point is the trimmed median of 31 runs. Each run is 100,000 SET operations and 100,000 GET operations per connection across 256 connections, with a warmup pass before the measured one, at pipeline depths of 1, 10, 25 and 50, with values of 1 to 1024 bytes.
Throughput comes out in Kops/sec, latency in microseconds at MIN, AVG, P50, P90, P99, P99.9, P99.99 and MAX, and CPU cycles in cycles per operation from perf. That is 154 charts, in linear and logarithmic scale.
Separately, and not part of any run. cache-bench mem starts each server, gives it a known number of distinct keys, lets it settle and reads the largest resident set it ever had. The key count is known rather than estimated: the filling pass writes one operation per key over a range the clients divide evenly, so nothing is written twice and nothing asks the server for a count it defines its own way.
It reports two numbers because there are two claims. Total bytes per entry is the peak divided by the keys in it, which is what a machine has to have. Overhead bytes per entry is what is left after the keys and the values themselves, which is what a design controls. At a hundred-odd bytes of payload per key an index that got twice as small halves the second and moves the first by a few percent, so quoting one of them is picking the flattering number.
It reports and does not judge. Garnet sizes its index at startup and Dragonfly preallocates per proactor, so the baseline each server held before a single key went in is recorded beside the peak rather than subtracted from it. Linux only, because there is no portable high water mark and no number here comes from a machine without /proc.
No expiry, no eviction, no mixed command set, no large values, no network, no replication, no persistence, no multi key operations. It is a hot path measurement of two commands. Numbers from it should not be used to say one engine is faster than another in general, and the results README will say so where the numbers appear.
crates/cb-core types, the JSON model, config, profiles
crates/cb-cache the eight cache adapters and process lifecycle
crates/cb-mem the memory measurement, its plan and its result file
crates/cb-memtier the memtier driver and its output parser
crates/cb-perf the perf driver, the PMU probe, CSV parsing
crates/cb-stats run selection and aggregation
crates/cb-chart the chart engine, axes and layout and its own rasterizer
crates/cb-docs generated LINEAR.md, LOGARITHMIC.md and README.md
crates/cache-bench the binary
tools/provision one script that turns a fresh Ubuntu box into one that can be swept
crates/cb-core/golden holds the original's own files, which is what every test in the workspace and cache-bench verify are checked against, and crates/cb-chart/assets/fonts holds the three faces the charts are drawn with. Both are in the crates that read them rather than beside them, because a published crate can only carry what is inside it.
cb-chart, cb-stats, cb-core and cb-docs build and test on any platform with no cache server installed, which is where the work that is hard to get right lives. cb-cache needs Linux.
cargo install cache-bench
That builds the binary from source and needs nothing else. A release also carries a tarball per platform on its releases page, with a SHA-256 next to each one, which is what to fetch on a box that has no Rust toolchain on it and no reason to grow one.
The eight cache servers and memtier_benchmark are not installed by any of this. They are named in config.jsonc by path, and cache-bench doctor says which ones it found. On Ubuntu, tools/provision/install.sh builds all nine of them at the versions pinned in tools/provision/versions.env and puts them where config.jsonc already looks, which is the half hour between a fresh box and a box that can be swept.
cache-bench doctor --profile wsl32 what this host can and cannot measure
cache-bench doctor --profile wsl32 --deep every server started once and stopped again
cache-bench run redis --threads 8 --pipeline 10 --perf no --run 1
cache-bench sweep --profile wsl32 the whole matrix, restartable
cache-bench mem --profile wsl32 what each engine costs to hold ten million keys
cache-bench recheck --dir results/wsl32 which runs today's checks would have refused
cache-bench choose --dir results/wsl32
cache-bench combine --dir results/wsl32
cache-bench spread --dir results/wsl32 which cells came out too noisy to quote
cache-bench chart --dir results/wsl32 --profile wsl32
cache-bench docs --dir results/wsl32
cache-bench archive --dir results/wsl32 --out wsl32.tar.gz
cache-bench archive --check wsl32.tar.gz --dir results/wsl32
cache-bench verify --against /path/to/cache-benchmarks/results
doctor refuses rather than warns. A machine with fewer cores than the profile names, a profile that sweeps cycles on a host with no counters, a working set that would not fit in memory, a load average that says somebody else is using the box: each of those is a sweep that produces numbers rather than an error, so each of them stops here instead. --write records what the machine is in host.json next to the results, and --deep starts each of the eight servers in turn and stops it again, which is the check no file can make. The load average is the one check that steps aside when --write points at a directory that already holds runs, because then this is describing a sweep that has happened rather than deciding whether to start one, and how busy the box is this minute is not a fact about when those runs were measured.
sweep takes days. It measures in the original's order, engine then threads then pipeline depth then counters then run number, so that all the runs of one cell happen together in time and a noisy hour shows up as one bad cell rather than as a tilt across the whole matrix. It is restartable: a cell whose file is there is skipped, a file that will not parse is measured again rather than trusted, and one results directory takes one sweep at a time. The other thing that stops a cell being measured is how many times it already has been, which failures.json counts. Four attempts and it is left alone, because a cell that has failed four times is one this machine cannot measure and the fifth attempt costs a full run to find that out again. --retry-failed throws those counts away, which is what to pass on a machine that has changed. --dry-run prints the cells it would measure and touches nothing.
Before every cell it checks whether anything else is on the machine, because doctor asks that once and a sweep runs for days. It samples the kernel's CPU counters over a window that starts after the last run's teardown, which is what the load average cannot do: between two runs a load average is mostly the run that just ended decaying out of it, so an idle box reads as busy and no threshold tells the two apart. Something else using more than half a core and the sweep waits for it rather than measuring a race, looking again every minute. Still there an hour later and the sweep stops, because that is another job rather than a blip and nothing is lost by stopping: the cells on disk stay and a sweep started again picks up from them.
A sweep keeps a record of itself next to the numbers. logs/sweep.jsonl is one line per attempt with the load average and the cores somebody else was using before it, which is what answers the question somebody asks a week later when one cell in one chart looks wrong. failures.json names every cell that was attempted and produced no file, with the reason verbatim and a count of how many times it has been tried, because the alternative to naming a missing cell is a chart that draws a zero and a zero is a claim while an absence is not. An engine that fails three times in a row without ever having produced a cell on this machine is put down for the rest of the session and named in that file, since the other six are still worth measuring and this is day three of eight. It has to be an engine that has never run here, because the runs of one cell sit next to each other in the order the matrix runs in, so a shape a box cannot measure is five consecutive failures and that rule on its own would put a working engine down and take the rest of its matrix with it. An engine with a cell on the disk has proved it runs, so the sweep walks past the bad shape, pays one run for each of its cells and leaves them alone on the next sweep once they have had their four tries. That is why a sweep of one engine on a busy box reaches the end instead of circling.
spread is the check to run before quoting anything. Every chosen file carries the coefficient of variation over the runs that went into its cell, and a cell that varies by half run over run is a picture of which run won rather than of how fast the engine is. It reports per engine how bad the worst cell was, how many were over the line and which thread counts those were at, so an engine that was steady sits in the table next to one that was not. --check fails instead of only saying so, and --over moves the line, which defaults to a coefficient of variation of 0.05. It throws nothing away and re-runs nothing, because what to do about a noisy cell is a decision with a machine and a deadline in it.
recheck is what to run when a check is added after the measuring is done. Every check the parser makes is a function of the bytes memtier wrote, and those bytes are kept: all three passes of every run sit in logs/ next to the run file they produced. So a sweep measured last month can be judged by this month's checks without measuring anything again, which is the difference between an hour and a week on a box that is busy. It reads each run's logs back through the parser a live run goes through, and with --apply a run whose logs no longer pass is moved into runs/refused/ and recorded in failures.json exactly as it would have been on the day. The runs above it are renumbered rather than orphaned, because a cell stops at its first missing number and leaving a hole would throw away measurements that passed. What it cannot do is make a cell whole again: a sweep that met the check on the day would have retried the cell, and a cell that comes out of here with two runs where the profile asked for five is under sampled and says so. choose will not reduce a cell of fewer than three runs, since over two runs the median is one of them and the best and worst are both of them, so four files carrying two numbers show a reader a spread that is the entire sample.
archive is how the evidence leaves the machine that produced it. What a results directory commits is the finished article, which is output.json, host.json, the charts and the generated markdown, and underneath that sit the run files and the load generator and server logs they were derived from. A reference sweep is 23808 of the first and about five times that many of the second, and tens of thousands of small files in git is a tax on every clone forever, so they go on the release instead. The archive is a gzipped tar with a MANIFEST in it holding the SHA-256 and the byte count of every member, --check recomputes all of them, and --check with --dir also asks whether every chosen file the committed output.json was built from is present, which is what ties the asset to the charts. The runs recheck refused go in it too, under runs/refused and counted separately from the runs, because a run that was refused is the evidence for why it was refused and the argument for where a threshold sits is an argument about what it kept and what it did not. The draft sweep in results/wsl32coarse comes to 4708 files and 11 MB, and checking it takes a tenth of a second.
verify is the claim this port makes about itself. With no arguments it checks the golden files committed here and runs anywhere in under a second. Pointed at a checkout of the original it reads all 20160 of its committed run files, reproduces all 2304 of its chosen files and its whole published output.json byte for byte, rebuilds all 154 of its charts down to the last bar, and then prints how far the corrected statistics sit from the original's.
The original was measured on a 32 core ARM64 AWS c8g.8xlarge with the cache pinned to 16 cores and the load generator pinned to the other 16. Profiles let the same harness run on a box that is not that, and every result file and every chart records which profile produced it. Mixing two profiles into one chart set without saying so is the failure this is built to prevent.
A profile can also say that its numbers are not fit to publish. smoke is one: two thread counts, pipeline one and ten, three runs, on a four core box, which answers whether a change helped in minutes and answers nothing about how these engines compare. docs refuses to write a results README from a profile marked that way, since a README is where a measurement stops being a note to oneself. The sweep runs as often as is useful.
Between the two there is a profile that publishes but samples coarsely. A run is a warmup plus fifty million operations, so a full wsl32 matrix of eight engines at twelve thread counts, four pipeline depths and thirty-one runs is weeks of machine time, and a first throughput number that arrives in weeks is one that in practice does not arrive. wsl32coarse is the same box, the same working set and the same four depths at three thread counts and five runs, which is a day. It gives up the shape of the thread curve and the tightness of the spread and gives up nothing else, so its cells compare directly with wsl32's.
epyc8coarse is the same trim applied to epyc8, and it drops the cycles half of the matrix as well, since that half doubles the time and a cycles question about one engine is better answered with perf than with a sweep of eight.
Recorded in divergences.md, with the reasoning for each one. The ones worth knowing about before reading any chart:
The statistics were rewritten. The original picks its median at the wrong index, sorts the SET results with a comparator that reads a different slice, never sorts the perf results at all before indexing into them as if it had, and carries the run count in a mutated global so that three of its four aggregates see fewer than 31 runs. All four are fixed, and --compat=upstream reproduces them exactly so the original's published output can still be regenerated byte for byte.
The charts use Jost and DejaVu Sans, which are open fonts embedded in the binary, rather than Futura and Verdana, which are not and which only resolve on macOS. Everything else about the chart layout is the original's.
A chart here is a function of its data and nothing else. The same numbers produce the same PNG, byte for byte, on Linux, macOS and Windows, and CI checks that on every push by drawing all 154 charts from a committed series on three operating systems and hashing every one of them. Every chart drawn from real measurements also carries a line along the bottom naming the profile and the machine, because two throughput charts from two machines are not comparable and the original's charts do not say which machine they came off.
The chart indexes are generated rather than maintained. The original writes LINEAR.md and LOGARITHMIC.md by hand, down to the anchor suffixes GitHub appends to repeated headings, and between them they link 120 of the 154 charts it publishes. Here both come out of the same table the charts are drawn from, so every chart that exists is linked, a chart that was not drawn is named rather than quietly dropped, and every image says what is on it.
The README in a results directory is generated too, and none of the facts in it are typed. The methodology bullets carry the profile's own numbers, the hardware table comes out of the host.json written before the sweep, and the version table is one row per engine read out of the results themselves. Two things go in it that the original has nowhere: the table of everything this port does differently, and what these numbers may and may not be used for in full, because a caveat that lives in a document nobody opens is a caveat that does not exist. CI regenerates every published document and fails if one of them was edited by hand.
The one thing that cannot be generated is what is true of one sweep and of no other: a build of an engine that turned out not to be steady, an engine that had to run over TCP on that box, a window where the machine was doing something else. A results directory may carry a NOTES.md saying that in somebody's own words, and the generated README links to it above the first chart when it is there, because a reader who has already read the bar has read it wrong.
Apache-2.0.
The three fonts in the binary are not. Jost is under the SIL Open Font License 1.1 and DejaVu Sans is under the Bitstream Vera Fonts Copyright, both permissive enough to embed, and both licence texts ship in the directory beside the font file they cover. See crates/cb-chart/assets/fonts/README.md for which release each file came from.