zu2: memory_pages bounds nothing while an async write is going, and the overshoot grows with the load

[tamnd]

Options::memory_pages says it is "pages of anonymous memory the log may hold". Under Durability::Async it is not a bound on anything. It is a target the flusher works towards afterwards, and while the writing is going the log holds whatever the writer has appended.

Measured with a probe that samples resident_pages on a thread while a single session writes 8 KiB values, memory_pages: 2, mutable_pages: 1, compact_below: 0 so nothing but eviction moves the head:

durability	records	written	bound	peak while writing
Async	10000	80 MiB	8 MiB	48 MiB, 12 pages
Async	40000	320 MiB	8 MiB	240 MiB, 60 pages
Durable	40000	320 MiB	8 MiB	12 MiB, 3 pages

Four times the load is five times the overshoot, so this is not a constant lag with a ceiling, it is the whole write staying in memory. At Durable the same load holds three pages, which is the bound plus the page being appended to and is exactly what the option promises.

The mechanism is not subtle once you look at it. Log::ensure_page allocates a page whenever an append reaches one, and it consults nothing:

let fresh = unsafe { alloc_zeroed(page_layout()) };
...
slot.store(fresh, Ordering::Release);
self.anonymous.fetch_add(1, Ordering::AcqRel);

The only thing that ever gives a page back is evict_behind, which runs from evict_settled, which is the flusher's. And evict_behind cannot take a page until it is durable:

if page_start(victim + 1) > self.flushed() {
    break;
}

which is right, a page that left memory before its bytes were on the device would be a reader preading a hole. So at Durable the writer is its own back pressure, because it does not return until the bytes are down, and eviction always has something to take. At Async nothing connects the writer to the flusher at all, and the log grows in memory at whatever rate the CPU can append.

The existing test knows this and works around it. the_page_bound_holds_after_the_writing_stops in crates/zu2/tests/end_to_end.rs is named for it and asserts after drop(s) and db.sync(), with a comment that what is being waited for is a thread noticing. Nothing asserts anything about during.

Why this matters beyond the doc being wrong. #767 is heading towards making a bound the default, since the anonymous total at defaults is the whole log and memory_pages is usize::MAX. A bound that a bulk load walks straight through would give a default that looks right in the options and does nothing for the load phase of every benchmark we run, which is the phase where the memory figure is worst.

The fix is the one the log already uses for its other ceiling. A writer that fills the log waits for a compaction pass, a_writer_that_fills_the_log_waits_for_a_pass in db.rs covers it. The same shape applies here: an append that would take the anonymous count past memory_pages wakes the flusher and waits for it, rather than allocating and moving on. That turns the option into what it says, at the cost of turning a write burst that outruns the device into a wait, which is what a bound means.

• Make the append path respect the bound rather than the flusher chase it
• A test that samples during the write and not after it, with the numbers above as the thing it catches
• Say in Options::memory_pages which durability modes it holds under, until it holds under both
• Re-measure, because a writer that waits for the flusher is a throughput change and it has to be on a host

Numbers above are from the laptop, and they are a behaviour observation rather than a throughput one: what is being claimed is that the peak scales with the load, which it does by a factor of five across a factor of four. Nothing published comes from this.

[tamnd]

Fixed in 54c9359.

One correction to what I wrote above: this was not an oversight, it was a stated position. Log::evict_settled's own doc says it, and has since #636:

A page can only leave memory once its bytes are durable, so a burst of async appends outruns the flusher and opened_page stops at the first page that is not flushed yet. That is fine while the burst is going on.

The numbers are the argument against that last sentence. It is fine if the overshoot has a ceiling. It does not: four times the load gave five times the overshoot, so what "while the burst is going on" costs is the burst, and a load phase is one long burst.

What went in. Log::throttle, called from allocate before the claim: an append that finds the anonymous count over the bound wakes the flusher, takes whatever is already durable with evict_behind, and then waits on the synced condition variable that a landed device write signals. Same load, same options, peak goes from 60 pages to 3.

Where it is called from matters more than what it does. It runs before slot.appending_at, so the writer has no frontier published while it waits. Anywhere below that and a writer waiting for a flush would be waiting for the flusher to wait for it, since sync_range calls wait_for_writers on everything below its target. That rules out putting this in opened_page, which is where eviction already runs and which was the obvious place to reach for.

Two other things it does not do. It never waits without a ceiling: twenty turns of half a millisecond each, then the writer takes the memory anyway. A database that holds too much for a moment is worse; one whose writers cannot be woken is not a database. And it does not throttle the engine's own sessions, since compaction copies a record to the tail before it can reclaim the page the record was in, so holding a pass because memory is tight holds the one thing that makes memory less tight.

• Make the append path respect the bound rather than the flusher chase it
• A test that samples during the write and not after it. the_page_bound_holds_while_the_writing_is_going in crates/zu2/tests/end_to_end.rs, with a ceiling of eight pages against a load of eighty, which is set where it is because the writer's patience does run out and the claim being tested is that the overshoot does not scale with the load
• Say which durability modes it holds under, which is now both, so the doc on Options::memory_pages needed nothing after all
• Re-measure on a host

The cost, and it is the point rather than a regression: the same load went from 668 ms to 1.25 s on the laptop, an 8 MiB bound forcing a device write every 8 MiB. Laptop, so that is a shape and not a number. The four arm A/B queued on server3 for #757 is the run that says what it is worth, and it now measures this too.

[tamnd]

Followed the counter through to the process, because everything above is resident_pages, and resident_pages counts slots rather than bytes. A page that eviction has taken out of its slot is not counted from the moment of the swap, but its memory does not go back until the epoch passes, so in principle the count could fall while the process holds the same memory.

It does not. Same test, same options, run alone under /usr/bin/time -l on the laptop:

	max rss	peak footprint
with the throttle	70.8 MiB	22.1 MiB
without it	133.9 MiB	116.3 MiB

Peak footprint is the number to read here, since max rss on macOS carries the binary and the harness with it. 116.3 MiB to 22.1 is 5.3x, against 320 MiB of writes at an 8 MiB bound, and it says the retirement queue is not quietly holding what the counter says has gone. That is what one would expect during a write burst, since every session operation moves the epoch on and the deferred frees drain behind it, and it is worth having measured rather than assumed: a long running scan pins an epoch and would let the queue build, which is a property of the design that predates this and is not what this changes.

The run without the throttle also fails the test, which is the point of the test:

a database with a bound of 2 pages held 23 while writing 320 MiB