Overlap log appends with the previous sync: the sync is 92 percent of a commit

[martian56]

Profiled on an SSD, eight page batches, comparing write_pages_with durable true against durable false:

• commit with sync: 2.982 ms
• same commit without the sync: 0.241 ms
• the sync alone: 2.741 ms, about 92 percent

That explains why every attempt to shave per group CPU or syscalls failed to move throughput, including coalescing the three writes per record (#63) and the various pacing and locking experiments on #46. There is almost nothing else in the commit to win back.

Also ruled out: the four MiB zero fill ahead was suspected of leaving a dirty backlog that every later sync pays for. Sweeping it says otherwise, smaller is worse. 4 MiB gives 2.86 ms, 256 KiB gives 5.98 ms, 64 KiB gives 3.26 ms. Leave it alone.

With the sync fixed and dominant, write throughput is group size divided by sync time, so the only remaining lever is fitting more commits behind each sync. Our commit worker is strictly serial: it collects a batch, appends its records, syncs, replies, and only then starts collecting again. Nothing is appended to the log while a sync is in flight, so a group can never be larger than the number of clients that happened to be waiting when it started.

Postgres does not have that limit. Its backends write their own log records while another backend is flushing, and one flush covers everyone who arrived during it, so its effective group grows with flush latency instead of being capped by it.

The change is to let appending continue during a sync: a writer that appends records and tracks a durable LSN, with commits waiting for a sync at or past their own LSN rather than owning the sync themselves. That is a real restructuring of group_commit.rs and should be measured against the 1cpu profile first, which is only 14 percent behind.

[martian56]

Implemented and rejected. The premise was that Postgres lets backends append while another flushes, so its group grows with flush latency while ours is capped by it.

Built it: a dedicated sync thread taking a handle on the active segment, the commit worker appending and handing off without waiting, commits acked only once their segment is synced, and checkpoints draining outstanding syncs before rotating. Whole workspace green, fifteen suites.

It does not help. Eight writers, five seconds: 510 ops/s against 558 for the serial version, and the average group stayed at 7.98 requests, exactly what it was before.

The reason is that the benchmark clients are synchronous. Each waits for its own reply before sending again, so while a sync is in flight every client is blocked on that sync and there is nobody left to arrive and enlarge the next group. In flight requests are bounded by the client count no matter how the sync is scheduled. Postgres is under the same constraint at the same concurrency, so this was never the difference between us.

Reverted. What it does rule in: since group size and sync count are identical for both engines, the whole remaining gap has to be the per group work outside the sync, which measured 0.241 ms here against a 0.25 ms per group deficit to Postgres. Those matching to two digits is worth taking seriously. The next measurement should break that 0.241 ms down rather than assume it, because it is apparently the entire difference.

[martian56]

Third syscall reduction experiment, rejected like the other two.

Framed a whole commit batch into one buffer and issued a single positional write, replacing twenty seven writes for an eight page group. Locally the non-sync part of a commit fell from 0.100 ms to 0.034 ms, a 66 percent cut, exactly as intended.

The matrix disagreed: arm64 minus 26.2 percent, 2cpu minus 15.7 percent, Windows minus 12.9 percent, against only plus 4.9 and plus 4.1 percent on the other two. Reverted.

That result matters more than the change. The non-sync work per group measured 0.241 ms and our per group deficit to Postgres measured about 0.25 ms, which looked like the answer. Cutting that work by two thirds produced no gain at all, so the two numbers matching was a coincidence. The gap is not the work around the sync.

Which leaves the sync itself being more expensive for us than for Postgres on the same device, at the same group size, with both doing one flush per group. The next hypothesis worth testing is block alignment: Postgres writes its log in aligned 8 KiB blocks, while our records are variable length and unaligned, so a record straddling a block boundary forces the kernel into a read modify write before the flush can complete. Padding each batch out to a block boundary is a contained change and would test it directly.