performance: same-branch writers starve under object-store latency

[ragnorc]

Problem

With several writers on one branch against an object store at realistic latency, one writer makes every commit and the others make none for the whole run.

Measured with the concurrent-writes benchmark on main at b14c22c5: 8 closed-loop writers on one branch, RustFS behind toxiproxy adding about 30 ms per round trip, 30 s window after a 5 s warmup, 3 runs:

Run	Commits in the window	Commits per writer
0	12	[0, 0, 0, 0, 0, 0, 0, 12]
1	12	[0, 0, 0, 0, 12, 0, 0, 0]
2	12	[0, 12, 0, 0, 0, 0, 0, 0]

The seven losing writers' operations finish only in the roughly 21 s drain after the window. The driver samples only operations acknowledged inside the window, so the reported service time (p50 2.46 s, p95 2.8 s) describes the winning writer alone, and the starved writers' waits never appear in it.

Local disk shows the same bias at smaller scale: within one run, per-writer commit counts range from 13 to 219.

Writers on different branches are not affected: with 8 writers on 8 branches, every writer makes the same number of commits (±2) in every run, locally and at +30 ms.

Likely mechanism (not yet confirmed in code)

A writer prepares outside the gates, then takes the branch gate and revalidates before it publishes. When one writer publishes, each queued writer revalidates, finds the head moved, and starts preparing again, after paying the round trips of the failed revalidation. The writer that just published starts preparing its next operation immediately after its own publication, so it is always slightly ahead, reaches the branch gate first, and wins again. At +30 ms a preparation takes about 2.4 s, so the losers never catch up within the window. Each run records 56 re-prepares and no exhausted re-prepare budgets (authority_conflicts is 0).

Scope

This is main's behavior. #783 (shared schema gate) spreads the same 12 commits over two or three writers, because reads no longer wait behind a publish, but it does not make same-branch admission fair.

Reproduce

# RustFS, plus toxiproxy adding 15 ms each way in front of it
docker run -d --name rustfs -p 9000:9000 -e RUSTFS_ACCESS_KEY=omnigraphci -e RUSTFS_SECRET_KEY=omnigraphci-secret rustfs/rustfs:1.0.0-beta.12 /data
docker run -d --name toxiproxy -p 8474:8474 -p 19000:19000 ghcr.io/shopify/toxiproxy:2.9.0
curl -s -X POST localhost:8474/proxies -d '{"name":"rustfs","listen":"0.0.0.0:19000","upstream":"host.docker.internal:9000"}'
for s in upstream downstream; do curl -s -X POST localhost:8474/proxies/rustfs/toxics -d "{\"type\":\"latency\",\"stream\":\"$s\",\"attributes\":{\"latency\":15}}"; done
aws --endpoint-url http://127.0.0.1:9000 s3api create-bucket --bucket omnigraph-bench

AWS_ACCESS_KEY_ID=omnigraphci AWS_SECRET_ACCESS_KEY=omnigraphci-secret AWS_REGION=us-east-1 \
AWS_ENDPOINT_URL=http://127.0.0.1:19000 AWS_ENDPOINT_URL_S3=http://127.0.0.1:19000 \
AWS_ALLOW_HTTP=true AWS_S3_FORCE_PATH_STYLE=true \
RUSTFLAGS= cargo bench --locked -p omnigraph-engine --bench scenarios -- \
  --scenario concurrent-writes --writers 8 --write-branches 1 --duration-secs 30 \
  --rows 64 --dims 8 --history-commits 64 --manifest-layout uncompacted --runs 3 \
  --target-uri s3://omnigraph-bench/starvation --out /tmp/starvation.jsonl

Per-writer counts are in each record's metrics.per_worker[].measured_ops.

Possible directions

• Fair admission for a writer that just lost revalidation, for example by re-preparing it under the branch gate it already holds, or by granting the branch gate in order of each operation's first attempt rather than its latest.
• Group commit at the branch publisher (step 3 of RFC 0067's throughput path), which batches same-branch arrivals instead of arbitrating between them.

Acceptance criteria

• Under the reproduction above, every writer on the branch commits within the window, and per-writer commit counts stay within a stated bound of each other.
• The benchmark reports each writer's longest wait, including operations that finish in the drain, so starvation is visible without reading per-writer counts.

[ragnorc]

Update after measuring on main with #783 (the shared schema gate):

• engine: shared schema gate and the write critical section (RFC 2026-09-18) #783 does not fix this. Same-branch writers still serialize on the branch gate, and the +30 ms one-branch cell reads 0.40 commits/s before and after.
• Rejecting stale attempts from memory, before revalidation's round trips, was prototyped and measured. At +30 ms it went from 0.40 to 0.53 commits/s; locally nothing changed; the starvation (one writer making nearly every commit) stayed. The shared gate RFC's decision log has the numbers.
• The ceiling is structural. Revalidation fails whenever the branch head moves, so every publication invalidates every other prepared write on the branch, and eight writers on one branch cannot beat one writer: 31.7 commits/s locally and 0.87 at +30 ms.

The proposed fix is #785, group commit. Writes queue in arrival order, and a refused write that re-prepares keeps its place, so a writer that just published cannot overtake the writers it made stale. Writes that don't conflict publish together in one compare-and-swap.