libaio: grow the staging array instead of aborting when it fills

[benjarvis]

Summary

evpl_libaio_read/write/flush staged each iocb into a fixed pending_iocbs array and killed the process once it filled:

evpl_libaio_abort_if(ctx->num_pending >= ctx->max_pending, "too many pending iocbs");

That treats a full submission queue as a caller bug, but it is ordinary backpressure. This PR grows the array instead, and adds a regression test that reproduces the abort deterministically.

Why the array fills

evpl_defer() only arms the flush — the callback does not run until the current callback chain returns (evpl.c, deferrals are drained at the top of the loop). So a caller that issues a batch inside one event-loop iteration stages all of it before a single io_submit() happens. Nothing drains in between.

The array is sized to libaio_max_pending (256 by default), which is the ring depth passed to io_setup() — not a bound on what a caller may hand us in one pass. Callers exceed it legitimately. In chimera's diskfs there are at least two independent paths that can:

• The tail-push thread caps home writes per device (DISKFS_PUSH_DEV_WATERMARK, 64), so its in-flight budget is 64 × device count — 640 across the ten devices its test harness configures.
• The commit thread's gate is checked once per batch rather than per write: diskfs_iq_process_batch() returns early only if redo_inflight >= DISKFS_COMMIT_WATERMARK (256) on entry, then admits a batch whose records each issue several chunk writes. Overshoot past the watermark is expected there by design — the adjacent retire ring is sized 1024 /* >= DISKFS_COMMIT_WATERMARK */.

Either way the caller's ceiling is above 256 while the staging array is exactly 256, and the process dies on the 257th iocb staged in a single pass.

This is the intermittent "too many pending iocbs" abort in chimera CI (chimera/posix/diskfs_evict_diskfs_aio, ~1–5 s into a ~60 s test). It is libaio-only because the io_uring backend's ring defaults to 8192 — 32× deeper — so the same burst fits. The io_uring backend has the same shape of hard failure at its own limit (evpl_io_uring_abort_if(!sqe, ...)); only the depth differs.

The change

• max_pending keeps its real meaning — the io_setup() nr_events ring depth. A new pending_capacity tracks allocated staging slots and doubles on demand. evpl_realloc() already fatals on OOM like the rest of the allocator API, so there is no new error path.
• The three copies of the stage-and-defer sequence collapse into one evpl_libaio_enqueue() helper.
• Draining needed no new machinery: a short or -EAGAIN io_submit() already keeps its unsubmitted tail queued, and evpl_libaio_complete() re-arms the flush after any completion batch, so the surplus goes out as ring slots free up.

Growth is bounded in practice by whatever in-flight cap the caller enforces, and the array holds one pointer per staged iocb — 640 entries is ~5 KB.

Verification

ubuntu24 / aarch64, Release, libaio via /usr/lib/aarch64-linux-gnu/libaio.so.

New test libevpl/libaio/burst sets the ring to 32 and stages 512 writes in one event-loop iteration, forcing submission to make partial progress and drain through completions:

	result
before this change	"too many pending iocbs" at libaio_block.c:133, SIGABRT (rc=134)
after this change	passes, all 512 completions delivered (5/5 runs)

libevpl/libaio/basic passes in both builds, so the abort is specific to the burst shape.

Full suite on a clean build of this branch: 162/162 pass.

Note on the chimera-side reproduction

The end-to-end diskfs_evict_diskfs_aio failure did not reproduce on my box, and it is worth saying so rather than claiming more than was shown. With the staging high-water mark instrumented, the observed peaks were:

condition	max staged
single run, devices on tmpfs	40
4 concurrent runs, devices on tmpfs	59
4 concurrent runs, devices on disk	41

All far below 256, and the test passes every time. Burst depth is a function of how long writes stay outstanding, and this box completes them too quickly for in-flight to accumulate toward either caller ceiling — consistent with the CI failure being Release-heavy (10 of 11 tracker events), since it takes an optimized build plus real device latency to hold 257+ writes outstanding. Reproducing it faithfully likely needs CI's degree of oversubscription, where the issuing thread is descheduled while other threads keep feeding its queue.

So what is demonstrated here is the mechanism and the fix, on the exact abort site, message and line seen in CI. The chimera-side attribution above is read off the caller's own constants and corroborated by the CI signature being libaio-only and never io_uring — it is not something I reproduced end to end, and the two candidate paths are not distinguished by the evidence I have.

Separately, chimera should probably not offer a backend more concurrent I/O than its submission path is configured for — a global in-flight ceiling derived from a queryable backend depth rather than constants maintained in parallel on both sides. That belongs in a chimera PR, not here; this change makes the backend correct regardless of what the caller offers.