--web: linear per-worker latency ramp is a per-request refcount leak (chained subscript reads on nested arrays); --max-requests recycles miscounted by the crash-loop guard

[mirchaemanuel]

Note: rewritten on 2026-07-10 with a source-level root-cause analysis and a minimal repro (the original symptom-level report is preserved in the edit history). The original title said "super-linearly"; the measured growth — of both latency and live heap blocks — is linear. The --max-requests crash-loop-guard bug at the bottom is unchanged.

Summary

Under --web, per-request latency on a single worker grows linearly with the number of requests served, until the worker dies (Fatal error: heap memory exhausted) and the master respawns it, resetting the ramp. The cause is a per-request refcount leak: live heap blocks accumulate linearly across requests; latency tracks the live count because the allocator's first-fit free-list scan lengthens with it; the crash is the exhaustion of the fixed heap. One leak, three symptoms (ramp, flat RSS, periodic crash).

The dominant leak source is a codegen bug that is not --web-specific: a chained subscript read $a[$i][$j] on a nested array materializes the inner array $a[$i] as a temporary that is never released (minimal repro below). Under --web it compounds request after request; in a CLI run it is masked by process exit.

Confirmed still present on current main (e63d5d337, v0.26.0-158) — slightly worse there than in the v0.26.0 release.

Two bugs remain bundled in this issue: the leak/ramp itself, and the crash-loop guard counting scheduled --max-requests recycles as startup deaths (unchanged, at the bottom).

Environment

• elephc v0.26.0 (release, 892ff162) and main (e63d5d337), macOS arm64, --workers 1, --heap-size=67108864 (64 MB)
• Real app: ivory, a pastebin compiled with --web; the measured route is GET /p/{id} on a 4159-byte syntax-highlighted paste (the bench.sh scenario)
• php -S on the identical handler stays flat at ~6 ms/req for 200+ requests

Instrumentation

--gc-stats prints nothing under --web: the report is emitted only in the exit-based main epilogue, which a --web worker never reaches. For these measurements I locally patched emit_web_handler_epilogue to emit the existing gc-stats counters at the end of every request (5 lines, gated on --gc-stats; happy to share), then tracked live = _gc_allocs − _gc_frees request-over-request. Making --gc-stats functional under --web would be independently useful.

Measurements: latency tracks the live-block count

Real app, sequential requests, single worker, no recycling.

v0.26.0 (892ff162):

request #	latency	live blocks after request
1	3.5 ms	4,025
10	27.6 ms	38,756
50	295 ms	193,116
100	579 ms	386,066
~102–105	worker dies: Fatal error: heap memory exhausted	—
after respawn	~6 ms	4,025 (reset), ramp restarts

allocs/request is constant (29,199); frees/request falls short by a constant amount → net leak ≈ 3,859 blocks/request. Both the live count and the latency grow linearly; response content stays byte-correct on every request until the crash.

main (e63d5d337): same shape, worse — allocs/request 39,815, net leak ≈ 6,252 blocks/request, request 90 already at 1,233 ms, crash at ~request 92.

RSS stays flat during the ramp (as in the original report) because the heap region is pre-reserved: the leak lives inside the fixed heap, invisible to ps.

Root cause (dominant): chained subscript read on a nested array leaks the inner array

Minimal repro, measured with the per-request instrumentation above (numbers are leaked blocks per request, worker serving the same handler repeatedly):

function make(int $n): array {
    $a = [];
    for ($i = 0; $i < $n; $i++) { $a[] = ['kw', 'word' . $i]; }
    return $a;
}

$a = make(2000);
$n = count($a);
for ($t = 0; $t < $n; $t++) {
    $x = (string) $a[$t][1];          // chained subscript: leaks ~3 blocks/iteration
}

variant	net leak per request
build + push only (never read back)	0
chained read $a[$t][1] (even one read per iteration)	6,000 (= 3 blocks × 2,000 iterations)
hoisted read: $inner = $a[$t]; $x = $inner[1];	0

So neither the nested-array push nor the element access per se leaks — specifically the chained form does: evaluating $a[$t][1] materializes the inner array $a[$t] as a temporary and never releases it (~3 heap blocks per evaluation: inner hash + elements). Binding the same temporary to a local first gets the normal local decref and leaks nothing. The suspected fix area is the lowering of an index read whose base is itself an index-read temporary — the intermediate refcounted temp is missing its release.

This repro does not depend on --web (it's general codegen); --web is where it becomes fatal because the worker survives across requests.

Attribution in the real app

• Bypassing the syntax highlighter entirely: leak drops from ~6,252 to 32 blocks/request → the highlighter accounts for ~99%.
• The app's only chained nested subscript is the highlighter's token output loop (app.php lines 246–249: $tokens[$t][0] / $tokens[$t][1], ~2,000 tokens/request). Hoisting $tok = $tokens[$t] removes ~58% of the leak (6,252 → 2,655 blocks/request on main) and the worker survives 120+ requests without crashing.
• Honest residual: the remaining ~2,655/request is still inside the highlighter but resists isolation — six faithful synthetic probes of its constructs (nested pushes with substr elements, explode per token, htmlspecialchars + span concat in a loop, the full scanner with hoisted reads, element-extraction→explode, and the combination) all measure 0. It appears context-dependent on the real scan()/tokenize() control flow; not yet minimized.

Why the latency ramps (allocator mechanics)

__rt_web_reset (src/codegen/web.rs) releases named state between requests (function statics, static properties, globals, superglobals, concat offset); transient request allocations are reclaimed purely by refcounting — so any refcount leak accumulates in the worker forever. The allocator (src/codegen_support/runtime/arrays/heap_alloc.rs / heap_free.rs) is first-fit with linear scans (segregated small bins ≤64 B + an ordered free list), an O(1) free-at-tail bump trim, and coalescing only for >64 B blocks. Leaked blocks pin the heap mid-region: they defeat the tail trim and block coalescing, so the free lists grow with every request and each allocation's scan cost grows linearly with requests served — the measured ~5–6 ms added per request. The crash is simply the fixed heap running out. The ramp and the crash are two symptoms of the same leak.

Workarounds

• App-side (partial): hoist inner arrays out of chained subscripts ($tok = $tokens[$t]; $tok[0]). Removes whatever fraction of the leak comes through that construct (~58% in ivory).
• Deploy-side: worker recycling. --max-requests N truncates the ramp every N requests; measured on the release (average ms/req including retries):

configuration	requests	ms/req	retries
php -S (baseline)	200	6	0
--max-requests 5	40	23	7
--max-requests 10	80	22	7
--max-requests 50	200	154	3
no recycling	100	316	0

With short recycling the native returns to the same order of magnitude as php -S (requests actually served: ~10–13 ms). Two caveats, both upstream bugs rather than defects of the mitigation:

Caveat 1: recycling is not graceful

At the recycle boundary the next request can find the connection reset (curl exit 56) and needs a client-side retry — observed OOOOOx pattern with --max-requests 5 (roughly 1 lost request per 5 served, timing-dependent).

Caveat 2: the crash-loop guard counts scheduled recycles as startup deaths

crates/elephc-web/src/server.rs (FAST_DEATH = 1000 ms, MAX_FAST_DEATHS = 10) treats any worker that lived less than 1 second as "died on startup", including an intentional --max-requests recycle. Under closely-spaced requests (~7 ms apart) a worker with --max-requests 5 lives ~35 ms: after 10 consecutive recycles the master prints 10 workers died on startup (likely a bad --listen or a handler crashing every request); giving up and shuts down the entire server — verified (~50–60 requests served, then everything down; identical with --workers 2). The counter only resets when a worker lives ≥ 1 s, so scattered traffic never trips it, but any benchmark or sustained burst with a short --max-requests kills the server outright.

Expected

• A chained subscript read must release the inner-array temporary it materializes (fix in codegen; the repro above should show live flat request-over-request).
• Per-request latency on a stable worker should not grow with request count for identical work.
• A worker exiting due to a configured --max-requests recycle should not count toward MAX_FAST_DEATHS.
• Nice-to-have: make --gc-stats emit per-request (or at-recycle) counters under --web — it made this diagnosis possible.

References

• Real-world app + benchmark used throughout: mirchaemanuel/ivory — bench.sh (the ms/req tables above), app.php (the highlighter hot loop), docs/bug-notes.md §16 (field notes; §12 documents the related returned-object corruption in the same refcount family)
• Related: Object returned by one class's method reads corrupted fields (or crashes) after an intervening heavy call to another class #512 (cross-class returned-object corruption — same "missing release/boxing" family, different symptom)

[mirchaemanuel]

PR for the crash-loop-guard half (Caveat 2): #523 — the worker now exits with a dedicated recycle code (86) on a planned --max-requests recycle and the master exempts those deaths from the fast-death accounting (neither incrementing nor resetting the streak, so churn can't trip the guard and can't mask a real crash loop either). Verified: the 80-request repro that used to kill the server at ~50 now serves 80/80 with the server alive, and a genuine crash loop (unbindable --listen) still trips the guard as before.

The leak/ramp itself (the main body of this issue) is being addressed separately — a codegen fix for the chained-subscript release is in progress.

[mirchaemanuel]

PR for the leak itself: #524 — the chained subscript read $a[$i][$j] was leaking its materialized inner container (+1 reference never dropped, since the intermediate never touches a local slot's release machinery). The fix releases the owned intermediate right after the consuming read, at the IR level (all targets), with string results excluded (borrowed). It also makes --gc-stats functional under --web (per-request counters), which is the instrumentation behind all the numbers in this issue.

Measured on the real app from this report: net leak drops from ~6,252 to ~2,655 blocks/request (−58%), and the heap memory exhausted crash that hit at ~request 92 no longer occurs within 120 requests. The minimal chained-subscript repro goes from 6,000 leaked blocks/request to zero.

This issue should stay open after #523/#524 land: the remaining ~2.6k blocks/request in the app come from a different, still-unisolated source (it does not reproduce through chained subscripts in synthetic probes — details and candidate paths in the PR notes).