server: RSS exceeds --memory-budget-bytes under concurrent Parquet SQL and is OOM-killed

Found in the #2055 M1 run (ADR-2040 D7, ClickBench Parquet lane), main at 01d9760, on a dedicated c6a.4xlarge (16 vCPU, 30 GB RAM). Nothing else ran on the box except RustFS on loopback.

What happens. Under the D7 concurrency phase (10 connections cycling the 43 ClickBench statements over a Parquet table), ravel-server grows past its own memory budget until the kernel OOM-kills it. Reproduced three times out of three, on both table layouts:

Run	Killed at	Anon RSS at the kill
arm A (100 files), first	t ≈ 210 s	30.0 GB
arm A (100 files), diagnostic rerun	t ≈ 220 s	29.96 GB
arm B (single file)	t ≈ 300 s	28 GB sampled peak

The budget was pinned with --memory-budget-bytes 27000000000. The server's startup stamps:

Setting	Value
`memory_overhead_reserve_bytes`	2,147,483,648
`memory_hard_caps_bytes`	12,150,000,000 (`cache_max_bytes` 10.8 GB + `catalog_cache_max_bytes` 1.35 GB)
`memory_remainder_bytes`	14,850,000,000
`sql_max_query_bytes`	13,365,000,000
`sql_tenant_max_bytes`	13,365,000,000

The same statements run one at a time complete on both layouts with no error: arm A in 100.3 s cold and 35.1 s hot, arm B in 102.8 s cold and 39.0 s hot.

Attribution. The diagnostic rerun scraped /metrics every 9 s. At t=184 s, the last snapshot before the kill:

Gauge	Value
`ravel_process_allocator_bytes{allocator="jemalloc",stat="resident"}`	30.05 GB
`ravel_memory_reserved_bytes{component="sql"}`	12.41 GB (cap 13.37 GB)
`ravel_cache_resident_bytes{cache="fetch"}`	10.58 GB (cap 10.8 GB)
`ravel_memory_reserved_bytes{component="fetch"}`	0.15 GB
`ravel_cache_resident_bytes{cache="catalog"}`	0

Every accounted component held its cap. One statement was refused cleanly at the SQL pool's limit (query memory budget exhausted: tenant memory budget exhausted). But the accounted total is 23.14 GB against 30.05 GB jemalloc-resident: about 6.9 GB that no budget sees. That untracked share moves with concurrent SQL work: about 4 GB at t=128 s, about 8.5 GB at t=147 s, about 6.9 GB at t=184 s. The budget is what the process is meant to stay under, so the process exceeds it by 3 GB or more, and on this box that is the difference between running and being killed.

Candidates, not yet checked:

memory a query allocates and never charges to its SQL reservation: Parquet pages decoded in the pinned reader before batches are reserved, the HTTP response's JSON encoding buffers, per-connection result buffering;
jemalloc keeping dirty pages resident after they are freed. The exporter reports only stat="resident"; allocated and active would separate live heap from retention.

To do:

Add jemalloc allocated and active to ravel_process_allocator_bytes, so live heap and retention are told apart under load.
Take a heap profile under the same 10-connection load (the heap-profiling feature) and name the unaccounted allocation sites.
Charge them to the SQL reservation, or bound them, so the process stays under --memory-budget-bytes under concurrency.
A test, or a bench assertion, that RSS under the concurrency phase stays at or below the budget.

The raw samples (mem.tsv, gauges.txt) and the four reports are attached to the #2055 results comment.

Refs: #2055, #2040

Correction to to-do item 1 above: ravel_process_allocator_bytes already exports stat="allocated" and stat="active" beside stat="resident", through render_allocator_family in services/ravel-server/src/metrics.rs. My diagnostic sampler kept only lines matching resident, so it dropped them; the data was there and I did not record it.

So item 1 is a measurement step, not a code change: the next run under the same 10-connection load records all three series. allocated far below resident points at allocator retention; allocated close to resident points at live heap that the budget does not charge. Items 2 to 4 (heap profile, charge or bound what it names, assert RSS against the budget) stand.

Pre-registration: scaled reproduction on a fleet executor

Where it runs. On an amd64 fleet executor (16 vCPU, 30 GB RAM), not a ClickBench box. The question is a memory-attribution one, so the executor's timing noise does not matter.

Setup.

ravel-server is built --release --features sql,heap-profiling from main at 2f7728f43 or later.
The store is RustFS or MinIO on loopback, as in M1.
--memory-budget-bytes 8000000000 (8 GB).
The ClickBench hits Parquet table: the first 20 of the 100 partitioned files (arm A layout), about 2.8 GB.
Load: the D7 concurrency phase, 10 connections cycling the 43 statements for 600 s, through clickbench_parquet_bench. Since bench: the concurrency phase spins on an unreachable engine instead of recording that it died #2632 the phase ends at the first connection-level failure, so an OOM shows up with a timestamp.

Sampling. Every 5 s, from /metrics:

all three ravel_process_allocator_bytes series: allocated, active, resident;
ravel_memory_reserved_bytes for every component;
ravel_cache_resident_bytes for every cache;
the process's anonymous RSS, from /proc/<pid>/status.

"Accounted" means the SQL and fetch reservations plus the fetch and catalog cache residency, the same sum as M1's 23.14 GB.

Heap profile. _RJEM_MALLOC_CONF=prof:true,lg_prof_sample:19,lg_prof_interval:31, plus one prof.dump at the sample where resident minus accounted is largest. Summarised with jeprof --text on the dump.

Expected figures and bands.

Reproduction. If M1's shape holds at this scale, the peak of resident minus accounted is at least 1.2 GB (15% of the budget) during the concurrency phase. If it stays under 0.4 GB (5%) for the whole phase, the reproduction failed at this scale. That result is reported as a miss, and the next step is a full-size box, not a conclusion that the gap is absent.
Attribution. At that peak sample, let gap = resident minus accounted:
- Allocator retention if resident minus allocated is at least half the gap.
- Live heap the budget does not charge if allocated minus accounted is at least half the gap.
- If neither half reaches 50%, that is reported as split, with both figures.
Sites. The top allocation sites in the profile at that sample, with their bytes. A site counts as the cause only if, together with the others, it covers at least 60% of the live-heap share of the gap.

Shape check. The 43 statements run once, serially, before the phase starts. Each statement must either answer or be refused with a typed memory-budget error before the phase result counts. Any other error makes the run invalid, because the smaller budget can change which statements hit the pool limit compared with M1's five.

wip: issue 2633 jeprof summaries, site attribution and final samples

Result of the scaled reproduction (against the pre-registration above)

Fleet task 6cd1ef60 ran this on an amd64 executor (16 vCPU, 32 GB). Main was at 2f7728f43. Store: RustFS 1.0.0 on loopback. Budget 8 GB; 20 hits files, 2.71 GB; 360 s at 10 connections.

The full write-up, samples, heap dumps and attribution scripts are on branch task/6cd1ef60-6bf8-4058-830c-884fcdd4f760/result, under investigations/2633/. They are not merged to main.

Validity: valid. All 43 serial statements answered. The concurrency phase completed 2,626 queries. It had 46 errors, all typed 422 memory-budget refusals. The server was not OOM-killed; peak VmRSS was 9.01 GB on a 32 GB host.

Band	Pre-registered	Result
1. Reproduction	peak gap of at least 1.2 GB passes	4.22 GB at +357 s. Pass. Every concurrency sample was at least 1.2 GB; the median was 2.68 GB.
2. Attribution at the peak-gap sample	retention, live heap, or split, at the 50% mark	resident − allocated 2.54 GB (60%); allocated − accounted 1.68 GB (40%). Split.
3. Sites	named sites cover at least 60% of the live share	Pass at group level. About 1.27 GB is DataFusion query state above the SQL reservation; about 0.51 GB is hyper `BytesMut` read buffers on the S3 GET path.

What the series shows beyond the bands:

Retention does not decay at idle. 2.0 GB of resident − active stayed resident 457 s after the last query. That fits jemalloc purging dirty pages only on allocator activity, with no background thread. This is observed, not tested.
Live heap peaks elsewhere. The uncharged live share peaks at a different sample: 3.22 GB at +157 s. There it is mostly string-keyed group tables, ArrowBytesViewMap and parquet decode_page under ravel_parquet::boundary::BoundaryStream::poll_next.
About 0.5 GB of hyper buffers is uncharged even at idle. It is present before any query runs. The profile cannot separate two causes:
- connection read buffers retained by the pool;
- cache entries charged at value.len() (crates/ravel-cache/src/cache.rs:372) while pinning a larger hyper allocation.
The pre-registered "accounted" sum double-counts the handoff overlap. This is ravel_memory_handoff_overlap_bytes, which was 1.25 GB at +157 s. The live-heap figure there is therefore understated by up to that amount. A deviation, as noted.

Deviations, all listed in the README:

The phase ran 360 s, not 600 s. This was the pre-registered fallback.
The bench was patched locally to accept 20 files for arm A, and the source was reverted.
The reference was a dummy, so answers were not compared.
The dumps were interval dumps, not on-demand.

Fix plan

Two causes act independently, so two tasks run in parallel. They are in different crates.

Retention. Turn on jemalloc's background purge thread in ravel-server. Set a bounded dirty and muzzy decay, compiled in, so idle and between-query retention is returned without anyone setting _RJEM_MALLOC_CONF. Acceptance: in a test or startup stamp, the effective opt.background_thread and decay values read back from mallctl and appear in the startup stamps.
Cache entries pinning a larger allocation. The cache charges what an entry actually holds. Where a fetched body's backing allocation is larger than its length, the entry is stored in an exact-size copy. Acceptance: a test inserts a Bytes slice of a larger buffer, and asserts both the charge and that the larger buffer is released.

After both land, the same scaled run is repeated, pre-registered again. Expected bands for that run:

retention at the peak-gap sample under 1.0 GB;
the idle 0.4 GB allocated − cache floor gone.

The DataFusion query-state share (1.27 GB at 3.96 GB per query cap) gets a third task. Its form depends on what is left after 1 and 2: either a headroom on the SQL pool's share of the remainder, or charging what DataFusion under-reports. It is decided on the rerun's numbers, not now.

Pre-registration: rerun after #2652 and #2653

#2652 (jemalloc background purge thread) and #2653 (cache stores an exact-size copy) are merged. This rerun repeats the scaled reproduction exactly, on main at the merge of both:

8 GB budget;
20 hits files;
RustFS on loopback;
10 connections.

The only intended variable is the two fixes. The first run is the baseline: task 6cd1ef60, branch task/6cd1ef60-6bf8-4058-830c-884fcdd4f760/result.

Same method:

the same sampler and attribution scripts, taken from that branch;
the same "accounted" sum.

Also recorded:

the allocator_background_thread startup stamp, and the ravel_process_allocator_background_thread gauge in every sample;
ravel_memory_handoff_overlap_bytes beside each sample, so the double count noted last time is visible.

Expected figures and bands

#	Figure	Baseline	Pass	Miss
0	Validity: serial pass answered or typed budget-refused, with no other error	valid	as before	any other error makes the run invalid
1	Background thread	(off)	stamp `true` and gauge `1` in every sample	anything else invalidates the run; it is not a measurement of the fix
2	Retention at the peak-gap sample (`resident` − `allocated`)	2.54 GB	under 1.0 GB	1.5 GB or more means #2652 did not take effect as expected
3	Idle retention, 300 s or more after the last query	2.0 GB	under 0.3 GB	—
4	Idle `allocated` minus cache-resident, 300 s or more after the last query	0.40 GB	under 0.15 GB	0.3 GB or more means the hyper buffers are not cache-pinned slices, and candidate (a), pooled connection buffers, is next
5	Peak gap (`resident` − accounted) over the phase	4.22 GB	reported against the baseline, no band	—

For row 4, between 0.15 GB and 0.3 GB is reported as partial.

Row 5 has no band because it still contains the DataFusion query-state share (1.27 GB at the baseline peak), which neither fix targets. The third task, for that share, is sized from what this run leaves.

Rerun result, against the pre-registration above

Fleet task db9e7a3c measured on the baseline host (ravel-clickbench-ubuntu-2) at main 5378afd2a, which includes #2652 and #2653.

Afterwards, the jeprof/addr2line attribution step pushed the executor's cgroup to its 28.8 GB limit, and the executor was OOM-killed before the task could commit. The fleet then requeued the task to a 15 GB host, where the memory tripwire stopped it.

The measured data was intact on the first host, so I copied it off by hand. It is on branch task/db9e7a3c-73f0-4fd9-ac9b-8bcb2327aa8d/result, under investigations/2633-rerun/, with a README. Nothing was re-measured. The executor service on that host is running again.

Row	Figure	Baseline	Rerun	Band	Verdict
0	Serial pass	valid	43/43 answered	no other error	valid
1	Background thread	off	stamp `true`, gauge 1 in 186/186 rows	as stated	valid
2	Retention at peak-gap sample	2.54 GB	5.43 GB	pass < 1.0, miss ≥ 1.5	MISS
3	Idle retention (≥ 300 s after last query)	2.0 GB	0.084 GB	pass < 0.3	PASS
4	Idle `allocated` − fetch cache	0.40 GB	0.023 GB	pass < 0.15	PASS
5	Peak gap	4.22 GB	6.30 GB	reported	worse

Idle is fixed.

Retention drops from 1.89 GB to 0.085 GB within 64 s of the last query.
The idle hyper BytesMut read buffers are gone, so the 0.4 GB was cache-pinned slices, and fix(cache): store an owned copy when an entry pins a larger buffer #2653 removed it.

Under load it is not fixed.

At the peak-gap sample, 5.40 GB of the 6.30 GB gap is resident minus active: freed pages jemalloc has not returned yet.
Peak VmRSS is 10.44 GB (baseline 9.01 GB).
Median retention over the phase is 3.60 GB (baseline 1.60 GB).

The background thread is running. So this is decay timing, not purging that never happens: jemalloc returns freed pages over dirty_decay_ms (10 s by default), and 10 concurrent aggregations free several GB inside one window. That mechanism is inferred, not tested.

One run each cannot separate the fixes from run-to-run variation in which heavy statements overlap. The baseline's peak came at +357 s; this one came at +85 s. Row 2 says #2652 alone does not bound retention under load. It does not say #2652 made it worse.

Next: a decay experiment, pre-registered here

One fleet task on the same 30 GB host, with one binary built from current main (--features sql, no heap profiling, no jeprof). The same scaled workload runs four times, in the order A1, B1, A2, B2:

A: default decay. _RJEM_MALLOC_CONF unset.
B: _RJEM_MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0. The background thread is still enabled by the server, and the stamp must show source="server".

Figure	Pass	Miss
B's retention at its peak-gap sample, both runs	< 1.0 GB	≥ 1.5 GB
B's peak VmRSS vs A's peak VmRSS (mean of two each)	at least 2.0 GB lower	under 1.0 GB lower
B's concurrency qps vs A's (mean of two each)	within 10 %	20 % or more lower (decay cost too high)
Each A run's retention at its peak-gap sample	reported, to size the noise between runs of the same arm	—

If B passes, the server sets these decay values itself at startup and stamps them, the same way #2652 handles the background thread. The opt.* values are read-only through the safe API, so the change is either compiled into the build's jemalloc configuration or made through a writable arenas.* control. Which one is decided after the experiment.

The DataFusion query-state share stays the separate third task. At the max-uncharged sample, allocated minus accounted is 1.87 GB. Up to 1.27 GB of that is handoff-overlap double counting, so the third task is sized against a smaller real figure than the baseline suggested.

wip: #2633 decay arm B1

wip: #2633 decay arm A2

wip: #2633 decay arm B2

docs(investigations): measure jemalloc decay against #2633 retention

Decay experiment result, scored against the pre-registration above

Fleet task c3d2ad79 ran on the same 30 GB host. It used one server binary built at 5ad3f6a with --features sql, and ran four arms in the order A1, B1, A2, B2:

A: default decay;
B: _RJEM_MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0.

All four arms are valid. Each answered 43 of 43 serial statements, and every startup stamp read allocator_background_thread=true source="server". The write-up and data are on branch task/c3d2ad79-81ac-4554-8d4f-28a3be9c9ff0/result, under investigations/2633-decay/.

Row	Figure	A1	A2	B1	B2	Band	Verdict
1	B retention at its peak-gap sample			4.33 GB	4.65 GB	pass < 1.0, miss ≥ 1.5	MISS
2	Mean peak VmRSS, B vs A	9.89	10.19	8.83	8.82	pass ≥ 2.0 GB lower, miss < 1.0 GB lower	neither: 1.21 GB lower
3	Mean qps, B vs A	8.00	7.26	6.65	6.94	pass within 10 %, miss ≥ 20 % lower	neither: 10.9 % lower
4	A retention at peak-gap sample	6.54	6.11			reported	0.43 GB apart

A 1 s decay works on average and leaves the peak in place:

Measure	A (default decay)	B (1 s decay)
Median retention over the phase	3.58 / 3.80 GB	1.27 / 1.39 GB
Mean VmRSS over the phase	8.46 / 8.32 GB	6.24 / 6.51 GB
Phase samples with retention ≥ 1.5 GB	72 / 72	23 / 33

B's retention is spiky. In B1, five consecutive samples read 0.9, 2.3, 4.3, 2.1 and 0.9 GB. When several heavy aggregations free their memory inside one window, 4 to 5 GB is still resident at that moment, and the peak is what an OOM sees. The throughput cost cannot be separated from noise: the two A runs alone differ by 9.2 % in qps.

So decay tuning is not the bound #2633 needs. The budget accounts reservations, and the process's real footprint can run 4 to 6 GB past them for seconds at a time. Within those seconds, memory freed but not yet returned dominates. Under the default decay, the uncharged live heap is small at those moments (0.16 to 0.96 GB).

The fix shape is a decision for the owner, asked separately. The options are:

admission that reads the allocator's own resident figure;
a derated budget;
shipping the shorter decay as a partial measure.

One more finding, outside #2633's scope: a fetch-memory refusal reaches the client as HTTP 503 "upstream storage temporarily unavailable" (1 to 3 per run). It is a budget refusal and should arrive as a typed 422 like the others.

docs(adr-2633): restate the uncharged-heap basis and fold catalog 503s

Pre-registration: where DataFusion's reservation falls short of its allocations

The owner's decision is to charge DataFusion's uncharged state before setting the gate's mark. This measurement finds what is uncharged. It runs one statement at a time, with no concurrency, so the process-wide allocated figure belongs to that statement alone.

Setup

Host: the 30 GB host.
Data: the same 20-file hits table.
Budget: 8 GB.
Binary: ravel-server built --features sql at current main, plus a temporary instrumentation patch that is never merged. The SQL pool (TenantDelegatingPool) records each consumer's peak reservation, keyed by DataFusion's MemoryConsumer name, and dumps the table at the end of each statement.
Statements: q13, q14, q17, q18, q19, q31, q32, q33, q34 and q35, each run alone three times (hot). These are the high-refusal and string-keyed group-by statements from the earlier runs. q1 and q2 run as controls.
Sampling: process stats.allocated every 20 ms during each statement. Its peak above the pre-statement value is the statement's allocation peak.

Expected figures and bands

#	Figure	Expectation	Miss
1	Peak allocation above baseline minus peak SQL reservation, per statement	at least 0.5 GB for the string-keyed group-bys (q33, q34, q35, q18, q19)	under 0.2 GB on all of them: the gap is not per-statement state, and the concurrency runs' gap has another source
2	Same, controls q1 and q2	under 0.1 GB	0.3 GB or more: a fixed per-statement overhead nothing charges
3	Per-consumer peaks against the allocation peak	one or two consumers' reported peak is below their share of the profile (heap profile of one run of the worst statement, built with heap-profiling, used only for attribution)	—

Report

per statement: peak allocated delta, peak SQL reservation, the gap, and its run-to-run spread;
per consumer: name, peak reported, and the profile bytes its allocation sites hold;
the named operators to charge: under-reported size() such as hash-table capacity versus length, output batches held by downstream operators, repartition and coalesce buffers, or Parquet decode buffers outside any reservation.

The heap profile is taken only for the single worst statement. Its jeprof step runs with its parallelism capped, because the last uncapped addr2line fan-out OOM-killed the executor.

wip: #2633 charging matrix runs

wip: #2633 profile attribution and analysis scripts

wip: #2633 hold run R2

Result: concurrency rerun with the aggregate hold (PR #2685), scored against the pre-registration above

Fleet task 4f2f129b ran two runs, R1 and R2, on the 30 GB host at main 6b794db90, with the hold in place. Both runs are valid (43/43 serial statements answered). Write-up and data: branch task/4f2f129b-2a5f-4560-8e5b-70a9034a53f8/result, investigations/2633-hold/.

Row	R1	R2	Verdict
1. Per-sample maximum of `allocated − accounted + overlap`	2.48 GB (serial pass; 1.33 GB maximum in the concurrency phase)	1.82 GB (concurrency phase)	R1 between the bands, R2 pass
2. Median of the same over the phase	0.60 GB	0.54 GB	decay A: 0.82 / 0.73
3. qps, mean of the two	8.03		pass (+5.2 % against 7.63)
4. Typed 422 refusals	53	87	decay A: 37 / 42 typed. The extra are mostly "query pool full", the expected cost of keeping shrunk bytes charged.
5. Peak retention / peak VmRSS	6.63 / 9.58 GB	6.00 / 9.77 GB	inside the decay A range; the hold does not target retention

What this establishes:

Concurrency phase: the uncharged live heap stayed under 2.0 GB in both runs (1.33 and 1.82 GB), and its median fell by about a quarter. Decay A had runs at 3.43 and 1.91 GB. With two runs per arm, a single comparison cannot fully separate the hold's effect from noise, but every concurrency-phase sample with the hold is under the band.
R1's 2.48 GB: this is one sample from the serial pass, taken while the fetch reservation was 1.10 GB, all of it counted as handoff overlap. It is a single-statement figure, not concurrency.
Handoff overlap: in nearly every sample of every run, handoff_overlap equals fetch_reserved. So row 1 is in practice allocated − sql_reserved − cache resident.
What the gate still has to bound: the uncharged heap is no longer the driver of the remaining overshoot. Retention is: resident − allocated still peaks at 6.0 to 6.6 GB. That is exactly what ADR-2633's resident gate and forced purge are for.

Next: revise ADR-2633 (PR #2661) with G taken from this run.

Mark derivation with the worst sample, G = 2.48 GB: 8.00 + 0.54 − 0.06 − 2.48 − 0.27 = 5.73 GB = 0.716 B, so a 70 % default.
With the concurrency-phase maximum, G = 1.82 GB, it is 0.80 B.
The revision also corrects the earlier overlap sign error, and records that DataFusion 55's default aggregation path already reserves emitted batches (found while preparing fix: charge RepartitionExec coalescer buffers to the memory pool apache/datafusion#26167). The hold becomes removable on that upgrade.

docs(adr-2633): re-derive the gate mark after the aggregate hold

docs(adr-2633): pre-register throughput and split sampler CPU

fix(sql): drop the aggregate hold now DataFusion 55 reserves emitted …

Finding: the resident gate closes for the whole concurrency phase at a 27 GB budget on a 32 GB host

Seen on the fresh c6a.4xlarge (ravel-df55-ab, 16 vCPU, 32 GB) running the #2720 parquet A/B, arm A = main 154c947 (ADR-2633 tasks 1 and 2 included, DataFusion 54.1), stock flags plus --memory-budget-bytes 27000000000. Not a pre-registered run of the ADR's acceptance test; it is the first real workload the gate has seen.

Stamp: memory gate resolved enabled=true high_water_bytes=18900000000 high_water_percent=70 memory_budget_bytes=27000000000 interval_ms=100 wait_ms=2000 source="default". Resolved cache_max_bytes 10,800,000,000 (loopback 40 %), memory_remainder_bytes 14,850,000,000, sql_max_query_bytes = sql_tenant_max_bytes 13,365,000,000.

Serial pass: 40 of 43 statements pass (q33 422 query pool; q34, q35 503), hot sum 22.4 s, cold sum 87.6 s. Unaffected.
Concurrency phase (10 connections, 600 s): 1,105 completed, 8,123 errors, error ratio 0.880, qps 1.84. Of the 8,126 server-side refusals, 7,819 are segment fetch ... upstream storage temporarily unavailable (fetch-memory refusals, the 503 class task 4 moves to 422), 140 query memory pool is full and spill is disabled, 167 process memory budget exhausted.
Peak VmRSS 19.31 GB against the 18.9 GB mark, so the gate was closed for most of the phase; with a 10.8 GB cache and 13.4 GB SQL pools the mark can be reached by accounted memory alone, which the startup check (caps 10.8 GB < 18.9 GB) does not cover.

What this says about ADR-2633 section 2: the default derivation and "on the current code the gate is expected to purge in every run and to refuse in none" were argued for the 8 GB scaled run (budget = host). At a budget of 27 GB on a 32 GB host, 70 % of the budget (18.9 GB) sits below cache plus one SQL pool (24.2 GB), so a concurrent workload that fills the pool closes the gate and the purge cannot reopen it: the bytes are live, not retained. The mark has to leave room for the accounted ceilings (caps plus the SQL tenant pool), or the gate has to exempt accounted growth, or the startup check has to count the SQL pool as it counts the caps.

For the #2720 A/B the gate is a confound landed today on both arms, so the parquet and RLOG passes rerun with --disable-memory-gate on both arms, stamped. Evidence kept on the box under /root/ab/gated/passes/p1A/ (server-errors.txt, vmrss.tsv, bench-report.json) and will be attached to the task 6 ticket (#2734).

Second data point from the same box, same arm A binary, same 27 GB budget, gate OFF (--disable-memory-gate): the server was OOM-killed by the kernel 34.4 s into the concurrency phase (the bench reports the engine became unreachable 34.4 s into the phase; the shell reports Killed). So at this host shape the gate traded an OOM kill for an 88 % refusal rate, which is the ADR's intended direction but not a usable operating point: the budget itself is too large for the host once the SQL pool, the caches and jemalloc retention are summed. The A/B continues at a 20 GB budget with the gate off on both arms; the gate-on run at 20 GB is a separate task 6 measurement.

docs(adr-2633): amend the gate to close on host available memory

Task 2 follow-up #2806 merged as PR #2810 (main 1590f94): the gate now closes on host available memory below a floor (default max(512 MiB, 2 % of host) clamped to host/8; purge floor twice that clamped to host/4; purge at most once per 5 s; cgroup term is working-set headroom). The 2026-10-11 amendment in the ADR records the arithmetic, the RSS counter-case, the decision and the new acceptance for task 6 (#2734). Tasks 3 to 6 proceed against this gate; task 3 (#2731) still waits on #2774.

fix(sql): drop the aggregate hold now DataFusion 55 reserves emitted …

ADR-2633 task 4 landed: PR #2841, main 3c47dde. Fetch-memory and catalog-decode refusals answer 422 with the budget class on PromQL HTTP, SQL HTTP and Flight SQL; the DDL footer-read path stays at 503 under #2840. Task 3 (#2731) dispatches after #2774; task 5 (#2733, docs incl. the ADR-1170 amendment) can go now; task 6 (#2734) is the acceptance ravel-perf is running on #2842.

fix(sql): drop the aggregate hold now DataFusion 55 reserves emitted …

Navigation Menu

Repository navigation

server: RSS exceeds --memory-budget-bytes under concurrent Parquet SQL and is OOM-killed #2633

Pre-registration: scaled reproduction on a fleet executor

Result of the scaled reproduction (against the pre-registration above)

Fix plan

Pre-registration: rerun after #2652 and #2653

Rerun result, against the pre-registration above

Next: a decay experiment, pre-registered here

Decay experiment result, scored against the pre-registration above

Pre-registration: where DataFusion's reservation falls short of its allocations

9 remaining items

Result: concurrency rerun with the aggregate hold (PR #2685), scored against the pre-registration above

Finding: the resident gate closes for the whole concurrency phase at a 27 GB budget on a 32 GB host

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

server: RSS exceeds --memory-budget-bytes under concurrent Parquet SQL and is OOM-killed #2633

Description

Activity

pmoust commented on Oct 7, 2026

pmoust commented on Oct 7, 2026

Pre-registration: scaled reproduction on a fleet executor

pmoust commented on Oct 7, 2026

Result of the scaled reproduction (against the pre-registration above)

Fix plan

pmoust commented on Oct 8, 2026

Pre-registration: rerun after #2652 and #2653

pmoust commented on Oct 8, 2026

Rerun result, against the pre-registration above

Next: a decay experiment, pre-registered here

pmoust commented on Oct 8, 2026

Decay experiment result, scored against the pre-registration above

pmoust commented on Oct 8, 2026

Pre-registration: where DataFusion's reservation falls short of its allocations

9 remaining items

pmoust commented on Oct 10, 2026

Result: concurrency rerun with the aggregate hold (PR #2685), scored against the pre-registration above

pmoust commented on Oct 10, 2026

Finding: the resident gate closes for the whole concurrency phase at a 27 GB budget on a 32 GB host

pmoust commented on Oct 10, 2026

pmoust commented on Oct 11, 2026

pmoust commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions