Old-gen growth from large-object births never arms a full collection: RSS climbs unbounded on repeated wide-object parse

[proggeramlug]

Old-generation growth from large-object births does not arm a full collection

A workload that repeatedly allocates and drops a large pointer-bearing object grows old-gen without bound. Every minor promotes essentially the whole nursery, copies nothing and frees nothing, and the full collection that would reclaim it never arrives.

records_array_wide_1m:parse — a flat object of 50 000 integer keys, parsed in a loop:

old_in_use=6578096    survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=28272432   survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=59873112   survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=112274032  survival_permille=997  copied_objects=0  freed_bytes=0

Peak RSS 228 MiB against Node's 121 and Bun's 95.

The same signature on records_array_16k:scan before #10098's lazy-array work, with the trigger state visible:

old_in_use=48385432  old_threshold=50331648  old_pending=false
[gc-arena-rebaseline] step=134217728 -> 268435456 -> 536870912 -> 1073741824

An old-gen threshold of 48 MB exists and old_in_use reaches 48.3 MB with old_pending=false; meanwhile the arena trigger rebaselines itself upward exponentially. So the arm that would notice old-gen filling keeps stepping out of the way of the arm that is filling it.

Why this is one cause, not two

Each object is individually reasonable. arena_alloc_gc puts a pointer-bearing object over LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB) directly into non-moving old-gen, and the reasoning in its own comment is sound: for a pointer-bearing object the cost of tenuring "is not its own bytes, it is every object it can reach, held live through the remembered set by a container nothing refers to any more".

That reasoning assumes a full collection eventually runs. When it does not, the policy inverts: the objects chosen for old-gen because they are expensive to hold are exactly the ones nothing reclaims.

Two of these are now fixed by moving the allocation rather than the policy — #10098 made the lazy JSON array cluster nursery-resident and took records_array_16k:scan from 205 MiB to 51 MiB and records_array_1m:scan from 159 to 75, both now below Node and Bun. That works because a lazy cluster's cache is small enough to sit under the line. A 50 000-key object's field storage is ~400 KB and cannot.

What to look at

• Whether old-gen growth from large-object births feeds the full-collection trigger at all, or only growth observed through promotion.
• Whether the arena trigger's exponential rebaseline should be bounded while old_in_use is climbing — the two arms currently move in opposite directions.
• old_pending=false at old_in_use ≈ old_threshold is the specific observation to explain first.

Scope note

This is GC policy, which CLAUDE.md routes separately and warns about, so it wants its own change with the ratchet corpus run before and after rather than riding along with JSON work. Filing it with the evidence rather than attempting it inside a JSON performance pass.

Rows currently affected: wide_1m:parse RSS 228 vs 95 MiB (2.4x), and the residual records_array_8m:scan 187 vs 134 and records_array_20m:scan 256 vs 226, whose clusters are genuinely over the pointer-bearing line.

Refs #10098, #793.

[proggeramlug]

Root cause found: the RETAINING veto is fed by the survival its own pinning creates

A full collection reclaims all of it. PERRY_GEN_GC=0 on wide_1m:parse, same binary:

iterations	generational	full-sweep-only
8	56 MiB	54 MiB
64	222 MiB	55 MiB (flat)

So nothing is unreclaimable and no allocation needs to move. The generational regime simply never runs the collection that would free it. At 55 MiB Perry would be below Node (121) and Bun (95) on this row.

The veto

gc::policy::old_reclaim_pressure_due:

let crossed_absolute_threshold = old_in_use >= threshold
    && baseline < threshold
    && !GC_MAJOR_PACING_RETAINING.with(|c| c.get());

GC_MAJOR_PACING_RETAINING is armed from measured young-survival, which on this workload reads survival_permille=997 on every minor. #7937's reasoning for the exemption is that "a heap whose young generation is not dying is retaining live data, and a full mark-sweep cannot lower the number being watched."

That reasoning inverts here. The young generation is not dying because large old-gen containers pin it through the remembered set — every parse's 400 KB field storage is born tenured, and a minor never sweeps old-gen, so it holds its 50 000 key strings live. High survival is the symptom of the condition a full collection would clear, and the veto reads it as proof that one would be futile. The measurement above is the counter-example: the full sweep lowers exactly the number being watched, by 4x.

The proportional arm does still bound the exposure, but the RETAINING multiplier widens the band to 128 MB, and old_in_use reaches ~112 MB over 64 iterations — so the bound is real but sits above where this workload ends.

The width cliff confirms the container

RSS growth from 8 to 64 iterations, by object width — the step is exactly at LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB), i.e. 16 384 keys at 8 bytes each:

keys	keys array	growth
8 000	62 KiB	16 MiB
14 000	109 KiB	21 MiB
17 000	132 KiB	67 MiB
24 000	187 KiB	85 MiB
50 000	390 KiB	166 MiB

Under the line the storage is nursery-resident and a minor reclaims it. Over the line it is born tenured and only a full collection can, which is the one that never comes.

Suggested shape

Young-survival cannot distinguish "live data" from "dead data pinned by a born-tenured container", so it should not be the sole veto on the arm that would tell them apart. Two options, either of which is testable against this fixture:

1. Exempt old-gen growth that came from births rather than promotion. credit_promoted_bytes_to_old_baseline credits the baseline only from promotion, so a born-tenured object moves old_in_use without moving baseline — the two quantities the arm compares already disagree in exactly the informative way. Tracking born-tenured bytes since the last full collection and letting them bypass the RETAINING veto is a small change with a direct signal.
2. Make the veto self-correcting: apply it only while the previous full collection actually failed to reclaim. Here the first one would succeed and disarm it.

Either needs the gc-ratchet corpus run before and after, which is why this is filed rather than attempted inside a JSON pass.

Note this is the same root cause as the residual records_array_8m:scan (187 vs 134 MiB) and records_array_20m:scan (256 vs 226), whose clusters are also genuinely over the pointer-bearing line. #10098 fixed the two rows whose clusters fit under it by making them nursery-resident; that approach cannot reach these.

[proggeramlug]

Attempted the trigger fix. It does not work, and the measurements say why: the trigger is only half the chain.

Three attempts, each measured on wide_1m:parse (64 iterations, Node 121 MiB, Bun 96 MiB):

change	full collections	peak RSS
main	0	222 MiB
exempt births from the RETAINING veto	1	222 MiB
+ recurring births arm, shared predicate	2	284 MiB
+ births arm split off the shared predicate	2	284 MiB

Making the collection fire is achievable — a born-tenured byte counter reset at each reclaim, exempted from the RETAINING veto, produces collections where main has none. It does not lower peak RSS, and two fulls make it worse than zero.

Why: the arena never returns capacity

[gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0
                      low_collections=0 fulls_remaining=0 disarmed=false
                      arena_live=81407888 arena_capacity=90177536
[gc-arena-rebaseline] total=141557760 step=1073741824 -> 1073741824

Right-sizing is armed (disarmed=false) and never starts an episode. So the chain is:

1. Old-gen fills with born-tenured storage that no minor sweeps.
2. The arena grows to hold it — total reaches 141 MB, rebaselining its own trigger 536 MB → 1 GB along the way.
3. A full collection, when it finally runs, frees objects but returns no capacity.
4. Peak RSS is set by arena capacity, which only ever grows.

That is why PERRY_GEN_GC=0 holds 55 MiB flat: with full sweeps throughout, old-gen never accumulates, so the arena never grows in the first place. It is not that fulls shrink the heap — it is that they stop it expanding.

Two late fulls are therefore strictly worse than none: the arena has already grown, and the collections add their own cost on top.

What this means for the fix

The trigger work above is necessary but not sufficient. Either:

1. Fire much earlier, before the arena expands — the born counter makes that expressible, but the floor would have to be well below the current threshold/2, and that trades CPU against a benchmark this issue has not measured. The gc-ratchet corpus is the right instrument.
2. Make arena-right-size actually run after a productive full. episodes=0 with disarmed=false is the specific thing to explain: arena_live=81 MB against arena_capacity=90 MB is only 10% slack, so a ratio-based gate may simply never see enough headroom — while the 141 MB the arena reached, versus the 55 MB the workload needs, is where the memory actually is.

Option 2 looks like the higher-value half, and it is independent of the trigger.

Also worth noting

old_reclaim_pressure_due has three consumers, one of them
copied_minor_promotion_handoff_pressure_due. I split the births arm out of the
shared predicate on the theory that scheduling a full before a promotion was the
#7592 futile-handoff shape. It made no difference — 284 MiB either way — so the
handoff is not implicated here, and a future change need not carry that
complication.

Everything above is reverted; nothing from this investigation is on a branch.

[proggeramlug]

The complete chain: two independent heuristics, both fed by the pathology they would cure

Same fixture, same binary, only PERRY_GEN_GC differs:

regime	arena_live	arena_capacity	utilization	peak RSS
generational	158 MB	175 MB	90%	222 MiB
full-sweep-only	6.4 MB	14.7 MB	44%	55 MiB

The workload's true live set is 6.4 MB. Everything above that in the generational column is dead storage that no collection has adjudicated: born tenured, over LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, and a minor never sweeps old-gen.

That inflated number is then consumed by two separate gates, and both read it as "this heap is busy":

1. old_reclaim_pressure_due — GC_MAJOR_PACING_RETAINING is armed from young-survival, which reads 997permille because the born-tenured containers pin the young graph through the remembered set. The veto blocks the collection that would disprove it.
2. arena_right_size — opens an episode only at or below ARENA_RIGHT_SIZE_TRIGGER_PCT (50%) live/capacity. It sees 90% and declines, so episodes=0 starts=0 released_capacity_bytes=0 with disarmed=false. Capacity is never returned.

So the chain is: born-tenured storage accumulates -> arena_live reads 25x the true live set -> the reclaim trigger declines AND right-sizing declines -> the arena grows to hold the garbage and never gives it back -> peak RSS is 4x the true requirement.

PERRY_GEN_GC=0 holds 55 MiB not because full sweeps shrink the heap, but because the arena never grows in the first place: live stays at 6.4 MB, so nothing ever asks for 175 MB of capacity.

Why the trigger fix alone cannot work

This is the measured reason attempts 2 and 3 made things worse rather than better. By the time a births-driven trigger fires, the arena has already expanded, and right-sizing still will not open because arena_live is still inflated -- the two fulls left it at 81 MB against a 6.4 MB true live set. Two late fulls are strictly worse than none: no capacity comes back, and the collections cost their own time.

Where a fix has to start

The shared defect is that both gates consult a liveness number that includes old-gen nothing has visited since it was born. Options, in the order I would try them:

1. Give the right-size gate a birth-aware path. Born-tenured bytes since the last reclaim are already trackable (a counter at the two birth sites in arena/allocators.rs costs one add). A heap whose capacity is dominated by un-adjudicated births should be able to open an episode regardless of apparent utilization, because apparent utilization is precisely what cannot be trusted there.
2. Fire the reclaim early enough that the arena never expands, which means a floor far below threshold/2. That trades CPU, and the gc-ratchet corpus is the instrument.

Option 1 looks independent of the trigger work and addresses the half that actually holds the memory.

Dead ends, so nobody re-derives them

• Exempting births from the RETAINING veto: produces collections (0 -> 1), no RSS change.
• A recurring births arm in old_reclaim_pressure_due: 2 collections, RSS 222 -> 284 MiB.
• Splitting that arm off the shared predicate so copied_minor_promotion_handoff_pressure_due keeps the original: identical 284 MiB, so the handoff is not implicated and a future change need not carry that complication.

All reverted; nothing is on a branch.

[proggeramlug]

Root cause found: promotion is structurally invisible to old-gen pressure

Correcting my three earlier comments on this issue. I framed this as a trigger-tuning
problem and measured three dead ends against that framing. It is not a tuning problem,
and that is why none of them moved.

The mechanism

credit_promoted_bytes_to_old_baseline (gc/policy.rs) adds a minor's promoted bytes
straight onto the old-reclaim baseline:

GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|b| b.set(b.get().saturating_add(promoted_bytes)));

So old_in_use - old_baseline is invariant under promotion, by construction. Promotion
raises both sides in lockstep. Only allocation born directly into old-gen can ever create
old-gen pressure.

That is deliberate and I am not proposing to change it — #7902 and #7965 both settled it,
and the argument in the code is right: the baseline is the base of a growth measurement,
not a liveness claim, and withholding the credit pins that base at 0 on exactly the
workloads that reach it.

But it leaves one question unanswerable: how much of old-gen arrived because a minor
could not tell garbage from survivors? On wide_1m:parse every minor promotes at
999 permille. Nothing dies. The entire nursery is laundered into old-gen, where no
minor will ever look at it again — and the only thing scheduling fulls was incidental
born-tenured allocation elsewhere in the parser.

How that was proved

JSON.parse re-mints every key of a wide object as a born-tenured string on every call:
PARSE_KEY_CACHE holds 4096 entries, a 50k-field object overflows it on the first parse,
and the parse-boundary clear then wipes it before the second parse can hit it. So 50,000
js_string_from_bytes_longlived allocations per iteration.

Fixing that — raising the bound so the schema stays cached — makes the row much worse:

	peak RSS @256 it	fulls
main	387 MiB	8
key-cache bound raised	833 MiB	0

Removing the garbage removed the only thing that was scheduling collections. That is the
cleanest possible demonstration of the mechanism, and it is a trap for anyone who fixes
the parser side first.

The fix

Record a second, independent signal alongside the baseline credit rather than correcting
it. Bytes promoted by a minor at >= 950 permille survival accumulate in
GC_OLD_GARBAGE_SUSPECT_BYTES; 16 MB of it makes an old reclaim due on its own; a reclaim
resets it. The baseline is untouched, so #7902 and #7965 both still hold.

Split off old_reclaim_pressure_due on purpose rather than folded into it — that predicate
is also what copied_minor_promotion_handoff_pressure_due asks, and an arm there schedules
a full immediately before a promotion, over survivors it cannot reclaim, which is the
futile-handoff shape #7592 removed. (That was one of my earlier dead ends.)

Floor is the measured knee of a 2/4/8/16/32/64/128 MB sweep at 256 iterations:

floor	peak RSS	CPU	fulls
8 MB	243 MiB	3.14 s	14
16 MB	243 MiB	3.00 s	12
32 MB	274 MiB	3.00 s	11
64 MB	510 MiB	2.71 s	5
(unpatched)	387 MiB	3.03 s	8

wide_1m:parse peak RSS 387 MiB -> 243 MiB (-37%) at baseline CPU. With the pacing arm
in place the key-cache fix is measurably redundant (243 MiB either way), so it is not
included.

Branch gc/old-born-bytes-arm-full, commit 14ceaef3d6. Not opened as a PR — this is a
GC pacing change and needs the gc-ratchet corpus run before/after by someone with that setup.

What still stands between this row and parity

Not pacing. RSS floors at 243 MiB however many fulls run — because freed pages are never
returned to the OS:

arena_live=56678536  arena_capacity=67108864      <- 56.7 MB live in a 67 MB arena
[gc-general-reclaim] examined=41 released=0 rejected: no_snapshot=40 ...
[gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0

The process holds 243 MiB while its arena is 67 MB and its live set is 56.7 MB. Block
release rejects 40 of 41 blocks at snapshot.data == 0 (arena/reset.rs:700). Targets are
Node 121 MiB / Bun 96 MiB, and 56.7 MB of live data is comfortably inside both — so this is
the whole remaining gap on the row, and it is independent of everything above.

Dead ends, so nobody re-derives them

1. Exempting births from the RETAINING veto — 1 full, 222 MiB, unchanged.
2. A recurring births arm on the shared predicate — 2 fulls, 284 MiB (worse; the
   futile-handoff shape).
3. Splitting that arm off the shared predicate — identical 284 MiB, so the handoff was
   never implicated.
4. A born-tenured-bytes floor — this does work (387 -> 154 MiB) but costs +7.6% CPU,
   because it buys RSS with extra whole-heap marks. The suspect-promotion arm gets most of
   the same RSS for free. Superseded.
5. Raising the parse-key cache bound alone — 833 MiB, as above.

[proggeramlug]

Correction, and the actual root cause

First, a correction to my previous comment. I said the suspect-promotion arm delivered
"387 MiB -> 243 MiB at baseline CPU". That CPU claim was measured on wide_1m:parse only.
Measured across the rest of the JSON matrix against clean main, it is not baseline CPU:

row	CPU main -> arm	RSS main -> arm
records_array_1m:scan	0.59s -> 1.30s (+120%)	410M -> 118M
records_array_8m:scan	1.73s -> 2.60s (+50%)	444M -> 178M
records_array_20m:parse	0.73s -> 1.05s (+44%)	267M -> 196M

Worse, the regression does not respond to the floor — swept at 16/64/256 MB,
records_array_1m:scan stays at 1.26–1.30s even at 256 MB where the arm should be inert. So
the cost is the predicate, not the extra collections: old_reclaim_pressure_or_suspect_due
sits in gc_budgeted_due_trigger, which runs at every safepoint, and the arm adds a
thread-local read on the common path. Reverted in 9e1c80e060. The diagnosis in that
commit stands (promotion credits the baseline, so only born-tenured allocation can create
old-gen pressure); the remedy does not.

The real cause: wide objects are born tenured, and a minor never sweeps old-gen

arena/allocators.rs already states the failure mode exactly:

It is TYPE-DEPENDENT, because the price of crossing it is: this object is also stamped
GC_FLAG_TENURED, and a minor never sweeps old-gen — so for a POINTER-BEARING object the
cost is not its own bytes, it is every object it can reach, held live through the
remembered set by a container nothing refers to any more.

A wide JSON document's shape-keys array is that container. Above 16,384 fields its keys
array crosses LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB), is born tenured, and
then holds its whole key set live after every instance has died.

Confirmed to the byte — PERRY_GC_CENSUS, 64 parses, fixtures straddling 131,072 bytes:

fields	keys array	shapes retained	live	peak RSS
16,000	128,016 B	0	0.0 MB	29 MiB
16,300	130,416 B	0	0.0 MB	29 MiB
16,500	132,016 B	30	22.7 MB	98 MiB
50,000	400,016 B	15	34.3 MB	177 MiB

The census names it directly: shape_keys_arrays {bytes: 6000240, count: 15},
slot_tags {string: 750000} — 15 arrays × 50,000 keys = 750,000 live strings, on a document
whose live set is ~0. Same binary under PERRY_GEN_GC=0: live 336 bytes, 50 MiB RSS.

Ruled out by direct experiment (each rebuilt and re-measured, all zero effect):
transition-cache size (raised 16,384 -> 65,536 on both the runtime constant and the mask
codegen bakes in proxy_reflect.rs); parse-key identity (PARSE_KEY_CACHE bound raised
4,096 -> 65,536); the shape table (table_len=12 < 15 live arrays); registered mutable root
scanners (all report slots=0 ptr_roots=0).

What a fix is worth, and why I am not proposing one yet

Raising the threshold 128 KB -> 512 KB (still inside the copier's 1 MiB MAX_YOUNG_MOVE_BYTES
and nursery-block ceilings) on wide_1m:parse:

0.62s / 284 MiB -> 0.15s / 40 MiB — −76% CPU and −86% RSS, better on both axes, against
Node 121 MiB and Bun 96 MiB. perry-runtime is green (3628 passed) after retargeting two
tests that hardcoded a field count and asserted old-gen residency — they now derive their
width from the constant, so the old-gen path stays covered at any threshold.

But globally it is not shippable: the same change moves ordinary 128 KB–512 KB data arrays
into the nursery, and on rows that do not need it that costs RSS —
records_array_8m:scan 444M -> 710M (+60%), records_array_20m:parse 267M -> 314M (+18%)
(CPU on those two improves, −29% and −44%).

So the widening has to be scoped to wide-object JSON construction, not global. I built
half of that: a WideShapeKeysBirthScope around allocate_parse_shape_keys_array, gated in
the cold large-object branch of arena_alloc_gc so the hot path is untouched. It fires
correctly (born young past threshold: 400016 bytes every parse) but only gets
wide_1m to 176 MiB, because the keys array is half the story — a 50,000-field object's own
property storage is also 400 KB and also born tenured. The scope needs to cover the
object-storage allocation on the JSON construction path too; records_array_* is unaffected
by that because its large allocation is array element storage, not wide-object storage.

That is the next step, and it wants the quiet-mini matrix plus the gc-ratchet corpus rather
than more measurement on a box at load 42.

[proggeramlug]

Fixed — PR #10145

The scoped fix is finished and open as draft #10145 (gated on the gc-ratchet corpus, since it changes birth generation).

The half I built earlier didn't work because the scope was open in the wrong place. A mask probe at each large allocation showed the keys array was admitted (mask=2) but the object's property storage saw mask=0. That storage is minted in object/json_construction.rs via js_object_alloc_class_inline_keys_stamped, not inside js_json_parse_result. I found the site by tagging all nine GC_TYPE_OBJECT allocation sites in object/alloc.rs. Scoping it there completes the fix.

Measured on current main against a same-tree baseline (the commit reverted and rebuilt):

row	node	bun	baseline	#10145
wide_1m:parse	136 MiB	71 MiB	220 MiB / 0.20s	40 MiB / 0.15s
records_array_8m:scan	156	166	643 / 0.94s	643 / 0.93s
records_array_20m:parse	422	170	313 / 0.40s	313 / 0.41s
records_array_1m:scan	93	95	410 / 0.58s	410 / 0.59s
records_array_16k:scan	57	77	117 / 0.13s	117 / 0.13s

One more correction to my earlier comments. The "main" figures I quoted before (283 MiB for wide_1m, 444 MiB for 8m:scan) came from a frozen binary whose runtime turned out to be a different source state. The compiler refused to cross-link it (library source 111389820894 vs a511bd884a7f). The numbers above use a baseline built from the same tree as the fix. That also shrinks the global-widening regression I reported on 8m:scan from "+60%" to 643 → 710 MiB. It's still a regression, and it's still why this is scoped.

[proggeramlug]

Closed by #10171 and #10145 ("Closes #10123"), landed via merge train #10188 (main 5cec2fbbc9).