perf(gc): walk an inline slot mask instead of re-entering the iterator (#10362)

changelog.d/10552-residual-prototype-relocation.md

@@ -0,0 +1,9 @@
+### Fixed
+
+- **A relocated non-object owner lost its explicit `[[Prototype]]` (#10493).** `Object.setPrototypeOf` on a receiver that is not meta-capable records the prototype in the residual address-keyed registry (#9304), and relocation owes that entry two things. Neither happened outside arrays and ordinary objects: `layout_transfer` reached the rekey only *below* its layout-kind early return, which a `GcLayoutSlotKind::None` cell never passes, and the recorded value was emitted as a child edge from the Array and Object arms alone. So a lazy JSON array, Map, Set, Error, Promise, Date, RegExp, Temporal cell or `dyn_eval` closure silently lost its prototype at its first relocation — correct before a collection, wrong after, exit code 0 and no warning — and the prototype value itself was neither retained nor rewritten. Fixing only the rekey is worse than fixing neither: it turns "prototype lost" into "entry names a stale address", a state measured between the two halves.
+
+  Both obligations now follow the registry's population, stated once in `prototype_chain::residual_prototype_owner_type` (everything except strings, bigints, meta records and compiled regex programs) rather than being wired to two kinds by hand. The rekey runs before the layout-kind return for every owner kind, latch-gated, with the move in a `#[cold]` call; the array-arm and move-hook copies are deleted. The recorded value is emitted ahead of the kind arms, so no arm's early return can skip it.
+
+  Each half has its own sabotage witness: removing the rekey fails at "the registry entry did not follow its owner", and restricting the value visit back to arrays and objects fails at "the recorded prototype still names its pre-collection address". Cost is +0.04% to +0.16% instructions where the registry is never armed, and +0.68% on a fixture that arms it and churns Errors/Maps/Dates — the per-owner cost arrays and objects have always paid.
+
+  Not a regression: the funnel before #10381 returns on the same check. Found from a CodeRabbit review comment on #10381 that landed unactioned; the population turned out to be six kinds wider than the one it named.

changelog.d/10584-inline-mask-walk.md

@@ -0,0 +1,44 @@
+Walk an inline slot mask directly instead of re-entering the slot iterator once
+per slot. `visit_gc_layout_slot_descriptors` called
+`HeapChildSlotIterator::next` for every payload slot; for the common case — a
+`Masked` selection whose mask is `LayoutSlotMask::Inline` — each of those calls
+re-dispatched the selection, re-decoded the mask's niche and rebuilt the limit
+and cursor masks, for about eight instructions of work.
+
+The mask's set bits are the slot indices, in ascending order, so the arm takes
+the word once and walks it with `trailing_zeros` and `word &= word - 1`. Every
+other selection, including a `Heap` mask (more than 64 payload slots), keeps the
+iterator. The helper carries the iterator's two side conditions with it: the
+one-shot raw-numeric accounting that `next`'s first call performs, and the
+cursor, left at the end so a later `next` yields nothing. The prefix and meta
+edges belong to the caller, which takes them before the payload; the helper
+asserts they are gone rather than arguing it.
+
+Measured on a control whose pointer fields target DISTINCT objects, because the
+older shared-child control let the collector's one-entry address memo answer
+83.3% of its classifications against 0.0% on the real fixtures, and so hid the
+cost of everything downstream of that memo. On it, `next` costs 75.8 of the
+417.1 instructions a pointer-slot visit costs, and the walk removes 69.7 of
+them. On the same control the shared-child version reports 75.9 — the iterator's
+own cost is what the blind control did NOT distort.
+
+The descriptor walk serves the copying minor, the full mark and the
+remembered-set rebuild. Inclusive instructions for the walk on gc3, exact, by
+caller: copying minor -7.95%, full mark -16.62%, remembered-set rebuild -17.59%,
+dirty scan unchanged. On `oldyoung`, whose masked population is mostly one
+`Heap` mask, the remembered-set rebuild and the dirty-coverage restore each pay
+one failed `take_inline_mask_word` dispatch per visit: +0.25% and +0.33%, about
+two instructions per object visit, against -4.49% on that fixture's copying
+minor and -1.07% on the program.
+
+Whole program, instructions:u, min of 5: gc3 -6.75%, w20000 -5.86%, w5000
+-4.90%, w1000 -2.36%, oldyoung -1.06%, and an allocation-only fixture flat to
+298 instructions in 320 million. No fixture regresses in instructions, peak RSS
+or max GC pause.
+
+The equivalence between the walk and the iterator is a property, and is tested
+as one: identical index sequences for every mask word (empty, one bit at each
+end, full width, both alternations, and 64 pseudo-random words) crossed with
+every live slot count from 0 to 128, with a sabotaged twin that drops the mask's
+top bit and must be caught, plus a real collection whose only young child hangs
+off the highest masked slot and its own sabotaged twin.

crates/perry-runtime/src/gc/layout/transfer.rs

@@ -17,13 +17,27 @@
 //! `GC_OBJ_TYPED_LAYOUT_INTACT`. What cannot ride a header is a record keyed
 //! by the object's ADDRESS, and that is all this funnel moves:
 //!
+//! * the residual static-prototype owner registry (#9304), gated by its
+//!   process-global latch (#7733/#7737) — for EVERY kind that can own an
+//!   entry, not only the layout kinds (see below);
 //! * the element-shape proof record (#7480), gated by the header bit that is
 //!   authoritative for it;
-//! * the residual static-prototype owner registry (#9304), gated by its
-//!   process-global latch (#7733/#7737);
 //! * the per-object `TYPED_LAYOUTS` and `LAYOUT_SLOT_MASKS` entries, gated by
 //!   #7510's emptiness flag and address filter.
 //!
+//! # The prototype registry is not layout metadata
+//!
+//! Its population is every receiver `object::prototype_chain::
+//! meta_capable_object` turns away — a Map, Set, Error, Promise, Date, RegExp,
+//! Temporal cell, lazy JSON array or closure as much as an array — and most of
+//! those kinds have no layout slots at all. The rekey used to sit in the array
+//! arm of the record path, below the layout-kind return, and in the ordinary
+//! object's move hook; every other movable owner kept its entry under the
+//! address it had just left, and the dead-owner prune then dropped it. So it
+//! runs first, keyed on `prototype_chain::residual_prototype_owner_type`, the
+//! same population predicate the collector's value visit uses
+//! (`gc/layout_slot_visit.rs`).
+//!
 //! Until #10362 the funnel re-derived the header half too — rewriting bits
 //! that were already equal, and re-resolving the intact bit through a
 //! ShapeId-keyed `SHAPE_LAYOUTS` probe — once per relocated object. Measured
@@ -64,30 +78,47 @@ use crate::gc::layout_tables::per_object_layouts_may_hold_either;
 /// `old_user` and `new_user` are user pointers of live allocations, and the
 /// caller has already made the destination header a copy of the source's (see
 /// the module docs). The precondition is asserted in test and debug builds.
-#[inline]
+///
+/// `inline(always)`: every relocation of every object runs this and all it
+/// keeps inline is gates — each record move is a cold out-of-line call — but
+/// three gates are enough for the heuristic to outline it from `move_young`,
+/// and then the call costs more than the gates.
+#[inline(always)]
 pub(crate) unsafe fn layout_transfer(old_user: *mut u8, new_user: *mut u8) {
     if old_user.is_null() || new_user.is_null() || old_user == new_user {
         return;
     }
+    if (old_user as usize) < GC_HEADER_SIZE + 0x1000 {
+        return;
+    }
+    // Before the layout-kind return, because the registry's owners are not the
+    // layout kinds (module docs). The latch first: it is one byte load, false
+    // for any process that never re-prototyped a non-object, and the move is
+    // out of line.
+    if crate::object::prototype_chain::object_static_prototypes_maybe_nonempty()
+        && crate::object::prototype_chain::residual_prototype_owner_type(
+            (*header_from_user_ptr(old_user as *const u8)).obj_type,
+        )
+    {
+        transfer_residual_prototype(old_user as usize, new_user as usize);
+    }
     // Kinds with no layout metadata at all (strings, meta records, RegExps)
-    // leave before anything else, exactly as before #10362. The destination
-    // carries the same `obj_type`, so one classification answers for both.
+    // have no layout record to move. The destination carries the same
+    // `obj_type`, so one classification answers for both.
     let Some(old_header) = layout_header_for_user(old_user as usize) else {
         return;
     };
     assert_relocation_copied_the_header(old_header, new_user);
 
     let reserved = (*old_header)._reserved;
     let is_array = (*old_header).obj_type == GC_TYPE_ARRAY;
-    // Three gates, all answered from words already in registers or in the one
+    // Two gates, both answered from words already in registers or in the one
     // hot thread-local slot #7510 keeps them in. Each is the same question the
     // record mover behind it asks first, hoisted so the common case — no
     // record anywhere near either address — never leaves this function.
     let per_object = per_object_layouts_may_hold_either(old_user as usize, new_user as usize);
     let element_shape = is_array && reserved & GC_ARRAY_ELEMENT_SHAPE != 0;
-    let static_prototype =
-        is_array && crate::object::prototype_chain::object_static_prototypes_maybe_nonempty();
-    if per_object || element_shape || static_prototype {
+    if per_object || element_shape {
         transfer_address_keyed_records(
             old_user as usize,
             new_user as usize,
@@ -102,9 +133,18 @@ pub(crate) unsafe fn layout_transfer(old_user: *mut u8, new_user: *mut u8) {
     header_clear_typed_layout_intact(old_header);
 }
 
-/// The record moves themselves. Cold: on a workload holding no per-object
-/// layout record, no element-shape proof and no re-prototyped array — the
-/// steady state of every monomorphic program — it is never reached.
+/// The residual prototype registry's rekey. Cold and out of line so the funnel
+/// stays small enough to inline into every relocation site: it is reached only
+/// once something in the process has been re-prototyped.
+#[cold]
+#[inline(never)]
+fn transfer_residual_prototype(old_user: usize, new_user: usize) {
+    crate::object::prototype_chain::object_static_prototype_owner_moved(old_user, new_user);
+}
+
+/// The layout record moves themselves. Cold: on a workload holding no
+/// per-object layout record and no element-shape proof — the steady state of
+/// every monomorphic program — it is never reached.
 #[cold]
 #[inline(never)]
 unsafe fn transfer_address_keyed_records(
@@ -119,10 +159,6 @@ unsafe fn transfer_address_keyed_records(
         // from both headers and fails closed — it clears the destination bit
         // when no record follows the move.
         crate::array::transfer_element_shape(old_user, new_user);
-        // #9304: a real array keeps an explicit [[Prototype]] in the residual
-        // address-keyed registry; moving GC and growth both replace the owner
-        // allocation through this hook.
-        crate::object::prototype_chain::object_static_prototype_owner_moved(old_user, new_user);
     }
     // #7510's two per-object maps. Both re-test the gate above for their own
     // address pair, so calling them when only a sibling gate fired costs one