perf(gc): exact-start bitmap + page-class-first pointer validation; header-first buffer-registry rejection

cc-perf-campaign/codex/REPORT_gc-overhead.md

@@ -0,0 +1,120 @@
+# GC pointer-validation / buffer-registry report
+
+## Branch and implementation SHA
+
+- Branch: `perf/gc-pointer-validation`
+- Base: `origin/main` at `504e180d085f37aaf0b9189e0c1bdb328d508f70`
+- Runtime implementation commit: `8d49faa556004b87e57e29a22ca822d3369a8ef3`
+
+The checkout's shared `.git` metadata was read-only in this environment. The
+branch was therefore fetched and created in an isolated Git directory while
+the work tree remained `/Users/amlug/projects/perry/agent-trees/gc-overhead`.
+
+## What changed
+
+- The synchronous exact `ValidPointerSet` census clears each arena block's
+  existing object-start bitmap and stamps every walkable header as it visits
+  it. This includes headers written by codegen's inline allocator. The normal
+  allocation path remains Map-only and gains no per-allocation bitmap write.
+- `ValidPointerSet::contains` asks the direct page-class metadata and exact-
+  start bitmap before its two-level sorted-run floor lookup. Non-arena and
+  page-table-control cases retain the sorted-run fallback. Malloc membership
+  remains an exact `BTreeSet` union and is always considered after an arena
+  miss.
+- `ValidPointerSet::enclosing_object` still uses the sorted census runs; the
+  bitmap is exact-start membership only.
+- `buffer::is_registered_buffer` now uses
+  `try_read_tracked_gc_header` to reject a tracked non-`GC_TYPE_BUFFER` object
+  before the address filter and exact registries. A `GC_TYPE_BUFFER` match is
+  not accepted from the header: managed buffers, headerless external buffers,
+  and SharedArrayBuffer backings retain the authoritative registry path.
+- `PERRY_GC_DIAG` gained one whole-run `[gc-pointer-validation]` line and one
+  `[gc-runtime-handles]` line. `PERRY_BUFFER_DIAG` gained
+  `header_rejects=` and renamed its downstream count `filter_probes=`.
+- Added named sabotage tests for the page-class exact-start arm and Buffer
+  header-class rejection.
+
+## Handle-scope source audit
+
+No handle-scope behavior changed.
+
+- A successful `js_segments_view_open` creates one `RuntimeHandleScope` and
+  pushes one input root across the cursor allocation. `js_segments_view_next`
+  and `js_segments_view_code_point_at` create no scope and push no roots; both
+  are allocation-free by their implementation contract. The other view entry
+  points do not open a scope per call.
+- A successful `js_regexp_new` creates one scope and pushes exactly two roots:
+  the pattern plus either the caller's already-canonical flags string or the
+  newly allocated canonical flags string.
+- Both public entry points receive raw values/pointers. A caller-owned scope
+  cannot let either callee refresh a relocated argument after an allocation
+  without changing the ABI to pass a mutable root/slot. This is not a one-line
+  scope removal.
+- Actual cc-turn scope and push counts were not measured locally. The added
+  diagnostic reports total scopes, pushes, pushes-per-scope, and maximum live
+  slots for the perrymaster run.
+
+## Gates and checks
+
+Run locally:
+
+- `rustfmt --edition 2021 --check` over all 12 changed Rust files: pass.
+- `./scripts/check_file_size.sh`: pass; zero Rust files over 2,000 lines.
+- `python3 scripts/addr_class_inventory.py`: address-class audit pass over
+  1,357 files, 315 allowlisted and 521 known sites. It separately reported two
+  pre-existing stale baseline reductions in `set.rs` and `crypto/kdf.rs`; no
+  baseline was rewritten.
+- `git diff --check`: pass.
+
+Not run:
+
+- Cargo invocations: 0. Immediately before the intended gates, `df -g /`
+  reported 24 GB available, but the binding wrapper
+  `secret-tests/cc-perf-campaign/measure_lock.sh` is absent. The task requires
+  every Cargo invocation to go through that wrapper, so no Cargo command was
+  issued.
+- `cargo build --release -p perry` with default features: not run.
+- `cargo test -p perry-runtime --release --lib -- --test-threads=1`: not run.
+- Sabotage tests executed: 0 of 2; both were authored but require the blocked
+  Cargo test gate.
+- `nm -g libperry_runtime.a`: not run. No exported symbol or exported function
+  signature was changed; no local archive exists because `target/` was
+  deleted and no build was permitted.
+
+## Counters and measurements
+
+- Pre-fix cc-rig counter run: not measured. The private cc harness and campaign
+  files named in the task are absent from this checkout, and the mandatory
+  build wrapper is absent.
+- Post-fix cc CPU/RSS: not measured.
+- Locally observed performance numbers: none.
+- The source map supplied with the task is not restated as a branch result.
+
+New counter line schemas to collect:
+
+```text
+[gc-pointer-validation] contains=... page_hits=... page_rejects=... run_fallbacks=... malloc_probes=... malloc_hits=... enclosing=...
+[gc-runtime-handles] scopes=... pushes=... pushes_per_scope=... max_slots=...
+[buffer-diag] header_rejects=... filter_probes=... admits=... rejected=... true_positives=...
+```
+
+## Exact perrymaster measurement request
+
+Fetch `fork/perf/gc-pointer-validation` and relink the runtime-only implementation
+SHA `8d49faa556004b87e57e29a22ca822d3369a8ef3` on main6's cache. Run the I7-view
+cc rig for the pushed SHA and its `origin/main` base: **5×3300 + 2×400 + idle
+rows**. Record turn CPU, main-thread samples, collector share, RSS, and the
+named pointer-validation/buffer leaves. Add the existing page-table control
+arm with `PERRY_GC_PAGE_CLASS_TABLE=0`. In counter runs enable
+`PERRY_GC_DIAG=1` and `PERRY_BUFFER_DIAG=<path>` and return the counter lines
+whole, including every `[gc-pointer-validation]`, `[gc-runtime-handles]`, and
+`[buffer-diag]` line; do not summarize or truncate them.
+
+## Could not verify
+
+- Compilation, linking, exported archive contents, unit/sabotage test execution,
+  cc-turn behavior, CPU parity, and RSS impact.
+- The task-referenced `cc-perf-campaign/BRIEF_COMMON.md`, `STATUS.md`,
+  `ARCHITECTURE.md`, `BRIEF_gc-overhead.md`, and
+  `RESULT_page_class_table.md` were not present under this worktree or the
+  stated `/Users/amlug/projects/perry/perry` repository.

changelog.d/gc-pointer-validation.md

@@ -0,0 +1,16 @@
+### Reduce GC pointer-validation and Buffer probe overhead
+
+The synchronous collector now rebuilds each arena block's exact-start bitmap
+from its own object census and uses the direct page-class metadata for constant-
+time `ValidPointerSet` membership. This covers codegen's inline allocator
+without adding work to the allocation path; the existing sorted census runs
+remain the kill-switch fallback and continue to serve interior-pointer lookup.
+
+Generic Buffer identity checks now reject tracked non-Buffer GC objects from
+their header class before consulting the address filter and exact side
+registry. Positive matches and headerless external Buffer/SharedArrayBuffer
+storage still use the authoritative registries.
+
+`PERRY_GC_DIAG` now reports pointer-membership and runtime-handle scope/push
+counts, and `PERRY_BUFFER_DIAG` reports header-class rejections, so the cc turn
+rig can attribute the affected hot paths directly.

crates/perry-runtime/src/arena/mod.rs

@@ -56,7 +56,8 @@ pub(crate) use block::{
     reset_gc_trigger_arena_probe,
 };
 pub(crate) use page_meta::{
-    address_span_overlaps_pages, defer_old_object_page_registration, page_class_table_report,
+    address_span_overlaps_pages, defer_old_object_page_registration, page_class_exact_object_start,
+    page_class_exact_start_enabled, page_class_table_report, record_censused_arena_object_start,
     register_block_space_with_object_starts, register_old_object_pages,
     unregister_block_generation, unregister_old_block_pages, OLD_GEN_RECLAIM_POOLED_BYTES,
     OLD_GEN_RECLAIM_RETURNED_BYTES, OLD_GEN_RECLAIM_REUSABLE_BYTES,

crates/perry-runtime/src/arena/page_meta/mod.rs

@@ -986,6 +986,62 @@ pub(crate) fn record_arena_object_start(header_addr: usize, obj_type: u8) {
     }
 }
 
+/// Stamp an object start discovered by the exact ValidPointerSet census.
+///
+/// The census first clears every block bitmap, then calls this for every
+/// walkable header—including objects created by codegen's inline allocator,
+/// which deliberately bypasses [`record_arena_object_start`]. Keeping the
+/// work here, rather than on allocation, exchanges one bitmap RMW per census
+/// object for constant-time membership on every subsequent pointer probe.
+#[inline(always)]
+pub(crate) fn record_censused_arena_object_start(header_addr: usize) {
+    if !page_class::page_class_table_enabled() {
+        return;
+    }
+    let Some((_space, range_base, bitmap)) = classify_heap_space_in_range(header_addr) else {
+        debug_assert!(false, "censused arena header was not in a registered block");
+        return;
+    };
+    if bitmap.is_null() || header_addr < range_base {
+        return;
+    }
+    let slot = (header_addr - range_base) >> super::OBJECT_START_SHIFT;
+    let word = slot / u64::BITS as usize;
+    let bit = slot % u64::BITS as usize;
+    unsafe {
+        *bitmap.add(word) |= 1u64 << bit;
+    }
+}
+
+/// Exact object-start membership for the direct page-table arm.
+///
+/// `Some` means page metadata answered authoritatively for an arena range;
+/// `None` leaves the caller on its census-run fallback (the 4-way control,
+/// synthetic ranges without a bitmap, and non-arena addresses). The exact
+/// ValidPointerSet builder clears each bitmap and repopulates it from the same
+/// census that fills its sorted runs, so this index has identical membership.
+#[inline(always)]
+pub(crate) fn page_class_exact_object_start(addr: usize) -> Option<bool> {
+    if !page_class::page_class_table_enabled() {
+        return None;
+    }
+    let (_space, range_base, bitmap) = classify_heap_space_in_range(addr)?;
+    if bitmap.is_null() {
+        return None;
+    }
+    let Some(header_addr) = addr.checked_sub(crate::gc::GC_HEADER_SIZE) else {
+        return Some(false);
+    };
+    if header_addr < range_base || !arena_header_is_object_start(header_addr, range_base, bitmap) {
+        return Some(false);
+    }
+    Some(true)
+}
+
+pub(crate) fn page_class_exact_start_enabled() -> bool {
+    page_class::page_class_table_enabled()
+}
+
 /// True only when `header_addr` is a recorded allocation boundary in the
 /// registered block beginning at `range_base`.
 #[inline(always)]

crates/perry-runtime/src/arena/page_meta/page_class.rs

@@ -389,7 +389,7 @@ struct PageClassStats {
 /// the positive control: both arms live in ONE binary so no build difference
 /// can be confounded with the change.
 #[inline(always)]
-fn page_class_table_enabled() -> bool {
+pub(super) fn page_class_table_enabled() -> bool {
     static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
     *ENABLED.get_or_init(|| crate::gc::env_default_on_enabled("PERRY_GC_PAGE_CLASS_TABLE"))
 }

crates/perry-runtime/src/arena/walk.rs

@@ -42,6 +42,10 @@ enum ArenaObjectCursorBlocks {
 
 pub(crate) struct ArenaObjectCursorBuilder {
     order: ArenaWalkOrder,
+    /// Clear each block's exact-start bitmap while taking the census snapshot.
+    /// Only the exact ValidPointerSet census requests this; other arena walks
+    /// retain the allocation-authored Map boundaries used by copying GC.
+    reset_object_starts: bool,
     blocks: Vec<ArenaWalkBlock>,
     address_blocks: std::collections::BTreeMap<usize, ArenaWalkBlock>,
     initialized: bool,
@@ -61,8 +65,16 @@ const ARENA_CURSOR_OLD: usize = 4;
 
 impl ArenaObjectCursorBuilder {
     pub(crate) fn new(order: ArenaWalkOrder) -> Self {
+        Self::new_with_object_start_reset(order, false)
+    }
+
+    pub(crate) fn new_with_object_start_reset(
+        order: ArenaWalkOrder,
+        reset_object_starts: bool,
+    ) -> Self {
         Self {
             order,
+            reset_object_starts,
             blocks: Vec::new(),
             address_blocks: std::collections::BTreeMap::new(),
             initialized: false,
@@ -91,7 +103,9 @@ impl ArenaObjectCursorBuilder {
             self.inspected_blocks = self.inspected_blocks.saturating_add(1);
             *remaining -= 1;
 
-            if let Some(block) = snapshot_cursor_block(region, block_pos, block_idx) {
+            if let Some(block) =
+                snapshot_cursor_block(region, block_pos, block_idx, self.reset_object_starts)
+            {
                 self.push_block(block);
             }
 
@@ -170,26 +184,52 @@ fn snapshot_cursor_block(
     region: usize,
     block_pos: usize,
     block_idx: usize,
+    reset_object_starts: bool,
 ) -> Option<ArenaWalkBlock> {
     let block = match region {
         ARENA_CURSOR_GENERAL => ARENA.with(|arena| unsafe {
-            let blocks = &(*arena.get()).blocks;
+            let blocks = &mut (*arena.get()).blocks;
+            if reset_object_starts {
+                if let Some(block) = blocks.get_mut(block_pos) {
+                    block.clear_object_starts();
+                }
+            }
             blocks.get(block_pos).map(snapshot_block_fields)
         }),
         ARENA_CURSOR_SURVIVOR0 => SURVIVOR_ARENA_0.with(|arena| unsafe {
-            let blocks = &(*arena.get()).blocks;
+            let blocks = &mut (*arena.get()).blocks;
+            if reset_object_starts {
+                if let Some(block) = blocks.get_mut(block_pos) {
+                    block.clear_object_starts();
+                }
+            }
             blocks.get(block_pos).map(snapshot_block_fields)
         }),
         ARENA_CURSOR_SURVIVOR1 => SURVIVOR_ARENA_1.with(|arena| unsafe {
-            let blocks = &(*arena.get()).blocks;
+            let blocks = &mut (*arena.get()).blocks;
+            if reset_object_starts {
+                if let Some(block) = blocks.get_mut(block_pos) {
+                    block.clear_object_starts();
+                }
+            }
             blocks.get(block_pos).map(snapshot_block_fields)
         }),
         ARENA_CURSOR_LONGLIVED => LONGLIVED_ARENA.with(|arena| unsafe {
-            let blocks = &(*arena.get()).blocks;
+            let blocks = &mut (*arena.get()).blocks;
+            if reset_object_starts {
+                if let Some(block) = blocks.get_mut(block_pos) {
+                    block.clear_object_starts();
+                }
+            }
             blocks.get(block_pos).map(snapshot_block_fields)
         }),
         ARENA_CURSOR_OLD => OLD_ARENA.with(|arena| unsafe {
-            let blocks = &(*arena.get()).blocks;
+            let blocks = &mut (*arena.get()).blocks;
+            if reset_object_starts {
+                if let Some(block) = blocks.get_mut(block_pos) {
+                    block.clear_object_starts();
+                }
+            }
             blocks.get(block_pos).map(snapshot_block_fields)
         }),
         _ => None,

crates/perry-runtime/src/buffer/header.rs

@@ -278,13 +278,35 @@ thread_local! {
 /// registry it guards is thread-local and `cargo test` gives each case its own
 /// thread inside one process.
     static TEST_BUFFER_REGISTRY_PROBES: std::cell::Cell<u64> = const { std::cell::Cell::new(0) };
+    static TEST_DISABLE_BUFFER_HEADER_CLASS: std::cell::Cell<bool> = const { std::cell::Cell::new(false) };
 }
 
 #[cfg(test)]
 pub(crate) fn test_buffer_registry_probe_count() -> u64 {
     TEST_BUFFER_REGISTRY_PROBES.with(|c| c.get())
 }
 
+#[cfg(test)]
+pub(crate) fn test_disable_buffer_header_class(disabled: bool) -> bool {
+    TEST_DISABLE_BUFFER_HEADER_CLASS.with(|state| state.replace(disabled))
+}
+
+#[inline(always)]
+fn managed_header_rejects_buffer(addr: usize) -> bool {
+    #[cfg(test)]
+    if TEST_DISABLE_BUFFER_HEADER_CLASS.with(std::cell::Cell::get) {
+        return false;
+    }
+    // A real managed Buffer must carry GC_TYPE_BUFFER. Page/registry ownership
+    // makes the header read safe; only a negative class answer is authoritative
+    // here. A class match still falls through to exact Buffer registries, so an
+    // aligned interior pointer cannot become a false positive.
+    unsafe {
+        crate::value::addr_class::try_read_tracked_gc_header(addr)
+            .is_some_and(|header| (*header.as_ptr()).obj_type != crate::gc::GC_TYPE_BUFFER)
+    }
+}
+
 #[cfg(test)]
 pub(crate) fn test_buffer_addr_window_bounds() -> Option<(usize, usize)> {
     BUFFER_LIKE_ADDR_WINDOW.bounds_for_tests()
@@ -493,6 +515,17 @@ pub fn is_registered_buffer(addr: usize) -> bool {
     if BUFFER_LIKE_EVER_REGISTERED.is_idle() {
         return false;
     }
+    // Since the 2026-07-09 allocator audit, every runtime-owned Buffer has a
+    // real `GC_TYPE_BUFFER` header. Reject a tracked non-Buffer before the
+    // address-filter hashes, TLS resolution, RefCell borrow and registry hash.
+    // Headerless external buffers and positive class matches retain the exact
+    // registry path below.
+    if managed_header_rejects_buffer(addr) {
+        if crate::hot_diag::buffer_on() {
+            crate::hot_diag::buffer_note_header_reject();
+        }
+        return false;
+    }
     // An address outside the registered window cannot be in any of the three
     // tables the slow path consults, so reject it here — inline, without the
     // call, the thread-local resolution, the `RefCell` borrow or the hash.