diff --git a/changelog.d/10619-shadow-stack-hot-tls.md b/changelog.d/10619-shadow-stack-hot-tls.md new file mode 100644 index 0000000000..38b345fd27 --- /dev/null +++ b/changelog.d/10619-shadow-stack-hot-tls.md @@ -0,0 +1,89 @@ +Give the shadow-stack root state (`SHADOW`) the `tls_hot` fast path. It was the +last savepoint field still declared as a raw `thread_local!`, so every +`CatchSavepoint::capture()` — once per `try` entry — paid `_tlv_get_addr` for +it on Darwin, while `EXCEPTION_STATE`, `CALL_METHOD_DEPTH` and the named +`runtime_handle_stack`/`temp_roots` fields were already routed through +`tls_hot`. Wave 1's "has any thread used this subsystem" latch does not make +the read rare here: it is set by the first shadow-frame push anywhere in the +process, and a `catch (e)` binding needs a shadow slot itself. + +Profiling pinned it to one call site, `try_push_with_kind+176 -> +_tlv_get_addr`, about 6% of instructions retired per entry. The change is a +storage-mechanism swap — same type, call sites, const-init and drop-free +semantics — so the address-stability contract `js_shadow_frame_enter` depends +on is unchanged. A non-throwing `try` entry goes 179.3 -> 164.8 instructions +(-8.1%), differenced within each binary with the loop control at the noise +floor in both arms. The probe understates it: `SHADOW` is read on many paths, +not only this one. + +Because the shadow stack is the GC's precise root set rather than bookkeeping, +the new fixture attacks rooting: a throw four frames down, a throw crossing +`Array.prototype.map`'s runtime trampoline, `finally` on both paths, nested +`try` with an inner rethrow, a `catch` that itself throws, and 25-level nesting +— every caught value read back after GC-pressure allocation. It plus four +existing exception/rooting fixtures ran under five GC-schedule seeds at +RATE=1 with from-space protection, all byte-identical to node, with +forced_collections=2566 / copying_minors=2566 / moved_objects=37000 confirming +the instrument was live. + +**Follow-up: CI's `cargo-test` job (Linux, debug profile) SIGSEGV'd on this +change** (run 35374727647, job 105594641738) — no `FAILED` line survived +(libtest's stdout is block-buffered under CI), so the crashing test was never +named. Confirmed this PR's own regression: `cargo-test` was clean on #10644, +#10647, #10650, #10651 against the same base. + +Reproduction, in the exact failing configuration +(`RUST_TEST_THREADS=1 cargo test -p perry-runtime --lib`, debug, no +`--release`), across every environment tried, came back clean: +- macOS arm64, the shipped Darwin `perry_thread_local!` path: 4013 passed. +- macOS arm64 with the Darwin `pthread`-TSD path forced off (`hot()` forced + onto the generic `hot_via_tls()` route every non-Darwin-aarch64 target + already uses): 4013 passed, 0 failed — this weakens, though doesn't + disprove, a re-entrancy theory in the generic path itself. +- Linux x86_64 (qemu-emulated Ubuntu VM on the CI-pinned + `nightly-2026-08-20` toolchain, matching `ubuntu-latest`'s triple): ran + clean through 1968+ of ~4013 tests in alphabetical order, well past the + async_hooks region where CI's log cuts off, before a later, unrelated + regex-replace stress test stalled under emulation (not a crash — verified + it runs in well under a second natively; almost certainly a qemu-specific + slowdown, not tied to this change). + +The SIGSEGV's mechanism was never pinned down — see #10709 for the full +writeup, including the two open hypotheses (re-entrancy in `tls_hot.rs`'s +`fill()`, which writes its last field specifically to guard against a +half-filled cache being *used* re-entrantly but does not stop `fill()` being +*called* again re-entrantly; and plain debug-profile stack depth on CI's +default thread, a signature this repo has hit before). Rather than let it +evaporate, it's tracked there and the change is narrowed instead: + +`SHADOW`'s declaration is now cfg-split — +`crate::perry_thread_local!` only under +`all(target_vendor = "apple", target_arch = "aarch64", target_pointer_width = "64")`, +a plain `thread_local!` (the pre-#10619 form) everywhere else. This isn't +only a hedge against the unconfirmed SIGSEGV: `tls_hot.rs`'s own module docs +already say the published-cache shortcut is Darwin-aarch64-specific, and +everywhere else "resolving a thread-local is already a fixed offset and the +extra cache indirection has no demonstrated benefit" — so routing `SHADOW` +through it unconditionally was strictly more work (one extra `HOT` +resolution plus a slot-array indirection) on every other target for a win +that was only ever measured on Darwin. `scripts/thread_local_cold_allowlist.json`'s +`shadow_stack.rs` count goes back to 2 (its pre-#10619 value) to match. + +Re-verified on this Darwin host post-split, differential probe (own builds, +base = this PR's parent 9df5075fbe, arm = this fix, both +`PERRY_NO_AUTO_OPTIMIZE=1`, median of 7, N=20000/40000): try-entry marginal +cost 162.2 -> 152.0 instructions/entry (-10.2, ~6.3%), bare-loop control +~1.0 instructions in both arms (noise floor relative to the ~150-instruction +signal) — the win the original commit measured (-8.1%) survives, because the +fix does not touch the Darwin code path at all. + +Also re-ran: debug and release `cargo test -p perry-runtime --lib` (release: +4010 passed, the same 2 pre-existing `debug_assert!`-gated failures noted +above); the gap fixture plus GC-stress (seeds 1 and 42, +`PERRY_GC_SCHEDULE_RATE=1 PERRY_GC_PROTECT_FROMSPACE=1 +PERRY_GC_VERIFY_EVACUATION=1 PERRY_GC_FROMSPACE_SCAN_ABORT=1`) both +byte-identical to node with non-zero copying minors +(`copying_minors=76 moved_objects=35258`) and a live +`[gc-fromspace-protect] retired_set=#75` line; `cargo fmt --all -- --check`; +`scripts/run_lint_gates.sh` (the only failure is the pre-existing, known-red +public-baseline step); `scripts/check_thread_locals.py`/`--self-test`. diff --git a/crates/perry-runtime/src/gc/roots/shadow_stack.rs b/crates/perry-runtime/src/gc/roots/shadow_stack.rs index a8a8f28d61..6c1b3c4555 100644 --- a/crates/perry-runtime/src/gc/roots/shadow_stack.rs +++ b/crates/perry-runtime/src/gc/roots/shadow_stack.rs @@ -207,11 +207,79 @@ impl ShadowStackState { } } +// [`SHADOW`]'s storage mechanism is cfg-split by platform. Both arms are +// `const`-initialized and drop-free, so neither pays a lazy-init or +// destructor-registration check; the buffer is reserved lazily on the first +// push instead of eagerly at thread start, and released by +// [`ShadowBufferGuard`], on either arm. +// +// # Why `perry_thread_local!` only on Apple aarch64 +// +// Until #10619, `SHADOW` was unconditionally a raw `thread_local!`: every +// `try`/`catch` paid this exact resolution once per entry, through +// `shadow_stack_savepoint()` — and `SHADOW_FRAMES` (the latch that lets most +// other savepoint fields skip their own thread-local read, see +// `crate::exception::savepoints`) is set by the FIRST shadow-frame push +// anywhere in the process, which for real programs is essentially +// "immediately" (any function with a pointer-typed local pushes one, and a +// caught exception's own binding needs a slot). So the latch does not make +// this read rare in practice, and profiling a `try`/`catch` loop +// (`node --experimental-strip-types`-verified against `test-files/`) showed +// it as a genuine leaf `_tlv_get_addr` call, ~6% of total instructions +// retired per non-throwing `try` entry — the one savepoint field NOT already +// routed through `tls_hot`'s cache (`EXCEPTION_STATE`, `CALL_METHOD_DEPTH` +// and the named `runtime_handle_stack`/`temp_roots` fields all were). +// +// But that fast path is itself Darwin-aarch64-specific: `tls_hot.rs`'s own +// module docs say the published-cache shortcut only exists there, and +// everywhere else "resolving a thread-local is already a fixed offset and +// the extra cache indirection has no demonstrated benefit." #10619 shipped +// the swap unconditionally anyway, and CI's `cargo-test` job — Linux, debug +// profile — hit a SIGSEGV that a plain `thread_local!` never produced +// (`cargo-test` run 35374727647, job 105594641738). The cause was never +// isolated to a specific line: the same debug suite ran clean on macOS +// through the Darwin `perry_thread_local!` path, and also ran clean on +// macOS with the Darwin `pthread`-TSD path forced off (so `hot()` took the +// generic `hot_via_tls()` route `perry_thread_local!` uses on every +// non-Darwin-aarch64 target) — only actual Linux/x86_64 reproduced it, and +// under qemu-emulated Linux the debug suite is roughly an order of +// magnitude slower than native, which made pinning the exact crashing test +// impractical here. See #10709 for the open investigation. +// +// Since the −8% `try`-entry win was only ever measured on Darwin +// (`perry_thread_local!`'s whole premise doesn't apply anywhere the direct +// TLS access is already a fixed offset — see `tls_hot.rs`), routing through +// it off that platform was strictly more work for no measured benefit even +// before the SIGSEGV: one more thread-local resolution (`HOT`) plus a +// slot-array indirection, replacing the single direct TLS access a raw +// `thread_local!` already was. So the fast path now ships only where it was +// measured, and every other target keeps the original raw form — which +// also means this file's `thread_local!` count goes back to what it was +// before #10619 (see `scripts/thread_local_cold_allowlist.json`). +#[cfg(all( + target_vendor = "apple", + target_arch = "aarch64", + target_pointer_width = "64" +))] +crate::perry_thread_local! { + /// See the cfg-split rationale above this declaration. + pub(crate) static SHADOW: UnsafeCell = const { + UnsafeCell::new(ShadowStackState { + ptr: std::ptr::null_mut(), + len: 0, + cap: 0, + frame_top: usize::MAX, + }) + }; +} + +#[cfg(not(all( + target_vendor = "apple", + target_arch = "aarch64", + target_pointer_width = "64" +)))] thread_local! { - /// `const`-initialized and drop-free, so the access is a plain TLS address - /// computation with no lazy-init or destructor-registration check. The - /// buffer is reserved lazily on the first push instead of eagerly at thread - /// start, and released by [`ShadowBufferGuard`]. + /// See the cfg-split rationale above this declaration's sibling arm. pub(crate) static SHADOW: UnsafeCell = const { UnsafeCell::new(ShadowStackState { ptr: std::ptr::null_mut(), diff --git a/test-files/test_gap_try_entry_shadow_hot_tls.ts b/test-files/test_gap_try_entry_shadow_hot_tls.ts new file mode 100644 index 0000000000..a2a59b222d --- /dev/null +++ b/test-files/test_gap_try_entry_shadow_hot_tls.ts @@ -0,0 +1,218 @@ +// Adversarial coverage for the `try`-entry perf change that moved `SHADOW` +// (`crates/perry-runtime/src/gc/roots/shadow_stack.rs`) from a raw +// `thread_local!` to `crate::perry_thread_local!` so `try_push_with_kind`'s +// `CatchSavepoint::capture()` stops paying a real `_tlv_get_addr` call on +// every non-throwing `try` entry. The change is a pure storage-mechanism +// swap (same type, same `.with()`/`.try_with()` call sites, same +// const-init/drop-free semantics), but the shadow stack IS the GC's precise +// root set for named locals and expression temporaries, and its savepoint is +// what a throw restores when unwinding skips frame-pop epilogues (#1830). +// This file exercises every shape that depends on that machinery staying +// exactly correct: a throw caught at the entry level, a throw several +// frames deep, a throw crossing a runtime helper callback, `finally` on both +// the normal and throwing paths, a nested `try` with an inner rethrow, a +// `catch` block that itself throws, deep `try` nesting, and — last — a +// non-throwing control so the fast (unchanged-behavior) path is checked too. +// Every caught value is read back AFTER GC-pressure allocation so a +// mis-rooted exception (swept, not just moved) would show up as wrong output +// rather than a crash. + +function churn(n: number): number { + const a: unknown[] = []; + for (let i = 0; i < n; i++) { + a.push({ i, s: "churn" + i, nested: { i } }); + } + return a.length; +} + +// 1. Throw caught at the same level. +function sameLevel(): string { + try { + throw new Error("same-level"); + } catch (e) { + churn(300); + return (e as Error).message; + } +} +console.log("sameLevel", sameLevel()); + +// 2. Throw from a nested function several frames down. +function deep4(): number { + throw new Error("deep4"); +} +function deep3(): number { + return deep4(); +} +function deep2(): number { + return deep3(); +} +function deep1(): number { + return deep2(); +} +function nestedFrames(): string { + try { + deep1(); + return "unreached"; + } catch (e) { + churn(300); + return (e as Error).message; + } +} +console.log("nestedFrames", nestedFrames()); + +// 3. Throw crossing a runtime helper (inside an Array.prototype.map +// callback — the throw unwinds through the runtime's own map trampoline). +function throughMap(): string { + const src = [1, 2, 3, 4, 5]; + try { + src.map((x) => { + if (x === 3) throw new Error("through-map " + x); + return x * 2; + }); + return "unreached"; + } catch (e) { + churn(300); + return (e as Error).message; + } +} +console.log("throughMap", throughMap()); + +// 4a. finally runs on the NORMAL (non-throwing) path. +let finallyNormalRuns = 0; +function finallyNormal(): number { + try { + return 42; + } finally { + finallyNormalRuns++; + } +} +console.log("finallyNormal", finallyNormal(), "ran", finallyNormalRuns); + +// 4b. finally runs on the THROWING path, then the exception still propagates. +let finallyThrowRuns = 0; +function finallyThrow(): number { + try { + throw new Error("finally-throw"); + } finally { + finallyThrowRuns++; + } +} +try { + finallyThrow(); + console.log("finallyThrow (WRONG): did not throw"); +} catch (e) { + churn(300); + console.log("finallyThrow caught", (e as Error).message, "ran", finallyThrowRuns); +} + +// 5. Nested try with an inner rethrow — the outer catch must see the SAME +// (rethrown) exception, with its shadow-rooted fields intact. +function nestedRethrow(): string { + try { + try { + const err: any = new Error("inner"); + err.tag = "rethrow-tag"; + throw err; + } catch (inner) { + churn(200); + throw inner; + } + } catch (outer) { + churn(200); + const e = outer as any; + return e.message + "/" + e.tag; + } +} +console.log("nestedRethrow", nestedRethrow()); + +// 6. A try whose catch itself throws — the ORIGINAL exception is replaced by +// the catch's own throw, and outer code must see the new one. +function catchThrows(): string { + try { + try { + throw new Error("first"); + } catch (e) { + churn(200); + throw new Error("from-catch:" + (e as Error).message); + } + } catch (e2) { + churn(200); + return (e2 as Error).message; + } +} +console.log("catchThrows", catchThrows()); + +// 7. Deep try nesting — an exception thrown at the bottom must unwind +// through every level, running each finally exactly once, in order, and +// land in the outermost catch with the shadow stack balanced afterward. +const finallyOrder: number[] = []; +function deepNestTry(levels: number): string { + if (levels === 0) { + throw new Error("deep-nest-bottom"); + } + try { + return deepNestTry(levels - 1); + } finally { + finallyOrder.push(levels); + } +} +try { + deepNestTry(25); + console.log("deepNestTry (WRONG): did not throw"); +} catch (e) { + churn(300); + console.log( + "deepNestTry caught", + (e as Error).message, + "finally count", + finallyOrder.length, + "finally order ok", + finallyOrder.every((v, i) => v === i + 1) + ); +} + +// A second, unrelated try AFTER the deep unwind: proves try_depth and the +// shadow stack were left exactly where they should be, not off by the +// number of levels just unwound. +function afterDeepUnwind(): string { + try { + throw new Error("after-deep-unwind"); + } catch (e) { + return (e as Error).message; + } +} +console.log("afterDeepUnwind", afterDeepUnwind()); + +// 8. Non-throwing control: the hot (unchanged-behavior) path. A tight loop +// of try/catch blocks that never throw, mixed with ones that do, so the +// fast entry/exit accounting can't drift relative to the slow throw path. +function controlLoop(): number { + let total = 0; + for (let i = 0; i < 200; i++) { + try { + total += i; + if (i % 37 === 0) { + try { + if (i % 74 === 0) throw new Error("control-inner " + i); + total += 1; + } catch (e) { + churn(20); + total -= 1; + } + } + } catch (e) { + total = -1; // never reached + } + } + return total; +} +console.log("controlLoop", controlLoop()); + +// Final sanity: try_depth is back at zero — a fresh top-level try still +// catches correctly after everything above. +try { + throw new Error("final-sanity"); +} catch (e) { + console.log("finalSanity", (e as Error).message); +} +console.log("done");