test(freestanding): probe the parse-depth guard, and correct the soak's stale premise

CHANGELOG.md

@@ -70,7 +70,30 @@ All notable changes to EigenScript are documented here.
   import, and the fix is structural — define it in the host TU. The
   matching extraction for `lint.c`'s 6 blocks (the E003 binding base,
   `--api`, `check_undefined_names`) is tracked separately.
-
+- **The embedded-stack soak states what its 64 KiB budget is, and now
+  tests the depth guard instead of assuming it (#758).** The gate's
+  header called 64 KiB "the EigenOS boot-stack size"; per #758 EigenOS
+  raised its boot stack as part of the #361 fix, so that reading was
+  stale (EigenOS is a separate repo, so that is the issue's measurement,
+  not one this repo's CI can check), and the budget was left tight on
+  purpose — it is a per-AST-level regression tripwire, not a worst-case
+  bound. Both the script header and
+  docs/FREESTANDING.md now say so, and the doc records the policy the
+  stale comment obscured: `PARSE_MAX_DEPTH` / `COMPILE_MAX_DEPTH` are
+  fixed counts sized for a roomy boot stack, so on a constrained profile
+  the binding constraint is the stack (caught by the guard page hosted,
+  the canary bare-metal), not the depth count. Deriving the guards from
+  the live stack was considered and rejected: it is a behavioural change
+  to shipped guards on every profile, in exchange for a bound only as
+  good as its per-level cost estimate, while the soak's value is exactly
+  as a per-level regression detector. Because the guards are therefore
+  *not* reachable under the soak's rlimit — measured, deeply nested
+  source is SIGSEGV 20/20 at 64 KiB and needs ~144 KiB before the guard
+  rejects it reliably — "the guard fires cleanly" had never been tested.
+  The soak binary gains a `depth-guard` mode (source nested past
+  `PARSE_MAX_DEPTH`, run at 256 KiB) and the gate now requires the
+  guard's own diagnostic, a non-crashing exit, and a still-usable state
+  afterwards.
 - **Build variants now coexist: per-variant objdirs with dependency
   tracking (#740).** Every runtime variant (`build`/`full`/`http`/`zlib`/
   `net`/`gfx`/`asan`/`asan-http`/`tsan`/`valgrind`/`poison`) was a single

docs/FREESTANDING.md

@@ -208,7 +208,7 @@ belongs there by construction. The entry point is
 `eigs_embed.h` with **source strings** — `main.c` (the POSIX CLI) is not part
 of the profile.
 
-Two CI gates keep it honest:
+Three CI gates keep it honest:
 
 - **`make freestanding-check`** (`tools/freestanding_check.sh`) — stage 1
   compiles the profile with `-ffreestanding -fno-stack-protector
@@ -222,6 +222,53 @@ Two CI gates keep it honest:
   behind an embed-API harness and proves the core language (functions, lists,
   dicts, f-strings, observer predicates) still runs while the carved surfaces
   fail loudly (undefined variable / profile-specific import error).
+- **`tools/embed_stack_soak.sh`** — builds the same hosted binary behind the
+  REPL-soak harness (`tools/embed_stack_soak_main.c`) and runs it under a
+  clamped stack rlimit, where per-AST-level C stack cost stops being invisible.
+  Two runs: the REPL soak at 64 KiB, and a depth-guard probe at 256 KiB. What
+  those two numbers mean is the next section.
+
+### Stack budget and the depth guards
+
+The soak's 64 KiB rlimit is a **per-level regression budget, not a stack
+size anyone ships on.** It is deliberately tighter than the real target: its
+job is to make a jump in stack cost per AST level fail loudly (that is what
+caught #361 — a by-value `Compiler` costing ~12.7 KiB of C stack per AST
+level), and a budget with little slack detects that fastest. Read it as a
+tripwire, not as a worst-case bound, and do not derive anything from it. It
+used to be described as "the EigenOS boot-stack size"; per #758 the EigenOS
+boot stack was raised (to 1 MiB) as part of the #361 fix, so that reading is
+stale. EigenOS lives in its own repository, so that figure is #758's
+measurement rather than one this repo's CI can check — which is exactly why
+the soak budget should not be described in terms of it.
+
+**The depth guards are sized for the roomy boot stack, and the constrained
+profile relies on the stack canary, not on the guards.** `PARSE_MAX_DEPTH`
+(256, `src/parser.c`) and `COMPILE_MAX_DEPTH` (128, `src/compiler.c`) bound
+C recursion so that deeply nested source — `eval` of untrusted input, say —
+is rejected instead of running off the stack. They are fixed counts, chosen
+against a stack with room, and they are **not** reachable on an
+arbitrarily small stack: measured on the soak harness, the 64 KiB run
+SIGSEGVs on deeply nested source long before `PARSE_MAX_DEPTH` trips, and
+the guard needs roughly 144 KiB to reject cleanly and reliably. So on a
+genuinely tiny stack the guard is not the binding constraint — the stack
+is, and an embedder there is protected by the platform's overflow detection,
+not by a depth count: hosted, the guard page (an instant SIGSEGV); bare
+metal, the boot stack's own canary word (#758). Note that is a canary the
+*kernel* places, not a compiler-inserted one — the profile compiles with
+`-fno-stack-protector`, as the gate above shows.
+
+That is a deliberate choice, not an oversight. The alternative — deriving
+the guards from the stack at runtime — is a behavioural change to shipped
+guards on every profile, in exchange for a bound that is still only as good
+as its estimate of per-level cost. Anyone lowering the soak budget, or
+relying on a depth guard on a constrained target, needs to know which of the
+two is actually holding.
+
+Both halves are tested rather than assumed: run 2 of the soak feeds the
+parser source nested past `PARSE_MAX_DEPTH` at a stack with the headroom to
+reach it, and requires the guard's own diagnostic plus a still-usable state
+afterwards — a SIGSEGV there fails the gate.
 
 ## Mini-libc/libm (EXISTS since 2026-07-02)
 

tools/embed_stack_soak.sh

@@ -1,14 +1,35 @@
 #!/usr/bin/env bash
 # Embedded-stack soak gate: build the freestanding-profile runtime hosted
 # (as freestanding_smoke.sh does) with the REPL-soak harness, then run it
-# with the stack rlimit clamped to 64 KiB — the EigenOS boot-stack size.
+# twice — once with the stack rlimit clamped to 64 KiB, once roomier.
 #
 # Hosted 8 MiB stacks hide per-AST-level C stack cost completely; this gate
 # is what catches a regression like the by-value Compiler in
 # compile_node_inner (~12.7 KiB of stack PER AST LEVEL — a 5-deep AST
-# overflowed EigenOS's 64 KiB boot stack and silently trampled .bss, the
-# mn-repl "#UD heisenbug"). Under the rlimit, an overflow is an instant
-# SIGSEGV instead of a layout-dependent delayed fault.
+# overflowed the 64 KiB boot stack EigenOS used at the time and silently
+# trampled .bss, the mn-repl "#UD heisenbug"). Under the rlimit, an overflow
+# is an instant SIGSEGV instead of a layout-dependent delayed fault.
+#
+# Run 1 — 64 KiB, the REPL soak. That 64 KiB is a deliberately tight
+# PER-LEVEL REGRESSION BUDGET, not a measured EigenOS stack size (#758):
+# per that issue EigenOS raised its boot stack as part of the #361 fix, and
+# this number was left where it was on purpose. A budget below the real
+# target is the right design for the job this gate does — detect a jump in
+# stack cost per AST level, the way it caught #361 — but it is NOT a
+# worst-case bound, and the depth guards are not sized against it (that
+# policy: docs/FREESTANDING.md). Measured on this harness: the soak
+# passes 20/20 down to a 32 KiB rlimit, is flaky at 28 and dead at 20, so
+# roughly half the 64 KiB budget goes to startup + parse + VM before any
+# recursive descent, and the rest survives only because a REPL soak runs
+# shallow programs.
+#
+# Run 2 — 256 KiB, the depth-guard probe (#758). Deeply nested source, at a
+# stack where the guard is genuinely the binding constraint, asserting the
+# guard's clean diagnostic instead of a SIGSEGV. At 64 KiB the same input is
+# SIGSEGV 20/20 — the guard cannot be reached there at all, which is why
+# "the guard fires cleanly" was assumed and never tested. 256 KiB is chosen
+# with margin: measured on this harness the guard trips cleanly 20/20 from
+# 144 KiB, 18/20 at 128 KiB, and 0/20 at 112 KiB.
 set -euo pipefail
 cd "$(dirname "$0")/.."
 
@@ -38,3 +59,23 @@ else
     printf '%s\n' "$out" | tail -5
     exit 1
 fi
+
+set +e
+gout=$(bash -c "ulimit -s 256; '$BUILD/embed_stack_soak' depth-guard" 2>&1); grc=$?
+set -e
+
+# All three conditions matter: rc 0 rules out the SIGSEGV (139) this is here to
+# distinguish from; the diagnostic proves it was the depth guard that rejected
+# the source and not some unrelated parse failure; the marker proves the
+# runtime was still usable afterwards.
+if [ "$grc" = 0 ] \
+   && printf '%s' "$gout" | grep -q "nesting too deep" \
+   && printf '%s' "$gout" | grep -q "embed_stack_soak: depth-guard OK"; then
+    echo "  PASS: parse-depth guard rejects cleanly in a 256 KiB stack"
+else
+    echo "  FAIL: depth-guard probe under 256 KiB stack (rc=$grc)"
+    # Drop the cascade of follow-on "expected ')'" lines the rejected source
+    # produces, so the tail shows the diagnostic that matters.
+    printf '%s\n' "$gout" | grep -v "expected ')'" | tail -5
+    exit 1
+fi

tools/embed_stack_soak_main.c

@@ -1,6 +1,12 @@
 /* embed_stack_soak_main.c — hosted twin of the EigenOS boot-to-REPL session,
- * run by tools/embed_stack_soak.sh under a 64 KiB stack rlimit (the EigenOS
- * boot-stack size).
+ * run by tools/embed_stack_soak.sh under a 64 KiB stack rlimit (a deliberately
+ * tight per-level regression budget, NOT a measured EigenOS stack size — see
+ * the header of that script and docs/FREESTANDING.md).
+ *
+ * Two modes, both driven by the script:
+ *   (no argument)  the REPL soak — shallow programs, tiny stack.
+ *   `depth-guard`  the parse-depth guard probe — deep source, roomier stack
+ *                  (see guard_probe below).
  *
  * One eigs_open, then a long sequence of eigs_eval_string calls on the SAME
  * state — the accumulation pattern the test suite never exercises (every
@@ -33,10 +39,79 @@ static void eval_line(const char *src) {
     if (v) eigs_value_release(v);
 }
 
-int main(void) {
+/* --- the depth-guard probe (#758) ----------------------------------
+ *
+ * The REPL soak above only ever runs shallow programs, so it proves the
+ * per-level stack cost is small — it never reaches a depth guard, and under
+ * the soak's 64 KiB rlimit it cannot: the C stack runs out long before
+ * PARSE_MAX_DEPTH does (this probe's input is SIGSEGV 20/20 there). "The
+ * guard fires cleanly" was therefore assumed, not tested.
+ *
+ * This mode tests it, at a stack size where the guard IS the binding
+ * constraint. GUARD_LEVELS is past PARSE_MAX_DEPTH (256) counted either as
+ * source levels or as parser recursion levels, so the guard trips whichever
+ * way that counter is bumped — measured, it is bumped twice per parenthesis
+ * level (parse_expression and parse_unary each bump), so 256 parser levels
+ * is the 129th '(' and the guard rejects from 128 source levels up. The
+ * assertions:
+ *
+ *   1. the eval is REJECTED (NULL) rather than crashing the process — an
+ *      expression that parsed would evaluate to 1;
+ *   2. the state still works afterwards, i.e. the guard unwound cleanly
+ *      instead of leaving the runtime wedged;
+ *
+ * and the script additionally requires the guard's own diagnostic on stderr,
+ * so a rejection for some unrelated reason cannot pass this vacuously.
+ *
+ * The nested source lives in .bss on purpose: building it on the stack would
+ * charge the probe for its own input.
+ */
+#define GUARD_LEVELS 300
+static char guard_src[2 * GUARD_LEVELS + 2];
+
+static int guard_probe(void) {
+    int n = 0;
+    for (int i = 0; i < GUARD_LEVELS; i++) guard_src[n++] = '(';
+    guard_src[n++] = '1';
+    for (int i = 0; i < GUARD_LEVELS; i++) guard_src[n++] = ')';
+    guard_src[n] = '\0';
+
+    EigsValue *v = eigs_eval_string(guard_src);
+    if (v) {
+        fprintf(stderr, "depth-guard: %d nested levels were ACCEPTED — the "
+                        "parse-depth guard did not fire\n", GUARD_LEVELS);
+        eigs_value_release(v);
+        return 1;
+    }
+    /* A parse rejection returns NULL without setting the embed error slot
+     * (eigs_eval_string bails before the VM runs), so this clears nothing
+     * today — it is here so the probe stays correct if that ever changes. */
+    eigs_clear_error();
+
+    /* The guard is only "clean" if the runtime survives it. */
+    v = eigs_eval_string("6 * 7");
+    if (!v || eigs_value_as_num(v) != 42.0) {
+        fprintf(stderr, "depth-guard: state unusable after the guard fired\n");
+        if (v) eigs_value_release(v);
+        return 1;
+    }
+    eigs_value_release(v);
+
+    printf("embed_stack_soak: depth-guard OK (%d levels rejected, state live)\n",
+           GUARD_LEVELS);
+    return 0;
+}
+
+int main(int argc, char **argv) {
     EigsState *st = eigs_open();
     if (!st) { fprintf(stderr, "eigs_open failed\n"); return 2; }
 
+    if (argc > 1 && strcmp(argv[1], "depth-guard") == 0) {
+        int rc = guard_probe();
+        eigs_close(st);
+        return rc;
+    }
+
     /* --- the EigenOS M5 demo pair --- */
     eval_line("6 * 7");
     eval_line("n is 10\n"