Transpiler cleanup final

.github/workflows/transpile-pure.yml

@@ -0,0 +1,131 @@
+name: Transpile Pure
+
+# Verifies that this branch produces byte-identical .zig output for every
+# .cht file in `transpile-tests/` compared to the merge base. Triggered
+# by including `#TRANSPILE_PURE` in the PR description.
+#
+# Use this when you've made a refactor that should NOT change the
+# transpiler's output — e.g., the schema-extraction refactors in #2 or
+# the FnSig/PipelineSite extractions in #4/#3. Editing the PR body to
+# add or remove the marker re-triggers the check.
+#
+# Scope: `transpile-tests/` only. The 408 standalone .cht files there
+# are designed to round-trip through the transpiler without external
+# package paths or runtime args. `examples/` is intentionally excluded
+# — many examples in that tree are driven by bc_emitter or need
+# specific --pkg flags that emit-zig can't infer.
+
+on:
+  pull_request:
+    types: [opened, synchronize, edited, reopened]
+
+concurrency:
+  group: transpile-pure-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  RUBY_VERSION: "3.2"
+
+jobs:
+  diff-zig:
+    name: Byte-diff generated .zig vs merge base
+    runs-on: ubuntu-latest
+    if: contains(github.event.pull_request.body, '#TRANSPILE_PURE')
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # Need the full history so we can find and check out the merge
+          # base. Default `fetch-depth: 1` would fail on `git checkout`.
+          fetch-depth: 0
+
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ env.RUBY_VERSION }}
+          bundler-cache: true
+
+      - name: Compute merge base
+        id: base
+        run: |
+          BASE_SHA=$(git merge-base HEAD origin/${{ github.event.pull_request.base.ref }})
+          echo "sha=$BASE_SHA" >> $GITHUB_OUTPUT
+          echo "Comparing branch HEAD ($(git rev-parse HEAD)) against merge base $BASE_SHA"
+
+      - name: Emit .zig from branch HEAD
+        run: |
+          mkdir -p /tmp/zig-branch
+          ./clear emit-zig transpile-tests -o /tmp/zig-branch || true
+
+      - name: Stage branch tooling for merge-base run
+        # The `clear emit-zig` subcommand may have been introduced or
+        # patched on this branch. Stash the branch's `clear` script and
+        # `tools/` so we use the SAME emitter against both checkouts —
+        # any diff is then purely a src/ change, not an emitter change.
+        run: |
+          cp clear /tmp/clear-branch
+          cp -r tools /tmp/tools-branch
+
+      - name: Check out merge base
+        run: |
+          git checkout ${{ steps.base.outputs.sha }}
+
+      - name: Restore branch tooling
+        run: |
+          cp /tmp/clear-branch ./clear
+          chmod +x ./clear
+          rm -rf tools && cp -r /tmp/tools-branch tools
+
+      - name: Reinstall gems for merge base
+        run: bundle install
+
+      - name: Emit .zig from merge base
+        run: |
+          mkdir -p /tmp/zig-base
+          ./clear emit-zig transpile-tests -o /tmp/zig-base || true
+
+      - name: Normalize auto-generated counter identifiers
+        # The transpiler uses node.object_id.abs to disambiguate nested-WITH
+        # guard variables, MATCH binding aliases, etc. — the IDs shift
+        # whenever Ruby allocation order changes (refactors that add/remove
+        # objects), even though the generated Zig is functionally identical.
+        # tools/normalize_zig.rb maps each unique counter to a stable
+        # first-occurrence index per file (`__c_guard_3220` -> `__c_guard_N1`)
+        # so the diff surfaces real semantic changes, not allocation drift.
+        # See the script's docs for collision-safety reasoning.
+        run: |
+          ruby tools/normalize_zig.rb /tmp/zig-base /tmp/zig-branch
+
+      - name: Diff trees
+        run: |
+          # `diff -r` exits 0 (no diff) / 1 (diffs) / 2 (error). We want
+          # to fail the job only on 1, so allow-fail and inspect.
+          set +e
+          diff -r --brief /tmp/zig-base /tmp/zig-branch > /tmp/diff.txt
+          status=$?
+          set -e
+
+          if [ "$status" = "0" ]; then
+            echo "::notice::Output is byte-identical between merge base and branch HEAD."
+            exit 0
+          fi
+
+          echo "::error::Generated .zig differs between merge base and branch HEAD."
+          echo ""
+          echo "=== Summary of changed files ==="
+          cat /tmp/diff.txt
+          echo ""
+          echo "=== Full unified diffs (first 500 lines per file) ==="
+          # Iterate over differing files and print a bounded unified diff.
+          while IFS= read -r line; do
+            # Lines look like: "Files /tmp/zig-base/X and /tmp/zig-branch/X differ"
+            #                 or: "Only in /tmp/zig-...: <file>"
+            if [[ "$line" == Files* ]]; then
+              base_file=$(echo "$line" | awk '{print $2}')
+              branch_file=$(echo "$line" | awk '{print $4}')
+              echo "----- $base_file vs $branch_file -----"
+              diff -u "$base_file" "$branch_file" | head -500
+              echo ""
+            else
+              echo "$line"
+            fi
+          done < /tmp/diff.txt
+          exit 1

.gitignore

@@ -23,11 +23,13 @@
 !/docs/
 !/examples/
 !/manifesto/
+!/sorbet/
 !/spec/
 !/src/
 !/stdlib/
 !/syntaxes/
 !/testdata/
+!/tools/
 !/transpile-tests/
 !/zig/
 

Gemfile

@@ -16,5 +16,15 @@ group :development do
   gem 'simplecov', require: false
   # Cobertura XML output for Codecov / coveralls / GitLab integration
   gem 'simplecov-cobertura', require: false
+
+  # Gradual typing — staged adoption per the self-host prep tracker
+  # (TODO.md "Self-host preparation" P1 / tasks #10 + #20). `sorbet`
+  # provides the static type checker (`srb tc`); `sorbet-runtime` is
+  # the inline `T::Sig` API (no runtime cost when not invoked because
+  # files start at `# typed: false`); `tapioca` generates RBI files
+  # for our gems so Sorbet sees their public APIs.
+  gem 'sorbet', require: false
+  gem 'sorbet-runtime'
+  gem 'tapioca', require: false
 end
 

Gemfile.lock

@@ -8,6 +8,7 @@ GEM
       descendants_tracker (~> 0.0.4)
       ice_nine (~> 0.11.0)
       thread_safe (~> 0.3, >= 0.3.1)
+    benchmark (0.5.0)
     bigdecimal (4.1.2)
     byebug (12.0.0)
     childprocess (5.1.0)
@@ -69,7 +70,9 @@ GEM
       logger (~> 1.6)
     logger (1.7.0)
     msgpack (1.8.0)
+    netrc (0.11.0)
     ostruct (0.6.3)
+    parallel (1.28.0)
     parallel_rspec (3.0.0)
       rake (> 10.0)
       rspec
@@ -82,12 +85,20 @@ GEM
     racc (1.8.1)
     rainbow (3.1.1)
     rake (13.3.1)
+    rbi (0.3.10)
+      prism (~> 1.0)
+      rbs (>= 4.0.1)
+    rbs (4.0.2)
+      logger
+      prism (>= 1.6.0)
+      tsort
     reek (6.5.0)
       dry-schema (~> 1.13)
       logger (~> 1.6)
       parser (~> 3.3.0)
       rainbow (>= 2.0, < 4.0)
       rexml (~> 3.1)
+    require-hooks (0.4.0)
     rexml (3.4.4)
     rspec (3.13.2)
       rspec-core (~> 3.13.0)
@@ -119,6 +130,7 @@ GEM
       simplecov (>= 0.22.0)
       tty-which (~> 0.5.0)
       virtus (~> 2.0)
+    rubydex (0.2.1-x86_64-linux)
     sexp_processor (4.17.5)
     simplecov (0.22.0)
       docile (~> 1.1)
@@ -129,7 +141,36 @@ GEM
       simplecov (~> 0.19)
     simplecov-html (0.13.2)
     simplecov_json_formatter (0.1.4)
+    sorbet (0.6.13196)
+      sorbet-static (= 0.6.13196)
+    sorbet-runtime (0.6.13196)
+    sorbet-static (0.6.13196-x86_64-linux)
+    sorbet-static-and-runtime (0.6.13196)
+      sorbet (= 0.6.13196)
+      sorbet-runtime (= 0.6.13196)
+    spoom (1.7.13)
+      erubi (>= 1.10.0)
+      prism (>= 0.28.0)
+      rbi (>= 0.3.3)
+      rbs (>= 4.0.0.dev.5)
+      rexml (>= 3.2.6)
+      sorbet-static-and-runtime (>= 0.5.10187)
+      thor (>= 0.19.2)
+    tapioca (0.19.1)
+      benchmark
+      bundler (>= 2.2.25)
+      netrc (>= 0.11.0)
+      parallel (>= 1.21.0)
+      rbi (>= 0.3.7)
+      require-hooks (>= 0.2.2)
+      rubydex (>= 0.1.0.beta10)
+      sorbet-static-and-runtime (>= 0.6.12698)
+      spoom (>= 1.7.9)
+      thor (>= 1.2.0)
+      tsort
+    thor (1.5.0)
     thread_safe (0.3.6)
+    tsort (0.2.0)
     tty-which (0.5.0)
     virtus (2.0.0)
       axiom-types (~> 0.1)
@@ -138,7 +179,6 @@ GEM
     zeitwerk (2.7.5)
 
 PLATFORMS
-  ruby
   x86_64-linux-gnu
 
 DEPENDENCIES
@@ -153,6 +193,9 @@ DEPENDENCIES
   rubycritic
   simplecov
   simplecov-cobertura
+  sorbet
+  sorbet-runtime
+  tapioca
 
 BUNDLED WITH
    2.7.2

TODO.md

@@ -74,6 +74,58 @@ Vision is clear: Gradual Typing (Ruby/Python) -> Typed (Rust) -> STRICT Typed (H
 
 `./clear doctor ` + profiling on replay can take you ~95% of the way from an untyped correct app to ADA-level safety and HFT-C speed automatically.
 
+## Self-host preparation (`src/` Ruby cleanup)
+
+Source signal: 70 .rb files / ~57k LOC in `src/`. Anti-pattern census from grep + `reek` + `debride`:
+
+  - 1409 `is_a?` checks (141 `is_a?(Hash)` = hash-as-struct; 99 `is_a?(Array)` = array-as-tuple; 67 mixed `is_a?(String)`/`is_a?(Symbol)`)
+  - 635 `respond_to?` (duck-typing escape hatches; 289 surface as reek `ManualDispatch`)
+  - 228 `.nil?` checks (164 surface as reek `NilCheck`)
+  - 64 reek `DataClump` clusters (top: PipelineHost/PipelineGenerator with `(list_node, smooth_node)` through 18 methods, Formatter::Emitter `(start, toks)` through 17, etc.)
+  - 91 `ControlParameter` + 54 `BooleanParameter` (flag args switching behavior)
+  - 313 `Struct.new` already in use → good baseline
+
+Each P0 item is mechanical and self-verifying via existing tests; doing them BEFORE typing lands keeps the typing pass cheap. Each P1 item depends on at least one P0 item.
+
+### P0 — must complete before declaring "ready to type"
+
+ - [ ] Extract top hash-as-struct schemas (target: 141 `is_a?(Hash)` → ~40): `Schema` for the `schema.is_a?(Hash) && schema[:kind] == :union` pattern in promotion_plan.rb / type.rb / annotator.rb / mir_lowering.rb (clustered: 36+25+24+14 sites). **Subsumes String/Symbol normalization:** ~15 of the 65 `is_a?(String|Symbol)` sites are schema mixed-key filters (`k.is_a?(Symbol)` rejecting `:kind`/`:field_defaults` metadata vs String field names) — typed Schema (`kind: Symbol`, `field_names: Array[String]`) kills them at the root. The other ~50 sites are either correct primitive-rejection in AST walkers or syntactic dispatch on `node.name`, NOT normalization.
+ - [ ] `PipelineSite`/`PipelineFrame` struct: collapse the `(id, options, rt_name, workers_code, list_node, smooth_node, stream_node, conc_op, lhs)` clump that DataClump-flags 25+ methods across PipelineHost / PipelineGenerator / PipelineRewriter
+ - [ ] `Formatter::Emitter::EmitterState` struct: collapse the `(toks, start, out, pc, po, arrow_idx)` clump (17-method DataClump cluster, single-class, smallest blast radius — good warm-up extraction)
+ - [ ] Tighten nilable fields where field is always set in practice: top targets `sync` (13), `node` (7), `out` (6), `schema` (5), `entry`/`code`/`expr_type`/`layout`/`mir`/`name`/`ownership`/`path`/`rl`/`rt`/`syn` (3 each); replace `.nil?` guards with construction-time invariants
+
+### P1 — high-leverage, depends on P0
+
+ - [ ] AST dispatch via Ruby 3 pattern matching: convert `if node.is_a?(AST::Identifier) ... elsif node.is_a?(AST::BinaryOp) ...` chains (~600 sites) to `case node in AST::Identifier(...)`. Maps directly to CLEAR union-tag dispatch later.
+ - [ ] Array-as-tuple → `Data.define(:kind, :value, :loc)` etc. (99 `is_a?(Array)` sites). Token tuples in lexer/parser, MIR ops with positional payload, multi-return method results.
+ - [ ] `MIRPass::WalkState` + `OwnershipDataflow::DataflowStep` struct extractions (`(bindings, result, stmt)` 9× / `(node, state, consumed)` 6× DataClump clusters)
+ - [ ] `respond_to?` purge (635 → ~50): each site is one of three things — missing trait (introduce module/interface), wrong type (fix caller), genuine duck-typing on heterogeneous external input (keep, document)
+ - [ ] Add Sorbet (or rbs+steep) to Gemfile; bootstrap with `tapioca init`; flip files from `# typed: false` to `# typed: true` per file. Acceptance gate per file: `# typed: strict` before it's a self-host candidate.
+
+### P2 — structural cleanup
+
+ - [ ] Eliminate `ControlParameter` (91) + `BooleanParameter` (54): every flag arg should be a separate method or a sealed enum input
+ - [ ] `LongParameterList` audit (135): residual after P0 PipelineSite / EmitterState extractions; remaining are either missing structs or genuinely 5+ orthogonal inputs
+ - [ ] Method-return-type uniformity audit: methods that return `value | nil` use Optional; methods that return different value types in different paths split or use sealed union
+ - [ ] Custom Rubocop cops as CI signal: `Project/HashAsStruct` (flags `is_a?(Hash)` near `[:key]` access), `Project/UnnecessaryNilCheck` (flags `.nil?` on locals never assigned nil), `Project/SymbolOrString` (flags the 2-arm dispatch). ~50 lines each.
+ - [ ] FsmTransform::RecursiveSplitter `(after_idx, builder, lowering)` 12-method DataClump cluster
+
+### P3 — nice-to-have / cosmetic
+
+ - [ ] `UncommunicativeVariableName` cleanup pass (890 reek hits) — names like `n`, `t`, `p` in non-trivial scopes
+ - [ ] `TooManyStatements` method splits (993 hits) — only where NOT intrinsic AST/MIR dispatch (those should stay as one method)
+ - [ ] `flog` triage: top-30 by score, refactor only the ones that aren't dispatch-shaped (rest are accidentally-flagged intrinsic complexity)
+ - [ ] `DuplicateMethodCall` cleanup (3637 reek hits): mostly local-let extractions, low-leverage but reduces noise
+
+### Acceptance criteria for "this file is self-host ready"
+
+A file passes when ALL of:
+ - [ ] `# typed: strict` (Sorbet or equivalent rbs)
+ - [ ] Zero `is_a?(Hash)` calls
+ - [ ] Zero `respond_to?` calls (or each remaining one is documented as genuine external-input duck-typing)
+ - [ ] Zero unguarded `.nil?` checks (every survivor is on a typed-nilable field, locally justified)
+ - [ ] Reek output for that file shows no `DataClump`, `LongParameterList`, `ControlParameter`, or `BooleanParameter` warnings
+
 ## v0.4 (December 25)
 
  - [ ] EXTREME STRICT compilation pt 1: en route to ADA-levels of safety (just blocking compilation where required, ensuring the syntax is forward compatible)

docs/agents/benchmark-fix-findings.md

@@ -0,0 +1,71 @@
+# Benchmark-Fix Branch: Tech Debt and Bug Findings
+
+This document summarizes the technical debt, architectural violations, potential bugs, and partial implementations identified in the `benchmark-fix` branch (relative to `master`).
+
+## 1. Architectural Violations
+
+### MIREmitter Type Inspection (Tech Debt)
+- **File:** `src/mir/mir_emitter.rb`
+- **Issue:** The emitter uses regex to parse Zig type strings (e.g., `node.zig_type[/ArrayListUnmanaged\((.+)\)/, 1]`) to determine element types.
+- **Violation:** Violates the "Dumb Transpiler" rule (Role 3 in `GEMINI.md`). The emitter should not perform semantic analysis or string parsing to make code-gen decisions.
+- **Recommendation:** Explicitly pass the element type in the MIR node (`MIR::EscapePromote`, `MIR::ContainerInit`, etc.).
+
+### Transpiler Semantic Decisions
+- **File:** `src/backends/pipeline_generator.rb`
+- **Issue:** Contains logic to determine cleanup requirements (e.g., `src_needs_cleanup = list_node.is_a?(AST::MethodCall) && ...`).
+- **Violation:** All allocator and cleanup decisions must reside in Pass 2 (`MIRLowering`) per `GEMINI.md`. Pass 4 should be a passive consumer of these decisions.
+- **Recommendation:** Move this logic into `MIRLowering` and communicate it via MIR nodes.
+
+### Legacy Pipeline Path
+- **File:** `src/mir/mir_lowering.rb` (`lower_smooth`)
+- **Issue:** Complex pipeline operators still fall back to `pipeline_legacy_host`, which generates `RawZig`.
+- **Violation:** `RawZig` is an unsafe escape hatch that bypasses `MIRChecker` (INV-12). Chained pipelines using this path are unverified for memory safety.
+- **Recommendation:** Complete the migration of all pipeline operators to structural MIR.
+
+## 2. Potential Bugs and Safety Risks
+
+### SmartEventFd.notify Optimization
+- **File:** `zig/runtime/scheduler.zig`
+- **Issue:** Re-introduces an optimization to skip the `write()` syscall if the target is not `WakeParked`.
+- **Risk:** A previous version of this optimization was removed because it caused deadlocks. While `prepareSleep` now uses atomics, this is a high-risk change for a core synchronization primitive.
+- **Recommendation:** Verify with aggressive Hammer and VOPR tests that no wake-up signals are lost.
+
+### Incomplete FsmTask Sequence Bumping
+- **File:** `zig/runtime/scheduler.zig`
+- **Issue:** `FsmTask.seq` (the park/wake transition counter) is only bumped on lock timeouts. It is NOT bumped during regular `WaitForIO` or `WaitForLock` transitions.
+- **Risk:** This breaks the `detectCycleFsm` protocol, which relies on `seq` to validate that a task hasn't been recycled or moved during a cross-scheduler chain walk.
+- **Recommendation:** Bump `task.seq` in `submitFsmResume` and `submitFsmSpawn` (or in the scheduler's FSM dequeue/dispatch loop).
+
+### FSM Liveness for IoSuspend Async Access
+- **File:** `src/mir/fsm_transform/liveness.rb`
+- **Issue:** For `IoSuspend` (e.g., async `read`), the kernel may access arguments after the FSM body has yielded.
+- **Risk:** If those arguments (like a buffer) are stack-allocated, they must be promoted to the FSM `ctx` struct. The current analysis charges these reads to the "next" segment, which should trigger promotion, but this needs rigorous validation to ensure no Use-After-Free (UAF) windows exist.
+- **Recommendation:** Add explicit tests for async IO with stack-allocated buffers in FSM tasks.
+
+## 3. Partial Implementations and Stubs
+
+### Missing detectCycleFsm
+- **File:** `zig/runtime/scheduler.zig`, `zig/runtime/fsm.zig`
+- **Issue:** References to `detectCycleFsm` and the UAF-safe "slab-pin" protocol exist, but the actual cycle detection logic for FSM tasks is missing.
+- **Status:** Partial implementation.
+
+### FSM Next-Bind RawZig
+- **File:** `src/mir/fsm_transform/suspend_resolvers.rb`
+- **Issue:** `resolve_next` uses a block of `RawZig` to handle `fsm_next_bind` logic.
+- **Status:** A "small follow-up" is noted to migrate this to structured MIR (`IfStmt` + `ReturnStmt`).
+
+### Disabled Liveness Safety Check
+- **File:** `src/mir/fsm_transform/liveness.rb`
+- **Issue:** `return unless uses_by_seg.key?(next_idx) || true` (line 143).
+- **Status:** The `|| true` effectively disables the check. Likely a debug leftover.
+
+## 4. Testing Gaps
+
+### FSM Stealing Stress
+- **Issue:** `fsm-steal-test.zig` covers basic functionality, but interactions between stealing and status transitions (Blocked/Ready) under heavy load need more coverage.
+- **Recommendation:** Implement a Hammer test specifically for FSM work-stealing with oversubscribed workers and high IO/Lock contention.
+
+### ArrayListUnmanaged String Matching
+- **Issue:** The `MIREmitter` logic for `ArrayListUnmanaged` relies on string-pattern matching on `zig_type`.
+- **Risk:** Fragile; could fail on complex nested types or if Zig's internal type naming changes.
+- **Recommendation:** Add unit tests in `spec/mir_emitter_spec.rb` covering complex nested collections to verify the regex robustness.