foreign: pass Prologos lambdas across the FFI as live closures#35
Draft
kumavis wants to merge 3 commits into
Draft
foreign: pass Prologos lambdas across the FFI as live closures#35kumavis wants to merge 3 commits into
kumavis wants to merge 3 commits into
Conversation
Extends the foreign marshaller to recognise Pi (function) types in
argument positions. When a foreign call passes a Prologos value of
function type, the marshaller wraps it as a Racket procedure that
marshals Racket-side arguments back into the IR, applies the lambda,
reduces via `nf`, and marshals the result back out.
The marshalling spec becomes recursive: each arg/ret position is either
a base-type symbol (existing behaviour) or `(cons 'fn parsed-type)` for
a function-typed slot. Backward-compatible — existing parsed types use
the same symbol-only shape they did before.
`foreign-type-tokens->sexp` now recurses into list-shaped tokens so a
sub-bracket form like `[Nat -> Nat]` parses as `(-> Nat Nat)` rather
than as a function application of `Nat`.
Removed the dead `(require "foreign.rkt")` from reduction.rkt to
permit foreign.rkt → reduction.rkt as the natural import direction
(needed `nf` in the wrapper).
Unblocks items 1 & 2 in 2026-04-28_ETPROP_PITFALLS.md (propagator
fire-fns and cell merge-fns no longer required to live in Racket).
The ETPROP doc is updated; eigentrust example is unchanged.
See docs/tracking/2026-04-28_FFI_LAMBDA_PASSING.md for the design,
mantra-alignment self-assessment (off-network scaffolding with
retirement plan toward on-network propagator-native callbacks),
and SRE lattice lens.
Tests:
- tests/test-foreign-callback.rkt: 10 new (parse-foreign-type for
function args, wrapper as procedure, integration via foreign decl).
- lib/examples/lambda-ffi-test.prologos: Level 3 acceptance, 4
end-to-end cases including Posit32 callback and two-callback call.
- tests/test-foreign.rkt, test-foreign-block.rkt, test-pvec.rkt,
test-io-opaque-01.rkt: pre-existing pass, no regression.
Builds on the lambda-FFI track (PR #35): the affine combination out[j] := bias[j] + Σ_i weight[j][i] · prev[i] is now a Prologos `defn affine-step` lambda passed across the FFI on each `net-add-prop` install. The Racket shim's broadcast item-fn invokes it once per peer per fire. What stays in Racket (the truly irreducible core): * cell-value carrier (gen-tagged immutable Posit32 vector) * the propagator's fire-fn (a Racket closure that reads input cell, invokes the Prologos kernel via the FFI bridge, and writes the output cell — pure plumbing, NO algorithmic content) * FFI marshalling glue (cons/nil chain walking, Posit32 bit-pattern extraction, per-row IR list construction) * handle/cell registries What's now in Prologos (the entire algorithm): * matrix transpose, decay scaling, bias computation * the per-row affine kernel itself * iteration driver * initial-zero vector `net-add-prop` is purpose-AGNOSTIC — it's a generic broadcast affine propagator wired to a domain-specific Prologos kernel. The same shim would serve any algorithm whose per-row update is `bias + Σ w·x`. CI integration: * New `tests/test-eigentrust.rkt` runs the .prologos file via `process-file` and asserts the converged scores match the Python reference within 1e-2 — auto-picked up by `tools/run-affected-tests --all` (the CI test command). * 5 power iterations lands within ~6e-3 of the steady-state eigenvector; runs in ~16s, under the test runner's 30s first-result guard. New pitfalls captured (docs/tracking/2026-04-28_ETPROP_PITFALLS.md): * #0 — FFI-call AST caching collapses identical side-effecting calls onto the same physical cell. Fixed via a "freshness tag" arg on `net-new-cell`. Surfaced when distinct layer cells turned out to be the same physical cell (self-loop). * #16 — FFI-callback overhead per fire is the bottleneck. Each kernel invocation runs `nf` on the lambda body. Documented as expected scaffolding cost of the off-network FFI bridge; future work is propagator-native callbacks via cell subscription. Test results: * tests/test-eigentrust.rkt: 2/2 pass (15.4s) * regression set (foreign / foreign-block / pvec / foreign-callback / eigentrust): 111/111 pass (21s) * Final converged scores match Python reference within 1e-2: [0.0712, 0.4288, 0.0712, 0.4288] (steady state ≈ [0.0652, 0.4348])
Merges the FFI marshalling work from `claude/add-posits-ints-ffi-oM6pF` (commits b9a6461 + a4c2004) into the lambda-passing track. The two features are orthogonal but both edit foreign.rkt's marshal dispatch; this commit reconciles them. The merged foreign.rkt now centrally handles four extra cases: * `[List T]` ↔ Racket list of marshalled elements (recursive) * `Posit8/16/32/64` ↔ exact rational (semantic value) * `Int` ↔ exact integer * `[A -> B]` arg pos ↔ Racket procedure (lambda passing — pre-existing) Recursive composition: `[List Posit32]` arrives at the Racket side as a Racket list of rationals; `[List [List Posit32]]` arrives as nested Racket lists of rationals. The marshaller does the cons/nil walking, posit decoding, and IR list construction so callers never have to. `parse-foreign-type` now uses a separate `arg-spec` helper for argument positions: an arrow-typed argument becomes `('fn arg-specs . ret-spec)` preserving the function-typed marshalling path. Non-arrow argument types delegate to `base-type-name` as before. `marshal-racket->prologos` for `'fn` errors clearly — Racket procedures returned to Prologos as live lambdas remain out of scope. Eigentrust shim refactored to use it: * Dropped ~60 lines of cons/nil walking helpers (`cons-name?`, `nil-name?`, `try-cons-decomp`, `is-nil?`, `prologos-list->list`). * Dropped ~15 lines of bit-vec / IR list conversion (`list-of-posit32->bits-vec`, `matrix->vec-of-vecs`, `bits-vec->prologos-list`). * Cell-value carrier now stores exact rationals directly (was: posit bit patterns); foreign.rkt's Posit32 marshaller does the encode / decode at the FFI boundary. * `net-add-prop` / `net-new-cell` / `net-run-read` declare `[List Posit32]` and `[List [List Posit32]]` types directly; arguments arrive as Racket lists of rationals. Result: eigentrust-prop.rkt is 99 non-comment LOC (was 166), a ~40% reduction. Algorithmic content is essentially zero — just plumbing between cells and the Prologos kernel via the FFI bridge. Updated `2026-04-28_ETPROP_PITFALLS.md` "what stays in Racket" section to reflect the FFI-marshalling-glue category being centralised rather than per-shim boilerplate. Tests: * 174 tests pass across foreign / foreign-block / pvec / foreign-callback / foreign-marshal-ext / eigentrust (28.6s). * test-foreign-marshal-ext.rkt (49 unit tests) ported from the parallel branch. * test-foreign.rkt gains 14 integration tests for Int / Posit / List FFI types, also from the parallel branch. Eigentrust converged values unchanged: [0.0712, 0.4288, 0.0712, 0.4288] (within 1e-2 of steady state). Credit: List/Posit/Int marshalling and the new test files are from `claude/add-posits-ints-ffi-oM6pF` (commits b9a6461 + a4c2004 by Claude); merged here with the lambda-passing additions in this branch.
hierophantos
approved these changes
Apr 28, 2026
hierophantos
left a comment
hierophantos
left a comment
Contributor
There was a problem hiding this comment.
Approving. Substantive but well-scoped extension to FFI marshaling — function-typed FFI parameters now work via wrap-prologos-fn-as-racket driving nf reduction. Dependency cycle break in reduction.rkt is principled (removed dead (require "foreign.rkt") whose imports were unused; expr-foreign-fn lives in syntax.rkt).
Particularly appreciated the explicit mantra-alignment self-assessment in the PR body — labeling the marshaller as "off-network scaffolding by design" with a named retirement direction (propagator-native callbacks via cell subscription) is exactly the disclosure discipline our workflow.md adversarial framing rule asks for. The "one-way only with reverse explicitly erroring" scope discipline is also correct — synthesizing typed Prologos values at marshal time would cross into territory that needs deliberate design.
Filing a follow-up tracking issue for the retirement direction so it's visible to future maintainers, and starting an FFI track series in our roadmap to capture this and future FFI work (forward vision: Prologos as polyglot hub).
Contributor
|
Approved. Note for merge: PR is currently in draft state — once you lift draft we'll merge. We're starting an FFI track series in our roadmap (with this PR as Track 1 landing) and filing the retirement-direction tracking issue alongside, so the scaffolding labeling in your design doc has a permanent home. |
hierophantos
added a commit
that referenced
this pull request
Apr 28, 2026
Three artifacts establishing FFI as a tracked Series: 1. docs/tracking/2026-04-28_FFI_MASTER.md (NEW) — FFI Series Master doc following established pattern (PM/SRE/BSP-LE/CIU/PPN). Polyglot-hub thesis, cross-series connections, progress tracker with Tracks 0-3 + future placeholders, design discipline, key invariants. 2. Issue #37 — "FFI: retire lambda-passing scaffolding via propagator-native callbacks (cell subscription)" — captures retirement direction from PR #35's design doc as a permanent tracked obligation. Names design questions deferred to phase-time mini-design. 3. MASTER_ROADMAP.org — added FFI Series entry between PReductions and Completed Standalone Tracks. Status: Track 0 ✅ pre-existing, Track 1 ✅ landed (PR #33 9ef3420), Track 2 🔄 pending merge (PR #35 awaiting lift-from-draft), Track 3 ⬜ tracked as #37. Forward vision: Prologos as polyglot hub. Each FFI track integrates with propagator-network discipline rather than bolting on parallel off-network bridges; pragmatic scaffolding when needed gets named and retirement-tracked (Track 3 / #37 is the canonical example). Dailies entry under "FFI Series started + retirement issue filed" captures the work + three lessons distilled (off-network-with- retirement-direction discipline, track-series-as-permanent-home for evolving infrastructure, external contributor work surfacing architectural opportunities).
Contributor
|
Quick ping — re-verified the PR today against current main: still CLEAN/MERGEABLE, CI green, our approval standing. The PR #33 dependency (Posit + List marshaling) is now merged, so the marshaling foundation this PR builds on is in place. Whenever you're ready, lifting from draft would let us merge. We've also captured the retirement direction you flagged (off-network scaffolding → propagator-native callbacks) as issue #37 alongside the new FFI track series in our roadmap, so the scaffolding labeling has a permanent tracking home regardless of merge timing. No rush — happy to wait if you're still iterating or it's blocked on something else. |
kumavis
pushed a commit
that referenced
this pull request
May 6, 2026
…it, `->` in identifiers silently fails Two new Prologos elaborator/reader pitfalls discovered during the refr-import track (Phase 34a + 34d). Both cost ~15 minutes each to debug because the failure modes were silent or had misleading error messages. Pitfall #34 — `data` constructor signatures have IMPLICIT return type. Writing `ctor : T1 -> T2 -> Result` is INTERPRETED as "takes 3 args (T1, T2, Result), returns Result." The user usually means "takes 2 args, returns Result." The convention in this codebase (Listener, QEntry, etc) is to drop the trailing `-> Result`: `ctor : T1 -> T2`. Discovered Phase 34a (commit 3d8c069). My initial `refr : Nat -> Nat -> Refr` caused smart constructors to fail with "Type mismatch [Pi Nat Refr -> Refr]". Fix: drop the trailing `-> Refr`. Pitfall #35 — Function names containing `->` silently fail. Identifiers with `->` (like `refr->syrup`, common in ML/Lisp converter naming) are SILENTLY DROPPED by the elaborator. The WS-mode reader parses `->` as the function-arrow type operator inside the identifier, splitting the symbol. spec/defn forms can't bind anything sensible and produce no output at all — callers get "Unbound variable" downstream. Discovered Phase 34d (commit 7c797e6). My `refr->syrup` was silently dropped; renamed to `refr-to-syrup` worked. Codebase convention: use `-to-` for converters. Also documented (no new pitfall, recurrence-only): - #18 (multi-arity defn cons-pattern) hit on add's `suc a b -> ...`; fix: bracket as `[suc a] b -> ...`. - #21 (multi-line clause body → ??__match-fail) hit on extract-refrs-from-list; fix: `defn name [arg] (match arg ...)` same shape as data/list's `concat`. - #16 (mutual recursion) hit on extract-refrs-from-args ↔ extract-refrs-from-list; fix: one-way recursion via shallow-refr. - Issue #60 (multi-constructor cross-module inference) hit on 5-arg helpers; fix: 2-3 arg shape via BridgeStep. Updated .claude/rules/prologos-syntax.md: - Added "NEVER use `->` in identifiers" to the Naming section - Added a new "Data type definitions" section with the implicit-return-type convention These two are filed in the pitfalls doc but not as GitHub issues yet — both are reasonably easy elaborator/reader fixes that upstream Prologos work could close. Triage decision deferred.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Until this change, a
foreign racket "..."declaration could marshal first-order values (Nat, Int, Rat, Bool, Char, String, Posit32, Posit64, Path, Keyword, Passthrough, Opaque) but not callable functions. A signature likeapply-twice : [Nat -> Nat] Nat -> Natwas rejected because the function-typed parameter had no marshaller. Two consumers needed this:docs/tracking/2026-04-28_ETPROP_PITFALLS.md§ "What MUST stay on the Racket side"): items 1 and 2 of the four irreducible-Racket responsibilities — propagator fire functions and cell merge functions — were both blocked solely on "Prologos lambdas can't cross the FFI boundary as live closures."This PR removes that blocker.
Design summary
parse-foreign-typeis now recursive. Each arg/return position in the parsed spec is either a base-type symbol (existing behaviour, fully backward compatible) or(cons 'fn parsed-foreign-type)for a function-typed slot. When a Prologos valuepfof function type is marshalled,wrap-prologos-fn-as-racketreturns a Racket procedure that, on each Racket-side call:marshal-racket->prologoson the corresponding arg spec (recursively — nested function specs install deeper bridges).(((pf arg1) arg2) ... argN)via left-foldedexpr-app.nf. The reducer's existing whnf/nf caches accelerate repeated calls.foreign-type-tokens->sexpnow recurses into list-shaped tokens via anormalize-sub-tokenhelper so a sub-bracket form like[Nat -> Nat]parses as(-> Nat Nat)rather than as a function application ofNat.Reverse direction (returning a Racket procedure to Prologos as a callable lambda) is explicitly not supported and errors with a clear message — that path requires fabricating an
expr-foreign-fnat marshal time, which crosses the type-checked surface and is out of scope here.A dead
(require "foreign.rkt")was removed fromreduction.rkt(it imported nothing actually used;expr-foreign-fnlives insyntax.rkt) so thatforeign.rktcan naturally requirereduction.rktfornf.Full design + mantra-alignment self-assessment + SRE lattice lens + edge cases: see
docs/tracking/2026-04-28_FFI_LAMBDA_PASSING.md.Mantra alignment
The marshaller is off-network scaffolding by design, named as such with a retirement plan:
nf; cache memoises subexpressionsThe full self-assessment is in the design doc. Refusing to build the bridge until propagator-native callbacks land would block items 1 & 2 of the EigenTrust pitfalls indefinitely and leave every Racket HOF unreachable. The bridge buys real freedom while the on-network path matures.
What changed in the four "must stay in Racket" responsibilities
(from
docs/tracking/2026-04-28_ETPROP_PITFALLS.md)equal?integration.The pitfalls doc is updated in this PR; the eigentrust example header is updated to match. The example itself is not refactored — the design doc's § "Stretch" sketches how it would work, but staying scoped to the marshaller keeps this PR focused.
Pipeline checklists applied
Per
.claude/rules/pipeline.md:make-parametercalls; bridge is a closure over its specs).expr-foreign-fnfield count unchanged; onlymarshal-incontents admit structured specs, which other consumers already treat as opaque procedures of one argument).Tests run
tests/test-foreign-callback.rkt(new, 10 tests): parse-foreign-type for function args (incl. nested), wrapper-as-procedure, arity checking, integration via the foreign decl with single + named + composed lambda. All pass (0.5s).tests/test-foreign.rkt(47 pre-existing): all pass, no regression.tests/test-foreign-block.rkt,tests/test-pvec.rkt: pre-existing pass, no regression.tests/test-io-opaque-01.rkt(13 pre-existing): direct user ofmarshal-prologos->racketwith symbol specs — confirms backward compat. All pass.lib/examples/lambda-ffi-test.prologos(Level 3 acceptance viaprocess-file): runs end-to-end with all 4 expected results (Nat callback5N; named-lambda3N; Posit32 callback4.0; two-callbackcompose-and-call3N).lib/examples/eigentrust.prologos(Level 3 regression): runs to completion with the same converged output.claude/setup-racket-grammar-tests-OekFZ).Test plan checklist
Generated by Claude Code
Generated by Claude Code