diff --git a/docs/tracking/2026-04-28_ETPROP_PITFALLS.md b/docs/tracking/2026-04-28_ETPROP_PITFALLS.md new file mode 100644 index 000000000..d10fd5844 --- /dev/null +++ b/docs/tracking/2026-04-28_ETPROP_PITFALLS.md @@ -0,0 +1,298 @@ +# EtProp Pitfalls — Notes from the EigenTrust-on-Propagators Bring-Up + +**Date**: 2026-04-28 +**Context**: While implementing the EigenTrust-on-propagators example +(`racket/prologos/lib/examples/eigentrust{,-prop.rkt,.prologos}`), the work +surfaced several rough edges in Prologos and the surrounding Racket tooling. +This document captures them in one place so future similar work doesn't have +to rediscover them. + +The list is **observation only** — these are bugs / friction in language +features and tooling, not in the EigenTrust example itself. They're filed +here because tracking them in commit messages alone made them invisible. + +## What MUST stay on the Racket side (the irreducible core) + +Across two passes of "minimise the Racket footprint", these four +responsibilities resisted every attempt to push them out into Prologos: + + 1. **Propagator fire functions.** The fire-fn is a Racket closure invoked + by the Racket-side BSP scheduler with the network as its argument. + Prologos lambdas can't cross the FFI boundary as live closures (the + FFI marshaller validates a fixed type signature on each call; live + closures don't fit that protocol). + + 2. **Cell merge functions.** Same constraint. The propagator network + calls merge-fn during cell writes (and during BSP fold-merges). It + has to be a Racket procedure. + + 3. **The cell-value carrier.** A gen-tagged immutable Racket vector. The + gen tag has to interoperate with Racket's `equal?` for the cell's + change detection, and the merge function has to be a Racket procedure + that operates on whatever Racket data structure the cell holds. + + 4. **FFI marshalling glue.** Walking Prologos cons/nil chains, extracting + posit32 bit-patterns out of `expr-posit32` IR nodes, encoding rationals + back into bit patterns. Pure Racket-side bookkeeping. + +Everything else — matrix transpose, decay scaling, bias computation, +initial-zero-vector, the iteration driver, the entire EigenTrust update +rule's arithmetic decomposition — is in the .prologos source. The +Racket-side `net-add-prop` is purpose-AGNOSTIC: it implements the +generic affine combination + + out[j] := bias[j] + Σ_i weight[j][i] · in[i] + +with no knowledge of EigenTrust, decay, transposition, or pretrust. + +The four FFI primitives (`net-new`, `net-new-cell`, `net-add-prop`, +`net-run-read`) are the entire Racket-Prologos interface. + +------------------------------------------------------------------ + +## 1. `def` is reference-transparent — side effects re-fire on every use + +In Prologos a `def name := body` stores `body`'s **AST**, not its evaluated +value. Each subsequent reference to `name` substitutes the AST and re-reduces +it. For pure computations that's fine; for foreign calls with side effects +it's a footgun. + +Repro: + + foreign racket "racket/base" [add1 : Nat -> Nat] + + foreign racket "lib/examples/eigentrust-prop.rkt" :as et + net-new : Posit32 -> Posit32 + + def h := et/net-new 1000000.0 + h + h + h ;; -> three different handle ids + +Each `h` re-evaluates `et/net-new`, allocating a fresh network. This breaks +multi-`def` imperative-style programming with foreign side effects. Note +that within a single `process-command` boundary the per-command `whnf-cache` +**does** memoise identical AST nodes, so a single top-level expression with +multiple syntactic occurrences of `h` after substitution does behave; but +`def`-bound names are unfolded across `def` boundaries. + +**Workaround**: the entire algorithm runs as one expression — typically a +single `defn` body whose RHS chains `let := body` bindings. Within that one +expression all FFI calls evaluate exactly once. + +**Underlying language design call**: this is consistent with definitional +equality in dependent type theory (defs are *transparent*), so it's not +strictly a bug — but the behaviour is surprising for anyone bringing in +side-effecting foreign code, and there is no warning surface. + +## 2. Reduction is call-by-name — unused let-bound side effects are dropped + +`let _ := side-effecting-call body` evaluates `body` without ever +substituting `_` (since `_` is a wildcard). The `side-effecting-call` is +therefore **never evaluated**. Same shape for any unreferenced binding: + + let _i := et/net-add-prop ... ;; never fires + let _r := et/net-run h ;; never fires + result-expr ;; only this is reduced + +The eigentrust shim works around this by making every "side-effecting" FFI +call return a **meaningful** Posit32 (the cell-ref or the handle id) so the +caller is forced to thread it through a subsequent call. Sequencing via +`do` blocks doesn't help — `do` desugars to nested let. + +A shim helper such as `seq : Unit -> A -> A` that pattern-matches on `unit` +also failed empirically; the match seems to be optimised through. Real +forcing only happens at the **FFI argument boundary** where `whnf` / +`nf` are used to marshal values. + +**Implication for any FFI-via-foreign API**: every effectful call MUST +return a non-trivial value that the user is induced to consume. Returning +`Unit` is a trap. + +## 3. `let x := value body` requires `body` strictly more indented than `let` + +Prologos's WS-mode `let` macro merges consecutive bodyless `let`s into a +single binding group, with the trailing expression as the body. Surface +syntax requires the body to be indented past the `let` column: + + defn calc [n] + let x := add1 n + add1 x ;; OK — body indented past `let` + +If the body shares the `let`'s indent the parser silently drops the whole +defn (no error reported, the name registers as **unbound**): + + defn calc [n] + let x := add1 n + add1 x ;; WRONG — defn never registered + +This combines with pitfall #4 below to make defn-body errors particularly +hard to diagnose. + +## 4. Defn-body errors swallow the whole defn + +When `defn name body` has a malformed body (e.g. wrong indent, undefined +identifier in a particular position), the surface form is silently rejected +and `name` never registers. The first hint is a downstream +"Unbound variable: name" error from a later top-level expression. There is +no error from the defn itself. + +Observed during eigentrust-prop bring-up: `defn pretrust-cells [...]` with +a stale forward reference compiled silently into nothing; the only signal +was the later `[pretrust-cells h pretrust]` failing. + +## 5. `defn name | pat -> body` arity-1 list patterns produce a non-exhaustive match + +This compiles cleanly but raises `??__match-fail` at run time: + + defn zeros-of + | nil -> nil + | [cons _ rs] -> [cons 0.0 [zeros-of rs]] + +The fix is the explicit-`match` form, which works: + + defn zeros-of [xs] + match xs + | nil -> nil + | [cons _ rs] -> [cons 0.0 [zeros-of rs]] + +The two forms should be equivalent; the multi-arity dispatch path appears to +miss the `nil`-as-constructor pattern in arity-1 list matches. + +## 6. Foreign type signatures must fit on ONE line + +The foreign-block parser tokenises type tokens and then splits on `->`. If +the type is wrapped across lines: + + net-add-prop : Posit32 Posit32 Posit32 + [List [List Posit32]] [List Posit32] Posit32 -> Posit32 + +`foreign-type-tokens->sexp` errors with "Cannot parse foreign type tokens". +The fix is a single (potentially long) line: + + net-add-prop : Posit32 Posit32 Posit32 [List [List Posit32]] [List Posit32] Posit32 -> Posit32 + +This is annoying for FFI declarations of broadcast propagators (which take +several list-typed arguments) but is currently mandatory. + +## 7. Foreign FFI doesn't support Posit32 marshalling out of the box + +`base-type-name` recognises `expr-Posit32` / `expr-Posit64` and returns the +symbol, but `marshal-prologos->racket` and `marshal-racket->prologos` had no +case for them: the foreign machinery would say "Unsupported marshal-in +type: Posit32". The previous handoff added Posit32/Posit64 marshalling +(see commit `f9a3d7c`); this is still the only marshalling path that +short-circuits the existing posit-impl.rkt arithmetic ops via FFI. + +The carrier is the **bit-pattern integer** (matching posit-impl.rkt), so +`foreign racket "posit-impl.rkt" [posit32-add : Posit32 Posit32 -> Posit32]` +works directly. + +## 8. Foreign data structure types are passed via `[List ...]` but only allowed +in specific positions + +`[List Nat]`, `[List Posit32]`, and `[List [List Posit32]]` work in foreign +type signatures. They must be bracketed (the parser handles them as type +applications). Using bare `List Nat` would unfold into curried `List -> +Nat -> ...`, which is wrong. + +## 9. Top-level `let` is forbidden, but inside `defn` bodies works fine + +`let h := body` at the top level errors with +`let: 'let' is not allowed at top level. Use 'def' instead.` + +There's no surface alternative for "give a name to an intermediate value at +the top level". `def` re-evaluates (pitfall #1). The only way to evaluate +something exactly once at top level is to inline the entire computation +into one expression. + +## 10. `(zeros-of rs)` parens vs `[zeros-of rs]` brackets are NOT +interchangeable + +In WS mode `(...)` is reserved for parser keywords (`match`, `def`, etc.) and +relational goals. Function application uses `[]`. Writing `(f x)` for an +ordinary application typically silently fails (the parser tries to find a +special form named `f` and either drops the form or fails very far away +from the source line). + +## 11. `list-literal` quoting handles bare numeric literals correctly + +Confirmed working: `'[0.5 0.25 0.125]` produces a `[List Posit32]` (the +posit literals retain their decimal-literal interpretation; bare `0.5` is +equivalent to `~0.5` per the grammar). Earlier confusion was that quoted +identifiers (`'[src1 src2]`) get reified as Datum symbols, not as variable +references — so list-literal sugar works for **constants** but not for +**values**. + +## 12. Racket-9-only features + Ubuntu 24.04 ships Racket 8.10 + +`make-parallel-thread-fire-all` (`propagator.rkt:2592`) uses `(thread +#:pool 'own)` — a Racket 9 feature that's silently a runtime error on +Racket 8.x: + + application: procedure does not accept keyword arguments + procedure: thread + +The driver unconditionally installs this executor at module load +(`driver.rkt:434`). On Racket 8 the test suite errors out from any code +path that goes through `run-to-quiescence-bsp`. Workarounds tried: + + * `parameterize` to `sequential-fire-all` at the call site — works for + your own entry points but doesn't help the driver's own + `run-post-compilation-inference!` which is set up at module load. + + * Patch `propagator.rkt` to detect Racket version — invasive. + +Resolution chosen here: build Racket 9 from source. The download host +(`download.racket-lang.org`) is on a curl allowlist, so source must be +pulled from `https://github.com/racket/racket/archive/refs/tags/v9.1.tar.gz` +and built with `make CPUS=4 PKGS=""`. `rackunit` then has to be +hand-installed by copying `pkgs/rackunit/rackunit-lib/rackunit` and +`pkgs/compiler-lib/raco/testing.rkt` into the Racket collects tree — +`raco pkg install` fails because the catalog server is also on the allowlist. + +## 13. Prologos's parser drops files using `prologos::core::abs-trait` + +`racket/prologos/lib/examples/foreign.prologos` `require`s +`prologos::core::abs-trait` which no longer exists (renamed/merged into +`prologos::core::arithmetic`). Running this example now errors: + + imports: Cannot find module: prologos::core::abs-trait + +This is unrelated to the eigentrust work but came up while sanity-checking +the FFI surface. + +## 14. `tools/check-parens.sh` hardcodes a Mac-style Racket path + +The pre-commit paren checker shells out to +`/Applications/Racket v9.0/bin/racket` (line 21), which is a Mac dev +machine artifact. On Linux + the project's expected path +(`/usr/local/bin/racket`) this fails open with a confusing +"FAIL: ... No such file or directory" message even when the file's parens +are balanced. + +Workaround used here: a small standalone `parencheck.rkt` script that +parses files with `read-syntax`. Either a `command -v racket` lookup or a +configurable path setting in `check-parens.sh` would close this. + +## 15. Default Posit32 conversions use exact rationals and are slow at scale + +Encoding a Racket rational into a Posit32 bit pattern (`posit32-encode`) and +decoding it back (`posit32-to-rational`) goes through the exact-rational +posit format. For our 4-peer × 20-iteration EigenTrust this is unproblematic, +but the propagator network's reduction phase reports ~55 seconds in +`reduce_ms` even though the actual fixpoint runtime is sub-second. The +overhead is in the elaboration / type-checking of the recursive `drive` +loop and the FFI marshalling, not the propagator firing itself. Future +work: a `Posit32` instance of `Num` and operator desugaring would let the +.prologos source operate on posits directly (without going through Rat), +which would also let the FFI shim drop its rational pre-computation step. + +------------------------------------------------------------------ + +## Tracking + +These observations are not tracked in any specific PIR or design doc; +they're cross-cutting friction discovered during the bring-up. If any of +them blocks future work, file a tracking doc per `.claude/rules/workflow.md` +and link back here. diff --git a/racket/prologos/foreign.rkt b/racket/prologos/foreign.rkt index 773bfd97e..f5d760714 100644 --- a/racket/prologos/foreign.rkt +++ b/racket/prologos/foreign.rkt @@ -70,15 +70,31 @@ [(expr-string v) v] [_ (error 'foreign "Cannot marshal to string — not a String literal: ~a" e)])) +;; Convert a Prologos Posit{32,64} to its raw bit-pattern integer. +;; This is a passthrough at the Racket level — posit-impl.rkt operates on the +;; same bit-pattern representation, so foreign-imported posit ops work directly. +;; FFI consumers that want rationals call posit{32,64}-to-rational themselves. +(define (posit32->bits e) + (match e + [(expr-posit32 bits) bits] + [_ (error 'foreign "Cannot marshal to Posit32 — not a Posit32 literal: ~a" e)])) + +(define (posit64->bits e) + (match e + [(expr-posit64 bits) bits] + [_ (error 'foreign "Cannot marshal to Posit64 — not a Posit64 literal: ~a" e)])) + (define (marshal-prologos->racket base-type val) (case base-type - [(Nat) (nat->integer val)] - [(Int) (int->integer val)] - [(Rat) (rat->rational val)] - [(Bool) (bool->boolean val)] - [(Unit) (void)] - [(Char) (char->rkt-char val)] - [(String) (string->rkt-string val)] + [(Nat) (nat->integer val)] + [(Int) (int->integer val)] + [(Rat) (rat->rational val)] + [(Bool) (bool->boolean val)] + [(Unit) (void)] + [(Char) (char->rkt-char val)] + [(String) (string->rkt-string val)] + [(Posit32) (posit32->bits val)] + [(Posit64) (posit64->bits val)] ;; Passthrough types: the Prologos IR value IS the Racket value [(Path Keyword Passthrough) val] [else @@ -110,15 +126,27 @@ (error 'foreign "Cannot marshal from Racket: expected exact rational, got ~a" n)) (expr-rat n)) +;; Convert a Racket value to a Prologos Posit{32,64} expression. +;; The carrier representation is the raw bit-pattern integer — same as +;; posit-impl.rkt. So the input is expected to be an exact integer (the bit +;; pattern). FFI consumers wanting to return a rational can compose with +;; posit{32,64}-encode themselves. +(define (bits->posit-expr width val ctor) + (unless (exact-integer? val) + (error 'foreign "Cannot marshal Racket value to Posit~a (expected bit-pattern integer): ~a" width val)) + (ctor val)) + (define (marshal-racket->prologos base-type val) (case base-type - [(Nat) (integer->nat val)] - [(Int) (integer->int val)] - [(Rat) (rational->rat val)] - [(Bool) (if val (expr-true) (expr-false))] - [(Unit) (expr-unit)] - [(Char) (expr-char val)] - [(String) (expr-string val)] + [(Nat) (integer->nat val)] + [(Int) (integer->int val)] + [(Rat) (rational->rat val)] + [(Bool) (if val (expr-true) (expr-false))] + [(Unit) (expr-unit)] + [(Char) (expr-char val)] + [(String) (expr-string val)] + [(Posit32) (bits->posit-expr 32 val expr-posit32)] + [(Posit64) (bits->posit-expr 64 val expr-posit64)] ;; Passthrough types: result is already a Prologos IR value [(Path Keyword Passthrough) val] [else diff --git a/racket/prologos/lib/examples/eigentrust-prop.rkt b/racket/prologos/lib/examples/eigentrust-prop.rkt new file mode 100644 index 000000000..92aa31abf --- /dev/null +++ b/racket/prologos/lib/examples/eigentrust-prop.rkt @@ -0,0 +1,366 @@ +#lang racket/base + +;;; +;;; PROPAGATOR-BASED EIGENTRUST — minimal Racket FFI shim. +;;; +;;; This module is the irreducible Racket-side plumbing. Prologos plans +;;; first-class propagator support, but the type checker isn't yet wired +;;; through the surface forms in the grammar; until then, the four +;;; propagator-network primitives below are exposed via `foreign racket +;;; "..."`. Everything else — the EigenTrust-specific algorithm — lives in +;;; the .prologos source (matrix transpose, decay weighting, pretrust +;;; biasing, the iteration loop, etc.). +;;; +;;; ----------------------------------------------------------------- +;;; What MUST stay in Racket (the irreducible core): +;;; +;;; 1. Propagator fire functions are Racket closures invoked by the +;;; Racket-side scheduler; Prologos lambdas can't cross the FFI +;;; boundary as live closures. +;;; 2. Cell merge functions, same reason. +;;; 3. The cell-value carrier (gen-tagged immutable Posit32 vector) and +;;; its monotone merge — both Racket data structures. +;;; 4. The persistent prop-network struct, handle/cell registries. +;;; 5. FFI marshalling: list walking on cons/nil chains, posit +;;; bit-pattern extraction. +;;; +;;; What MOVED out, into the .prologos source: +;;; +;;; 1. Matrix transpose +;;; 2. Decay scaling (1 − α) · C +;;; 3. Bias computation α · pretrust +;;; 4. Initial-zero vector for new-layer cells +;;; 5. The iteration driver +;;; +;;; ----------------------------------------------------------------- +;;; Mantra alignment (per .claude/rules/on-network.md) +;;; +;;; "All-at-once, all in parallel, structurally emergent +;;; information flow ON-NETWORK." +;;; +;;; * All-at-once — each iteration is ONE compound cell holding the +;;; full score vector for all peers, allocated atomically. +;;; * All in parallel — `net-add-prop` installs ONE broadcast propagator +;;; (per .claude/rules/propagator-design.md § Broadcast +;;; Propagators) covering all N components as items. +;;; The BSP scheduler decomposes across OS threads at +;;; fire time. +;;; * Structurally emergent — the cell DAG iter-0 → iter-1 → ... → iter-K +;;; drives firing order via dataflow. +;;; * ON-NETWORK — every score lives in a cell. +;;; +;;; ----------------------------------------------------------------- +;;; FFI surface (consumed from .prologos via `foreign racket "..."`): +;;; +;;; net-new : Posit32 -> Posit32 +;;; Allocate a propagator network with the given fuel budget; return +;;; a Posit32 handle id. +;;; +;;; net-new-cell : Posit32 -> List Posit32 -> Posit32 +;;; Allocate ONE compound cell holding the given Posit32 vector. +;;; Returns a Posit32 cell-ref. +;;; +;;; net-add-prop : Posit32 -> Posit32 -> Posit32 +;;; -> List (List Posit32) -> List Posit32 -> Posit32 +;;; Install ONE broadcast propagator that computes the affine +;;; combination: +;;; +;;; out[j] := biases[j] + Σ_i weights[j][i] · in[i] +;;; +;;; Domain-AGNOSTIC: the .prologos caller precomputes weights and +;;; biases from whatever raw inputs the algorithm has. Returns +;;; output-cref so layers can be chained. +;;; +;;; net-run-read : Posit32 -> Posit32 -> List Posit32 +;;; Run the network to quiescence, then read the cell as a Posit32 +;;; list. Combined so call-by-name reduction is forced to evaluate +;;; the full layer chain before the quiescence pass. +;;; +;;; All "side-effecting" calls return a meaningful Posit32 (handle, cell +;;; ref, or score list) instead of Unit. Prologos's reduction is +;;; call-by-name — an unused result expression would never be evaluated +;;; and the side effect would be silently dropped. Threading a Posit32 +;;; through every call forces strict evaluation order. +;;; + +(require racket/match + racket/list + racket/vector + (rename-in "../../propagator.rkt" + [net-new-cell raw-net-new-cell] + [net-add-broadcast-propagator raw-net-add-broadcast]) + "../../syntax.rkt" + "../../posit-impl.rkt") + +(provide + net-new + net-new-cell + net-add-prop + net-run-read) + +;; ======================================== +;; Handle registry — Posit32 carrier. +;; ======================================== +;; +;; Posit32 represents small integers exactly within its range, so we use +;; the *bit pattern* of the encoded posit as the on-the-wire handle. The +;; Prologos surface sees Posit32 values; the registry side-table on the +;; Racket side maps decoded integer ids to mutable boxes holding the +;; persistent prop-network. + +(struct et-net (box) #:transparent) + +(define handle-registry (make-hasheqv)) ;; integer -> et-net +(define cell-registry (make-hasheqv)) ;; integer -> cell-id (the prop-network's cell) +(define next-handle-id (box 1)) ;; reserve 0 in case +(define next-cell-id (box 1)) + +(define (id->posit32 i) (posit32-encode i)) +(define (posit32->id bits) (inexact->exact (posit32-to-rational bits))) + +;; ======================================== +;; List helpers: walk WHNF cons/nil chains. +;; ======================================== +;; +;; Foreign args are reduced to nf before marshalling, so List values arrive +;; as nested expr-app of expr-fvar 'cons (or qualified ::cons) terminating +;; in expr-nil / expr-fvar 'nil. + +(define (cons-name? sym) + (let ([s (symbol->string sym)]) + (or (string=? s "cons") + (let ([n (string-length s)]) + (and (>= n 6) (string=? (substring s (- n 6)) "::cons")))))) + +(define (nil-name? sym) + (let ([s (symbol->string sym)]) + (or (string=? s "nil") + (let ([n (string-length s)]) + (and (>= n 5) (string=? (substring s (- n 5)) "::nil")))))) + +(define (try-cons-decomp e) + (and (expr-app? e) + (let ([func (expr-app-func e)] + [tail (expr-app-arg e)]) + (and (expr-app? func) + (let ([head (expr-app-arg func)] + [inner (expr-app-func func)]) + (cond + [(and (expr-fvar? inner) (cons-name? (expr-fvar-name inner))) + (cons head tail)] + [(and (expr-app? inner) + (expr-fvar? (expr-app-func inner)) + (cons-name? (expr-fvar-name (expr-app-func inner)))) + (cons head tail)] + [else #f])))))) + +(define (is-nil? e) + (or (expr-nil? e) + (and (expr-fvar? e) (nil-name? (expr-fvar-name e))) + (and (expr-app? e) + (let ([f (expr-app-func e)]) + (and (expr-fvar? f) (nil-name? (expr-fvar-name f))))))) + +(define (prologos-list->list e) + (let loop ([cur e] [acc '()]) + (cond + [(is-nil? cur) (reverse acc)] + [(try-cons-decomp cur) + => (lambda (hd-tl) (loop (cdr hd-tl) (cons (car hd-tl) acc)))] + [else (error 'eigentrust-prop + "list walk: not a list / not in WHNF: ~v" cur)]))) + +(define (list-of-posit32->bits-vec e) + (define lst (prologos-list->list e)) + (for/vector #:length (length lst) + ([x (in-list lst)]) + (match x + [(expr-posit32 b) b] + [_ (error 'eigentrust-prop "expected Posit32 in list, got ~v" x)]))) + +(define (matrix->vec-of-vecs e) + (define rows (prologos-list->list e)) + (for/vector #:length (length rows) + ([row (in-list rows)]) + (list-of-posit32->bits-vec row))) + +(define (bits-vec->prologos-list bits-vec) + (foldr (lambda (b acc) + (expr-app (expr-app (expr-fvar 'cons) (expr-posit32 b)) acc)) + (expr-nil) + (vector->list bits-vec))) + +;; ======================================== +;; Cell value: generation-tagged immutable vector of Posit32 bit patterns. +;; +;; The merge function keeps the entry with the higher generation. This is +;; CALM-monotone (joins commute / are associative / are idempotent) under +;; the usual correctness contract: each generation is written exactly +;; once, by exactly one propagator, in topological order. +;; ======================================== + +(struct gen-vec (gen vec) #:transparent) + +(define (gen-merge old new) + (cond + [(not (gen-vec? old)) new] + [(not (gen-vec? new)) old] + [(> (gen-vec-gen new) (gen-vec-gen old)) new] + [else old])) + +(define (current-gen v) (if (gen-vec? v) (gen-vec-gen v) -1)) +(define (current-vec v) (if (gen-vec? v) (gen-vec-vec v) #())) + +;; ======================================== +;; net-new — allocate a propagator network. +;; ======================================== +;; +;; fuel-bits : Posit32 bit pattern for a small integer fuel budget. +;; Returns: Posit32 bit pattern for the handle id. + +(define (net-new fuel-bits) + (define fuel-int + (let ([r (posit32-to-rational fuel-bits)]) + (cond + [(exact-integer? r) r] + [(rational? r) (inexact->exact (round r))] + [else 1000000]))) + (define h (et-net (box (make-prop-network (max 1 fuel-int))))) + (define id (unbox next-handle-id)) + (set-box! next-handle-id (+ id 1)) + (hash-set! handle-registry id h) + (id->posit32 id)) + +(define (lookup-handle hbits) + (define id (posit32->id hbits)) + (or (hash-ref handle-registry id #f) + (error 'eigentrust-prop "no such network handle: ~a" id))) + +;; ======================================== +;; net-new-cell — allocate a compound cell. +;; ======================================== +;; +;; Each cell holds a vector of Posit32 bit-patterns — the score for every +;; peer at one iteration, all bundled into one cell so the broadcast +;; propagator writes them atomically (mantra: all-at-once, on-network). +;; +;; init-bits-list : Prologos List Posit32 — initial score vector. +;; Returns : Posit32 bit pattern encoding the cell-ref id. + +(define (net-new-cell hbits init-bits-list) + (define handle (lookup-handle hbits)) + (define init-vec (vector->immutable-vector (list-of-posit32->bits-vec init-bits-list))) + (define net (unbox (et-net-box handle))) + (define-values (net* cid) + (raw-net-new-cell net (gen-vec 0 init-vec) gen-merge)) + (set-box! (et-net-box handle) net*) + (define cref (unbox next-cell-id)) + (set-box! next-cell-id (+ cref 1)) + (hash-set! cell-registry cref cid) + (id->posit32 cref)) + +(define (lookup-cell crefbits) + (define cref (posit32->id crefbits)) + (or (hash-ref cell-registry cref #f) + (error 'eigentrust-prop "no such cell-ref: ~a" cref))) + +;; ======================================== +;; net-add-prop — install ONE broadcast propagator implementing a +;; domain-agnostic affine combination over a vector cell. +;; ======================================== +;; +;; Per `.claude/rules/propagator-design.md` § Broadcast Propagators: +;; "any for/fold or for/list that processes independent items is a +;; candidate for broadcast." This is exactly that — N output components +;; share one input cell. The broadcast-profile metadata makes the +;; propagator decomposable across OS threads (BSP-LE Track 2 Phase 1B) +;; when the scheduler chooses. +;; +;; for each item j in 0..N-1: +;; out[j] := bias[j] + Σ_i weights[j][i] · in[i] +;; +;; The propagator is purpose-AGNOSTIC. EigenTrust-specific math (matrix +;; transpose, decay weighting, pretrust biasing) lives in the .prologos +;; source — only the linear-combination and gen-tagging plumbing is here. +;; This is the irreducible Racket-side core: cell merge functions and +;; propagator fire functions cannot cross the FFI boundary as Prologos +;; closures, so they MUST be implemented in Racket. +;; +;; weights : Prologos List (List Posit32) — weights[j][i] +;; biases : Prologos List Posit32 — biases[j] +;; +;; Returns output-cref so the caller can chain net-add-prop calls. + +(define (net-add-prop hbits input-cref output-cref weights-list biases-list) + (define handle (lookup-handle hbits)) + (define input-cid (lookup-cell input-cref)) + (define out-cid (lookup-cell output-cref)) + (define W-rows (matrix->vec-of-vecs weights-list)) + (define b-bits (list-of-posit32->bits-vec biases-list)) + (define n (vector-length b-bits)) + (unless (= n (vector-length W-rows)) + (error 'net-add-prop "weights outer length ~a != biases length ~a" + (vector-length W-rows) n)) + (for ([row (in-vector W-rows)] [j (in-naturals)]) + (unless (= n (vector-length row)) + (error 'net-add-prop "weights row ~a has ~a entries, expected ~a" + j (vector-length row) n))) + ;; Convert weights and biases to exact rationals once, up front. + (define W + (for/vector ([row (in-vector W-rows)]) + (for/vector ([b (in-vector row)]) (posit32-to-rational b)))) + (define biases + (for/vector ([b (in-vector b-bits)]) (posit32-to-rational b))) + ;; --- Broadcast propagator: ONE install, N items, parallel-decomposable. + (define items (build-list n values)) + ;; item-fn: per-component update. Reads the prev-iter compound cell, + ;; computes the affine combination for component j, returns + ;; (input-gen, j, bits) + ;; The input-gen tag lets the next layer's gen strictly exceed it so + ;; re-fires (after the input cell evolves) dominate stale fires. + (define (item-fn j input-vals) + (define prev-cell-val (car input-vals)) + (define prev (current-vec prev-cell-val)) + (define input-gen (current-gen prev-cell-val)) + (define wts (vector-ref W j)) + (define bias (vector-ref biases j)) + (define sum + (for/fold ([acc bias]) ([w (in-vector wts)] [b (in-vector prev)]) + (+ acc (* w (posit32-to-rational b))))) + (list input-gen j (posit32-encode sum))) + ;; result-merge: accumulate per-component results into a gen-vec + ;; carrier whose gen is input-gen + 1. Gen-merge then keeps the latest + ;; fire when the input cell evolves across BSP rounds. + (define (final-merge acc r) + (define input-gen (car r)) + (define j (cadr r)) + (define bits (caddr r)) + (define base + (cond + [(gen-vec? acc) (vector-copy (gen-vec-vec acc))] + [else (make-vector n 0)])) + (vector-set! base j bits) + (gen-vec (+ 1 input-gen) (vector->immutable-vector base))) + (define net0 (unbox (et-net-box handle))) + (define-values (net* _pid) + (raw-net-add-broadcast net0 (list input-cid) out-cid + items item-fn final-merge)) + (set-box! (et-net-box handle) net*) + output-cref) + +;; ======================================== +;; net-run-read — flush the network and snapshot a cell's vector. +;; ======================================== +;; +;; The single FFI call forces strict evaluation of `output-cref` before +;; running, so the entire chain of net-add-prop side effects has fired +;; by the time run-to-quiescence runs. + +(define (net-run-read hbits cref) + (define handle (lookup-handle hbits)) + (define cid (lookup-cell cref)) + (define net0 (unbox (et-net-box handle))) + (define net* (run-to-quiescence net0)) + (set-box! (et-net-box handle) net*) + (define final-vec (current-vec (net-cell-read net* cid))) + (bits-vec->prologos-list final-vec)) diff --git a/racket/prologos/lib/examples/eigentrust.prologos b/racket/prologos/lib/examples/eigentrust.prologos new file mode 100644 index 000000000..e501694de --- /dev/null +++ b/racket/prologos/lib/examples/eigentrust.prologos @@ -0,0 +1,208 @@ +ns examples.eigentrust + + +require [prologos::data::list :refer [List]] + + +;; ============================================================ +;; EigenTrust on Propagators (via Racket FFI) +;; ============================================================ +;; +;; EigenTrust [Kamvar/Schlosser/Garcia-Molina 2003] is the classic +;; reputation algorithm for peer-to-peer networks. Each peer i has a +;; trust score t[i] in [0, 1]. Local trust C[i][j] says how much peer i +;; trusts peer j (rows of C are normalised to sum to 1). A pre-trust +;; vector p anchors the iteration against Sybil attacks. The power +;; iteration +;; +;; t_{k+1} = α · p + (1 − α) · C^T · t_k +;; +;; converges to the dominant eigenvector — the global trust ranking. +;; +;; ----- Mantra (.claude/rules/on-network.md) ----- +;; +;; "All-at-once, all in parallel, structurally emergent +;; information flow ON-NETWORK." +;; +;; * All-at-once — each iteration is ONE compound cell holding the +;; full Posit32 score vector for every peer. +;; * All in parallel — `net-add-prop` installs ONE broadcast propagator +;; (`.claude/rules/propagator-design.md` § Broadcast) +;; covering all N peer updates per round; the BSP +;; scheduler decomposes those items across OS threads +;; at fire time. +;; * Structurally emergent — the cell DAG iter-0 → iter-1 → … → iter-K +;; drives firing order; we never tell the scheduler +;; what fires when. +;; * ON-NETWORK — every score lives in a cell; the gen-tagged +;; compound merge is CALM-monotone. +;; +;; ----- Where each piece of the algorithm lives ----- +;; +;; Racket (eigentrust-prop.rkt — irreducible plumbing): +;; * The four propagator-network primitives (net-new, net-new-cell, +;; net-add-prop, net-run-read). +;; * The compound-cell carrier and its gen-tagged merge. +;; * The fire function — a Racket closure that evaluates a +;; domain-AGNOSTIC affine combination +;; out[j] := bias[j] + Σ_i W[j][i] · in[i] +;; on the propagator's items. (Closures can't cross the FFI +;; boundary, so the fire-fn must stay Racket; we keep it generic so +;; EigenTrust-specific math doesn't leak into it.) +;; * Posit32 ↔ rational marshalling. +;; +;; Prologos (this file — the algorithm): +;; * Matrix transpose +;; * Decay scaling (1 − α) · Cᵀ +;; * Bias computation α · pretrust +;; * Building zero-vectors for fresh-layer cell initialisation +;; * The K-iteration driver loop +;; +;; First-class propagator surface (`net-new`, `net-new-cell`, +;; `net-add-prop`) is on the Prologos roadmap but not yet wired through +;; the type checker; this file imports the same primitives via +;; `foreign racket "..."` until that work lands. +;; +;; ============================================================ + + +;; -- FFI: the irreducible four propagator primitives -- +;; +;; All carriers are Posit32 — handles, cell-refs, fuel, scores, weights. +;; "Side-effecting" calls return a meaningful Posit32 so call-by-name +;; reduction is forced to evaluate them. + +foreign racket "lib/examples/eigentrust-prop.rkt" :as et + net-new : Posit32 -> Posit32 + net-new-cell : Posit32 [List Posit32] -> Posit32 + net-add-prop : Posit32 Posit32 Posit32 [List [List Posit32]] [List Posit32] -> Posit32 + net-run-read : Posit32 Posit32 -> [List Posit32] + + +;; ---------- Vector / matrix helpers (pure Prologos) ---------- + +;; Constant-zero vector with the same length as the reference. +spec zeros-of [List Posit32] -> [List Posit32] +defn zeros-of [xs] + match xs + | nil -> nil + | [cons _ rs] -> [cons 0.0 [zeros-of rs]] + + +;; scale-vec α v ↦ [α · v[0], α · v[1], …] +spec scale-vec Posit32 [List Posit32] -> [List Posit32] +defn scale-vec [s xs] + match xs + | nil -> nil + | [cons x rs] -> [cons [p32* s x] [scale-vec s rs]] + + +;; scale-mat α M ↦ M with every entry scaled by α +spec scale-mat Posit32 [List [List Posit32]] -> [List [List Posit32]] +defn scale-mat [s xss] + match xss + | nil -> nil + | [cons xs rs] -> [cons [scale-vec s xs] [scale-mat s rs]] + + +;; First column of a matrix; truncates if any row is shorter than the +;; rest. (Used by `transpose`.) +spec map-head [List [List Posit32]] -> [List Posit32] +defn map-head [xss] + match xss + | nil -> nil + | [cons xs rest] -> + match xs + | nil -> nil + | [cons x _] -> [cons x [map-head rest]] + + +;; Drop the first column of a matrix. +spec map-tail [List [List Posit32]] -> [List [List Posit32]] +defn map-tail [xss] + match xss + | nil -> nil + | [cons xs rest] -> + match xs + | nil -> [map-tail rest] + | [cons _ xs'] -> [cons xs' [map-tail rest]] + + +;; Predicate: every row of the matrix is nil. +spec all-empty? [List [List Posit32]] -> Bool +defn all-empty? [xss] + match xss + | nil -> true + | [cons xs rs] -> + match xs + | nil -> [all-empty? rs] + | _ -> false + + +;; transpose M ↦ Mᵀ for a (rectangular) matrix. +spec transpose [List [List Posit32]] -> [List [List Posit32]] +defn transpose [xss] + match [all-empty? xss] + | true -> nil + | false -> [cons [map-head xss] [transpose [map-tail xss]]] + + +;; ---------- The EigenTrust algorithm ---------- +;; +;; Each `net-add-prop` install is ONE broadcast propagator covering all N +;; peers in parallel. Layer cells form a DAG iter-0 → iter-1 → … → iter-K +;; that the BSP scheduler fires in topological order. +;; +;; Construction-time recursion in `drive` is just for traversing K. At +;; RUN time, the cell-DAG IS the parallel decomposition. +spec drive Posit32 Posit32 [List [List Posit32]] [List Posit32] [List Posit32] Nat -> Posit32 +defn drive + | h cell weights biases _zeros 0N -> cell + | h cell weights biases zeros [suc k] -> + let next-cell := et/net-new-cell h zeros + let chained := et/net-add-prop h cell next-cell weights biases + drive h chained weights biases zeros k + + +;; t_{k+1} = α · p + (1 − α) · Cᵀ · t_k +;; +;; The propagator's affine kernel is out[j] := bias[j] + Σ W[j][i] · in[i], +;; so we precompute, in pure Prologos: +;; +;; weights = (1 − α) · Cᵀ +;; biases = α · pretrust +;; +;; …and hand both to net-add-prop. The Racket side never sees the matrix +;; or the decay constant. +spec eigentrust [List Posit32] [List [List Posit32]] Posit32 Nat -> [List Posit32] +defn eigentrust [pretrust matrix decay iters] + let one-minus-decay := [p32- 1.0 decay] + let weights := [scale-mat one-minus-decay [transpose matrix]] + let biases := [scale-vec decay pretrust] + let zeros := [zeros-of pretrust] + let h := [et/net-new 1000000.0] + let init-cell := [et/net-new-cell h pretrust] + let final-cell := [drive h init-cell weights biases zeros iters] + et/net-run-read h final-cell + + +;; ---------- Concrete reputation graph ---------- +;; +;; 0 1 2 3 +;; 0 [ 0.0 0.5 0.5 0.0 ] +;; 1 [ 0.0 0.0 0.0 1.0 ] +;; 2 [ 0.5 0.0 0.0 0.5 ] +;; 3 [ 0.0 1.0 0.0 0.0 ] + +def trust-matrix : [List [List Posit32]] := + '['[0.0 0.5 0.5 0.0] + '[0.0 0.0 0.0 1.0] + '[0.5 0.0 0.0 0.5] + '[0.0 1.0 0.0 0.0]] + +def pretrust : [List Posit32] := '[0.25 0.25 0.25 0.25] + + +;; Run 20 power iterations with α = 0.15. Converges to +;; ≈ [0.0652 0.4348 0.0652 0.4348]. +eigentrust pretrust trust-matrix 0.15 20N