diff --git a/README.md b/README.md index cb094d4..17dac4b 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,8 @@ gen-scope is generic. It has no knowledge of NixOS, aspects, policies, or system - [API Reference](#api-reference) - [eval](#eval) - [evalDebug](#evaldebug) + - [evalWarm](#evalwarm) + - [recordedDeps](#recordeddeps) - [buildNodes](#buildnodes) - [Algebraic Graph Construction](#algebraic-graph-construction) - [Attribute Combinators](#attribute-combinators) @@ -221,6 +223,40 @@ Returns `{ node, get, allNodes, allNodesWhere, subtreeOf, nodesOfType }`: Same interface as `eval`. Provides structured cycle traces instead of Nix's opaque "infinite recursion." Trade-off: defeats memoization. Use for diagnosing cycles only. +### `evalWarm` + +```nix +evalWarm { + roots; # { id = { id, type, parent, decls }; } + attributes; # { attrName = self: id: value; } + parseParent ? null; # id → parentId | null + priorResults; # { id = { attrName = cachedValue; }; } + isClean; # id → bool +} +``` + +Warm-cache variant of `eval` for incremental re-evaluation. A leaf attribute of a **clean** node (`isClean id == true`) whose value is present in `priorResults` is served from the cache **without forcing its compute function**; everything else evaluates cold. `children`/`derived-children` are **never** served warm — tree structure is always recomputed, so a dirty descendant stays reachable through freshly-materialized parents. Returns the same shape as `eval`. + +With `eval`'s defaults (`priorResults = {}`, `isClean = _: false`) the warm branch never fires, so `eval` and warm-off `evalWarm` are byte-identical — they share a single code path. + +```nix +result = engine.evalWarm { + inherit roots attributes; + priorResults = { "host:web" = { region = "us-east"; }; }; # from a prior eval + isClean = id: id != "host:db"; # only db changed +}; +result.get "host:web" "region" # "us-east" — served warm, compute fn not forced +result.get "host:db" "region" # recomputed (db is dirty) +``` + +### `recordedDeps` + +```nix +recordedDeps { declaredEdges } id # → [id] +``` + +First-class projection of a consumer's **declared** read-edges: it simply applies `declaredEdges id`. Pure and memo-free — it never runs through `get`. The *dynamic* read-set (the attributes a node actually `self.get`s) is only recoverable via `evalDebug`'s fresh-`self`-per-`get`, which defeats memoization; there is no pure, memo-preserving way to capture it, so the declared edges are the inspectable contract. Useful for incremental consumers that need an explicit dependency edge set (e.g. driving a rebuild). + ### `buildNodes` ```nix @@ -317,10 +353,12 @@ collectionAttr { traverse; extract; combine ? a: b: a ++ b; filter ? _: true; } Traverse modes: `"imports"`, `"children"`, `"siblings"`, `"ancestors"`, `"neron"`, `"label:"`, or custom function. -**`"neron"` traverse mode** — Specificity-ordered collection following D > I > P (declaration, import, parent) priority. Unlike `query`, which returns a single shadowed result, `"neron"` returns all contributions from reachable scopes in specificity order, suitable for fold-based composition (e.g., collecting all modules, merging config fragments). +**`"neron"` traverse mode** — Specificity-ordered collection following D over I over P (local shadows import shadows parent) priority. Unlike `query`, which returns a single shadowed result, `"neron"` returns all contributions from reachable scopes in specificity order, suitable for fold-based composition (e.g., collecting all modules, merging config fragments). Properties: cycle-safe via seen-set tracking, diamond-safe deduplication (each scope visited at most once), recursive parent resolution. Traversal order: self, then unseen imports, then parent — mirroring the Neron (2015) resolution calculus but collecting rather than shadowing. +The neron traversal order (self → imports → parent, imports in declaration order) is a public, stable contract: collection determinism for ordered-list channels rests on this pin plus a left fold, so changing the traversal order is a breaking change. + ```nix # Collect all config fragments from local scope, imports, and ancestors config-modules = engine.collectionAttr { @@ -409,7 +447,7 @@ just ci eval # run eval suite just ci eval.test-basic-root-attribute # specific test ``` -Requires nix-unit. 152 tests across 10 suites. +Requires nix-unit. 163 tests across 19 suites (12 test files). ## Theoretical Foundations @@ -425,3 +463,4 @@ Requires nix-unit. 152 tests across 10 suites. | Radul & Sussman (2009) "Art of the propagator" | **Informed by** | Monotonic convergence concept for `circular` attribute iteration; cells accepting information from multiple sources as design influence on scope graph merging | | Van Wyk et al. (2010) "Silver: extensible AG" | **Informed by** | Forwarding concept (productions defining default attribute values via translation); collection attributes with fold operators as design influence on `collectionAttr` | | Mokhov et al. (2018) "Build systems a la carte" | **Informed by** | Demand-driven evaluation as suspending scheduler (§4.1); Nix's lazy evaluation recognized as the scheduling mechanism — we do not build a scheduler, Nix is the scheduler | +| Acar et al. (2006) "Adaptive functional programming" | **Informed by** | Warm-cache incremental re-evaluation (`evalWarm`): reusing clean prior results and recomputing only dirty nodes; `recordedDeps` as the declared read-edge projection of a dynamic dependence graph | diff --git a/ci/tests/eval.nix b/ci/tests/eval.nix index d3221ae..aa81b7e 100644 --- a/ci/tests/eval.nix +++ b/ci/tests/eval.nix @@ -314,4 +314,282 @@ in ]; }; }; + + # appended inside the same `{ ... }` the module returns, as a sibling of flake.tests."eval" + flake.tests."eval-warm" = + let + mkRoots = + decls: + lib.mapAttrs (id: d: { + inherit id; + type = "t"; + parent = null; + decls = d; + }) decls; + poison = { + children = self: id: { }; + boom = self: id: throw "boom-${id}"; + val = self: id: (self.node id).decls.v or 0; + }; + in + { + # clean + prior present ⇒ warm-served, fn NEVER forced (poison doesn't throw) + test-warm-serves-clean-no-force = { + expr = + (genScope.evalWarm { + roots = mkRoots { + n = { + v = 5; + }; + }; + attributes = poison; + priorResults = { + n = { + boom = "cached"; + }; + }; + isClean = _: true; + }).get + "n" + "boom"; + expected = "cached"; + }; + # dirty ⇒ recompute ⇒ poison fn runs ⇒ throws + test-dirty-recomputes = { + expr = + (builtins.tryEval ( + (genScope.evalWarm { + roots = mkRoots { + n = { + v = 5; + }; + }; + attributes = poison; + priorResults = { + n = { + boom = "cached"; + }; + }; + isClean = _: false; + }).get + "n" + "boom" + )).success; + expected = false; + }; + # clean but attr absent from priorResults ⇒ falls through to fn ⇒ throws + test-warm-missing-attr-falls-through = { + expr = + (builtins.tryEval ( + (genScope.evalWarm { + roots = mkRoots { + n = { + v = 5; + }; + }; + attributes = poison; + priorResults = { + n = { }; + }; + isClean = _: true; + }).get + "n" + "boom" + )).success; + expected = false; + }; + # warm value is served verbatim (not the fresh computation) + test-warm-serves-prior-value = { + expr = + (genScope.evalWarm { + roots = mkRoots { + n = { + v = 5; + }; + }; + attributes = poison; + priorResults = { + n = { + val = 99; + }; + }; + isClean = _: true; + }).get + "n" + "val"; + expected = 99; + }; + # a dirty node reads a clean dep's WARM value + test-dirty-reads-clean-dep = { + expr = + (genScope.evalWarm { + roots = mkRoots { + a = { + base = 10; + }; + b = { + base = 0; + }; + }; + attributes = { + children = self: id: { }; + val = self: id: if id == "b" then (self.get "a" "val") + 1 else (self.node id).decls.base; + }; + priorResults = { + a = { + val = 99; + }; + }; + isClean = id: id == "a"; + }).get + "b" + "val"; + expected = 100; + }; + # children NEVER warm-served (stale prior children ignored; fresh structure used) + test-children-always-recomputed = { + expr = + let + attrs = { + children = + self: id: + if id == "p" then + { + c = { + id = "c"; + type = "t"; + parent = "p"; + decls = { }; + }; + } + else + { }; + label = self: id: "fresh-${id}"; + }; + w = genScope.evalWarm { + roots = mkRoots { p = { }; }; + attributes = attrs; + priorResults = { + p = { + children = { + BOGUS = { }; + }; + }; + }; + isClean = _: true; + }; + in + builtins.attrNames (w.get "p" "children"); + expected = [ "c" ]; + }; + # a dirty GRANDCHILD is reachable through freshly-recomputed children + test-dirty-grandchild-reachable = { + expr = + let + attrs = { + children = + self: id: + if id == "p" then + { + c = { + id = "c"; + type = "t"; + parent = "p"; + decls = { }; + }; + } + else if id == "c" then + { + g = { + id = "g"; + type = "t"; + parent = "c"; + decls = { }; + }; + } + else + { }; + label = self: id: "fresh-${id}"; + }; + w = genScope.evalWarm { + roots = mkRoots { p = { }; }; + attributes = attrs; + priorResults = { + g = { + label = "stale-g"; + }; + }; + isClean = id: id != "g"; # g dirty, p/c clean + }; + in + w.get "g" "label"; + expected = "fresh-g"; # g recomputed (dirty) AND reachable (children recomputed, never warm) + }; + # per-(node,attr) granularity: clean node, attr a warm / attr b cold + test-mixed-warm-cold = { + expr = + let + w = genScope.evalWarm { + roots = mkRoots { n = { }; }; + attributes = { + children = self: id: { }; + a = self: id: "fresh-a"; + b = self: id: "fresh-b"; + }; + priorResults = { + n = { + a = "cached-a"; + }; + }; # has a, not b + isClean = _: true; + }; + in + [ + (w.get "n" "a") + (w.get "n" "b") + ]; + expected = [ + "cached-a" + "fresh-b" + ]; + }; + # evalWarm with warm-off defaults == eval + test-evalWarm-defaults-equal-eval = { + expr = + (genScope.evalWarm { + roots = mkRoots { + n = { + v = 7; + }; + }; + attributes = poison; + priorResults = { }; + isClean = _: false; + }).get + "n" + "val"; + expected = + (genScope.eval { + roots = mkRoots { + n = { + v = 7; + }; + }; + attributes = poison; + }).get + "n" + "val"; + }; + }; + + flake.tests."recorded-deps" = { + test-recordedDeps-declared = { + expr = genScope.recordedDeps { declaredEdges = id: if id == "b" then [ "a" ] else [ ]; } "b"; + expected = [ "a" ]; + }; + test-recordedDeps-empty = { + expr = genScope.recordedDeps { declaredEdges = _: [ ]; } "x"; + expected = [ ]; + }; + }; } diff --git a/lib/eval.nix b/lib/eval.nix index 5787212..a456e96 100644 --- a/lib/eval.nix +++ b/lib/eval.nix @@ -15,41 +15,37 @@ let roots, attributes, parseParent ? null, + priorResults ? { }, + isClean ? (_: false), }: lib.fix ( self: let - # Wrap a child node with a lazy attribute cache (_eval). - # The cache propagates recursively: _eval.children wraps grandchildren. + # One per-attribute evaluator, shared by rootEval + wrapChild._eval. + # children/derived-children always recompute the structure (descendant + # _eval caches are built by recursing through wrapChild). A clean node's + # leaf attr present in priorResults is served without forcing fn (warm, + # relocatable); otherwise cold demand. With the defaults (isClean = _: false) + # the warm branch never fires, so eval stays byte-identical. + evalAttr = + nodeId: attrName: fn: + if attrName == "children" || attrName == "derived-children" then + let + raw = fn self nodeId; + in + builtins.mapAttrs (_: wrapChild) raw + else if isClean nodeId && (priorResults.${nodeId} or { }) ? ${attrName} then + priorResults.${nodeId}.${attrName} + else + fn self nodeId; wrapChild = childNode: childNode // { - _eval = builtins.mapAttrs ( - attrName: fn: - if attrName == "children" || attrName == "derived-children" then - let - raw = fn self childNode.id; - in - builtins.mapAttrs (_: wrapChild) raw - else - fn self childNode.id - ) attributes; + _eval = builtins.mapAttrs (attrName: fn: evalAttr childNode.id attrName fn) attributes; }; - - # Root memoization: each root gets a lazy attrset of its attribute computations. rootEval = lib.mapAttrs ( - id: _: - builtins.mapAttrs ( - attrName: fn: - if attrName == "children" || attrName == "derived-children" then - let - raw = fn self id; - in - builtins.mapAttrs (_: wrapChild) raw - else - fn self id - ) attributes + id: _: builtins.mapAttrs (attrName: fn: evalAttr id attrName fn) attributes ) roots; # Resolve a node by ID. @@ -234,7 +230,44 @@ let }; in mkSelf { } [ ]; + + # Relocatable warm-cache evaluator. Same interface as `eval`; a clean node's + # leaf attr present in priorResults is served without forcing its compute fn, + # while children/derived-children are never warm-served (structure always + # recomputed, so dirty descendants stay reachable). Thin wrapper over `eval` — + # a single code path, with `eval`'s defaults being the warm-off case. + # (Informed by Acar's self-adjusting computation: reusing clean prior results.) + evalWarm = + { + roots, + attributes, + parseParent ? null, + priorResults, + isClean, + }: + eval { + inherit + roots + attributes + parseParent + priorResults + isClean + ; + }; + + # First-class inspectable projection of the consumer's declared read-edges. Pure, + # zero memo cost (never runs through `get`). The dynamic read-set — the attributes + # a node actually self.get's — is only recoverable via evalDebug's fresh-self-per-get, + # which defeats the memo; there is no pure, memo-preserving way to capture it, so the + # declared edges are the inspectable contract. (Acar's self-adjusting computation: + # the read edges of a dynamic dependence graph.) + recordedDeps = { declaredEdges }: id: declaredEdges id; in { - inherit eval evalDebug; + inherit + eval + evalDebug + evalWarm + recordedDeps + ; }