Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2054ae4
WIP: term tables as relations + get-fresh! mint + cost on views
oflatt Jul 15, 2026
a04e1e1
WIP: proof/AST/ProofList constructors as relations + mint helper
oflatt Jul 15, 2026
110bee6
WIP: flatten all nested proof-construction sites to mint sequences
oflatt Jul 15, 2026
698657e
Rework proof extraction + container-rebuild for relations
oflatt Jul 15, 2026
4259ec9
Add set-if-empty canonicalization: term-only validated, proof mode WIP
oflatt Jul 15, 2026
c0717a4
Faithful term-construction threading for proof-mode canonicalization
oflatt Jul 16, 2026
e14c288
Thread connectors into union proofs via proof-of-max/min
oflatt Jul 16, 2026
ebd0d39
Fix proof-mode canonicalization: reflexive proof for canonical-childr…
oflatt Jul 16, 2026
b91b748
Fix proof-mode canonicalization: thread connector through let-bound t…
oflatt Jul 16, 2026
b5bc96d
Regenerate proof-testing snapshots after canonicalization fix
oflatt Jul 16, 2026
130b83f
Scope nat_conn per instrumentation unit
oflatt Jul 16, 2026
7a408ed
Canonicalize container elements via a UF-Aux table (uniform, no @Congr)
oflatt Jul 16, 2026
3459497
cargo fmt
oflatt Jul 16, 2026
8de6594
Children-free custom-merge proofs + reflexive rule premises
oflatt Jul 17, 2026
7fce91a
Run custom merges in the FD view's :merge (drop current+rule)
oflatt Jul 18, 2026
c8d59a0
DD backend: fix build + unify fresh-id minting with core-relations
oflatt Jul 18, 2026
33ddfe3
Route get-fresh!/set-if-empty through the backend SPI
oflatt Jul 18, 2026
1788cc3
Fix proof-extraction order-dependence + custom eq-sort output rebuild
oflatt Jul 18, 2026
ea2da6d
Make get-fresh! a single generic primitive (sort as string arg)
oflatt Jul 18, 2026
cc83e75
Drop @ prefix from set-if-empty/view-proof primitive names
oflatt Jul 18, 2026
e58ba7d
Batch (input …) fiat loading into one bodyless rule (avoid per-mint g…
oflatt Jul 19, 2026
5e15bb7
Load (input …) facts natively instead of compiling a loader rule
oflatt Jul 19, 2026
18f82f4
Desugar globals as functions (not constructor+union) in term/proof en…
oflatt Jul 20, 2026
c9425be
core-relations: reset only modified tables' indexes in merge_all
oflatt Jul 20, 2026
47baa22
egglog-bridge: make free_external_func O(1) via reverse id->message i…
oflatt Jul 20, 2026
2b24395
Make proof extraction deterministic (smallest witness row)
oflatt Jul 20, 2026
0842924
Update proof_tests snapshots + hoisting assertion for the relations e…
oflatt Jul 20, 2026
bd9da5d
docs: add "Building nested terms in actions" section (Neg/Add example)
oflatt Jul 20, 2026
448148b
docs: restructure proof_encoding.md around actions then queries
oflatt Jul 20, 2026
7ed414a
Trim CHANGELOG entries to concise bullets (drop impl detail)
oflatt Jul 20, 2026
56547fe
Address PR review: fix prove-exists panic, drop dead code, tidy comments
oflatt Jul 20, 2026
26660c5
core-relations: remove dead set_counter / Counters::set
oflatt Jul 20, 2026
c649d01
docs: clarify union-in-rule and use e/e'/e'' naming for nested terms
oflatt Jul 20, 2026
3418ec6
Fix global-value proofs; narrow skip-rebuild; drop dead Merge path
oflatt Jul 20, 2026
e18b4ad
docs: address review comments (fd_custom_funcs, displaced comment, ba…
oflatt Jul 20, 2026
f9ab314
docs: note backend needs no union/constructor when evaluating actions
oflatt Jul 20, 2026
3225c4e
Register get-fresh! once per e-graph instead of per eq-sort
oflatt Jul 20, 2026
4b738b1
Move run_merge_subexpr to proof_format (raw->proof conversion)
oflatt Jul 20, 2026
915707d
Restore Database::set_counter (used by the differential-dataflow back…
oflatt Jul 20, 2026
1c0761a
Load all (input ...) natively; make native_input annotation-driven
oflatt Jul 20, 2026
c8f34a5
Make fail take a list of commands
oflatt Jul 20, 2026
1e1a162
Review cleanups: local nat_conn map, output_is_eclass helper, dd add_…
oflatt Jul 20, 2026
5f63d52
Make :no-merge unsupported by the term/proof encoding
oflatt Jul 20, 2026
4a64eec
Support primitive/Unit :no-merge natively; drop fd_custom_funcs
oflatt Jul 20, 2026
a19517a
Clean up proof_encoding rebuilding rules; tighten comments
oflatt Jul 21, 2026
6b82140
Make @UF-Aux name fresh + annotation; mint-first in add_term_and_view
oflatt Jul 21, 2026
a809bea
Add env-gated DD perf diagnostics: phase timing, plan dump, W override
oflatt Jul 21, 2026
a90bd49
Compact DD physical layouts: per-ruleset row width + u128 join keys
oflatt Jul 21, 2026
b08f616
Make host-side bookkeeping O(delta): key index + view event logs
oflatt Jul 21, 2026
233c7ca
Share raw-relation arrangements across all join call sites
oflatt Jul 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

644 changes: 476 additions & 168 deletions egglog-experimental/dd/src/dd_native.rs

Large diffs are not rendered by default.

447 changes: 306 additions & 141 deletions egglog-experimental/dd/src/interpret.rs

Large diffs are not rendered by default.

628 changes: 440 additions & 188 deletions egglog-experimental/dd/src/lib.rs

Large diffs are not rendered by default.

14 changes: 11 additions & 3 deletions egglog-experimental/dd/tests/files.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,24 @@ const DEBUG_SUBSET: &[&str] = &[
/// - container rebuild read primitives (registered through egglog's
/// `ActionRegistry`; DD has direct container storage but no registry
/// execution state for read primitives over its term-encoded mirror):
/// `container-proofs`, `datatypes`, `nested-container-dirty-propagation`,
/// `repro-querybug3`.
/// `container-proofs`, `container-fail`, `datatypes`,
/// `nested-container-dirty-propagation`, `hardboiled_conv1d_32`.
/// - `input` from an external CSV whose path is relative to the corpus dir; the
/// DD harness runs from the `dd` crate root, so the file cannot be found:
/// `string_quotes`.
const KNOWN_UNSUPPORTED: &[(&str, &str)] = &[
("container-proofs.egg", "requires a backend action registry"),
("container-fail.egg", "requires a backend action registry"),
("datatypes.egg", "requires a backend action registry"),
(
"nested-container-dirty-propagation.egg",
"requires a backend action registry",
),
("repro-querybug3.egg", "requires a backend action registry"),
(
"hardboiled_conv1d_32.egg",
"requires a backend action registry",
),
("string_quotes.egg", "string_quotes.csv"),
];

/// Run `program` on DD and render its outputs with the SAME normalization
Expand Down
2 changes: 2 additions & 0 deletions egglog-experimental/src/set_cost.rs
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ impl Macro<Vec<Command>> for SetCostDeclarations {
term_constructor: None,
unextractable: false,
identity_vals: None,
cost: None,
});
}
}
Expand Down Expand Up @@ -159,6 +160,7 @@ fn generate_cost_table_commands_from_variants(variants: &[Variant]) -> Vec<Comma
term_constructor: None,
unextractable: false,
identity_vals: None,
cost: None,
}
})
.collect::<Vec<_>>()
Expand Down
10 changes: 8 additions & 2 deletions egglog-experimental/tests/eggcc_2mm_proof.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,14 @@ fn eggcc_2mm_bounded_export_uses_container_helpers() {
);
}

// `:no-merge` is unsupported by the term/proof encoding, so the bounded export
// rewrites its no-merge functions to `:merge old` to stay proof-supported.
assert!(
non_comment_program.contains(":no-merge"),
"bounded eggcc export should preserve native no-merge declarations"
non_comment_program.contains(":merge old"),
"bounded eggcc export should use `:merge old` for its former no-merge functions"
);
assert!(
!non_comment_program.contains(":no-merge"),
"`:no-merge` is unsupported by the encoding; the bounded export must not use it"
);
}
18 changes: 10 additions & 8 deletions egglog-experimental/tests/fixtures/eggcc-2mm-pass1.egg
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
; egraphs-good/eggcc#796 (https://github.com/egraphs-good/eggcc/pull/796).
; The fixture was checked into egraphs-good/egglog-experimental#56
; (https://github.com/egraphs-good/egglog-experimental/pull/56). Its original
; :no-merge declarations are preserved now that proof mode supports them.
; no-merge declarations are rewritten to `:merge old` here: the term/proof
; encoding does not support no-merge functions, and these length/succ functions
; are deterministic so keeping the old value on a (non-occurring) conflict is sound.

; Prologue
(datatype Expr)
Expand Down Expand Up @@ -93,7 +95,7 @@
(constructor TLConcat (TypeList TypeList) TypeList :unextractable)
(rewrite (TLConcat (TNil) r) r :ruleset type-helpers)
(rewrite (TLConcat (TCons hd tl) r) (TCons hd (TLConcat tl r)) :ruleset type-helpers)
(function TypeList-length (TypeList) i64 :no-merge)
(function TypeList-length (TypeList) i64 :merge old)
(constructor TypeList-ith (TypeList i64) BaseType :unextractable)
(rule () ((set (TypeList-length (TNil)) 0)) :ruleset type-helpers)
(rule ((= lst (TCons hd tl)) (= len (TypeList-length tl))) ((set (TypeList-length lst) (+ 1 len))) :ruleset type-helpers)
Expand Down Expand Up @@ -218,7 +220,7 @@
(rule ((TupleT tylist) (PureTypeList tylist)) ((PureType (TupleT tylist))) :ruleset type-analysis)
(rule ((TNil)) ((PureTypeList (TNil))) :ruleset type-analysis)
(rule ((TCons hd tl) (PureBaseType hd) (PureTypeList tl)) ((PureTypeList (TCons hd tl))) :ruleset type-analysis)
(function ListExpr-length (ListExpr) i64 :no-merge)
(function ListExpr-length (ListExpr) i64 :merge old)
(constructor ListExpr-ith (ListExpr i64) Expr :unextractable)
(constructor ListExpr-suffix (ListExpr i64) ListExpr :unextractable)
(constructor Append (ListExpr Expr) ListExpr :unextractable)
Expand All @@ -227,7 +229,7 @@
(rule ((= (ListExpr-suffix list n) (Nil))) ((set (ListExpr-length list) n)) :ruleset always-run)
(rewrite (Append (Cons a b) e) (Cons a (Append b e)) :ruleset always-run)
(rewrite (Append (Nil) e) (Cons e (Nil)) :ruleset always-run)
(function tuple-length (Expr) i64 :no-merge)
(function tuple-length (Expr) i64 :merge old)
(rule ((HasType expr (TupleT tl)) (= len (TypeList-length tl))) ((set (tuple-length expr) len)) :ruleset always-run)
(relation leading-Expr (Expr))
(relation leading-Expr-list (ListExpr))
Expand Down Expand Up @@ -649,7 +651,7 @@
(constructor AddIntInterval (IntInterval IntInterval) IntInterval)
(rewrite (AddIntInterval (MkIntInterval lo1 hi1) (MkIntInterval lo2 hi2)) (MkIntInterval (AddIntOrInfinity lo1 lo2) (AddIntOrInfinity hi1 hi2)) :ruleset always-run)
(datatype List<i64+IntInterval> (Nil-List<i64+IntInterval>) (Cons-List<i64+IntInterval> i64 IntInterval List<i64+IntInterval>))
(function Length-List<i64+IntInterval> (List<i64+IntInterval>) i64 :no-merge)
(function Length-List<i64+IntInterval> (List<i64+IntInterval>) i64 :merge old)
(rule ((= x (Nil-List<i64+IntInterval>))) ((set (Length-List<i64+IntInterval> x) 0)) :ruleset always-run)
(rule ((= x (Cons-List<i64+IntInterval> hd0 hd1 tl)) (= l (Length-List<i64+IntInterval> tl))) ((set (Length-List<i64+IntInterval> x) (+ l 1))) :ruleset always-run)
(rule ((= x (Nil-List<i64+IntInterval>))) ((set (Length-List<i64+IntInterval> x) 0)) :ruleset memory-helpers)
Expand Down Expand Up @@ -715,7 +717,7 @@
(relation PointsNowhere-PtrPointees (PtrPointees))
(rule ((= f (PointsTo x)) (IsEmpty-List<i64+IntInterval> x)) ((PointsNowhere-PtrPointees f)) :ruleset always-run)
(datatype List<PtrPointees> (Nil-List<PtrPointees>) (Cons-List<PtrPointees> PtrPointees List<PtrPointees>))
(function Length-List<PtrPointees> (List<PtrPointees>) i64 :no-merge)
(function Length-List<PtrPointees> (List<PtrPointees>) i64 :merge old)
(rule ((= x (Nil-List<PtrPointees>))) ((set (Length-List<PtrPointees> x) 0)) :ruleset always-run)
(rule ((= x (Cons-List<PtrPointees> hd0 tl)) (= l (Length-List<PtrPointees> tl))) ((set (Length-List<PtrPointees> x) (+ l 1))) :ruleset always-run)
(rule ((= x (Nil-List<PtrPointees>))) ((set (Length-List<PtrPointees> x) 0)) :ruleset memory-helpers)
Expand Down Expand Up @@ -753,7 +755,7 @@
(rewrite (ExprSet-union (ES set1) (ES set2)) (ES (set-union set1 set2)) :ruleset memory-helpers)
(constructor ExprSet-insert (ExprSet Expr) ExprSet)
(rewrite (ExprSet-insert (ES set1) x) (ES (set-insert set1 x)) :ruleset memory-helpers)
(function ExprSet-length (ExprSet) i64 :no-merge)
(function ExprSet-length (ExprSet) i64 :merge old)
(rule ((ES set1)) ((set (ExprSet-length (ES set1)) (set-length set1))) :ruleset memory-helpers)
(datatype Pointees (TuplePointsTo List<PtrPointees>) (PtrPointsTo PtrPointees))
(constructor UnwrapPtrPointsTo (Pointees) PtrPointees)
Expand Down Expand Up @@ -829,7 +831,7 @@
(rewrite (PointsToCells (Alloc id sz state ty) aps) (TuplePointsTo (Cons-List<PtrPointees> (PointsTo (Cons-List<i64+IntInterval> id (MkIntInterval (I 0) (I 0)) (Nil-List<i64+IntInterval>))) (Cons-List<PtrPointees> (PointsTo (Nil-List<i64+IntInterval>)) (Nil-List<PtrPointees>)))) :ruleset memory-helpers)
(constructor PointsToCellsAtIter (Pointees Expr Expr i64) Pointees)
(rule ((= e (DoWhile inputs pred-body)) (PointsToCells e aps)) ((union (PointsToCellsAtIter aps inputs pred-body 0) (PointsToCells inputs aps)) (union (PointsToCellsAtIter aps inputs pred-body 1) (UnionPointees (PointsToCellsAtIter aps inputs pred-body 0) (PointeesDropFirst (PointsToCells pred-body (PointsToCellsAtIter aps inputs pred-body 0)))))) :ruleset memory-helpers)
(function succ (i64) i64 :no-merge)
(function succ (i64) i64 :merge old)
(rule ((PointsToCellsAtIter aps inputs pred-body i)) ((set (succ i) (+ i 1))) :ruleset memory-helpers)
(rule ((= pointees0 (PointsToCellsAtIter aps inputs pred-body i)) (= pointees1 (PointsToCellsAtIter aps inputs pred-body (succ i))) (Resolved-Pointees pointees0) (Resolved-Pointees pointees1) (!= pointees0 pointees1)) ((union (PointsToCellsAtIter aps inputs pred-body (+ i 2)) (UnionPointees pointees1 (PointeesDropFirst (PointsToCells pred-body pointees1))))) :ruleset memory)
(rule ((= pointees (PointsToCellsAtIter aps inputs pred-body i)) (= pointees (PointsToCellsAtIter aps inputs pred-body (succ i)))) ((union (PointsToCells (DoWhile inputs pred-body) aps) pointees)) :ruleset memory)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,8 @@ expression: snapshot
(premises)
(substitution))
(let t0 (Add (Num 2) (Num 3)))
(let t1 (Add (Num 3) (Num 2)))
(Sym
(= t0 t1)
(Rule
(= t1 t0)
(name "(rewrite (Add a b) (Add b a) :ruleset optimization)")
(premises (Fiat (= t0 t0)))
(substitution (a (Num 2)) (b (Num 3)) (@rewrite_var__ t0))))
(Rule
(= t0 (Add (Num 3) (Num 2)))
(name "(rewrite (Add a b) (Add b a) :ruleset optimization)")
(premises (Fiat (= t0 t0)))
(substitution (a (Num 2)) (b (Num 3)) (@rewrite_var__ t0)))
13 changes: 13 additions & 0 deletions egglog/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,24 @@

## [Unreleased] - ReleaseDate

- The term/proof encoding supports `:no-merge` functions with a primitive or `Unit` output (encoded as an FD view declared native `:no-merge` with an identity-column guard on the output), but no longer supports `:no-merge` functions with an eq-sort output (whose conflict check needs union-find leaders). An eq-sort `:no-merge` program is reported unsupported and must run on the native backend, or give the function a `:merge` (e.g. `:merge old`). This removes the rule/`current`-helper machinery the encoding previously used to emulate `:no-merge`.
- `(fail <command>+)` now accepts multiple commands, running them in order and succeeding if any one fails (previously it wrapped a single command). Desugaring, global removal, and proof encoding keep the whole expansion of a wrapped command inside the `fail`, so `fail` now works over commands that expand to several — including `(fail (set …))` under the term/proof encoding.
- In the term/proof encoding, load `(input …)` for custom functions (with or without `:merge`, including `:no-merge` `Unit`-output ones) natively via `EGraph::native_input`, the same path already used for constructors and relations. This removes the per-input bodyless "loader rule" (and its fresh ruleset) that custom-function inputs used to compile to.
- Make proof extraction deterministic so proof-mode snapshot tests no longer flake on backends with nondeterministic row order (the differential-dataflow backend).
- Speed up freeing external functions, which was quadratic in the number of short-lived one-shot rules created (e.g. under the term/proof encoding).
- Speed up `core-relations`' `merge_all` by resetting only the tables that changed during the call instead of every table.
- Desugar global variables as functions instead of constructor + `union` in the term/proof encoding, and skip rebuilding after non-`union` top-level actions (removing a per-definition rebuild cost).
- Fix user-defined primitives (registered through the Rust API after construction) being reported as unbound under term encoding / proofs: primitive registration now also reaches the term-encoding typechecker, so the encoder can typecheck the encoded program. Previously callers had to manually register the primitive on `proof_state.original_typechecking` as well.
- **Pluggable backend SPI.** `EGraph::with_backend(Box<dyn Backend>)` lets a third party drive the egglog frontend with their own backend (see the `egglog-backend-trait` crate and the `egglog-experimental-dd` example). A backend without a native union-find declares `Backend::requires_term_encoding()`; `EGraph::with_term_encoding()` opts such an e-graph into the term-encoding pipeline (congruence and rebuild lower to rules over `@uf` tables), and running a term-encoding-only backend without it now errors with `Error::BackendRequiresTermEncoding` instead of silently dropping `union`s.
- Route the term/proof encoding's `get-fresh!` (id minting) and `set-if-empty` (view canonicalization) primitives through the backend SPI (`Backend::register_get_fresh` / `register_set_if_empty` / `register_view_proof`), so a backend can service them against its own storage instead of reaching into core-relations tables. The differential-dataflow backend implements them over its host-side mirror and now runs eq-sort programs under term/proof encoding.
- Add `make nightly` and `scripts/nightly_bench.py`, a hyperfine-based benchmark harness that measures every `tests/**/*.egg` program at 1/2/4/8 threads and (where supported) in proof-testing mode, caps each run at a 2-minute timeout, skips sub-50ms programs, and emits an HTML dashboard (one row per benchmark, one column per configuration) for nightly.cs.washington.edu. The dashboard uses [eval-live](https://github.com/oflatt/eval-live) for interactive filtering and sorting.
- Rework the term/proof encoding's union-find and congruence maintenance,
substantially reducing proof-mode time and memory.
- In the term/proof encoding, run a custom function's `:merge` in its
functional-dependency view's own `:merge` (like constructor congruence) instead
of a separate rule plus a `current` helper table. This computes the merge once
rather than twice, so encoded runs no longer mint over-merged extra term rows,
and the merged value is justified by a proper merge-function proof.
- **Tuple-output functions.** A function may declare more than one output sort, e.g.
`(function interval (Math) (i64 i64) :merge (values (max old0 new0) (min old1 new1)))`. Such a
function stores its outputs as separate value columns; the functional dependency is
Expand Down
Loading