Expand paper benchmark fixtures

AGENTS.md

@@ -104,10 +104,12 @@ or modify the JSONL.
 - Benchmark inputs should not contain executable `(prove ...)` commands. Use
   `(check ...)` so the selected treatment controls proof extraction, and cover
   strict proof validity in proof tests.
-- Benchmark files are resolved relative to the command invocation directory,
-  not relative to comparison targets.
-- Cache reuse is decided by binary SHA-256, file SHA-256, fact-directory
-  SHA-256, backend, treatment, and timeout.
+- Benchmark files and their relative `(include ...)` paths are resolved from
+  the command invocation directory, and workloads execute from that directory
+  rather than from comparison targets.
+- Cache reuse is decided by binary SHA-256, the source-closure SHA-256 covering
+  the top-level file and transitive includes, fact-directory SHA-256, backend,
+  treatment, and timeout.
 - The baseline and candidate must have different endpoint cache identities
   (binary SHA-256, backend, and treatment). They may use the same binary when
   backend or treatment differs.

README.md

@@ -187,15 +187,18 @@ selected workloads containing `(input ...)` commands:
 ```
 
 Paths are resolved relative to the command invocation directory, not relative
-to either target. Both endpoints therefore run the exact same file and fact
-directory contents. Their SHA-256 hashes are part of the cache identity.
+to either target. Relative `(include ...)` paths use that same directory, and
+workloads execute there. Both endpoints therefore run the exact same source
+closure and fact-directory contents. The file cache identity covers the
+top-level file and every transitively included file; the fact-directory hash is
+recorded separately.
 
 With no positional files, the representative suite is:
 
-- `egglog/tests/math-microbenchmark.egg`
+- `benchmarks/math-microbenchmark/math-run-010.egg`
 - `egglog-experimental/tests/fixtures/eggcc-2mm-pass1.egg`
-- `benchmarks/pointer-analysis-small.egg`, with
-  `benchmarks/data/pointer-analysis-small`
+- `benchmarks/pointer-analysis-initdb.egg`, with
+  `benchmarks/data/pointer-analysis-initdb`
 - `egglog/tests/hardboiled_conv1d_32.egg`
 - `benchmarks/luminal-llama.egg`
 - `egglog/tests/web-demo/herbie.egg`
@@ -205,13 +208,34 @@ corpus:
 
 | Workload | Adaptation or scope | Correctness signal |
 | --- | --- | --- |
-| Math | Existing synthetic stress fixture | Existing file-test snapshot |
+| Math | The paper artifact's Math rules plus the repository's existing seven-seed adaptation, sampled at ten unrestricted iterations by default; committed wrappers provide sparse checkpoints at 0, 10, ..., 100 iterations, with artifact scheduler parity left for follow-up work | The selected checkpoint checks an integration-by-parts equality at that depth (the zero checkpoint checks the seed) |
 | eggcc 2mm | Existing bounded container fixture | Generated `main` function type is checked |
-| Pointer analysis | First 100 rows from 23 relations; three legacy functions are constructors for current egglog compatibility | Known `constant_points_to` row is derived |
+| Pointer analysis | All 73,864 rows from the 23 `initdb.bc` relations consumed by the adapted program; three legacy functions are constructors for current egglog compatibility | Known `constant_points_to` row is derived |
 | Hardboiled | Dormant canonicalization rules using unsupported unstable helpers are omitted | Extracted WMMA store result is checked |
 | Luminal | Static Llama graph from [`egglog_repro` commit `7fb0194`](https://github.com/saulshanabrook/egglog_repro/blob/7fb0194812b5b11e41a286d8b55e48e3b0bfcd66/llama.egg) | `t712` is checked after kernel lowering |
 | Herbie | Static engine proxy without Racket orchestration or an FPCore corpus | All 14 checks exercise the selected treatment |
 
+The Math wrappers are generated by `scripts/generate_math_checkpoints.py` and
+share `benchmarks/math-microbenchmark/base.egg` through `(include ...)`. Run a
+small depth comparison explicitly with:
+
+```bash
+./bench.py --rounds 1 \
+  benchmarks/math-microbenchmark/math-run-000.egg \
+  benchmarks/math-microbenchmark/math-run-010.egg
+```
+
+See `benchmarks/math-microbenchmark/README.md` for the rule-set provenance and
+the exact adaptation. These wrappers currently use ordinary `(run N)` rather
+than the artifact's backoff scheduler. Checkpoints above ten are opt-in scaling
+cases and can exhaust memory before reaching the timeout.
+
+The pointer input is the complete `initdb.bc` input for the 23 relations read by
+this adaptation, not the artifact's full 30-program pointer-analysis matrix.
+See `benchmarks/data/pointer-analysis-initdb.PROVENANCE.md` for its archive and
+content hashes. Herbie remains the bounded static proxy because the current
+runner does not reproduce the artifact's backoff scheduler.
+
 Benchmark files must not contain executable `(prove ...)` commands. Use
 `(check ...)` in timed workloads so the selected treatment controls whether
 proof extraction is included in the timing boundary.
@@ -436,7 +460,7 @@ separately exports an HTML snapshot.
 Cache reuse is keyed by:
 
 - binary SHA-256;
-- file SHA-256;
+- workload source-closure SHA-256 (the top-level file and transitive includes);
 - fact-directory SHA-256;
 - backend;
 - treatment; and

benchmarking/collection.py

@@ -299,7 +299,7 @@ def run_process(
         summary_path = Path(directory) / "timing-summary.json"
         workload = workload_command(binary_path, file_spec, backend, treatment)
         command = [workload[0], "--timing-summary", str(summary_path), *workload[1:]]
-        result = run_command(command, checkout_path, timeout_sec)
+        result = run_command(command, file_spec.working_directory or checkout_path, timeout_sec)
         require_workload_unchanged(file_spec)
         if result.status != "success":
             return ProcessObservation(result=result, timing_summary=None)

benchmarking/models.py

@@ -73,6 +73,7 @@ class FileSpec:
     sha256: str
     fact_directory: Path | None = None
     fact_directory_sha256: str = ""
+    working_directory: Path | None = None
 
 
 def validate_unique_file_identities(files: Sequence[FileSpec]) -> None:

benchmarking/profile.py

@@ -287,7 +287,7 @@ def run_samply_record(
     try:
         process = subprocess.Popen(
             command,
-            cwd=checkout_path,
+            cwd=file_spec.working_directory or checkout_path,
             env=env,
             stdout=sys.stderr,
             stderr=sys.stderr,
@@ -396,7 +396,11 @@ def run_profile(args: argparse.Namespace, console: Console, invocation_cwd: Path
                     f" {request.backend}/{request.treatment} for {request.mode.profile_seconds}s",
                 )
             )
-            calibration = run_command(workload, checkout_path, request.timeout_sec)
+            calibration = run_command(
+                workload,
+                request.file.working_directory or checkout_path,
+                request.timeout_sec,
+            )
             if calibration.status != "success" or calibration.timing.wall_sec is None:
                 detail = calibration.error.message if calibration.error is not None else calibration.status
                 raise ValueError(f"profile calibration failed: {detail}")

benchmarking/workloads.py

@@ -8,9 +8,12 @@
 
 from __future__ import annotations
 
+import hashlib
+import json
 from collections.abc import Iterator, Sequence
 from dataclasses import dataclass
 from pathlib import Path
+from typing import Literal
 
 from .models import FileSpec, validate_unique_file_identities
 from .targets import sha256_directory, sha256_file
@@ -25,25 +28,40 @@ class WorkloadConfig:
 
 
 DEFAULT_WORKLOADS = (
-    WorkloadConfig("egglog/tests/math-microbenchmark.egg"),
+    WorkloadConfig("benchmarks/math-microbenchmark/math-run-010.egg"),
     WorkloadConfig("egglog-experimental/tests/fixtures/eggcc-2mm-pass1.egg"),
     WorkloadConfig(
-        "benchmarks/pointer-analysis-small.egg",
-        "benchmarks/data/pointer-analysis-small",
+        "benchmarks/pointer-analysis-initdb.egg",
+        "benchmarks/data/pointer-analysis-initdb",
     ),
     WorkloadConfig("egglog/tests/hardboiled_conv1d_32.egg"),
     WorkloadConfig("benchmarks/luminal-llama.egg"),
     WorkloadConfig("egglog/tests/web-demo/herbie.egg"),
 )
 
 
+@dataclass(frozen=True)
+class _EgglogToken:
+    kind: Literal["open", "close", "atom", "string"]
+    value: str = ""
+
+
+@dataclass(frozen=True)
+class WorkloadSourceIdentity:
+    """Content identity and files read by one top-level egglog workload."""
+
+    sha256: str
+    files: tuple[Path, ...]
+
+
 def resolve_files(
     raw_files: Sequence[str],
     invocation_cwd: Path,
     fact_directory: str | None = None,
 ) -> tuple[FileSpec, ...]:
     """Resolve selected or default workloads relative to the invocation directory."""
 
+    working_directory = invocation_cwd.resolve()
     if raw_files:
         chosen = tuple(WorkloadConfig(file, fact_directory) for file in raw_files)
     else:
@@ -59,6 +77,7 @@ def resolve_files(
         absolute_path = absolute_path.resolve()
         if not absolute_path.is_file():
             raise FileNotFoundError(f"benchmark file does not exist: {display_path}")
+        source_identity = workload_source_identity(absolute_path, working_directory)
 
         resolved_fact_directory: Path | None = None
         fact_directory_sha256 = ""
@@ -74,18 +93,19 @@ def resolve_files(
             FileSpec(
                 display_path=display_path,
                 absolute_path=absolute_path,
-                sha256=sha256_file(absolute_path),
+                sha256=source_identity.sha256,
                 fact_directory=resolved_fact_directory,
                 fact_directory_sha256=fact_directory_sha256,
+                working_directory=working_directory,
             )
         )
     resolved = tuple(files)
     validate_workloads(resolved)
     return resolved
 
 
-def _egglog_tokens(source: str) -> Iterator[str | None]:
-    """Yield parentheses and atoms while hiding comments and string contents."""
+def _egglog_tokens(source: str) -> Iterator[_EgglogToken]:
+    """Yield enough lexical structure to inspect top-level commands."""
 
     index = 0
     while index < len(source):
@@ -98,63 +118,142 @@ def _egglog_tokens(source: str) -> Iterator[str | None]:
             index = len(source) if newline == -1 else newline + 1
             continue
         if character == '"':
+            start = index
             index += 1
+            terminated = False
             while index < len(source):
                 if source[index] == "\\":
                     index += 2
                 elif source[index] == '"':
                     index += 1
+                    terminated = True
                     break
                 else:
                     index += 1
-            yield None
+            if not terminated:
+                raise ValueError("unterminated egglog string literal")
+            yield _EgglogToken("string", source[start:index])
             continue
-        if character in "()":
-            yield character
+        if character == "(":
+            yield _EgglogToken("open")
+            index += 1
+            continue
+        if character == ")":
+            yield _EgglogToken("close")
             index += 1
             continue
         end = index
         while end < len(source) and not source[end].isspace() and source[end] not in ";()":
             end += 1
-        yield source[index:end]
+        yield _EgglogToken("atom", source[index:end])
         index = end
 
 
-def file_contains_executable_prove_command(path: Path) -> bool:
-    """Return whether a workload contains a top-level ``prove`` command."""
+def _top_level_commands(source: str) -> Iterator[tuple[_EgglogToken, ...]]:
+    """Yield direct arguments for each top-level command."""
 
     depth = 0
-    expecting_command = False
-    for token in _egglog_tokens(path.read_text(encoding="utf-8")):
-        if token == "(":
+    command: list[_EgglogToken] = []
+    for token in _egglog_tokens(source):
+        if token.kind == "open":
             if depth == 0:
-                expecting_command = True
-            elif expecting_command:
-                expecting_command = False
+                command = []
             depth += 1
-        elif token == ")":
-            if depth == 1:
-                expecting_command = False
-            depth = max(0, depth - 1)
-        elif depth == 1 and expecting_command:
-            if token == "prove":
-                return True
-            expecting_command = False
-    return False
+        elif token.kind == "close":
+            if depth == 0:
+                continue
+            depth -= 1
+            if depth == 0:
+                yield tuple(command)
+        elif depth == 1:
+            command.append(token)
+
+
+def _include_paths(source: str, source_path: Path) -> tuple[str, ...]:
+    includes: list[str] = []
+    for command in _top_level_commands(source):
+        if not command or command[0] != _EgglogToken("atom", "include"):
+            continue
+        if len(command) != 2 or command[1].kind != "string":
+            raise ValueError(f"invalid include command in {source_path}")
+        try:
+            include = json.loads(command[1].value)
+        except json.JSONDecodeError as error:
+            raise ValueError(f"invalid include path in {source_path}: {error.msg}") from error
+        if not isinstance(include, str):
+            raise ValueError(f"invalid include path in {source_path}")
+        includes.append(include)
+    return tuple(includes)
+
+
+def _workload_source_identity(
+    path: Path,
+    working_directory: Path,
+    stack: tuple[Path, ...],
+) -> WorkloadSourceIdentity:
+    resolved_path = path.resolve()
+    if resolved_path in stack:
+        cycle = " -> ".join(str(candidate) for candidate in (*stack, resolved_path))
+        raise ValueError(f"egglog include cycle: {cycle}")
+    if not resolved_path.is_file():
+        raise FileNotFoundError(f"included egglog file does not exist: {resolved_path}")
+
+    source_bytes = resolved_path.read_bytes()
+    source = source_bytes.decode("utf-8")
+    includes = _include_paths(source, resolved_path)
+    if not includes:
+        return WorkloadSourceIdentity(sha256_file(resolved_path), (resolved_path,))
+
+    digest = hashlib.sha256()
+    digest.update(b"egglog-workload-includes-v1\0")
+    digest.update(source_bytes)
+    files: list[Path] = [resolved_path]
+    for include in includes:
+        include_path = Path(include).expanduser()
+        if not include_path.is_absolute():
+            include_path = working_directory / include_path
+        child = _workload_source_identity(include_path, working_directory, (*stack, resolved_path))
+        digest.update(b"\0include\0")
+        digest.update(child.sha256.encode("ascii"))
+        files.extend(child.files)
+    return WorkloadSourceIdentity(
+        f"sha256:{digest.hexdigest()}",
+        tuple(dict.fromkeys(files)),
+    )
+
+
+def workload_source_identity(path: Path, working_directory: Path) -> WorkloadSourceIdentity:
+    """Hash one file and the ordered transitive contents of its includes."""
+
+    return _workload_source_identity(path, working_directory.resolve(), ())
+
+
+def _source_contains_executable_prove_command(path: Path) -> bool:
+    source = path.read_text(encoding="utf-8")
+    return any(command and command[0] == _EgglogToken("atom", "prove") for command in _top_level_commands(source))
+
+
+def file_contains_executable_prove_command(path: Path, working_directory: Path | None = None) -> bool:
+    """Return whether a workload source closure contains a top-level ``prove``."""
+
+    root = path.parent if working_directory is None else working_directory
+    identity = workload_source_identity(path, root)
+    return any(_source_contains_executable_prove_command(source_path) for source_path in identity.files)
 
 
 def require_workload_unchanged(file_spec: FileSpec) -> None:
     """Reject an observation if its mutable inputs no longer match their cache identity."""
 
     try:
-        file_sha256 = sha256_file(file_spec.absolute_path) if file_spec.absolute_path.is_file() else None
+        working_directory = file_spec.working_directory or file_spec.absolute_path.parent
+        file_sha256 = workload_source_identity(file_spec.absolute_path, working_directory).sha256
         if file_spec.fact_directory is None:
             fact_directory_sha256 = ""
         elif file_spec.fact_directory.is_dir():
             fact_directory_sha256 = sha256_directory(file_spec.fact_directory)
         else:
             fact_directory_sha256 = None
-    except OSError as error:
+    except (OSError, UnicodeError, ValueError) as error:
         raise ValueError(f"workload changed during execution: {file_spec.display_path}") from error
     if file_sha256 != file_spec.sha256 or fact_directory_sha256 != file_spec.fact_directory_sha256:
         raise ValueError(f"workload changed during execution: {file_spec.display_path}")
@@ -165,7 +264,7 @@ def validate_workloads(files: Sequence[FileSpec]) -> None:
 
     validate_unique_file_identities(files)
     for file_spec in files:
-        if file_contains_executable_prove_command(file_spec.absolute_path):
+        if file_contains_executable_prove_command(file_spec.absolute_path, file_spec.working_directory):
             raise ValueError(
                 f"{file_spec.display_path} contains an explicit prove command; "
                 "benchmark files should use check so the selected treatment controls proof extraction"