Expand paper benchmark fixtures

[saulshanabrook]

Expands size of pointer analysis to full size of smallest egglog paper benchmark and adds multiple versions of math microbenchmark for different run lengths

Summary by CodeRabbit

• New Features

  • Added generated Math benchmark checkpoints from 0–100 iterations, with validation for missing or outdated files.
  • Benchmarks and profiling now run from each workload’s configured working directory.

• Bug Fixes

  • Cache reuse now detects changes in included and nested source files.
  • Added validation for missing, malformed, cyclic, or invalid included workload files.
  • Improved detection of proof commands in included sources.

• Documentation

  • Expanded benchmark guidance, provenance details, workload descriptions, and scaling limitations.
  • Documented the complete Pointer Analysis input dataset.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds invocation-directory-aware benchmark execution, transitive include hashing for cache identity and mutation checks, generated Math checkpoint workloads, updated pointer-analysis data, provenance documentation, and proof-mode regression coverage.

Changes

Benchmark workload and checkpoint updates

Layer / File(s)	Summary
Workload identity and invocation-directory execution benchmarking/models.py, benchmarking/workloads.py, benchmarking/collection.py, benchmarking/profile.py, tests/test_workloads.py, tests/test_collection.py, tests/test_profile.py, AGENTS.md, README.md	FileSpec stores the invocation working directory. Benchmark and profiling commands use it when configured. Workload hashes and mutation checks now cover ordered transitive includes and validate missing files, cycles, malformed input, and encoding errors.
Generated Math checkpoint workloads scripts/generate_math_checkpoints.py, tests/test_math_checkpoints.py, benchmarks/math-microbenchmark/README.md, scripts/__init__.py, README.md	The generator writes and validates checkpoints from 0 through 100 iterations in steps of 10. Tests verify generated content, annotations, and terminal checks.
Proof regressions and benchmark provenance egglog/tests/proof_mode_regression.rs, benchmarks/data/pointer-analysis-initdb.PROVENANCE.md	Proof tests use the pointer-analysis initdb.bc dataset and generated Math checkpoints. Tests cover zero and ten iterations, plus failure at nine iterations. Provenance records the source artifact and included CSV coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Invocation as Invocation directory
  participant Workloads as benchmarking.workloads
  participant Runner as benchmarking.collection
  Invocation->>Workloads: Resolve source and include paths
  Workloads->>Workloads: Hash transitive source closure
  Workloads->>Runner: Return FileSpec with working_directory
  Runner->>Invocation: Execute benchmark from working directory

Loading

Possibly related PRs

• saulshanabrook/egglog-encoding#5: Extends the benchmarking system, including workload specifications, execution, caching, documentation, and tests.
• saulshanabrook/egglog-encoding#14: Both changes update benchmark workload configuration and cache identity.
• saulshanabrook/egglog-encoding#25: Both changes update proof-checking regression tests.

Suggested reviewers: oflatt

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.69% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the expansion of pointer-analysis and math benchmark fixtures described in the pull request.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch codex/full-pointer-math-checkpoints

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/generate_math_checkpoints.py`:
- Around line 32-34: Update scripts/generate_math_checkpoints.py lines 32-34 to
emit a non-reflexive zero-depth assertion that verifies the initialized seed.
Update README.md lines 211-211 and benchmarks/math-microbenchmark/README.md
lines 22-24 to describe zero-checkpoint coverage accurately. Update
egglog/tests/proof_mode_regression.rs lines 216-225 to assert the zero-depth
property through the corrected checkpoint.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f63d3d27-7fe5-49e7-b162-ba5701556a55

📥 Commits

Reviewing files that changed from the base of the PR and between 6ef88f1 and a90ff9a.

⛔ Files ignored due to path filters (53)

• benchmarks/data/pointer-analysis-initdb/alloc_aliases.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/alloc_matches.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/alloc_subregion_at_path.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/allocation.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/allocation_by_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/assign_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/call_instruction_arg.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/call_instruction_fn_target.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/constant_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/constant_ptr_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/function.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/function_nparams.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/function_param.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/gep_var_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/getelementptr_instruction_base.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/instruction_assigns_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/instruction_in_function.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/load_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/load_instruction_address.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/ret_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/ret_instruction_value.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/store_instruction_address.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-initdb/store_instruction_value.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/alloc_aliases.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/alloc_matches.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/alloc_subregion_at_path.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/allocation.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/allocation_by_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/call_instruction_arg.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/call_instruction_fn_target.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/constant_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/constant_ptr_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/gep_var_points_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/getelementptr_instruction_base.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/instruction_assigns_to.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/instruction_in_function.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/load_instruction.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/load_instruction_address.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/store_instruction_address.csv is excluded by !**/*.csv
• benchmarks/data/pointer-analysis-small/store_instruction_value.csv is excluded by !**/*.csv
• benchmarks/math-microbenchmark/base.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-000.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-010.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-020.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-030.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-040.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-050.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-060.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-070.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-080.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-090.egg is excluded by !**/*.egg
• benchmarks/math-microbenchmark/math-run-100.egg is excluded by !**/*.egg
• benchmarks/pointer-analysis-initdb.egg is excluded by !**/*.egg

📒 Files selected for processing (15)

• AGENTS.md
• README.md
• benchmarking/collection.py
• benchmarking/models.py
• benchmarking/profile.py
• benchmarking/workloads.py
• benchmarks/data/pointer-analysis-initdb.PROVENANCE.md
• benchmarks/math-microbenchmark/README.md
• egglog/tests/proof_mode_regression.rs
• scripts/__init__.py
• scripts/generate_math_checkpoints.py
• tests/test_collection.py
• tests/test_math_checkpoints.py
• tests/test_profile.py
• tests/test_workloads.py

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Replace the reflexive zero-checkpoint assertion.

(check (= {seed} {seed})) succeeds by reflexivity. It does not show that base.egg initialized the seed before the check. The zero-depth regression therefore only verifies parsing and execution, despite the documentation claim.

• scripts/generate_math_checkpoints.py#L32-L34: emit a non-reflexive assertion that depends on the initialized seed.
• README.md#L211-L211: describe the zero checkpoint as parse/run coverage unless the assertion changes.
• benchmarks/math-microbenchmark/README.md#L22-L24: make the same documentation correction.
• egglog/tests/proof_mode_regression.rs#L216-L225: assert the zero-depth property through the corrected checkpoint.

📍 Affects 4 files

• scripts/generate_math_checkpoints.py#L32-L34 (this comment)
• README.md#L211-L211
• benchmarks/math-microbenchmark/README.md#L22-L24
• egglog/tests/proof_mode_regression.rs#L216-L225

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/generate_math_checkpoints.py` around lines 32 - 34, Update
scripts/generate_math_checkpoints.py lines 32-34 to emit a non-reflexive
zero-depth assertion that verifies the initialized seed. Update README.md lines
211-211 and benchmarks/math-microbenchmark/README.md lines 22-24 to describe
zero-checkpoint coverage accurately. Update
egglog/tests/proof_mode_regression.rs lines 216-225 to assert the zero-depth
property through the corrected checkpoint.