ERM

Evaluation entry points for Energy-guided Recursive Model (ERM) experiments on Sudoku, PPBench, and Maze.

Paper

This repository accompanies Energy-guided Recursive Model (arXiv:2607.10128).

This repository compares EqR candidate selectors under a shared K-candidate protocol and, optionally, under parallel tempering (PT):

• majority vote
• q-head selector
• confidence selector
• ERM energy selector
• oracle / pass@K

These files are intended as release scripts, not as machine-local server launchers. Paths are supplied explicitly by command line.

Repository Contents

File	Purpose
inference/sudoku_erm_eqr_inference.py	Sudoku shared-candidate and PT ERM evaluation.
inference/ppbench_erm_eqr_inference.py	PPBench 5-type shared-candidate and PT ERM evaluation.
inference/maze_erm_eqr_inference.py	Maze shared-candidate and PT ERM evaluation with rule-Hopfield path energy.
requirements.txt	Minimal Python package list.

Setup

Install the Python dependencies:

pip install -r requirements.txt

Required Inputs

Each script expects three external inputs:

1. An EqR source tree passed by --code_root. The path must contain the models/ and dataset/ packages used by the original EqR training code.
2. A trained checkpoint passed by --checkpoint.
3. The dataset root passed by --data_path.

Note

The scripts do not download checkpoints or datasets. Release repositories should provide download links or instructions separately.

Evaluation

Common Arguments

Argument	Meaning
--code_root	EqR source tree to import.
--checkpoint	Checkpoint .pth file.
--data_path	Dataset root.
--split	Evaluation split. Defaults are task-specific.
--phase	shared, pt, or all.
--K / --k	Number of candidates or PT replicas. Default: 128.
--D	Maximum recurrent steps. Default: 64.
--out_json	Output JSON path.
--max_batches	Debug subset. 0 means full split.
--allow_partial_load	Allow non-strict checkpoint loads. Do not use for paper numbers unless the missing/unexpected keys are understood.

Important

The checkpoint loader is intentionally strict by default. A run fails if no parameters match, or if there are missing/unexpected keys. This avoids silently reporting numbers from a bad checkpoint load.

Sudoku

python inference/sudoku_erm_eqr_inference.py \
  --code_root /path/to/sudoku_eqr_code \
  --checkpoint /path/to/sudoku_checkpoint.pth \
  --data_path /path/to/sudoku-lite-eval \
  --split test \
  --phase all \
  --K 128 \
  --D 64 \
  --out_json sudoku_erm_eval_result.json

The ERM score is a Sudoku factor-memory energy based on row, column, and box constraints, with clue cells clamped.

PPBench

python inference/ppbench_erm_eqr_inference.py \
  --code_root /path/to/ppbench_eqr_code \
  --checkpoint /path/to/ppbench_checkpoint.pth \
  --data_path /path/to/ppbench_5type_pid1 \
  --split golden \
  --phase all \
  --K 128 \
  --D 64 \
  --out_json ppbench_erm_eval_result.json

PPBench ERM is type-aware. The file below is required:

<data_path>/<split>/all__types.npy

Warning

If that file is missing, the script fails instead of falling back to an unknown puzzle type. This is deliberate: without type labels, the PPBench ERM selector is not the method reported in the paper.

Important

The PPBench architecture uses mlp_t=False with RoPE positional encoding. Do not evaluate a mlp_t=False checkpoint without positional encoding; that configuration is permutation-invariant for the relevant attention block and is known to collapse.

Maze

python inference/maze_erm_eqr_inference.py \
  --code_root /path/to/maze_eqr_code \
  --checkpoint /path/to/maze_checkpoint.pth \
  --data_path /path/to/maze-30x30-unique-1k \
  --split test \
  --phase all \
  --K 128 \
  --D 64 \
  --out_json maze_erm_eval_result.json

The Maze ERM selector uses rule-Hopfield path energy over legal simple S -> G paths, with local path-degree factors as tie-breakers. It does not use a precomputed target path helper.

Output

Each script writes a JSON payload with the checkpoint, dataset, split, K, D, baseline result, shared-candidate results, and PT results if requested. Metrics include:

Metric	Meaning
exact	Sequence-level exact accuracy.
token	Token accuracy over non-ignored labels.
exact_n	Exact count.
valid_n	Number of valid examples.

Reproducibility Notes

• Use the same EqR source version that produced the checkpoint.
• Use full precision settings and CUDA/PyTorch versions compatible with the original training environment.
• For paper numbers, run with --max_batches 0 and without --allow_partial_load.
• Keep the output JSONs from full runs in the release or artifact bundle so users can compare counts, not only percentages.

Acknowledgements

The training code used in this work is based on Equilibrium Reasoners (EqR). We thank the EqR authors for making their code publicly available.