Make parse results deterministic across runs

[vthorsteinsson]

Summary

Repeated parses of the same sentence could yield different trees whenever the parse forest contained subtrees with exactly equal reduction scores — the tie was effectively broken by malloc addresses. This was a long-standing issue, previously worked around in tests by comparing forest combination counts instead of reduced trees.

Root cause

State::getHash() in eparser.cpp XOR-ed raw heap pointers (m_pProd, m_pw) into the hash. Since a Column stores Earley states in hash bins and Column::nextState() enumerates them bin by bin, the state processing order varied with allocation addresses between runs. That changed the order of addFamily() calls when building the SPPF, and thereby the family indices used by the reducer's "lowest index wins" tie-breaker in reducer.py.

Changes

• eparser.h / eparser.cpp: State::getHash() is now content-based, using Production::getId() and a new Label::getHash()/Node::getHash() (nonterminal index, dot, production id, token span) instead of pointer values. State equality is unchanged; only bin placement needed determinism.
• grammar.py: Terminal.__hash__ and Nonterminal.__hash__ now use the creation-order sequence number instead of id(), so set/dict iteration order over grammar items is stable across processes and PYTHONHASHSEED values. The cached _hash snapshot stays separate from _index, which is renumbered after the grammar is read.
• test_parse.py: new test_deterministic_reduction with sentences that previously flipped between equal-score parses; verified to fail against the old parser core.
• test_native_matching.py: updated a comment documenting the old nondeterministic tie-breaking.

Verification

• Pre-fix, canonical forest dumps of the same sentence differed on every parse (same content, different family order); post-fix they are byte-identical across in-process runs, separate processes, and PYTHONHASHSEED values.
• The new regression test fails against the old _eparser build and passes with this change.
• Full test suite passes (120 tests); ruff and mypy clean; no measurable parse performance change (the new hash is a handful of integer multiply-adds, on par with the pointer XOR it replaces).

Note

This can change which of two equally-scored parses is returned compared to 3.7.1 — stably, from now on. Worth a mention in the release notes of the next release.

🤖 Generated with Claude Code