Rust ENDF-6 parser crate with Python bindings and a parity harness

[shimwell]

Moves the reader to Rust, so the same parser can serve consumers that never load Python — in particular a converter emitting Arrow tables directly.

Additive. No Python is changed or removed except to teach the package to read .xz files; the existing pytest suite still gives 168 passed, 2 skipped, and 235 passed once the extension module is built.

Layout

crates/
├── endf/      the parser. No Arrow, no Python, no dependencies at all.
└── endf-py/   PyO3 bindings. Thin: every type forwards to the Rust one.

endf describes the formats and nothing more. A simulation-ready projection — reconstructed resonances, unionised grids, an Arrow schema — is a consumer's concern, so it can depend on arrow-rs without that cost reaching everyone who just wants to read a file.

What is ported

Every module of the package, module for module, except where noted under Gaps:

Format layer	records, function, material, and MF 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 23, 26, 27, 28, 33, 34, 40
Processed formats	ace (Type 1 tables), urr
Distributions	univariate, angle_energy, plus AngleDistribution and EnergyDistribution spanning ENDF and ACE alike
Objects	product, reaction, incident_neutron, incident_photon, decay, radionuclide_production, chain
Support	data, fission_energy, njoy

Each covers every representation the file defines, not only the ones the fixtures happen to use. About 13,300 lines of Rust, with 2,300 of comment and 2,300 of unit test alongside.

The Python extension

endf-py exposes both layers a consumer reads.

The concrete types come across as classes: Material, Tabulated1D, Tabulated2D, CrossSection, Product, Reaction, IncidentNeutron, IncidentPhoton, Decay, Chain, AceTable, FissionProductYields, RadionuclideProduction. Beside them the free functions — float_endf, int_endf, get_materials, get_tables, ace_tables_from_string, reaction_name, reaction_mt, photon_reaction_name, photon_reaction_mt, gnds_name, zam, temperature_str, decay_modes, normalise_branch_ratios, radionuclide_production, isomer_table, level_to_isomeric_state — and the constant tables ATOMIC_SYMBOL, SUM_RULES, INTERPOLATION_SCHEME, FISSION_MTS, EV_PER_MEV, K_BOLTZMANN.

The sum types come across as dicts tagged with a kind key rather than as a class per variant:

>>> neutron[51].products[0].distribution[0]
{'kind': 'uncorrelated',
 'angle': {'energy': [...], 'mu': [{'kind': 'legendre', 'coefficients': [...]}, ...]},
 'energy': {'kind': 'level-inelastic', 'threshold': 1.4e6, 'mass_ratio': 0.98}}

kind is exactly the discriminant an Arrow union column needs, and the alternative is a dozen wrapper classes expressing nothing more.

Material.section_data is a drop-in. Every one of the 414 sections across the fixtures comes back as the same dictionary the Python reader builds, keyed by the same ENDF field names, so code reaching into material[3, 1]['sigma'] runs unchanged against either reader. Material.interpret() is there too, picking IncidentNeutron or IncidentPhoton by NSUB.

The upstream quirks come with them, because matching means matching: MT=458 reports ZA as a float since it is read from a CONT record, MF=7 MT=4 stores the outer LT on each additional temperature rather than the LI it read, an unresolved range with LRF=1 is dispatched past unread, and a decay record too short for its internal conversion coefficients yields an empty tuple rather than a zero. Each is commented at its site with the issue that tracks it.

How it is tested

tools/dump_golden.py writes what the Python reader produces for a fixture; crates/endf/tests/golden.rs reads each dump, runs the Rust reader over the file it names, builds the same path -> value map from its own parse, and compares the two maps whole. A field renamed, dropped or added shows up as a path on one side and not the other.

Values are compared bit-for-bit. The dump records the shortest round-tripping decimal and both readers round correctly, so any difference is real. Only computed values use a 1e-12 relative tolerance, and is_interpolated lists exactly which: sampled interpolation; the MF=10 yields, which divide two interpolated cross sections; the propagated uncertainties, where uncertainties accumulates a variance and takes its square root; and the forward-scattered fraction with the removal cross section that folds it in, because NumPy re-associated the Clenshaw recurrence between 2.2 and 2.4. Nothing that comes off the file is compared loosely.

The dump goes well past the parsed sections — every reaction, every nuclide, the MF=8/9/10 join, decay data with its source distributions, photon data with its atomic relaxation, and a whole depletion chain through a KIND chain golden naming ten decay and two neutron evaluations. For ACE it walks the AND block, the whole of DLW following each reaction's linked list, every reaction including elastic, and the nuclide they belong to. The NJOY input deck is held to Python's byte for byte.

The bindings are tested the same way: tests/test_rust_bindings.py runs the extension and the pure-Python reader over the same fixtures and compares, rather than asserting values written down by hand. It skips itself when the module is not built.

Value paths compared	38,129 across 28 goldens
Tests	131 Rust unit + 6 integration (debug and release), 67 binding
cargo clippy --all-targets -- -D warnings	clean
cargo fmt, ruff check, ruff format	clean
pytest	235 passed, 2 skipped

Three kinds of coverage are pinned and asserted rather than described: UNCOVERED_BY_ANY_FIXTURE, the parsers nothing exercises (MF40 alone, now); DISTRIBUTION_SHAPES, every angular, energy and joint angle-energy shape the dumpers can write; and SECTIONS_WITHOUT_A_DICT, now empty, so a section_data projection that stops being built fails rather than quietly disappearing. None can drift in either direction.

CI is in .github/workflows/rust.yml, three jobs: the crate (fmt, clippy, test in both profiles); the goldens, which python tools/dump_golden.py --check holds to what the reader produces; and the bindings, built as an abi3 wheel with maturin on 3.10 and 3.13 and then tested.

Fixtures

Twenty-three ENDF evaluations and four ACE tables, stored xz-compressed: an evaluation is highly repetitive and goes about six to one, the dumps about seven, so 9.4 MB of text is 1.5 MB of blobs and 80,000 lines leave the diff. Python reads them through endf.fileutils.open_text, which handles .xz and leaves anything else alone; Rust reads them with lzma-rs, a pure-Rust dev-dependency, so the crate stays dependency-free for anything that uses it. The extension carries it as a real dependency, so a path that reads in one reader reads in the other.

tools/trim_endf.py cuts an evaluation to the sections under test keeping the records valid — U235 is 36 MB whole and 528 KB here. Where no real file small enough to keep holds a shape, one is built: make_urr_ace.py for the unresolved resonance block, make_laws_ace.py for the ACE laws Li6 does not use, make_denormal_ace.py for the float form NJOY writes for a denormal, make_nfy_endf.py for the fission product yields, and make_shapes_endf.py for MF2 Breit-Wigner, MF5 LF=12, MF6 LANG=2 and LAW=6, and MF13. The last two share the record writer in tools/endf_writer.py. Values are invented; the layout is the format's, which is the part both readers are held to.

Gaps, deliberately not hidden

• MF40 has no fixture — it needs an evaluation with radionuclide production covariances. The MF33 subsection parser it delegates to is covered.
• MF2 has real Reich-Moore parameters, a Case C unresolved region and a synthetic Breit-Wigner section, but R-matrix limited (LRF=7) and unresolved Cases A and B are untested. Cases A and B are unreachable through the current dispatch anyway — see Unresolved resonance ranges with LRF=1 are silently dropped (parse_mf2 dispatches on LRF where it means LRU) #15.
• No fissile or photoatomic ACE table, so the ACE fission path, MFTYPE=13 photon production, IncidentPhoton::from_ace and a second temperature for add_temperature_from_ace are written but unexercised.
• Not ported, on purpose. _add_compton_profiles and _add_bremsstrahlung attach data from a shipped HDF5 file resampled with a cubic spline — auxiliary data, and it would cost the crate an HDF5 reader and a spline. The chain's XML serialisation needs an XML writer and form_matrix needs sparse linear algebra. None of the four reads a nuclear data format.
• Nothing emits Arrow yet. The types were shaped to map onto the target schema — MT=458's polynomial/tabulated split is exactly a kind column, AngleEnergy's four variants are the four shapes distributions.arrow has columns for — but the RecordBatch construction is not here.

Upstream issues found while porting

Twelve. Each is reproduced in Rust deliberately, with a comment linking the issue, so the two readers agree until they are settled:

• get_intg_record writes every element of a row to the same column #11 — get_intg_record writes every element of a row to the same column
• parse_mf33_subsection appends NC sub-subsections twice when LTY == 0 #12 — parse_mf33_subsection appends NC sub-subsections twice when LTY == 0
• get_tab2_record uses int() where the rest of the reader uses int_endf() #13 — get_tab2_record uses int() where the rest of the reader uses int_endf()
• MF=1 MT=458 reports ZA as a float where every other section reports an int #14 — MF=1 MT=458 reports ZA as a float where every other section reports an int
• Unresolved resonance ranges with LRF=1 are silently dropped (parse_mf2 dispatches on LRF where it means LRU) #15 — unresolved resonance ranges with LRF=1 are silently dropped (dispatches on LRF where it means LRU)
• parse_mf7_mt4 reads LI from each temperature record and then discards it #16 — parse_mf7_mt4 reads LI from each temperature record and then discards it
• parse_mf34 discards every subsection it parses, and fills LB with LS #18 — parse_mf34 discards every subsection it parses, and fills LB with LS
• AngleEnergy.from_ace raises AttributeError for ACE law 5: GeneralEvaporation has no from_ace #19 — AngleEnergy.from_ace raises AttributeError for ACE law 5
• AngleDistribution.forward_fraction returns uninitialized memory for ACE-derived distributions, so removal_xs is nondeterministic there #21 — forward_fraction returns uninitialized memory for ACE-derived distributions, so removal_xs is nondeterministic there
• chain.replace_missing walks off the end of ATOMIC_SYMBOL and raises KeyError: -1 #22 — chain.replace_missing walks off the end of ATOMIC_SYMBOL and raises KeyError: -1
• Decay.decay_constant raises ZeroDivisionError for a nuclide whose evaluated half-life is zero #23 — Decay.decay_constant raises ZeroDivisionError for a nuclide whose evaluated half-life is zero

#15, #18 and #21 are the ones worth attention. Case A and Case B are how most actinides write their URR, and the range comes back with only its header and no error — and with the file position wrong, so a later range is misparsed. In #18 every angular-distribution covariance is read off the stream and thrown away. In #21 the same removal_xs call gives a different answer depending on what ran before it.

Where there is no behaviour worth matching, the port does the defensible thing and says so in a comment: it refuses ACE law 5 by name (#19), computes the forward fraction properly for the ACE shapes (#21), stops the stand-in walk at the edge of the table (#22), and treats a zero half-life as unevaluated rather than as instant decay (#23).

Five differences were found in the other direction, in code written here, all by the exhaustive comparisons:

• temperature_str(1200.5) gave 1201K against Python's 1200K — Python's round breaks ties to even, f64::round away from zero. That string keys which temperature a table belongs to.
• The univariate CDFs differed in the last two digits for linear-log and log-linear, because Python subtracts logs (np.diff(np.log(x))) where the port took the log of a ratio.
• The Legendre recurrence differed in the last two digits of every forward-scattered fraction: NumPy 2 forms the Clenshaw ratios before multiplying, where the port multiplied then divided.
• MF=8 MT=457 discrete radiation records are written with NT = 6 or NT = 12 within a single spectrum. The port returned zeros for the absent internal conversion coefficients where Python returns nothing — a zero conversion coefficient is a physical claim, and the wrong one.
• The NJOY deck wrote a temperature as 900 where Python writes 900.0. Python's str always writes a fractional part; Rust's shortest form does not. The same trap breaks the decay mode encoding, which reads RTYP by stripping zeros off the formatted float, so python_float_str lives in data and both use it.

All five fixed and pinned.

Building

cargo test                                      # parser and parity tests
cargo clippy --all-targets -- -D warnings
python tools/dump_golden.py                     # regenerate the golden files
python tools/dump_golden.py --check             # or just check them, as CI does
maturin develop -m crates/endf-py/Cargo.toml    # the extension module
python -m pytest                                # including the binding tests