Skip to content

M3/E5/Issue #18 — HEL1OS Parsers - #23

Merged
Rexy-5097 merged 1 commit into
mainfrom
feat/m3-e5-issue-18-hel1os-parsers
Sep 11, 2026
Merged

Rexy-5097 merged 1 commit into
mainfrom
feat/m3-e5-issue-18-hel1os-parsers

Conversation

@slazyverse

Copy link
Copy Markdown
Collaborator

Canonical issue

M3 / E5 / Issue #18 — HEL1OS parsers. Milestone M3, Epic E5 (Ingest), TIS §15 row 18:
| 18 | 18 | E5 | 900 | L | 17 | per-field vs spec | no imputation | parse fixtures |.

Dependencies and eligibility evidence

Dependency State Evidence
#17 SoLEXS parsers (row 18's declared dependency) ✔ merged 6a96084 (PR #20)
#15 ingest contract (the boundary these parsers sit behind) ✔ merged 1d65c63 (PR #18)
#12 pure domain model (Observation, invariants used here) ✔ merged 530a412 (PR #15)
#10 provenance kernel, #11 contracts, #13 import rules, #14 manifest ✔ merged dfe068b, 6c26b33, 6beeeaa, 807bdf4

Eligibility recomputed from origin/main:docs/tis/TIS-v1.0.md §15 — complete {1…17},
eligible {#18, #22, #23, #25, #26, #27, #39}; lowest Order selects #18.
Base: origin/main = a2893dc.

Scope

The five HEL1OS product parsers plus orbit identity and version precedence:

orbit     §4    identity, precedence rules 1–4, overlap detection. No merge API.
lc        §2.6  per-detector band rates; band edges from EXTNAME; CTR unit read.
spectra   §2.7  PHA spectra, 341 CZT / 511 CdTe; R-1 epoch resolution.
hk        §2.8  housekeeping in archive order; every decisive column with its unit.
gti       §2.9  per-detector good time intervals; lowercase columns; F-12.
events    §2.5  photon events, row rules; exposed, never ingestible.

Supporting, approved changes: _fits.py moves from parsers/solexs/ to parsers/ so both
instrument packages share the no-default FITS accessors without either importing the other, and
gains declared_unit(); the three SoLEXS import lines follow it; parsers/__init__.py's stale
docstring is corrected.

Scientific contract — ε_t = 1 ms (SPEC-parsers r7 §5.1)

Verifying against all 391 orbits proved three contract checks undecidable at float64 resolution.
Implemented from the r6 text alone, R-1 H3 rejected 647 of 1,564 spectra products and §2.8's
header-span check rejected 94 of 391 housekeeping products — every rejection within about two
representable float64 steps of its bound.

r7 therefore defines col_span and header_span, adds the time-representation allowance
ε_t = 1 ms
with an explicit scope (§2.5 span and V-EVT-2, §2.7 R-1 H3, §2.8 header span — and
nowhere else), and assigns F-06 to the §2.5 and §2.8 time checks that carried no rule id.
V-EVT-2 receives a comparison rule derived from the data itself: ≤ ½·r_isot + ε_t.

ε_t is not invented here. It is the value the owner approved in CONTRADICTION-006 Defect A
as an implementation-only fix and the value the Milestone V–VII parsers already applied to R-1
(_FLOAT_EPS_S = 1e-3). r7 moves it from code into the contract, because an implementation-only
fix to a contract defect does not survive a re-implementation — which is exactly what happened.
ε_t never touches a physical time difference: §2.8 inversion statistics stay unthresholded,
§2.6's MJD stays strictly increasing, GTI durations are unchanged, and F-09's exact SoLEXS
identity is untouched.

Every replaced clause is quoted beside its amendment; nothing was overwritten silently.

Defect records

  • CONTRA-007 — CLOSED by r7. Defects A (§2.7 H3), B (§2.8 header span), C (§2.5 span and
    V-EVT-2), plus Observations D (bands need not share a time axis), E (§2.8's detector-health
    names are abbreviations), F (three different overlap-pair counts: 46 / 49 / 50), G (archive-wide
    HK inversions reaching 1,153 s).
  • CONTRA-008 — OPEN. §2.5's mjd non-decreasing rule, falsified archive-wide.

CONTRA-008 — the event parser remains fail-closed

§2.5's non-decreasing rule is falsified by 1,564 of 1,564 event HDUs (391 of 391 orbits;
37,912,843 backward steps; largest 1,156.36 s). The rule has NOT been amended, reinterpreted or
weakened.
The parser enforces it as written and terminates at F-16 on the first decrease,
after yielding only rows that passed every other §2.5 rule. Consequently no real event stream
completes today
, and that is the intended behaviour until the maintainer rules.

The specification issue is unresolved. CONTRA-008 records the measurement and a proposed
amendment that is deliberately not applied; the archive-wide findings are evidence for the
record, not permission to bypass the rule. An integration test pins the falsification, so the day
the rule changes, the test changes with it.

Archive-wide verification (all 391 orbits, real corpus)

Product Before this PR After
Spectra 917 ok, 647 F-06 1,564 / 1,564 — all resolve to H3
Housekeeping 295 ok, 94 F-05, 2 F-16 389 / 391 — the 2 are known F-16 archive defects
Light curves 1,564 / 1,564 1,564 / 1,564
GTI 1,564 / 1,564 1,564 / 1,564
Events (headers) 391 / 391 391 / 391
Fully clean orbits 134 / 391 389 / 391

The two exceptions are HLS_20260201_120005_43198sec_lev1_V111 and
HLS_20260202_000005_43183sec_lev1_V111, the duplicate-HK orbits the V&V plan lists as known
archive defects. They still terminate at F-16, by design. H3 residuals across all 1,564 spectra
lie within ±1.36 µs of the bin bound — about 735× inside ε_t.

Event verification

An independent read-only probe examined 1,352,158,522 event rows across 1,564 detector HDUs:

  • ener ≤ 0: 0 rows
  • utc-isot vs mjd: within ±0.5000 ms on every row, 0 parse failures — exactly half the
    resolution the string carries, which is what r7's V-EVT-2 rule encodes
  • span violations: all exactly one float64 step; 0 beyond 1 ms
  • backward mjd steps: 37,912,843 across every HDU → the ordering rule remains fail-closed

SoLEXS regression (after the _fits.py relocation)

With the real archive: 98 unit + 31 integration tests pass
(test_solexs_parsers, test_issdc_pradan, test_solexs_parse_with_platform,
test_issdc_pradan_acquire). The imports gate passes unchanged, and the test asserting neither
instrument imports the other still holds. Only the docstring, the new accessor and three import
lines changed; SoLEXS behaviour is identical.

Verification

Gate With real corpus Clean (no data)
links 74 documents, 510 citations same
contracts 12 schemas, additive same
imports 53 modules, 267 imports, 9 policies same
architecture 563 passed 563 passed
unit 583 passed 581 passed, 2 skipped
integration 146 passed 124 passed, 22 skipped
research (ruff + tests/v2) clean; 188 passed clean; 131 passed, 57 skipped
lint (ruff E9,F on the changed tree) clean —
web verify / test / build / budget 0 errors / 78 passed / 35 pages / all budgets satisfied —

Clean export

Built with git archive from the staged tree — 3,774 files, no .git, no corpus:
links pass, architecture 563, unit 581 passed + 2 skipped, integration 100 passed + 22 skipped,
research 131 passed + 57 skipped. Every real-data test skips rather than fails (12 of the
skips are HEL1OS), satisfying E5 §17 and STD-12.

The export also reports 24 errors, which are the pre-existing IMPL-012 condition and unrelated
to this PR
: tests/integration/test_v1_tag_restores.py and
tests/integration/test_salvage_provenance.py collect exactly 24 tests, and they assert on
repository refs — the v1-surya-final tag. A git archive export carries no .git and therefore
no tags, so they error there and pass in any real checkout (integration is 146/146 with the
corpus). The same 24 were recorded at #13, #15, #16 and #17.

Limitations

  1. CONTRA-008 remains open, and real event streams therefore terminate at F-16. Resolving it
    is a specification decision, not an implementation one.
  2. Security remediation is a separate track. ISSDC/PRADAN session cookies remain committed at
    origin/main in 8 files and in all 11 tags. This PR does not touch them, does not use them, and
    must not be read as addressing them. History rewriting is out of scope here.
  3. Milestone VIII obligations remain separate. A-11, A-12 and A-13 are not discharged; this
    work supplies evidence toward them.
  4. The archive-wide sweep is a verification run, not a committed test — CI has no corpus.
  5. §4's overlap-count discrepancy (46 / 49 / 50) is recorded, not reconciled; it belongs to the
    coverage map in M3/E5/#16 — ISSDC-PRADAN adapter #19.
  6. SoLEXS .hk (§2.4) remains unassigned to any issue — the Engineering Plan defect reported at M2/E4/#14 — Tier 2 manifest schema #17.

Explicitly out of scope

#19's grid and write; 1-minute aggregation; dual write; the §4 minute-level coverage map and
version-resolution log; T1–T7 canonical tables; curation; dataset/label/protocol/environment/method
registries; evaluation; evidence; portal; ML; and SoLEXS .hk.


Do not merge on my account: this is a contributor PR from slazyverse. Rexy-5097 reviews and
merges.

🤖 Generated with Claude Code

Implements `SPEC-parsers` §2.5–§2.9 and §4 for HEL1OS: band light curves,
PHA spectra, housekeeping, per-detector GTI, an event reader, and orbit
identity with version precedence. No merge API, no coverage map, no
aggregation and no dual write — those belong to the write path (#19).

GOVERNANCE FIRST. Verifying against all 391 orbits proved three contract
checks undecidable at float64 resolution: R-1 H3 rejected 647 of 1,564
spectra products and §2.8's header-span check rejected 94 of 391
housekeeping products, every one of them by one or two representable
steps. `SPEC-parsers` r7 therefore defines `col_span`, adds the §5.1
time-representation allowance ε_t = 1 ms — the value CONTRADICTION-006
Defect A approved in code and never recorded in the contract — and gives
the §2.5 and §2.8 time checks the F-06 id they lacked. Recorded as
CONTRA-007, CLOSED by r7.

§2.5's `mjd` non-decreasing rule is falsified by every event HDU in the
archive (1,564 of 1,564; 37.9 M backward steps; largest 1,156 s). It is
NOT amended: the parser enforces it and fails closed, and the
falsification is recorded OPEN as CONTRA-008 for the maintainer to rule
on.

D6: the missing §2.5 row checks (span, V-EVT-2, `ener>0`), the declared
`ener` and `CTR` units, CZT `pix`/`offsetchn`, and every §2.8 decisive
column with its declared unit and archive dtype. Only `czt1temp` and
`czt2temp` are required finite, as §2.8 states.

D3: `_fits.py` moves from `parsers/solexs/` to `parsers/` so both
instruments share the no-default FITS accessors without either importing
the other, and gains `declared_unit()`. SoLEXS behaviour is unchanged and
regression-tested.

Verified with the real 391-orbit corpus: links, contracts, imports,
architecture 563, unit 583, integration 146. Clean export from the staged
tree: unit 581 (2 skipped), integration 100 (22 skipped), every real-data
test skipping rather than failing. Archive-wide, 389 of 391 orbits now
parse end to end; the 2 that do not are the known duplicate-HK archive
defects, still terminating at F-16.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rexy-5097
Rexy-5097 merged commit 1a379f0 into main Sep 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants