Skip to content

Add an end-to-end runnable graphnet demo on the bundled Prometheus data - #8

Merged
sevmag merged 5 commits into
mainfrom
graphnet-example
Aug 2, 2026
Merged

Add an end-to-end runnable graphnet demo on the bundled Prometheus data#8
sevmag merged 5 commits into
mainfrom
graphnet-example

Conversation

@sevmag

@sevmag sevmag commented Aug 2, 2026

Copy link
Copy Markdown
Owner

What

The graphnet connection made demonstrable end to end on data everyone has: one example script holding every line of demo-specific code, runnable both as plain Python and through the Hydra launcher.

examples/graphnet_demo.py — the full SPINE loop on the 50-event Prometheus example file graphnet bundles (data/examples/sqlite/prometheus/prometheus-events.db):

  • Reader: a graphnet SQLiteDataset (EdgelessGraph + NodesAsPulses + the stock Prometheus detector with replace_with_identity, which keeps every column raw including sensor_id) adapted via spine_graphnet.readers.GraphNetRawDataset. sensor_id is globally unique in the file (503 sensors) and serves directly as the data-carried sensor key. Prometheus rows are single photons with no charge column, so a tiny wrapper appends unit charge to complete the (x, y, z, t, charge) layout.
  • Scaling: spine_graphnet.scaling.DetectorScaler (new, general) turns any graphnet Detector's feature_map into a SPINE FeatureScaler — unmapped columns pass through, query positions go through the detector's xyz entries. The demo pretrains in the exact standardization graphnet's own examples apply to this file (Prometheus()), so a downstream graphnet fine-tune sees the same feature space; the demo defines no scaler of its own.
  • Geometry asset: built from the file itself (distinct sensors, full sorted neighbour lists), loaded with load_geometry(..., sensor_key="sensor_id").
  • Selection: events filtered with can_always_split under the same knobs the task samples with, split 28 train / 7 val.
  • Task + backbone: occupancy-only CurtainTask, tiny DeepIceBackbone (92K params), trained via fit() with CurtainValAUC.
  • The payoff: the exported checkpoint loads back into a stock graphnet DeepIce with strict key matching, ready for supervised fine-tuning.

Hydra path for the same run — the script always stages a self-contained demo dir (db copy, geometry, selection parquets) and has --prepare-only; the experiment config (introducing the configs/experiment pattern) then drives the identical run. The data group targets graphnet_demo.demo_reader directly — running python examples/train_curtain.py puts examples/ on sys.path, so no demo code needs to live in the installed package:

python examples/graphnet_demo.py --prepare-only
python examples/train_curtain.py +experiment=prometheus_demo

Core changes riding along:

  • fit() passes accelerator="auto" instead of pinning "gpu", so the demo runs on CPU-only machines.
  • Random holdout mode removed — the temporal cutoff is the pretext; every benchmarked run trained with it. The never-used random sensor split cost a second branch in sample_event and can_always_split, two pass-through CurtainTask arguments, and a dead random_vis_frac knob in every caller's selection filter. Net −40 lines, no behavior change for any existing config.

Verification

Both paths ran end to end on a CPU compute node after every commit, most recently: plain script (selection 28/7, 15 epochs in about a minute, best val loss 0.678, strict load_state_dict into stock DeepIce passed) and Hydra experiment (staged dir, trained, exported checkpoint val loss 0.682, stored config confirms the graphnet_demo.demo_reader target and the nested DetectorScaler(Prometheus())). All pre-commit hooks pass.

🤖 Generated with Claude Code

Severin Magel and others added 5 commits August 2, 2026 00:14
examples/graphnet_demo.py runs the full loop on the 50-event example file
shipped in a graphnet checkout: a SQLiteDataset is adapted to the RawEvent
contract via an identity detector, the geometry asset and the
guaranteed-splittable event selection are built from the file itself, a tiny
DeepIce (92K params) is pretrained on CPU in about a minute, and the exported
checkpoint is loaded back into a stock graphnet DeepIce. fit() now selects
the accelerator automatically so the demo also runs without a GPU.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The temporal cutoff is the pretext: every benchmarked run trained with it,
and the never-used random sensor split cost a second branch in sample_event
and can_always_split, two pass-through CurtainTask arguments, and a dead
random_vis_frac knob in every caller's selection filter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The demo's graphnet chain moves to an importable home
(spine_graphnet.prometheus: identity detector, unit-charge wrapper, and a
demo_reader factory matching the data group's reader contract), and its
scaler joins spine.data.scaling next to HexagonScaler. graphnet_demo.py now
always stages a self-contained demo dir (db copy, geometry asset, selection
parquets) and gains --prepare-only, so the same run also launches from
config via the new configs/experiment pattern:

    python examples/graphnet_demo.py --prepare-only
    python examples/train_curtain.py +experiment=prometheus_demo

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DetectorScaler (spine_graphnet.scaling) turns any graphnet Detector's
feature_map into a spine FeatureScaler: named pulse columns are standardized
per the detector, unmapped columns (the demo's unit charge) pass through,
and query positions go through the detector's xyz entries. Pretraining in
the detector's own standardization keeps the encoder's feature space
identical to what a downstream graphnet fine-tune applies; the demo-specific
PrometheusDemoScaler leaves the core.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
spine_graphnet.prometheus is gone: the stock Prometheus detector already
reads raw through replace_with_identity (it adds identity entries even for
columns its feature_map lacks, e.g. sensor_id), and the Hydra data group can
target graphnet_demo.demo_reader directly because running
examples/train_curtain.py puts examples/ on sys.path. The integration
package keeps only general surface: backbone, reader adapter, DetectorScaler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sevmag
sevmag merged commit db55863 into main Aug 2, 2026
1 check passed
@sevmag
sevmag deleted the graphnet-example branch August 2, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant