Skip to content

Restructure im-calc onto dask and annotate its output - #139

Draft
lispandfound wants to merge 1 commit into
nz_checkfrom
nzvm/im-calc-metadata
Draft

Restructure im-calc onto dask and annotate its output#139
lispandfound wants to merge 1 commit into
nz_checkfrom
nzvm/im-calc-metadata

Conversation

@lispandfound

Copy link
Copy Markdown
Contributor

im-calc read the whole broadband into memory as float64 and fanned the
IM kernels out over a process pool. That caps the station count at what
one node's RAM holds, and the pool is redundant now that the kernels in
im-calculation are dask-native.

Instead, open the broadband chunked over stations only -- component
and time must each stay whole, as the kernels take them as core
dimensions -- and let each kernel return a lazy Dataset. Nothing is
computed until to_netcdf, which streams chunk by chunk. --cores and
--psa-step are gone; dask sizes chunks from array.chunk-size.

Output becomes a DataTree with one group per intensity measure rather
than a flat Dataset, which is what lets each IM carry its own components
and its own units. Every leaf is annotated: IM_UNITS on the measures,
COORDINATE_METADATA on the station coordinates, so the file describes
itself instead of relying on the reader to remember that pSA is in g and
rrup in km.

The distance block moves out of the command body into Distances /
calculate_distances unchanged -- rrup, rjb, hyp and epi are
bit-identical to before. One deliberate change: rx and ry are now
divided by 1000 like every other measure. They were previously emitted
in metres while documented and grouped with distances in km.

calculate_source_parameters collapses a multi-fault realisation to the
single rake, dip, ztor and zbot a rupture is described by, moment-
weighted, and hangs them off the root attributes. Rakes are averaged as
unit vectors (Rakes.as_vectors), since the mean of -179 and 179
degrees is 0, not 180.

lat/lon are normalised to latitude/longitude on read, so the
command accepts SW4 low-frequency files as well as EMOD3D ones.

Needs the dask-native kernels from im-calculation's no_parallel
branch; pinned here until that is released.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com


Stack created with GitHub Stacks CLIGive Feedback 💬

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the following dependencies for consistency.

Comment thread pyproject.toml
dependencies = [
# UCGMSim Dependencies
"im-calculation>=2025.12.5",
"im-calculation @ git+https://github.com/ucgmsim/IM_calculation@no_parallel",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"im-calculation @ git+https://github.com/ucgmsim/IM_calculation@no_parallel",
"im-calculation @ git+https://github.com/ucgmsim/IM_calculation.git@no_parallel",

`im-calc` read the whole broadband into memory as float64 and fanned the
IM kernels out over a process pool. That caps the station count at what
one node's RAM holds, and the pool is redundant now that the kernels in
im-calculation are dask-native.

Instead, open the broadband chunked over stations only -- `component`
and `time` must each stay whole, as the kernels take them as core
dimensions -- and let each kernel return a lazy Dataset. Nothing is
computed until `to_netcdf`, which streams chunk by chunk. `--cores` and
`--psa-step` are gone; dask sizes chunks from `array.chunk-size`.

Output becomes a DataTree with one group per intensity measure rather
than a flat Dataset, which is what lets each IM carry its own components
and its own units. Every leaf is annotated: `IM_UNITS` on the measures,
`COORDINATE_METADATA` on the station coordinates, so the file describes
itself instead of relying on the reader to remember that pSA is in g and
rrup in km.

The distance block moves out of the command body into `Distances` /
`calculate_distances` unchanged -- rrup, rjb, hyp and epi are
bit-identical to before. One deliberate change: `rx` and `ry` are now
divided by 1000 like every other measure. They were previously emitted
in metres while documented and grouped with distances in km.

`calculate_source_parameters` collapses a multi-fault realisation to the
single rake, dip, ztor and zbot a rupture is described by, moment-
weighted, and hangs them off the root attributes. Rakes are averaged as
unit vectors (`Rakes.as_vectors`), since the mean of -179 and 179
degrees is 0, not 180.

`lat`/`lon` are normalised to `latitude`/`longitude` on read, so the
command accepts SW4 low-frequency files as well as EMOD3D ones.

Needs the dask-native kernels from im-calculation's `no_parallel`
branch; pinned here until that is released.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant