Restructure im-calc onto dask and annotate its output - #139
Draft
lispandfound wants to merge 1 commit into
Draft
Conversation
| dependencies = [ | ||
| # UCGMSim Dependencies | ||
| "im-calculation>=2025.12.5", | ||
| "im-calculation @ git+https://github.com/ucgmsim/IM_calculation@no_parallel", |
Contributor
There was a problem hiding this comment.
Suggested change
| "im-calculation @ git+https://github.com/ucgmsim/IM_calculation@no_parallel", | |
| "im-calculation @ git+https://github.com/ucgmsim/IM_calculation.git@no_parallel", |
lispandfound
force-pushed
the
nzvm/im-calc-metadata
branch
from
September 2, 2026 02:23
9328139 to
6ce3c8c
Compare
lispandfound
force-pushed
the
nzvm/im-calc-metadata
branch
from
September 2, 2026 02:52
6ce3c8c to
accbecb
Compare
`im-calc` read the whole broadband into memory as float64 and fanned the IM kernels out over a process pool. That caps the station count at what one node's RAM holds, and the pool is redundant now that the kernels in im-calculation are dask-native. Instead, open the broadband chunked over stations only -- `component` and `time` must each stay whole, as the kernels take them as core dimensions -- and let each kernel return a lazy Dataset. Nothing is computed until `to_netcdf`, which streams chunk by chunk. `--cores` and `--psa-step` are gone; dask sizes chunks from `array.chunk-size`. Output becomes a DataTree with one group per intensity measure rather than a flat Dataset, which is what lets each IM carry its own components and its own units. Every leaf is annotated: `IM_UNITS` on the measures, `COORDINATE_METADATA` on the station coordinates, so the file describes itself instead of relying on the reader to remember that pSA is in g and rrup in km. The distance block moves out of the command body into `Distances` / `calculate_distances` unchanged -- rrup, rjb, hyp and epi are bit-identical to before. One deliberate change: `rx` and `ry` are now divided by 1000 like every other measure. They were previously emitted in metres while documented and grouped with distances in km. `calculate_source_parameters` collapses a multi-fault realisation to the single rake, dip, ztor and zbot a rupture is described by, moment- weighted, and hangs them off the root attributes. Rakes are averaged as unit vectors (`Rakes.as_vectors`), since the mean of -179 and 179 degrees is 0, not 180. `lat`/`lon` are normalised to `latitude`/`longitude` on read, so the command accepts SW4 low-frequency files as well as EMOD3D ones. Needs the dask-native kernels from im-calculation's `no_parallel` branch; pinned here until that is released. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lispandfound
force-pushed
the
nzvm/im-calc-metadata
branch
from
September 4, 2026 01:44
accbecb to
d5c3412
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
im-calcread the whole broadband into memory as float64 and fanned theIM kernels out over a process pool. That caps the station count at what
one node's RAM holds, and the pool is redundant now that the kernels in
im-calculation are dask-native.
Instead, open the broadband chunked over stations only --
componentand
timemust each stay whole, as the kernels take them as coredimensions -- and let each kernel return a lazy Dataset. Nothing is
computed until
to_netcdf, which streams chunk by chunk.--coresand--psa-stepare gone; dask sizes chunks fromarray.chunk-size.Output becomes a DataTree with one group per intensity measure rather
than a flat Dataset, which is what lets each IM carry its own components
and its own units. Every leaf is annotated:
IM_UNITSon the measures,COORDINATE_METADATAon the station coordinates, so the file describesitself instead of relying on the reader to remember that pSA is in g and
rrup in km.
The distance block moves out of the command body into
Distances/calculate_distancesunchanged -- rrup, rjb, hyp and epi arebit-identical to before. One deliberate change:
rxandryare nowdivided by 1000 like every other measure. They were previously emitted
in metres while documented and grouped with distances in km.
calculate_source_parameterscollapses a multi-fault realisation to thesingle rake, dip, ztor and zbot a rupture is described by, moment-
weighted, and hangs them off the root attributes. Rakes are averaged as
unit vectors (
Rakes.as_vectors), since the mean of -179 and 179degrees is 0, not 180.
lat/lonare normalised tolatitude/longitudeon read, so thecommand accepts SW4 low-frequency files as well as EMOD3D ones.
Needs the dask-native kernels from im-calculation's
no_parallelbranch; pinned here until that is released.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Stack created with GitHub Stacks CLI • Give Feedback 💬