Add ERA5 regional hindcast example by ewquon · Pull Request #233 · NumericalEarth/NumericalEarth.jl

ewquon · 2026-05-13T19:42:13Z

Summary

Adds examples/era5_breeze.jl — a building block for a regional modeling example that will eventually couple Breeze (compressible solver, in development) to forthcoming SlabLand and SlabOcean components.

Current scope is data ingest only: download ERA5 reanalysis over an SGP-centered LAM bounding box (HI-SCALE 2016-09-10 case day) and interpolate onto a LatitudeLongitudeGrid sized for ~3 km horizontal cells at the domain center latitude. Tᵥ is computed as a derived field using Breeze.ThermodynamicConstants for the Rᵥ/Rd − 1 coefficient.

Out of scope, to be added as the underlying capabilities come online:

Notes

Surface pressure is read onto a 3-D grid with Nz=1 rather than a 2-D (Bounded, Bounded, Flat) grid, sidestepping CliMA/Oceananigans.jl#5473 (Flat↔non-Flat interpolate! errors with a cryptic BoundsError; #5474 will convert that to a clear ArgumentError but does not lift the restriction). Mirrors the pattern in examples/ERA5_hourly_data.jl.
This example uses a LatitudeLongitudeGrid. A future variant on a projected Cartesian grid would benefit from the set! projection support proposed in Add map projection support to set! for RectilinearGrid targets #232.

Test plan

Run end-to-end against the CDS API on a machine with ~/.cdsapirc configured.

glwagner · 2026-05-13T20:30:08Z

+using NumericalEarth.DataWrangling: BoundingBox, Metadatum, download_dataset
+using NumericalEarth.DataWrangling.ERA5


can we add the appropriate exports so that these additional outputs are not needed?

also for the example, I recommend avoiding manual calls to download_dataset. I don't want to codify this workflow (we should instead develop the utility on #141). The issue is that people will adopt this style and not realize that it isn't recommended anymore once we have the utility

The utility in #141 is based on download_dataset, I think you mean #143

When the new utility lands, we can revise this example as well as ERA5_hourly_data.jl

Until then, let's avoid anything special (it will download automatically under the hood) so we don't have to revert it or change the documentation

glwagner · 2026-05-13T20:30:18Z

+using Oceananigans
+using Oceananigans.Fields: CenterField, XFaceField, YFaceField
+using Breeze
+using Breeze.Thermodynamics


what's this for?

vapor_gas_constant isn't exported from Breeze.Thermodynamics

NumericalEarth/Breeze.jl#699

glwagner · 2026-05-13T20:31:26Z

+set!(T,  Metadatum(:temperature;                         dataset=ds_pl, date=start_date, meta_common...))
+set!(qᵛ, Metadatum(:specific_humidity;                   dataset=ds_pl, date=start_date, meta_common...))
+set!(qᶜ, Metadatum(:specific_cloud_liquid_water_content; dataset=ds_pl, date=start_date, meta_common...))
+set!(qⁱ, Metadatum(:specific_cloud_ice_water_content;    dataset=ds_pl, date=start_date, meta_common...))


do we need a MetadataSet object? I can open an issue about this

This is something we've discussed before. That would clean things up here

codecov · 2026-05-13T21:14:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This reverts commit 5463396.

ewquon · 2026-05-14T06:44:40Z

Okay, the model state should be ready for dynamic initialization.

A couple of important notes:

As noted in ERA5 pressure-level ingest: per-column geopotential and surface-pressure masking #236, we need to better address geopotential height variability and we should discuss the approach to this. I've prototyped per-column vertical interpolation to unblock the ICs work and to have as a reference when developing the data wrangling improvement.
Because we don't have terrain yet, I'm using the height above ground level to interpolate onto the flat-bottomed Breeze grid. (Otherwise, portions of the Breeze domain would be essentially underground and I imagine this would be problematic for the initialization)

glwagner · 2026-05-14T13:15:51Z

@ewquon should we add terrain? Terrain following should work for the fully compressible formulation (just not substepping)

ewquon · 2026-05-14T14:35:34Z

@ewquon should we add terrain? Terrain following should work for the fully compressible formulation (just not substepping)

Hallelujah! I'm happy to add that in. Anything that gets us closer to reality sooner rather than later is a win in my book.

glwagner · 2026-05-14T15:12:01Z

    Example("Breeze over two oceans", "breeze_over_two_oceans", false),
-    Example("ERA5 hourly data", "ERA5_hourly_data", true),
+    Example("ERA5 hourly data retrieval", "ERA5_hourly_data", true),
+    Example("ERA5 hindcast simulation", "era5_breeze", true),


Suggested change

Example("ERA5 hindcast simulation", "era5_breeze", true),

Example("Atmosphere hindcast nested in ERA5", "era5_breeze", false),

This will be a heavy case so we shoudln't run every commit (change true to false). Also we should clarify, this is an atmosphere case (eventually atmosphere-land), and probably should also indicate it uses Breeze

glwagner · 2026-05-14T15:17:01Z

@ewquon I added the "build all examples" label so that it will build even if the "build always" tag is false.

glwagner · 2026-05-14T15:17:37Z

@giordano do you know what the GPU error is?

https://github.com/NumericalEarth/NumericalEarth.jl/actions/runs/25867031203/job/76011977037?pr=233#step:2:211

giordano · 2026-05-14T15:29:19Z

I believe

  failed to register layer: write /usr/local/share/julia/artifacts/75ae42466e7dd21641ff521f63cfb456d09433e2/lib/libcudnn_engines_precompiled.so.9: no space left on device
  Warning: Docker pull failed with exit code 1, back off 8.515 seconds before retry.

simply happens when the runner was reused from a previous run from somewhere else in the organisation, but the new build uses a new Docker image than the previous run, and then it fails to pull the second one because the combination of the two images fills the partition.

In summary, it's an offensive, albeit annoying, issue, restarting the job should succeed (unless you rerun in the same situation 😄)

ewquon added 2 commits May 13, 2026 13:41

Add ERA5 regional hindcast example

b0b2b79

Cleanup

18b6320

ewquon marked this pull request as draft May 13, 2026 19:53

Construct compressible Breeze model via atmosphere_simulation

ffc3670

glwagner reviewed May 13, 2026

View reviewed changes

Comment thread examples/era5_breeze.jl Outdated

glwagner reviewed May 13, 2026

View reviewed changes

Comment thread examples/era5_breeze.jl Outdated

glwagner reviewed May 13, 2026

View reviewed changes

Comment thread examples/era5_breeze.jl Outdated

Export download_dataset

5463396

ewquon force-pushed the eq/era5_breeze_land branch from 9d673f4 to 5463396 Compare May 13, 2026 20:43

Use Oceananigans' ReferenceToStretchedDiscretization for vertical grid

62ca809

glwagner reviewed May 13, 2026

View reviewed changes

Comment thread src/DataWrangling/DataWrangling.jl Outdated

glwagner mentioned this pull request May 13, 2026

Introduce MetadataSet (and a generic download) for multi-variable Metadata workflows #235

Open

10 tasks

ewquon added 7 commits May 13, 2026 20:48

Revert "Export download_dataset"

8e48f4b

This reverts commit 5463396.

Cleanup, remove download_dataset calls

71150bc

Minor code cleanup

a59f15b

Add per-column ERA5 interpolation with terrain workaround

d1dfc9b

Merge branch 'main' into eq/era5_breeze_land

925bd12

Fix intermediate_grid coordinate alignment with ERA5 native cells

83a76a1

Add profile plots at three sites of varying terrain elevation

5e73f8e

glwagner reviewed May 14, 2026

View reviewed changes

Comment thread examples/era5_breeze.jl

glwagner and others added 2 commits May 14, 2026 08:46

Merge branch 'main' into eq/era5_breeze_land

b3d9426

Literate

020d821

glwagner reviewed May 14, 2026

View reviewed changes

glwagner added the build all examples add this label to build all the examples in the PR label May 14, 2026

		using NumericalEarth.DataWrangling: BoundingBox, Metadatum, download_dataset
		using NumericalEarth.DataWrangling.ERA5

	Example("ERA5 hindcast simulation", "era5_breeze", true),
	Example("Atmosphere hindcast nested in ERA5", "era5_breeze", false),

Conversation

ewquon commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes

Test plan

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented May 13, 2026

Codecov Report

Uh oh!

Uh oh!

ewquon commented May 14, 2026

Uh oh!

glwagner commented May 14, 2026

Uh oh!

ewquon commented May 14, 2026

Uh oh!

Uh oh!

glwagner May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glwagner commented May 14, 2026

Uh oh!

glwagner commented May 14, 2026

Uh oh!

giordano commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ewquon commented May 13, 2026 •

edited

Loading

glwagner May 14, 2026 •

edited

Loading