Conversation
|
Just a minor organisational suggestion: don't merge the PR into |
|
|
||
| An atmosphere model built on Oceananigans is ideally suited to this approach. First, Oceananigans employs relatively simple C-grid numerical methods and sits within a well-established Julia ecosystem that supports powerful usability patterns for model configuration, simulation, visualization, and post-processing [@Oceananigans]. Second, Oceananigans has demonstrated world-leading performance for ocean simulations, including recent GPU-based mesoscale-resolving climate simulations, with reported speedups of \(O(10\text{--}100\times)\) relative to existing Fortran-based codes in comparable regimes [@OceananigansArxiv; @Silvestri2025]. As a result, an atmosphere model based on Oceananigans also has the potential to become a world-class code for regional weather forecasting applications. | ||
|
|
||
| Existing atmospheric models face several challenges. Many legacy codes are written in Fortran and can be difficult to extend, modify, or learn for new users. While these codes achieve excellent performance, their complexity often creates barriers to entry for students and researchers from adjacent fields. Modern codes may offer improved usability but sometimes lack either the physical fidelity or computational performance required for production simulations. |
There was a problem hiding this comment.
Worth mentioning difficulty with porting to accelerators, especially when portability across vendors is needed? Nowadays there are various tools for running Fortran code on GPU (like OpenMP GPU offloading), but vendor support is patchy. Although this topic could be slightly controversial.
There was a problem hiding this comment.
If we need references for this, I found
- Portability of Fortran’s ‘do concurrent’ on GPUs (from ~1.5 years ago) which shows difficulties back then with running the same code on AMD GPUs
|
|
||
| Breeze.jl addresses these challenges by combining high performance with accessibility. Key design principles include: | ||
|
|
||
| 1. **GPU-first architecture**: Breeze.jl is designed from the ground up for GPU computing. Leveraging KernelAbstractions.jl [@Churavy_KernelAbstractions_jl], the same code runs efficiently on both CPUs and GPUs, enabling researchers to utilize modern accelerated hardware without code modifications. This approach follows the successful model demonstrated by Oceananigans.jl [@OceananigansArxiv], which showed that high-level Julia code can achieve excellent performance across heterogeneous architectures. |
There was a problem hiding this comment.
Should we mention that KernelAbstractions.jl enables us to run code on multiple GPU backends? This would match the comment above about difficulties with porting legacy Fortran code to multiple GPU backends. But we should also do some tests with the other GPUs, I can help with that.
There was a problem hiding this comment.
Uhm, the FFT is a problem, trying to run https://numericalearth.github.io/BreezeDocumentation/dev/literated/bomex/ on an AMD GPU I get:
julia> model = AtmosphereModel(grid; dynamics, coriolis, microphysics, advection, forcing,
boundary_conditions = (ρθ=ρθ_bcs, ρqᵗ=ρqᵗ_bcs, ρu=ρu_bcs, ρv=ρv_bcs))
ERROR: UndefVarError: `plan_ifft!` not defined in `AMDGPU.rocFFT`😢
There was a problem hiding this comment.
Uhm, I think this is an error on the Oceananigans side, plan_ifft! seems to be tested in AMDGPU, it's just not in AMDGPU.rocFFT: https://github.com/JuliaGPU/AMDGPU.jl/blob/75e4a05364720762d3a5c2d69bdbcec5d9afb5a8/test/rocarray/fft.jl#L44
There was a problem hiding this comment.
Yeah, this should work when calling existing functions:
julia> using AMDGPU, AbstractFFTs
julia> M = ROCArray(randn(ComplexF32, 4, 4))
4×4 ROCArray{ComplexF32, 2, AMDGPU.Runtime.Mem.HIPBuffer}:
0.232252+0.28769im 0.169463-0.74924im -0.702146-0.158628im 0.284935-0.462465im
-0.589165+1.20839im 1.02383-0.123087im 0.124982-0.598176im -0.507629+0.883225im
-0.72208+0.740543im -0.320967-0.052425im -0.285868+1.71853im -0.207032+0.888445im
-0.849192-0.83669im 0.0440104-0.541001im 0.0156304-0.0877196im 1.11872+1.34555im
julia> plan = plan_ifft!(M)
0.0625 * rocFFT in-place complex inverse plan for 4×4 ROCArray of ComplexF32
julia> plan * M
4×4 ROCArray{ComplexF32, 2, AMDGPU.Runtime.Mem.HIPBuffer}:
-0.0731404+0.216434im 0.189983+0.0470791im -0.273808+0.067809im -0.325081+0.0186613im
0.00188965-0.290931im -0.277825+0.0433258im -0.126461+0.0458511im 0.129709+0.153547im
-0.12079+0.0601228im 0.0257463-0.142214im 0.0982778+0.302669im 0.233898-0.0564445im
0.188167-0.256287im 0.367389+0.134521im 0.0709181-0.0811373im 0.123379+0.0246834imThere was a problem hiding this comment.
For the record, this was fixed upstream in Oceananigans by CliMA/Oceananigans.jl#4593.
There was a problem hiding this comment.
we should definitely include AMD benchmarks!
There was a problem hiding this comment.
I added a sentence to highlight support for multiple GPU families, but also added a footnote to point out limitations.
Definitely agree |
|
I'll mention also here that JOSS today announced they updated their submission scope requirements, so we need to make sure we match the new ones. |
|
So, this work has fallen behind schedule (or rather by the wayside). We are currently working on maturing acoustic substepping. Should we reset our publication date goals until we have a reasonably complete acoustic substepping functionalty which can be benchmarked? I think that would push submission to the end of May, or June. |
4 participants
This PR builds a draft of a JOSS paper. I think we should target submitting this by the end of January, or thereabouts. I am starting the paper now so that we can create a list of TODOs that we would like to finish before submitting the paper.
Here is a preliminary TODO list for publication:
Conservative potential temperature (Bryan and Fritsch 2002)Total energy (Romps 2008, Yatunin 2025)Entropy for non-phase-equilibrium (?)Open boundary case (Open boundaries for AtmosphereModel #287)@navidcy
@mmr0
@bischtob
@kaiyuan-cheng
@giordano
@danny-rosenfeld
Note: anyone is welcome to be a JOSS author if they want to contribute to this package! We want as wide-ranging collaborations as possible. Please speak up if you would like to be added. I am happy to accept help pushing this over the finish line ;-)