Adding the LDCast architecture

[leifdenby]

The purpose of this issue is to plan and track the implementation of the LDCast architecture in the mlcast python package.

I think a good start would be summarise what work has gone on each of the meteorological centers so we can plan how to combine the pieces of work :)

Maybe @mafdmi and @franchg you can add that here?

Looking forward to getting started on this! 🚀

(link to slack channel discussing the implementation: https://mlcast.slack.com/archives/C0A0JCV36CR, we will use this issue to type up current status of previous work and plan for next steps)

[mfroelund]

I've made my reworked version of LDCast available here: https://github.com/dmidk/ldcast-dmi/. I'll describe below what I've done:

Config interface

I've introduced a new configuration interface based on YAML files.
An example configuration can be seen below.
I imagine it could be useful to also include parameters relevant to the model architecture like number of hidden layers, kernel sizes, etc. in the config file to make them easily adjustable during experiments - also if we want to use tools like DVC, it makes them easily trackable.
When loading the configuration, it is validated using Pydantic in ./ldcast/config_parser.py.

schema_version: v0.1.0

general:
  seed: 24
  max_nproc: 4
  model_dir: &model_dir /path/to/model_outdir/

datasets: 
  radar_composite: &radar_composite
    path: /path/to/radar_composite.zarr
  radar_predict: &radar_predict
    path: /path/to/radar_predict.zarr

preprocessing:
  sampling:
    sampler: EqualFrequencySampler
    sample_shape: [2, 2]
    timestep: PT10M
    num_timesteps: 2
    time_range_sampling: [-PT10M, PT20M]
    overwrite: False
    inputs:
      dataset: *radar_composite
    outputs: &radar_sampled 
      path: /path/to/sampled_radar.zarr

models:
  autoenc:
    batch_size: 4
    batches_per_epoch: 2 # Should be a multiple of the number of GPUs
    max_epochs: 2
    chpt_path: null
    inputs:
      datasets:
        sampled: *radar_sampled
        target: *radar_composite
        predictions: 
          radar_composite: *radar_composite
    outputs:
      autoenc_weights: &autoenc_weights
        filename: autoenc-test.pt
        directory: *model_dir

  genforecast:
    batch_size: 1
    batches_per_epoch: 2 # Should be a multiple of the number of GPUs
    max_epochs: 2
    chpt_path: null
    num_future_timesteps: 3
    num_past_timesteps: 3
    use_obs: True
    use_nwp: False
    lr: 1e-4
    inputs:
      datasets:
        sampled: *radar_sampled
        target: *radar_composite
        predictions: 
          radar_composite: *radar_composite
      autoenc_weights: *autoenc_weights
    outputs:
      genforecast_weights: &genforecast_weights
        filename: genforecast-test.pt
        directory: *model_dir
  
  forecast:
    num_diffusion_iters: 2
    timestep: PT5M
    past_timesteps: 3
    future_timesteps: 3
    autoenc_time_ratio: 1
    crop_box: [[128, 480], [160, 608]]
    inputs:
      dataset: *radar_predict
      autoenc_weights: *autoenc_weights
      genforecast_weights: *genforecast_weights
    outputs:
      path: /path/to/forecast.zarr

CLI adjustments

I've extended the usage of Fire to define the CLI. I've defined the structure of the CLI like this:

NAME
    ldcast - Main CLI class.

SYNOPSIS
    ldcast - GROUP | COMMAND | VALUE

DESCRIPTION
    Main CLI class.

GROUPS
    GROUP is one of the following:

     train
       Cli setup for executing training.

     visualize
       Cli setup for visualization.

COMMANDS
    COMMAND is one of the following:

     forecast
       Cli entry point for running forecast without training.

     sample
       Cli entry point for running sampling without training.

VALUES
    VALUE is one of the following:

     config

with the train subcommand having the following commands:

NAME
    ldcast train - Cli setup for executing training.

SYNOPSIS
    ldcast train COMMAND | VALUE

DESCRIPTION
    Cli setup for executing training.

COMMANDS
    COMMAND is one of the following:

     all
       Execute all training pipelines.

     autoenc
       Execute the autoencoder training pipeline.

     genforecast
       Execute the genforecast training pipeline.

VALUES
    VALUE is one of the following:

     config

     num_nodes

     save_model

I.e. to train the autoencoder, one would run:

uv run ldcast train autoenc --config path/to/config.yaml --save_model --num_nodes 1

Datasets

To rework the data loading to work directly from Zarr stores, we've created a zarr version of the patched DMI data and changed the EqualFrequencySampler to save the sampled data to a Zarr store as well. This way, the data loading can be done directly from the two Zarr stores without needing to do any preprocessing runtime. The loading of the DMI radar data, patching and saving to zarr is done in parallel and can be inspected in ./DataPreprocessing/radarToZarrML.py and ./DataPreprocessing/merge_datasets.py. The latter script merges multiple Zarr stores into a single one - it was necessary to split up the conversion into smaller parts to avoid running out of memory.

When reworking the dataloading part, I wanted to make it as simple as possible and to avoid adding sampler specific logic. Thus, to still respect the equal frequency sampling, I decided to save the sampled dataset by simply duplicating the data in each bin until every bin has the same number of samples as the bin with the maximum number of samples. After this, the bins are flattened and saved with a multiindex (patch_time, patch_icoord, patch_jcoord). This way, when loading the dataset, we can simply load the next N batches and even apply shuffling without worrying about the binning. I think this is a good requirement to set for every type of sampler we introduce, that the sampled dataset should be directly loadable.

Dataloading and batch generation

I've completely transformed the dataloading and batch generation part of the code into a pytorch structure. This can be inspected in ./ldcast/features/data_handling.py.
I've defined a PatchesDataset class that handles loading the patched data from the Zarr store, and a PredictDataset class that handles loading the data for prediction.

I've introduced a TrainDataModule that handles setting up the dataloaders for the different stages. The dataloaders are set up with a DistributedSampler to ensure correct data distribution when using multiple GPUs.

For prediction, I've defined the PredictDataModule to handle data loading for predictions, which uses an EnsembleBatchSampler to create batches for ensemble predictions. This is the case no matter the number of ensemble members, i.e. also for "deterministic" cases with only one prediction/ensemble member.

Models

I've moved the scripting logic from the ldcast/scripts/train_autoenc.py and ldcast/scripts/train_genforecast.py files into the __init__.py files of the respective model folders; i.e. ldcast/models/autoenc/__init__.py and ldcast/models/genforecast/__init__.py. For the forecast, I've decided to treat it as a model on its own, but since it is simpler, I've kept it in a single file: ldcast/models/forecast.py. This file contains much of the original code from the ldcast/scripts/forecast_demo.py file. The plotting part of the forecast_demo.py file, I've moved to the visualization directory: ldcast/visualization/plots.py.

[leifdenby]

I've made my reworked version of LDCast available here: https://github.com/dmidk/ldcast-dmi/. I'll describe below what I've done:

Thanks @mafdmi! This is really nice work, and your description of it is really clear. Thank you for taking the time to type this up. I will take some time to digest this and add some questions ☺️

In the meantime I have run the DMI LDCast repo through Devin's deepwiki code overview tool: https://deepwiki.com/dmidk/ldcast-dmi - this could be useful for others to dive deeper into the code.

@arjj8 if you have any questions to the implementation feel free to chime in here also!

Questions I asked of Devin:

• "how much of the data processing pipeline can be executed before training and how much is executed during training?": https://deepwiki.com/search/how-much-of-the-data-processin_9cbc0165-14e7-4fcc-9e21-0562f5df11f3?mode=fast

[mfroelund]

Wow! Amazing tool!

[leifdenby]

Copying what @franchg wrote on slack (28/11/2025) so we don't loose it:

Hi Martin! I had the time to check the work you have done on the LDCast fork!
There are a couple of important aspects that I want to point out, and that I should have put into writing a long ago about the original LDCast implementation by Jussi.
We discovered, also discussing with Daniele Nerini from meteoswiss, that the behavior of the model when trained with the equal frequency sampler becomes too geared toward convective schemes.
In practice what happens is that the model after the first 30 minutes start to convert any precipitation pattern into convective precipitation.
You can see an example of this in this animation that I presented at EGU last year to show the effect of the data distribution in training generative models

output_fps20.mp4

You can see the behavior of LDCast (the original checkpoint as released by Jussi) in comparison with the observation and also the other two models (GPTCast and DGMR)
LDCast quickly hallucinates towards thuderstorm everywhere, and this is due to the fact that the models basically gives the same probability to the evolution of the parttern towards either no rain and thuderstorms. This is clearly not desirable for a general pourpose nowcasting model like the ones that we want to release within MLCast

Of course this behavior is due to the equal frequency sampler and diappears when sampling the data in a less agressive way.
[9:03 AM] The other thing is a bug in the original implementation that affected the paper release
[9:03 AM] If you check the commit history: https://github.com/MeteoSwiss/ldcast/commits/master/
[9:05 AM] there is a commit after the paper release that fixes a bug about the fact that the extrapolator part of LDCast was not being trained properly: MeteoSwiss/ldcast@267ef65
[9:06 AM] this bug was discovered by Alberto Carpentieri that was repourposing LDCast for solar nowcasting (Shadecast).
[9:07 AM] https://github.com/EnergyWeatherAI/GenerativeNowcasting

[leifdenby]

To do a bit of planning I did some comparison of the DMI () and MeteoSwiss () repos with LDCast. Here's what I learnt: The DMI repo includes all commits in the MeteoSwiss fork, so the bug that you mentioned @franchg is included what @mfroelund has been working from.

I also scanned through all the commits on the DMI fork with the help of codex, and here's a table indicating for the changes

Codex summary	Martin’s write‑up (matching line)	Type	Related commits
Zarr pipeline (conversion, load/save via xarray; model runs from Zarr)	“rework the data loading to work directly from Zarr… created a zarr version… sampler saves to Zarr… loading can be done directly from the two Zarr stores”	Feature	dmidk/ldcast-dmi@15f9224 dmidk/ldcast-dmi@73c626f dmidk/ldcast-dmi@4830a15
Equal‑frequency sampling via bin duplication	“save the sampled dataset by simply duplicating the data in each bin…”	Feature	dmidk/ldcast-dmi@bec1f83
DDP‑ready data loading / DistributedSampler	“dataloading… pytorch structure… TrainDataModule… DistributedSampler”	Feature	dmidk/ldcast-dmi@18f94d8 dmidk/ldcast-dmi@cc2f40b dmidk/ldcast-dmi@07b2fa3
DataModule generalized for multiple prediction datasets	“PredictDataModule… EnsembleBatchSampler…”	Feature	dmidk/ldcast-dmi@cc2f40b
Fire‑based CLI w/ sampling as separate command	“extended the usage of Fire… CLI structure… sample command”	Feature	dmidk/ldcast-dmi@371d846
Forecaster rework	“forecast treated as a model on its own… ldcast/models/forecast.py”	Feature	dmidk/ldcast-dmi@208c1d0
Script‑to‑model refactor (train scripts into model init.py)	“moved scripting logic… into ldcast/models/autoenc/init.py and …/genforecast/init.py”	Feature	dmidk/ldcast-dmi@208c1d0
Visualization split	“plotting… moved to ldcast/visualization/plots.py”	Feature	dmidk/ldcast-dmi@208c1d0
Re‑patching workflows (data re‑patch scripts)	Not mentioned	Feature	dmidk/ldcast-dmi@f396a14 dmidk/ldcast-dmi@436a649 dmidk/ldcast-dmi@ac0e4fb
Seed handling / time handling cleanup	Not mentioned	Bugfix/Maint	dmidk/ldcast-dmi@a2934ff dmidk/ldcast-dmi@e60e43c
Dask worker limits to avoid hangs	Not mentioned	Bugfix/Maint	dmidk/ldcast-dmi@6fb942c
“Fix bug for third time!” (unspecified)	Not mentioned	Bugfix	dmidk/ldcast-dmi@2cf9d6a
“Adapt for Leonardo” environment	Not mentioned	Feature/Ops	dmidk/ldcast-dmi@80dbdfa

And here's a diagram of the features and bugfixes that @mfroelund I think you did (again with the help of codex), to give me an overview of the changes and decide what needs to go into mlcast (features in green, bugfix in orange):

     flowchart TD
    classDef feature fill:#d9f5e5,stroke:#2b7a5e,stroke-width:1px,color:#0f3d2e;
    classDef bugfix fill:#ffe7c7,stroke:#c56a1a,stroke-width:1px,color:#5a2d00;

    subgraph CLI["1) CLI interface"]
      CLI1["Fire-based CLI; sampling split into its own command"]:::feature
    end

    subgraph DATASET["2) Dataset preparation"]
      DP1["Zarr conversion + xarray save/load pipeline"]:::feature
      DP2["Re-patching workflow with patch-shape outputs"]:::feature
      DP3["Equal-frequency sampling via bin duplication"]:::feature
      DP4["Dask worker limits to avoid hangs"]:::bugfix
    end

    subgraph DATALOAD["3) Data loading"]
      DL1["PyTorch-style dataloading refactor"]:::feature
      DL2["DistributedSampler + DDP data split correctness"]:::feature
      DL3["Train/Predict DataModules + ensemble batch sampling"]:::feature
      DL4["Seed handling + time handling cleanup"]:::bugfix
    end

    subgraph MODEL["4) Model architecture"]
      MA1["Forecaster rework; forecast as dedicated model module"]:::feature
      MA2["Training scripts moved into model modules"]:::feature
    end

    subgraph TRAIN["5) Model training"]
      MT1["Sampling moved out of training flow"]:::feature
      MT2["DDP training correctness fixes"]:::bugfix
      MT3["Generic bugfix commits (unspecified)"]:::bugfix
    end

    subgraph INFER["6) Model inference"]
      MI1["Forecast pipeline refactor"]:::feature
      MI2["PredictDataModule supports ensembles"]:::feature
    end

    %% Relationships
    CLI1 --> DP1
    CLI1 --> DP3
    CLI1 --> DL1
    CLI1 --> MT1

    DP1 --> DL1
    DP3 --> DL1
    DP2 --> DL1
    DP4 --> DP1

    DL1 --> MT2
    DL2 --> MT2
    DL3 --> MI2

    MA1 --> MI1
    MA2 --> MT1

    MT2 --> MI1

Loading

[leifdenby]

With regards to how you changed the training dataset preparation @mfroelund, I've tried to make a diagram below of what I think is going on.

flowchart TD
  A[Source data is the DMI radar archive<p><i>Path example: /dmidata/projects/nwp/radar/.../YYYY/MMDD/nowcast.dk.com.YYYYMMDDHHMM...zip</i></p><p><i>Format ZIP files with HDF5 frames; one 2D image per timestamp</i></p>]

  B[Tiling step<p><i>DataPreprocessing/radarToZarrML.py</i></p><ul><li>Split each 2D image into fixed size tiles</li><li>Store tile coordinates and timestamp</li><li>Separate normal tiles and empty tiles</li></ul>]

  C[( ''patched Zarr'' tile store<p><i>Path example: /leonardo_work/.../radar_patched_*.zarr</i></p><ul><li>All tiles with coords and indices</li></ul> )]

  D[Optional merge<p><i>DataPreprocessing&#47;merge_datasets.py</i></p><ul><li>Combine multiple tiled stores</li></ul>]

  E[( merged tiled store<p><i>Path example: /dmidata/projects/.../merged_radar_patched_dmi_*.zarr</i></p> )]

  F[Sampling step<p><i>ldcast&#47;features&#47;sampling.py</i></p><ul><li>Bin tiles by intensity</li><li>Compute start_time start_i start_j</li><li>Duplicate bins for equal sampling</li></ul>]

  G[( ''sampled Zarr'' index store<p><i>Path example: /leonardo_scratch/.../sampled_radar_*.zarr</i></p><ul><li>Start indices for train valid test</li></ul> )]

  H[Training uses both<p><i>ldcast&#47;features&#47;data_handling.py</i></p><ul><li>Tile data from ''patched Zarr''</li><li>Sample indices from ''sampled Zarr''</li></ul>]

  A -->|read by| B
  B -->|writes to| C
  C -->|read by| D
  D -->|writes to| E
  C -->|read by| F
  E -->|optional input to sampling| F
  F -->|writes to| G
  C -->|read by| H
  G -->|read by| H

Loading

If I've understand correctly I wouldn't call the first dataset you create the "patched zarr" (since patching something means something else), rather I would call the process of splitting a large 2D array into smaller square 2D arrays tiling. As I understand it the process is:

1. Start from a collection of HDF5 files representing gridded radar observations over time
2. Create square spatially sampled tiles across entire dataset, save to "patched zarr" (I would call this "tiled zarr")
3. Compute tile-statistics (say sum of scalar field for each tile) and bin tile-statistics in equally spaced bins to give an measure of distribution. Create collection of tile-indices to use during training to ensure near-equal number of tiles from each scalar bin (your approach here is to oversample the bins with lowest counts I think), store tile indicies in "sampled zarr", I would call this the "tile sampling indicies zarr".

Does that sound right @mfroelund? Also could you add here a example of the contents of these two zarr datasets? I.e. the output of repr(xr.open_dataset(<tiled-zarr-path>)) and similarly for the zarr with the tile-indicies?

How does this overlap with what you've done for the sampling @franchg? I know you also create tile-indices, but rather as a CSV. But you don't create a zarr dataset of tiles, right? (I think you just expect to have a single zarr dataset covering the entire source data and index into that with sampling indicies stored into a CSV).

[mfroelund]

Amazing what these tools can do!

Yes, it sounds pretty much as what I've done, and I agree, it's probably more correct calling it tiling:)

W.r.t. the dataloading I've not spend much time on the radarToZarrML.py and the merge_datasets.py scipts. The former was actually made by @arjj8. I did some parallelization of the patching/tiling and made it save the tiled dataset to zarr using xarray. But these scripts should definitely have a makeover if we want to use them in the future.

Create collection of tile-indices to use during training to ensure near-equal number of tiles from each scalar bin (your approach here is to oversample the bins with lowest counts I think)
Yes, that's true. I did this to simplify the loading of the data, such that we didn't have to keep track of which bin to sample from when loading. But all this equal freq. sampling is probably not ideal as @franchg has pointed out, so my intension was also to make it optional to sample (didn't have the time).

I'll give you an example of the contents of the two datasets, just need to find them..

[arjj8]

Hej! all of this is amazing and I am really excited to start working on it! About the patching/tiling (@leifdenby you are right this is more a tiling process, sorry for the confusion), my idea can be understood through the following gif:

But I believe that it need to be changed to take into account mlcast database.

[mfroelund]

First of all, I want to say that I'm not an expert on xarray, so there is probably better ways of doing what I've done, that I didn't know off!

Patched/tiled dataset

Since I wanted to be able to get a patch/tile from the dataset using

patch = self.ds_patches.sel(
    patch_times_seconds=<time>,
    patch_icoord=<i>,
    patch_jcoord=<j>,
).patches

I needed to do something about the degeneracy of the patch_icoord and patch_jcoord for a given patch time, so I introduced a MultiIndex patch_index (same for the zero_patches, zero_patch_index) that simply runs from 0 to number_of_patches (0 to number_of_zero_patches).

To save and load a dataset with a MultiIndex, I used cf_xarray's decode_compress_to_multi_index and encode_multi_index_as_compress functions. So loading the dataset is done with

def get_patch_data(path: Path):
    logger.info("Opening patch dataset")
    ds_patches = xr.open_zarr(path)
    ds_patches = cfxr.decode_compress_to_multi_index(
        ds_patches, ["patch_index", "zero_patch_index"]
    )

    return ds_patches

in features/data_handling.py. An example of a patched/tiled dataset for the period 2017-07-02T13:50 to 2017-08-01T16:40 can be seen below:

<xarray.Dataset> Size: 27GB
Dimensions:                   (patch_index: 63217, x: 216, y: 248,
                               zero_patch_index: 19828)
Coordinates:
    patch_times               (patch_index) datetime64[ns] 506kB ...
  * patch_index               (patch_index) object 506kB MultiIndex
  * patch_times_seconds       (patch_index) int64 506kB 1499003400 ... 150160...
  * patch_icoord              (patch_index) int64 506kB 4 4 4 5 5 ... 6 6 6 7 7
  * patch_jcoord              (patch_index) int64 506kB 1 2 5 1 2 ... 3 4 5 2 3
  * zero_patch_index          (zero_patch_index) object 159kB MultiIndex
  * zero_patch_times_seconds  (zero_patch_index) int64 159kB 1499003400 ... 1...
  * zero_patch_icoord         (zero_patch_index) int64 159kB 2 2 3 3 ... 4 6 2 4
  * zero_patch_jcoord         (zero_patch_index) int64 159kB 2 3 2 3 ... 5 5 3 5
Dimensions without coordinates: x, y
Data variables:
    patches                   (patch_index, x, y) float64 27GB ...

Sampled dataset

The sampling consists of several steps:

1. Binning of the patches
2. Calculation of "indices with complete sample" (indices_with_complete_sample function in sampling.py)
3. Calculation of "starting indices for centers" (starting_indices_for_centers function in sampling.py)
4. Oversampling of the bins to have equal frequency, when loading from the dataset at random
5. Concatenation of all bins, so we end up with a dataset with one index

Bullet 2. and 3. comes from the original code, and it took me a while to understand what they were doing. I also did some changes to the functions to speed them up. I've tried to describe what they do below, as I understand them. But I'm not sure I completely understand why we need to do what they are doing.

The indices_with_complete_sample function basically builds up a dictionary with the structure {time: {i: {j: 0}}} for all triplets of time, i and j for which sufficient neighboring patches/tiles exists in both time and space. The number of patches that are sufficient is determined by

• the so called sample_shape, which is set in the config at `preprocessing.sampling.sample_shape
• the time_range_valid array, which is defined in the execute_sampling function in sampling.py based on the preprocessing.sampling.num_timesteps and the preprocessing.sampling.timestep config values.

Now, the starting_indices_for_centers function determines for every patch/tile of the binned patches/tiles, if the index of the patch/tile is in the dictionary of complete samples, as well as its neighbours in both time and space. This time the neighbourhood to consider is determined by

• the sample_shape
• the time_range_sampling, which is set in the config at preprocessing.sampling.time_range_sampling

If an index of a patch/tile, as well as its neighbours, are found in the dictionary of complete samples, the index is added to a list of starting indices. This then finally ends up in the sampled dataset (after bullet 4. and 5.).

I kept the notion of "starting indices" in the sampled dataset, although it might be more meaningfull to just call them tile indices like tile_i and tile_j:

<xarray.Dataset> Size: 20MB
Dimensions:     (index: 622510) **
Coordinates:
  * index       (index) int64 5MB 0 1 2 3 4 ... 622506 622507 622508 622509
Data variables:
    start_i     (index) int64 5MB 5 4 5 3 4 5 5 5 5 4 1 ... 2 2 1 1 0 6 0 4 3 6
    start_j     (index) int64 5MB 1 1 2 3 3 1 5 2 3 4 3 ... 1 1 1 1 2 0 2 6 1 5
    start_time  (index) int64 5MB 1499647800 1500457800 ... 1501047600
Attributes:
    preselected_samples:  None
    sample_shape:         [1, 1]
    split:                train
    time_range_valid:     [-600, 4800]
    timestep_secs:        600

** I'm a bit puzzled now why the index of the patch/tile dataset and the sampled dataset are not of equal size..

[franchg]

Hi all,

I've just published the first version of the mlcast sampling package / code :

👉 https://github.com/mlcast-community/mlcast-dataset-sampler

The architecture of the package should allow to add as many sampling strategies we want, currently the importance sampling streategy from DGMR is implemented.

Two CLI commands in sequence:

# Step 1: Filter valid datacubes (removes gaps and high-NaN regions)
mlcast.sample_dataset filter-nan /path/to/mlcast-radar.zarr \
    --start-date 2021-01-01 --end-date 2024-12-31 \
    --time-depth 24 --width 256 --height 256
# produces a csv of indexes `valid_datacubes_2021-01-01-2024-12-31_24x256x256_3x16x16_10000.csv`

# Step 2: Importance sampling (weights by rain intensity) 
mlcast.sample_dataset sample /path/to/mlcast-radar.zarr \
    valid_datacubes_2021-01-01-2024-12-31_24x256x256_3x16x16_10000.csv
# produces a csv `sampled_datacubes_2021-01-01-2024-12-31_24x256x256_3x16x16_10000.csv`

This addresses the equal-frequency issue I mentioned - instead of giving equal probability to all intensities (which causes thunderstorm hallucinations), we use a minimum probability + intensity-weighted bonus.

Unified DataLoader: I think we should use the same torch Dataset to train all models. The sampler outputs a simple CSV with (t, x, y) that indexes directly into the source Zarr - no intermediate tiled stores needed.

Envisioned workflow:
mlcast-datasets → mlcast-dataset-sampler → mlcast (training)

Example DataModule that works with the sampled CSV (i.e. the output of step 2 above plus the mlcast validated zarr)
👉 https://github.com/DSIP-FBK/ConvGRU-Ensemble/blob/main/convgru-ens/datamodule.py

@mfroelund or @arjj8 could you try running mlcast-dataset-sampler on the DMI dataset? Would be great to verify it works with the DMI Zarr. The data variable and time coordinate names are configurable via --data-var and --time-var.

The output CSV can be used together with the original Zarr and the DataModule to train LDCast. For multi-dataset training, we can run the sampler on each radar Zarr, create a SampledRadarDataset for each, and combine them using torch.utils.data.ConcatDataset.

If I have the time tomorrow I will follow up with a comple example on how to use this to train LDCast.

[arjj8]

Hej @franchg !

I tried to run the mlcast.sample_dataset with DMI zarr dataset, but I get the following error:
uv run mlcast.sample_dataset filter-nan /path/to/DMI_DataSet_500m_10min_v010.zarr --start-date 2025-01-01 --end-date 2025-12-31 --time-depth 24 --width 241 --height 216 --max-nan 100000 --data-var dbz --time-step-minutes 10

2026-02-06 11:46:35 | ERROR | mlcast_dataset_sampler.commands.filter_nan:run:302 - No data found between 2025-01-01 00:00:00 and 2025-12-31 00:00:00

I modified some of the arguments to try to fit to the DMI data, e.g. time-step-minutes 10, data-var dbz ... I also play around with others like max-nan, start and end date, but I get the same error. I also print the values of the DMIvdataset just to verify that it is not just nan values. Could it be I am using wrong some of the arguments, for example what does time-depth refers to?

Edit 1 So I think I have found the reason, I code the function myself, and realised that when using zarr, I don't know if it is due to my version, and then using the pd.to_datetime(zg['time'][:]) function the time variable it becomes:

while when reading with xarray and then using the same function:

I have thought about some potential reasons of this: 1) it can be that the way the DMI time variable was saved it is wrong and when read by the zarr library this happen, 2) there is something happening with the zarr version or 3) this is common when using zarr. Maybe we could try to another database and see if a similar thing happen.

[leifdenby]

I tried to run the mlcast.sample_dataset with DMI zarr dataset, but I get the following error:
uv run mlcast.sample_dataset filter-nan /path/to/DMI_DataSet_500m_10min_v010.zarr --start-date 2025-01-01 --end-date 2025-12-31 --time-depth 24 --width 241 --height 216 --max-nan 100000 --data-var dbz --time-step-minutes 10

@arjj8 I think you might be having the same problem as raised here: mlcast-community/mlcast-dataset-sampler#1 :) I suggest reposting your comment over there so the three of you can work out what's going wrong together. Then we can keep this issue to be about implementing LDCast (which I appreciate also could use the sampler, but doesn't necessarily need one)

[arjj8]

hej @franchg and @leifdenby !
Iran the new version of the mlcast dataset sampler and it works now. I got both the sampled_datacubes_****.csv and the metadata.json files! :D