Consider setting time and platform as dimensions

[melodb]

I used a function like this:

def restore_mindex(
    data: xr.Dataset,
    index_var: str = "index",
    platform_var: str = "platform",
    indicator_var: str = "number_of_identifier",
) -> xr.Dataset:
    platform_arr = data[platform_var].values[data[indicator_var].values.astype("int")]
    mindex = pd.MultiIndex.from_arrays([platform_arr, data.time.values])
    return data.assign_coords(xr.Coordinates.from_pandas_multiindex(mindex, index_var))

to make index a coordinate that is a stack of time and platform coordinates (and also creates dimensions for time and platform). I'm not 100% sure that this will let you select by time and platform... you might need to do .unstack("index") first (which might introduce NaNs if different platforms have different time values). We use something similar in OpenGHG Inversions to work with our old output format, and I found have the time and site (or platform) coordinates useful.

Also the platform_arr replaces the indices with the labels, but you could just use data[indicator_var].values instead if you want the indices.

(If you already have time and platform, then ds.stack(index=("time", "platform")).dropna("index") will give you an index coordinate like fluxy uses now. You can't save stacked dimensions/coordinates like this to netCDF, so either you need to unstack (and add NaNs), or convert to something like the format in the CDL templates.)

Since assimilation_flag isn't a dimension you'd need to use .where in that case anyway. But potentially you could do ds_all[m].sel(time=slice(start_date, end_date).sel(platform="TAC"). (If say, "TAC" == ds_all[m]["platform"][site_index])

Originally posted by @brendan-m-murphy in #234 (comment)

[melodb]

I have created this issue so that we don't forget about this. Everything is working fine in fluxy at the moment. To be evaluated if there is an advantage in restructuring the data within fluxy.

[brendan-m-murphy]

I think the main advantage is that it is more intuitive to directly select the site/platform you want, or directly select the times you want.

Also if the dictionary of datasets were converted to a xr.Datatree, you could select from all of the models simultaneously with one .sel call. (I mentioned this in a separate issue. A Datatree is probably not 100% exchangeable with a dictionary of datasets though.)

[lionel42]

EDIT: We should definitely NOT do this in the data reading part, see my comment below. If we do this we should do it in the specific plot functions only.

If it helps to use this Multiindex for accessing specific times or platfrom, we should include it in the data reading part.

[lionel42]

Also we now use masking and accessing at various places in the code, we need to make sure we are consistent everywhere.

[brendan-m-murphy]

Just a note: there is a CF convention for this https://cfconventions.org/cf-conventions/cf-conventions.html#compression-by-gathering

So the index variable should probably the attribute compress="platform number_of_identifier" (e.g. in the template). There is a method in cf_xarray to decode this attribute (which is essentially what the function I suggested does).

We already use cf_xarray in openghg as the base of our unit conversion code. I guess fluxy doesn't require openghg though, so it would be a new dependency.

[lionel42]

Reading the CF conventions I noticed that the advantage of having this index dimension instead of platform and time is to allow for a broader definition of sampling times and platform.

In particular if you have a measurements that you don't hourly average, you will end up with the exact datetimes which will be different across all platforms, thus leading to non gridded data on the time dimension or multiple data for the same hour in case of time rounding.

Another example can be satellite observations, which will have multiple observations for the same platform/datetime combination.

I would suggest to not use these mindex to change the general structure. If they are helpful in a plotting function, then you could still use it there with care.

[brendan-m-murphy]

I think if the "time" dimension has different interpretations for different platforms then those platforms should be different groups in the netcdf (which would mean using Datatrees).

In xarray you can still "stack" platform and time into "index" and drop NaNs along "index", which would give you effectively the same "index" dimension, but it would be linked to platform and time in a more useful way.