grib filtering and different ppt tables

[oloapinivad]

Working on PRIMAVERA WP5 I have noticed a current limitation in the grib filtering that I suspect it is not so easy to solve. Due to data volume limitation in WP5 it has been decided to reduce the number 3-hour variables. This means that some of the surface variable in the GG file are now at 3-hour freq, and some at 6-hour.

As a consequence, the grib filtering is broken - it is currently based on pressure levels vs. surface levels. Unfortunately, we cannot use the simple time-based approach since some of those variables are fluxes (for example keeping 6-hour only).

Probably the solution will be to split the GG file analysis in two blocks, one for 6hr data and one for 3hr data. But it requires quite a large rewriting of the ifs_*.sh files.

[plesager]

Got a similar problem with the added output variables for CORDEX (as done for a KNMI run).
In that configuration, sst ends up with a different time step in GG files, and z on all hybrid levels (instead of 1) and a different time step in the SH files. I ended up with a working solution by removing these variables (and also the model levels) through the filters. That lets me run hiresclim2 and timeseries. I pushed a dedicated branch (cordex) for that specific case.

But in your case, just removing the variables with the different time step is not an option. But I think a general solution is feasible. First the filters should just select the desired variable and write them into their own file. For the 2D filter, that would be something like:

    FILTERGG2D="if ( param is \"182.128\" || param is \"165.128\" || param is \"166.128\" || param is \"167.128\" || param is \"31.128\" || \
param is \"34.128\" || param is \"141.128\" || param is \"168.128\" || param is \"164.128\" || param is \"186.128\" || param is \"187.128\" || \
param is \"188.128\" || param is \"78.128\" || param is \"79.128\" || param is \"137.128\" || param is \"151.128\" || param is \"243.128\" || \
param is \"205.128\" || param is \"144.128\" || param is \"142.128\" || param is \"143.128\" || param is \"228.128\" || param is \"176.128\" || \
param is \"177.128\" || param is \"146.128\" || param is \"147.128\" || param is \"178.128\" || param is \"179.128\" || param is \"180.128\" || \
param is \"181.128\" || param is \"169.128\" || param is \"175.128\" || param is \"208.128\" || param is \"209.128\" || param is \"210.128\" || \
param is \"211.128\" ) { write \"[paramId]_[shortName].grb\"; }"

That means the cdo splitvar is not needed anymore. Need to remove the -o filename option from the grib_filter call.

Then the time steps (pptime in the code) should be an associative array (like dictionaries in python but for bash), so each variable can have its own step. The only remaining problem then is when variables with different time steps are combined (sub, add, ...), but hopefully this is not the case.