Skip to content

Support multiple interpolation windows in ForecastTrajectoryGapFiller - #2418

Open
gavinevans wants to merge 1 commit into
metoppv:masterfrom
gavinevans:mobt_783_gap_filler_multiple_windows
Open

Support multiple interpolation windows in ForecastTrajectoryGapFiller#2418
gavinevans wants to merge 1 commit into
metoppv:masterfrom
gavinevans:mobt_783_gap_filler_multiple_windows

Conversation

@gavinevans

Copy link
Copy Markdown
Contributor

Description
This PR extends the Forecast TrajectoryGapFiller plugin and CLI, so that it is possible to use different interpolation windows for different forecast source pairs. This is desirable because a shorter interpolation window might be desirable at nowcast timescales whilst a longer interpolation window might be desirable at longer lead times.

This PR also fixes a couple of edge cases that were noticed in testing:

  1. I noticed that the _identify_periods_to_regenerate wasn't identifying the expected leadtimes for regeneration correctly. Notably although later lead times were being identified fine, the periods being identified were e.g. [3600, 3600, ...]. This was surprising because 3600 (seconds) shouldn't have been identified as a transition, and this transition shouldn't have been included as a duplicate. _identify_source_transitions now groups sources by forecast period first, instead of walking source-by-source. That means if multiple sources share the same period, they are treated as one source state for that period. The transition check is then only between consecutive unique periods, and only when the source set actually changes. That is what removes the spurious 3600 transition and the duplicate 3600 entry. For example, period_to_sources variable could look something like below where a transition will only be identified where the set of sources changes.
{
    3600:   {"uk_det", "uk_ens"},
    21600:  {"uk_det", "uk_ens"},
    43200:  {"uk_ens"},
    86400:  {"uk_ens"},
    129600: {"gl_ens"},
    172800: {"gl_ens"},
}
  1. In the _create_regeneration_tasks method, I noticed that it was possible for regeneration tasks to be silently dropped. This was possible when, for example, the plugin had identified T+1 as a transition lead time, but an interpolation window of 3 hours was being used. This led to the plugin trying to find a T-2 forecast period. As a T-2 forecast period doesn't exist, it was silently dropping this regeneration. There is now support for this situation by a) finding the nearest forecast periods before and after the transition, so that the regeneration of particular lead times won't just be silently dropped, and b) the method is now much noisier if it has to fall back to finding a nearest lead time, so that this sort of edge case won't occur silently.

Testing:

  • Ran tests and they passed OK
  • Added new tests for the new feature(s)

…windows for difference forecast source pairs. Modify identification of forecast periods for regeneration to avoid identifying more transition points than expected. Add warnings and better support, rather than silently continuing, in the situation where the forecast periods required for regeneration are not available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants