Skip to content

Add dynamical as a priority/source - #88

Open
williamhobbs wants to merge 17 commits into
mainfrom
add_dynamical
Open

williamhobbs wants to merge 17 commits into
mainfrom
add_dynamical

Conversation

@williamhobbs

@williamhobbs williamhobbs commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Closes #68, #2, #56 (after it was re-opened), #87, and #89.

To-do list:

  • run a few more tests (and one day add actual tests...)
  • update documentation
  • add some filters with helpful error messages for incompatible combinations of inputs
  • consider condensing functions (e.g., merge get_solar_forecast and get_solar_forecast_fast into one with use_fastherbie as an input parameter, maybe also merge the ensemble function as well...) I decided not to do this, at least for now.
  • update remaining interpolation to center of hour (HRRR is maybe the only one left?)
  • shift 1min clear sky to the 30s mark? (1min clear sky calculations should be centered on the 30s mark #89)

@williamhobbs

Copy link
Copy Markdown
Owner Author

For filters/checks/warnings on input parameter combinations, combinations that don't work:

  • decomp_model == 'erbs' and model in ['hrrr', 'cams'] -> warnings.warn(f'model={model} does not require decomposition, but you entered decomp_model="{decomp_model}". This will not do anything.)
  • hrrr_hour_middle == False and model != 'hrrr' -> warnings.warn(f'You entered hrrr_hour_middle=False, which does not apply to the model you entered, "{model}". This will not do anything.)
  • hrrr_coursen_window not None and model != 'hrrr' -> warnings.warn(f'You entered hrrr_coursen_window={hrrr_coursen_window}, which does not apply to the model you entered, "{model}". This will not do anything.)
  • [model == 'cams' and priority not None...]
  • [model not an ensemble and entered member not None]
  • [model not 'cams' and (cams_area not None or cams_api_key not None)]

@williamhobbs

williamhobbs commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

[edit: this was fixed with 2c3adf1]

This bit of code is not working for non-dynamical sources:

hefty/src/hefty/solar.py

Lines 1746 to 1764 in 7613eba

if not get_ens_temp and not get_ens_wind:
_, _, _, search_str_2 = model_input_formatter(
init_date, run_length, lead_time_to_start, model,
resource_type, full_ens=False, get_ens_temp=False,
get_ens_wind=False, member=0)
# 2m temperature search string should be second
search_str_cust = "|".join(search_str_2.split('|')[1:2])
# use control member
member_cust = 0
df_rem = get_fcast_dataframe(
latitude, longitude, init_date, fxx_range, model,
search_str_cust, priority, product,
fast, attempts, resource_type,
member_cust)
# merge
df_merged = pd.merge(
df_merged,
df_rem.drop(columns=['lead_time', 'number']),
on=['valid_time', 'point'])

get_fcast_dataframe is getting temperatures for all members, not just the control. An example of this:

member_cust = 0
resource_type = 'solar'
latitude = [32, 30]
longitude = [-89, -90]
init_date = pd.Timestamp('2026-09-10 00:00:00')
fxx_range = range(9, 16, 3)
model = 'ifs_ens'
search_str_cust = ':2t'
priority = 'azure'
product = 'enfo'
fast=True
attempts=2

df_rem = get_fcast_dataframe(
                latitude, longitude, init_date, fxx_range, model,
                search_str_cust, priority, product,
                fast, attempts, resource_type,
                member_cust)

df_rem

returns a dataframe with 300 rows, but it should be 6.

To get just the control from IFS ens, options are:

  • before 2026-05-12, use a search string like ':2t:sfc:g'
  • after 2026-05-12, switch product to 'oper' and search string '':2t'' (or ':2t:sfc:g')

I think the solution is to use the product that is returned from model_input_formatter.

@williamhobbs

williamhobbs commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

Informal tests included:

  • run through all the example notebooks (no errors, results look consistent) [edit: apparently I didn't wait for everything to finish; there was an error with cams, but I fixed it with the next commit...]
  • check the fix for the last comment with something like:
resource = get_solar_forecast_ensemble(
    latitude=[32, 30], longitude=[-89, 90],
    init_date='2026-09-10', run_length=6,
    lead_time_to_start=12, model='ifs_ens',
    attempts=2, priority='azure')
  • running some internal applications that use hefty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add dynamical.org as a source, make a generic "get NWP and return dataframe" function

1 participant