auto-generate deflator files from inflation_default.csv#63
Open
merveturan wants to merge 12 commits into
Open
Conversation
- Remove static `deflator.csv` files; update references in runfiles.csv and sources.csv - Update `get_source_deflator_map()` function to build `deflator.csv` from `inflation_default.csv` in `copy_files.py` - Add current financial year (most recent inflation value) to scalar.csv - Update deflators.csv directory path in `output_calc.py` - Update `inflate_series()` function in reeds2.py to calculate deflator values from inflation_default.csv -Add base dollar year to postprocessing defaults.py script
| @@ -1,4 +1,5 @@ | |||
| DEFAULT_DOLLAR_YEAR = 2024 | |||
| DEFAULT_DOLLAR_BASED_YEAR = 2004 | |||
Contributor
There was a problem hiding this comment.
This value is already specified in the dollar_year switch in cases.csv, so should be read from there:
Line 64 in 3a3bd81
| cost_upgrade_gasct2h2ct,0.33,"--fraction-- Sets upgrade costs for H2-CT plants based relative to capital cost for Gas-CT." | ||
| cost_upgrade_gascc2h2cc,0.28,"--fraction-- Sets upgrade costs for H2-CC plants based relative to capital cost for Gas-CC." | ||
| cost_vom_psh,0.375,"--2004$/MWh-- VOM cost for PSH" | ||
| current_financial_year,2024,"--year-- most recent financial year to have inflation value (used for inflation adjustments)" |
Contributor
There was a problem hiding this comment.
It would be nice to avoid setting the same value in multiple places. This value is also set in bokeh:
)I would either use the value from bokeh or remove it from bokeh and read it from here.
Comment on lines
+173
to
+179
| # Set the initial deflator value for the base year to 1.0 | ||
| deflator_values: dict[int, float] = {int(sw.dollar_year): 1.0} | ||
|
|
||
| # Calculate the deflator values for each year, relative to the base year, using the inflation rates | ||
| for y in years[1:]: | ||
| rate = inflation_df.loc[inflation_df['t'] == y, 'inflation_rate'].values[0] | ||
| deflator_values[y] = deflator_values[y - 1] / rate |
Contributor
There was a problem hiding this comment.
You could instead use the existing get_inflatable() function:
inflatable = reeds.io.get_inflatable()
deflator = 1 / inflatable[2004].loc[2004:2024]
github-actions
Bot
added
data_changes
model_changes
input_processing
postprocessing
switches
labels
Updated the description for 'dollar_year' to clarify its purpose.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes the hand-maintained inputs/financials/deflator.csv and instead generates deflator.csv during input processing from inputs/financials/inflation_default.csv, while also centralizing “modeled dollar year” in inputs/scalars.csv and updating postprocessing to use explicit modeled/output dollar-year constants.
Changes:
- Auto-generate
inputs_case/deflator.csvininput_processing/copy_files.pyfrominflation_default.csv(and stop trackinginputs/financials/deflator.csvin-repo). - Move
dollar_yearfromcases.csvintoinputs/scalars.csvand update preprocessing/postprocessing codepaths to reference scalars / defaults accordingly. - Update BokehPivot and retail rate postprocessing to distinguish modeled vs output dollar year (
ReEDS_DOLLAR_YEARvsDEFAULT_DOLLAR_YEAR) and compute inflation adjustments accordingly.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sources.csv | Removes the inputs/financials/deflator.csv source entry since it’s no longer a repo input. |
| runfiles.csv | Treats deflator.csv as generated (no source filepath). |
| reeds/output_calc.py | Updates inflation helper to read inputs_case/deflator.csv (but current path logic is incorrect for run-directory inputs). |
| postprocessing/retail_rate_module/retail_rate_calculations.py | Imports modeled/output dollar-year constants and uses them for inflation conversion. |
| postprocessing/retail_rate_module/mean_bias_error.py | Replaces hardcoded modeled dollar year with ReEDS_DOLLAR_YEAR. |
| postprocessing/retail_rate_module/ferc_distadmin.py | Defaults dollar_year to ReEDS_DOLLAR_YEAR. |
| postprocessing/input_plots.py | Uses scalars['dollar_year'] as the modeled-dollar-year for inflation conversion. |
| postprocessing/compare_cases.py | Uses ReEDS_DOLLAR_YEAR instead of a hardcoded modeled dollar year. |
| postprocessing/bokehpivot/reeds2.py | Switches to computing inflation multiplier from inflation_default.csv (but currently lacks correct handling for dollar years earlier than the modeled year). |
| postprocessing/bokehpivot/defaults.py | Adds ReEDS_DOLLAR_YEAR constant (currently duplicates inputs/scalars.csv and is inconsistently named). |
| inputs/scalars.csv | Adds dollar_year scalar (modeled dollar year). |
| inputs/financials/deflator.csv | Removes the hand-maintained deflator table from the repo. |
| input_processing/transmission.py | Uses scalars['dollar_year'] to select the appropriate USD column. |
| input_processing/copy_files.py | Implements deflator generation into inputs_case/deflator.csv (currently relies on a global inputs_case). |
| input_processing/calc_financial_inputs.py | Uses scalars['dollar_year'] instead of a case switch. |
| cases.csv | Removes dollar_year from case switches. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
217
to
225
| def inflate_series(case,dfin, to_dollar_year=DEFAULT_DOLLAR_YEAR): | ||
| """ | ||
| Inflate a series of financial data to a specified dollar year. | ||
| """ | ||
| df_deflator = pd.read_csv( | ||
| os.path.join(reeds_path, 'inputs', 'financials', 'deflator.csv'), | ||
| index_col=0, | ||
| os.path.join(case, '..', '..', 'inputs_case', 'deflator.csv'), | ||
| index_col=0 | ||
| ) | ||
| return dfin * 1 / df_deflator.loc[to_dollar_year, 'Deflator'] |
Comment on lines
+75
to
+76
| deflator_multiplier = prod( inflation_dict[y] | ||
| for y in range(int(ReEDS_DOLLAR_YEAR) + 1, int(core.GL['widgets']['var_dollar_year'].value) + 1 )) |
| @@ -1,4 +1,5 @@ | |||
| DEFAULT_DOLLAR_YEAR = 2024 | |||
| ReEDS_DOLLAR_YEAR=2004 | |||
wesleyjcole
approved these changes
May 26, 2026
Contributor
wesleyjcole
left a comment
wesleyjcole
left a comment
There was a problem hiding this comment.
Looks good. Thanks for making the updates.
| rb_globs = {'output_subdir':'/outputs/', 'test_file':['cap.csv','outputs.h5'], 'report_subdir':'/reeds2'} | ||
| this_dir_path = os.path.dirname(os.path.realpath(__file__)) | ||
| df_deflator = pd.read_csv(os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'inputs/financials/deflator.csv')), index_col=0) | ||
| df_inflation = pd.read_csv(os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, 'inputs/financials/inflation_default.csv')), index_col=0) |
Contributor
There was a problem hiding this comment.
Since you're already reading from the ReEDS repo here, I think it would be cleaner to just read dollar_year from scalars.csv too. Then you wouldn't need to hard-code a copy of it as REEDS_DOLLAR_YEAR in defaults.py.
Comment on lines
+15
to
+16
| sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) | ||
| from bokehpivot.defaults import REEDS_DOLLAR_YEAR |
Contributor
There was a problem hiding this comment.
Suggested change
| sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) | |
| from bokehpivot.defaults import REEDS_DOLLAR_YEAR | |
| from pathlib import Path | |
| sys.path.append(str(Path(__file__).parent.parent.parent)) | |
| import reeds | |
| dollar_year = int(reeds.io.get_scalars().dollar_year) |
- similar suggestions below. (Most of the functions like
reeds.io.get_scalars(), if called without an argument, return the default from the repo.)
| from reeds import reedsplots | ||
| from reeds.report_utils import SLIDE_HEIGHT, SLIDE_WIDTH | ||
| from bokehpivot.defaults import DEFAULT_DOLLAR_YEAR, DEFAULT_PV_YEAR, DEFAULT_DISCOUNT_RATE | ||
| from bokehpivot.defaults import DEFAULT_DOLLAR_YEAR,REEDS_DOLLAR_YEAR, DEFAULT_PV_YEAR, DEFAULT_DISCOUNT_RATE |
Contributor
There was a problem hiding this comment.
Same here; reeds is already imported, so you can just get it from reeds.io.get_scalars().dollar_year
Comment on lines
+178
to
+179
| if write_out: | ||
| deflator.to_csv(os.path.join(inputs_case, 'deflator.csv'), index=False) |
Contributor
There was a problem hiding this comment.
For a function named get_...(), I think its clearest for it to just return the value and not write files. This line can be moved after the function call below.
|
|
||
| source_deflator_map = get_source_deflator_map(reeds_path) | ||
| source_deflator_map = get_source_deflator_map(reeds_path,inputs_case) | ||
|
|
Contributor
There was a problem hiding this comment.
You can move the write here:
Suggested change
| source_deflator_map.to_csv(os.path.join(inputs_case, 'deflator.csv'), index=False) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR resolves #46 by auto-generating deflator values directly within
copy_files.pyfrominputs/inflation_default.csv, eliminating the need to manually maintain a separateinputs/deflators.csvfile. Previously,deflators.csvhad to be updated by hand each year whenever inflation values were revised, creating a risk that users would update one file but forget the other. Now, deflator values are computed programmatically at runtime.It also automates model dollar year definition on model and postprocessing.
Technical details
Implementation notes
The core change is in
copy_files.py, which now readsinputs/inflation_default.csvand derives the full deflator time series from it automatically. The previously hand-maintainedinputs/deflators.csvfile has been removed from the repository entirely. In addition, the current financial year, most recent year that we have inflation value, is determined as a scalar which makes the reference year explicit and easier to update in a single place.The
dollar_yearswitch has been relocated from cases toscalers.csv, reflecting that it characterizes the model itself rather than individual case runs. On the modeling side, all dollar year variables are now defined automatically from the scalardollar_year.On the postprocessing side, two distinct dollar year variables are introduced to clarify their separate roles:
DEFAULT_DOLLAR_YEAR— the output dollar year used when presenting resultsReEDS_DOLLAR_YEAR— the modeled dollar year as defined in the modelBoth are centralized in
\postprocessing\bokehpivot\defaults.py. Going forward, if the modeled dollar year needs to change, developers only need to updatescalers.csvand\postprocessing\bokehpivot\defaults.py.Additional changes
inputs/deflators.csvremoved from the repository.Switches added/removed/changed
The
dollar_yearswitch has been relocated from cases toscalers.csv.Issues resolved
Closes #46
Known incompatibilities
Relevant sources or documentation
Validation, testing, and comparison report(s)
The auto-generated deflator values differ from the original hand-maintained values by less than 0.2% across all years (2004–2024), as shown in the table below. These small differences are rounding errors which causes very small difference in model comparisons under Pacific and Nation cases .
Checklist for author
Details to double-check
hourlize/resource.pywas rerun to regenerate the existing/prescribed VRE capacity dataGeneral information to guide review
Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how
Yes, It used in preparing text and coding.
Tag points of contact here if you would like additional review of the relevant parts of the model