Skip to content

Create fake hourly profiles for github tests; fix logs#26

Merged
patrickbrown4 merged 14 commits into
mainfrom
pb/fake
Apr 22, 2026
Merged

Create fake hourly profiles for github tests; fix logs#26
patrickbrown4 merged 14 commits into
mainfrom
pb/fake

Conversation

@patrickbrown4
Copy link
Copy Markdown
Contributor

@patrickbrown4 patrickbrown4 commented Apr 17, 2026

Summary

This PR addresses #22 and #19:

  • A function is added to recf.py to create fake CF profiles with the new GSw_FakeData switch is active, allowing us to avoid ~12 GB of data download and caching for the runner tests
  • The logs had inadvertently been reformatted by bokeh; now they're back to normal

Technical details

Implementation notes

Fake profiles

  • A new make_fake_profiles() function is added to recf.py. To make the profiles somewhat realistic in terms of min/mean/max (but not temporal shape), we randomly sample from the two-state [0,1] array, where probability(1) = CF [fraction].
    • These profiles should NOT be used for any kind of analysis; they are just for testing when you want to avoid downloading the actual data via reeds/remote.py.
    • Some example histograms are below in the validation section.
  • In the current implementation, we only use it for the site-level upv, wind-ons, and wind-ofs profiles.
    • Those are by far the biggest files we download from Zenodo, so avoiding their download speeds up the test and reduces the peak storage needs of the runner
    • It's probably good to keep automatically testing the other Zenodo downloads anyway
      • Though we might want to extend the new functionality to distPV, since that file is still kind of large at ~440 MB
  • In runbatch.py, we check to make sure all the cases in a batch use the same GSw_FakeData setting (since we download the files shared by all runs)

Logs

The verbose logging came from this line: https://github.nrel.gov/ReEDS/ReEDS-2.0/pull/1927/files#diff-a48c953794de025c1a499dbd77153a1da54ebb6bcd68499d4509b32bc4b8477fR21. Because a bokehpivot import was added to reeds.output_calc, and because bokeh uses its own debug-level logging, all the python scripts started using debug-level logging. This PR reverts that change, so now the logs are back to normal.

Additional changes

  • Fixed the git metadata at the top of meta.csv (it wasn't working since we don't have any tags on this repo yet)

Switches added/removed/changed

  • GSw_FakeData (default 0): Use fake capacity factor profiles (for TESTING ONLY when you do not want to download the real CF data)

Issues resolved

Validation, testing, and comparison report(s)

Zero change for the Pacific test case: results-v20260416_mainM0_Pacific,v20260416_fakeM0_Pacific.pptx

The github_Pacific case finished for me locally.

Here are histograms of the CF for a couple random samples:

image image

Checklist for author

Details to double-check

  • Included comparison reports for appropriate test cases
  • Documentation updated if necessary
  • Code formatting standardized
  • Reusable functions used where possible instead of copy/pasted code

General information to guide review

  • Zero impact on results of default case
  • No large data file(s) added/modified
  • No substantive impact on runtime for full-US reference case
  • No substantive impact on folder size for full-US reference case
  • No change to process flow (runbatch.py, d_solve_iterate.py)
  • No change to code organization
  • No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

No

Copy link
Copy Markdown

@ahamilton5 ahamilton5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verbose logs fix works great. Thanks, Patrick.

Copy link
Copy Markdown
Contributor

@kodiobika kodiobika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this! LGTM

Comment thread docs/source/faq.md Outdated
patrickbrown4 and others added 3 commits April 21, 2026 09:09
Co-authored-by: kodiobika <35176195+kodiobika@users.noreply.github.com>
@patrickbrown4 patrickbrown4 merged commit 975f998 into main Apr 22, 2026
10 checks passed
@patrickbrown4 patrickbrown4 deleted the pb/fake branch April 22, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants