Speed up scheduling problem preparation for flex-models with many devices - #2516
Draft
Ahmad-Wahid wants to merge 5 commits into
Draft
Speed up scheduling problem preparation for flex-models with many devices#2516Ahmad-Wahid wants to merge 5 commits into
Ahmad-Wahid wants to merge 5 commits into
Conversation
The scheduling problem carries commitments as DataFrames whose name, class, commodity and deviation prices are the same value repeated down every row. Building the model read those back one `df[column].iloc[0]` at a time: pandas constructs a Series and positionally indexes it for each, costing a few microseconds where a dict lookup costs a tenth of one. That is cheap until the flex-model has a device per charging session. A site with 20 charge points of 2 EVSEs each produces roughly 7,400 sub-commitments per schedule, because a commitment is split into one sub-commitment per group and each time step is usually its own group. The reads then run into the tens of thousands per solve. The constants are now gathered in convert_commitments_to_subcommitments, which already has the parent frame in hand, and carried on SchedulingProblem as `commitment_scalars`. Gathering them there rather than from the split result is the point: the parent is read once, not once per time step, and the group prices come from one vectorized `groupby.first()` instead of a lookup per group. Measured on a week of EV charging (20 charge points of 2 EVSEs each, PV, building and battery), profiled, one simulated day: _validate_commitments_are_enforceable 1.79s -> 0.02s _price_of (59,176 calls) 2.64s -> 0.01s aggregate_commodity_costs 0.98s -> 0.00s convert_commitments_to_subcommitments 2.10s -> 2.39s (now builds them) prepare_scheduling_problem 11.11s -> 9.77s pandas self-time 26.73s -> 20.25s Unprofiled, that is 24.6s -> 20.8s for a simulated day and 172s -> 138s for the week, with the resulting schedules unchanged. _identify_commitment and aggregate_commodity_costs now take the gathered constants. aggregate_commodity_costs still accepts frames, which is what the Pyomo backend passes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rames
Two more places read the sub-commitment frames back a lookup at a time.
Splitting a commitment produces one sub-commitment per group, and a group is
usually a single time step, so both ran thousands of times per schedule on
frames holding a single row.
The device grouping (stock key, device column, device_group column) is now
sliced from the parent while splitting, using the group positions the
groupby already knows, and carried alongside the other constants. The loop
that builds device_group_lookup reads those instead of the frames.
The convexity check summed the deviation prices over the sub-commitments,
which meant concatenating thousands of one-row frames. It sums over the
commitments as passed in instead. Splitting partitions a commitment's rows
between groups, and a group that yields both a downwards and an upwards
sub-commitment contributes each price exactly once, just as the row it came
from does, so the sums are identical -- verified by computing both and
asserting equality across the planning tests and a full week of EV charging
before removing the check.
Profiled, one simulated day of the same benchmark:
pd.concat (convexity check) 3.78s -> 0.10s
DataFrame.__getitem__ 1.41s/61,443 -> 0.16s/811
convert_commitments_to_subcommitments
2.39s -> 2.79s (now gathers both)
pandas self-time 20.25s -> 17.80s
profiled total 39.0s -> 36.8s
Unprofiled: 20.8s -> 19.7s for a simulated day, 138s -> 130s for the week,
schedules unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…emand
Splitting a commitment produces one sub-commitment per group, and a group is
usually a single time step, so a schedule with a device per charging session
holds thousands of one-row frames. The HiGHS backend then read each one back
for its quantity and time-step columns.
Those are now sliced from the commitment while it is split, alongside the
constants gathered there already, including the row bounds -- which follow
from whether the sub-commitment kept the upwards or the downwards price
column, and so are known without inspecting the result. With the commodity
also coming from the gathered constants, the HiGHS backend no longer looks
at the frames at all.
That leaves them needed only by the Pyomo backend, so they are no longer
built up front: SubCommitmentFrames records the frame each came from, its
group's row positions and the price column its half drops, and materialises
one the first time it is asked for. Verified by building both ways and
comparing frame by frame with assert_frame_equal, across the planning tests
and a full week of EV charging, before removing the check.
Profiled, one simulated day of the same benchmark:
_column 2.90s/31,451 -> 0.14s/1,863
groupby get_iterator 1.57s -> not reached
convert_commitments_to_subcommitments
3.29s -> 1.87s
prepare_scheduling_problem 11.11s -> 3.73s
pandas self-time 26.73s -> 13.56s
profiled total 46.9s -> 31.7s
Unprofiled: 19.7s -> 16.5s for a simulated day and 130s -> 104s for the
week, schedules unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Documentation build overview
3 files changed± genindex.html± _autosummary/flexmeasures.data.models.planning.scheduling_problem.html± api/v3_0.html |
Ahmad-Wahid
had a problem deploying
to
dockerhub
September 10, 2026 19:29 — with 
GitHub Actions
Error
GitHub Actions
Error
4.3.0 stops building ORM objects that the bulk-save path never reads, and writes large belief batches with COPY rather than a multi-row INSERT, so storing beliefs gets substantially cheaper. The Docker image installs from uv.lock with "uv sync --frozen", so the lock is what decides the version; the floor in pyproject.toml is raised to match, so a later re-resolve cannot quietly drop back to 4.2.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Building the scheduling problem gets slow when a flex-model has many devices — for
example one device per EV charging session. This makes it faster. Schedules are
unchanged.
Why it was slow
A commitment is a pandas DataFrame. Several of its columns hold the same value in
every row: the name, the class, the commodity, the deviation prices.
When we split a commitment into sub-commitments, we make one per group — and a group
is usually a single time step. So a site with 20 charge points of 2 EVSEs each ends up
with about 7,400 sub-commitments per schedule, each a DataFrame with a single row.
Building the model then read those values back one at a time, with
df[column].iloc[0]. Every such read makes pandas build a Series and index into it:a few microseconds, where a plain dict lookup would be a tenth of one. Tens of
thousands of times per solve, that became most of the preparation time.
What we do instead
The same idea three times: read each thing once, from the commitment we already
have in hand, instead of reading it back off thousands of small frames.
SchedulingProblemascommitment_scalars.as they were passed in, instead of concatenating every sub-commitment — splitting
partitions a commitment's rows, so the sums come out the same.
during the split too. The HiGHS backend then never looks at the sub-commitment
frames at all, so they are only built when something asks for one. In practice that
means only Pyomo builds them.
Results
Measured on a week of EV charging (20 charge points of 2 EVSEs each, plus a PV, a
building and a battery). Profiled, one simulated day:
prepare_scheduling_problem_columnpd.concatfor the convexity checkEnd to end, without the profiler: a simulated day went from 19.7 s to 16.5 s, and
the week from 130 s to 104 s.
Small problems are unaffected — there is nothing to save when there are only a handful
of commitments.
How we know schedules did not change
Each step was checked by computing the result both the old way and the new way and
asserting they matched — across the planning tests and a full week of real EV
charging — before removing the check. For the lazily built frames that comparison was
assert_frame_equal, frame by frame.Tests on this branch:
flexmeasures/data/models/planning— 359 passed, 3 xfailedflexmeasures/data— 1,078 passed (one pre-existing error intest_automation_scheduling_fresh_db, which fails the same way onmain)Notes for review
mainwas merged into this branch. The testresults are from after the merge.
aggregate_commodity_costsstill accepts commitment frames, which is what the Pyomobackend passes it.
SubCommitmentFramesis a small read-only sequence: it remembers which frame eachsub-commitment came from and which rows are its own, and builds the frame on first
access.
🤖 Generated with Claude Code