diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index 58681e6c43..a5716176fe 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -16,6 +16,8 @@ v3.0-32 | July XX, 2026 * Legacy ``schedule`` field (in scheduling endpoints) and ``forecast`` field (in forecasting endpoints) remain in responses, unchanged, for backward compatibility. New clients should prefer ``job``; see :ref:`api_background_jobs` for the full response format. These fields are not (yet) formally deprecated — see the "Planned API v4" discussion linked from :ref:`api_deprecation` for where and when their removal is being tracked. - ``GET /api/v3_0/jobs/`` now returns ``202 Accepted`` while a job is queued or running, ``422 Unprocessable Entity`` for failed jobs, and ``200 OK`` for finished jobs. See :ref:`api_background_jobs` for the response format and polling flow. - ``GET /api/v3_0/jobs/`` now also returns kebab-case metadata fields such as ``func-name`` and ``enqueued-at``, alongside the existing snake_case fields (``func_name``, ``enqueued_at``, etc.), which remain unchanged for backward compatibility. New clients should prefer the kebab-case fields. +- Breaking: ``relax-soc-constraints`` now defaults to ``True`` (set it or ``relax-constraints`` to ``False`` to keep ``soc-minima``/``soc-maxima`` hard), and the built-in storage fallback scheduler has been retired. ``GET /api/v3_0/sensors//schedules/`` now returns ``400`` with the failure reason for an infeasible storage schedule instead of a ``303`` redirect to (or an automatic follow of) a fallback schedule. ``FLEXMEASURES_FALLBACK_REDIRECT`` remains relevant only for custom schedulers that still define a fallback scheduler. +- ``soc-min`` and ``soc-max`` in the storage flex-model are now canonical: besides a fixed quantity, they also accept a sensor reference or time series, making them dynamic storage SoC boundaries. Sensor references (on any field that accepts them) may now include a ``default`` fallback quantity to fill time slots where the referenced sensor has no value. ``soc-minima`` and ``soc-maxima`` remain supported as deprecated legacy aliases, and a fixed ``soc-min``/``soc-max`` now follows the default SoC relaxation behavior instead of always being a hard constraint. - Added a ``group`` field to the storage flex-model, accepted by the `/assets/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_ (POST) endpoint, referencing a power sensor representing a group of devices (e.g. a shared inverter or feeder). The group's ``power-capacity`` is enforced as a hard constraint on the group's aggregate power, while its ``consumption-capacity``/``production-capacity`` are enforced as soft constraints with default breach prices; the group's scheduled aggregate power is saved to the group sensor. - The ``group`` field also accepts a ``{"asset": }`` reference (besides ``{"sensor": }``), pointing at an asset whose own (DB-stored) flex-model defines the group's constraints. Such a group defines no power sensor of its own; its aggregate schedule is instead saved via its ``consumption``/``production`` output sensor references, following the same conventions as any other asset-only flex-model entry. This lets the entire flex-model for a device tree (including groups) live in the DB, with ``flex-model`` omitted or empty on the trigger request. - Fixed: `/assets/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_ (POST) now also accepts the legacy ``force_new_job_creation`` field name (in addition to ``force-new-job-creation``), matching the sensor-level trigger endpoint. Previously, only the sensor-level endpoint accepted both spellings. diff --git a/documentation/api/introduction.rst b/documentation/api/introduction.rst index 760d902ee8..30350dd7e6 100644 --- a/documentation/api/introduction.rst +++ b/documentation/api/introduction.rst @@ -116,18 +116,19 @@ See Other (303) --------------- Some API responses return ``HTTP status 303 (See Other)`` to redirect the client to a different resource. -This happens, for example, when a scheduling job fails and a fallback schedule has been computed instead. +This can happen when a custom scheduler defines a fallback scheduler, the original scheduling job fails, and the fallback schedule has been computed instead. In that case, the response includes a ``Location`` header pointing to the fallback schedule endpoint, so clients can automatically retrieve the fallback result. The response body will contain a JSON message with a ``status`` field set to ``"UNKNOWN_SCHEDULE"`` and a ``message`` field explaining the reason for the redirect. .. note:: - The fallback schedule mechanism activates when the main scheduler encounters an infeasible problem (i.e. when constraints cannot be satisfied). - This is less likely to happen when ``"relax-constraints": true`` is set in the ``flex-context``, as constraint relaxation softens most infeasibility-causing constraints. - The hard constraints that remain even after constraint relaxation are ``soc-min``, ``soc-max``, ``soc-targets`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``. + FlexMeasures' built-in storage scheduler no longer computes a fallback schedule for infeasible problems. + Instead, ``soc-min`` and ``soc-max`` boundaries, including the legacy aliases ``soc-minima`` and ``soc-maxima``, are relaxed by default through ``"relax-soc-constraints": true`` (setting either it or ``"relax-constraints"`` to ``false`` keeps them hard). + The hard constraints that remain even after constraint relaxation are ``soc-targets`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``. + If hard constraints cannot be satisfied, the scheduling job fails and clients receive the failure reason when requesting the schedule. - Server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default). + For custom schedulers that still define a fallback scheduler, server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default). Here is a client-side code example in Python for handling 303 redirects (this merely follows the redirect and should be revised to make use of the client's monitoring tools): diff --git a/documentation/changelog.rst b/documentation/changelog.rst index a734982fe4..8af6607776 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -40,6 +40,8 @@ New features * Commodity contexts that omit grid-connection fields (prices and site capacities) now get smart defaults instead of failing or silently leaving the grid unconstrained — for instance, a bare ``{"commodity": "gas"}`` is treated as having no grid connection; see :ref:`commodity_context_defaults` for the full rules [see `PR #2272 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] * Improve chart axis domain for event values not around zero, with a per-sub-chart ``y-axis`` option in ``sensors_to_show`` (default ``zero``, which pads the axis out to include zero) that can be set to ``data`` to fit a sub-chart's y-axis to the values shown, to an explicit ``[min, max]`` domain that the axis will cover at least (expanding to fit data beyond it), or to a strict ``{"min": min, "max": max}`` domain that the axis will never exceed (clamping data beyond it, with a warning when that happens), editable from the graph editor [see `PR #2244 `_] +* Breaking behaviour change: storage SoC constraints (``soc-minima``/``soc-maxima``) are now relaxed by default (``relax-soc-constraints`` defaults to ``True``; set it or ``relax-constraints`` to ``False`` to keep them hard), and the built-in storage fallback scheduler has been retired, so infeasible storage problems now fail with their failure reason instead of silently saving a fallback schedule; ``FLEXMEASURES_FALLBACK_REDIRECT`` is now only relevant for custom schedulers that define a fallback scheduler [see `PR #2252 `_] +* Support defining ``default`` fallbacks on sensor references, and support dynamic storage ``soc-min`` and ``soc-max`` boundaries with sensor references or time series; ``soc-minima`` and ``soc-maxima`` remain supported as legacy aliases, and scalar ``soc-min`` / ``soc-max`` follow the default SoC relaxation behavior [see `PR #2267 `_] * Extended ``GET /api/v3_0/jobs/`` with a ``result`` field containing ``unresolved`` and ``resolved`` soft state-of-charge constraint analysis (``soc-minima``/``soc-maxima`` violations or satisfied constraints, keyed by asset ID) for scheduling jobs; both arrays are empty when no SoC constraints were defined [see `PR #2072 `_] * New storage flex-model field ``operation-modes`` confines a device's power to one of several power bands, following the S2 standard's operation modes — for example, a device that is either off or running at one fixed power [see `PR #2278 `_] * New ``FLEXMEASURES_LP_SOLVER_OPTIONS`` config setting to pass solver options to the scheduling solver, validated against the installed HiGHS build so that unknown or unsupported options raise instead of being silently ignored [see `PR #2283 `_] diff --git a/documentation/concepts/commitments.rst b/documentation/concepts/commitments.rst index d185492e6d..7a72f6a379 100644 --- a/documentation/concepts/commitments.rst +++ b/documentation/concepts/commitments.rst @@ -138,7 +138,7 @@ Typical translations include: - tariffs (``consumption-price``, ``production-price``) → an ``"energy"`` FlowCommitment with zero baseline so net consumption/production is priced; - peak/excess limits (``site-peak-production``, ``site-peak-production-price``, etc.) → dedicated peak FlowCommitment(s); -- storage-related fields (``soc-minima``, ``soc-minima-breach-price``, etc.) → StockCommitment(s). +- storage-related fields (``soc-min``, ``soc-max``, legacy ``soc-minima`` and ``soc-maxima``, and the SoC breach prices) → StockCommitment(s). Let us look at some concrete examples. @@ -179,9 +179,9 @@ commitments the scheduler constructs. 6. **SOC minima / maxima (storage preferences)** - - *Fields used*: ``soc-minima``, ``soc-minima-breach-price``, ``soc-maxima`` and ``soc-maxima-breach-price``. + - *Fields used*: ``soc-min``, ``soc-minima``, ``soc-minima-breach-price``, ``soc-max``, ``soc-maxima`` and ``soc-maxima-breach-price``. - *Commitment*: StockCommitment(s) that price deviations below minima or - above maxima. Hard storage capacities are set through ``soc-min`` and ``soc-max`` instead and are modelled as Pyomo constraints. + above maxima. Set ``relax-soc-constraints`` to ``False`` to keep these SoC bounds as hard Pyomo constraints instead. 7. **Power bands per device** diff --git a/documentation/configuration.rst b/documentation/configuration.rst index 79afcdce86..de1b6156b1 100644 --- a/documentation/configuration.rst +++ b/documentation/configuration.rst @@ -994,7 +994,9 @@ Default: ``None`` (defaults are set internally for each sunset API version, e.g. FLEXMEASURES_FALLBACK_REDIRECT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Control how the API handles a failed scheduling job when a fallback schedule has been computed. +Control how the API handles a failed scheduling job when a custom scheduler has computed a fallback schedule. + +FlexMeasures' built-in storage scheduler no longer computes fallback schedules, but custom schedulers may still define fallback schedulers. If ``True``, the API returns ``HTTP status 303 (See Other)`` with a ``Location`` header pointing to the fallback schedule endpoint. Clients must follow this redirect themselves to obtain the fallback schedule (see :ref:`api_see_other`). diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index a653211690..bb81cd3b25 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -1,6 +1,6 @@ .. _scheduling: -Scheduling +Scheduling =========== Scheduling is the main value-drive of FlexMeasures. We have two major types of schedulers built-in, for storage devices (usually batteries or hot water storage) and processes (usually in industry). @@ -387,19 +387,20 @@ If you model devices that *buffer* energy (e.g. thermal energy storage systems c However, here are some tips to model a buffer correctly: - Describe the thermal energy content in kWh or MWh. - - Set ``soc-minima`` to the accumulative usage forecast. + - Set dynamic ``soc-min`` values to the accumulative usage forecast. - Set ``charging-efficiency`` to the sensor describing the :abbr:`COP (coefficient of performance)` values. - Set ``storage-efficiency`` to a value below 100% to model (heat) loss. For a hands-on example of a heat buffer fed by multiple devices, see :ref:`tut_multi_feed_storage`. -What happens if the flex model describes an infeasible problem for the storage scheduler? Excellent question! -It is highly important for a robust operation that these situations still lead to a somewhat good outcome. -From our practical experience, we derived a ``StorageFallbackScheduler``. -It simplifies an infeasible situation by just starting to charge, discharge, or do neither, -depending on the first target state of charge and the capabilities of the asset. +If the flex model describes an infeasible problem for the storage scheduler, the failure should remain visible. +By default, ``soc-min`` and ``soc-max`` boundaries are relaxed into soft constraints, so the scheduler can still return a useful schedule when these boundaries cannot be fully met. +The legacy ``soc-minima`` and ``soc-maxima`` aliases follow the same behavior. +Setting either ``relax-soc-constraints`` or ``relax-constraints`` to ``false`` in the flex-context keeps them as hard constraints. +Exact ``soc-targets``, ``power-capacity`` (in the flex-model), and ``site-power-capacity`` (in the flex-context) remain hard constraints. +If those hard constraints make the problem infeasible, the scheduling job fails instead of producing a fallback schedule. -Of course, we also log a failure in the scheduling job, so it's important to take note of these failures. Often, mis-configured flex models are the reason. +It is important to take note of these failures. Often, misconfigured flex models are the reason. For a hands-on tutorial on using some of the storage flex-model fields, head over to :ref:`tut_v2g` use case and `the API documentation for triggering schedules <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_. For further hands-on examples, see :ref:`tut_multi_feed_storage` (multiple devices feeding one shared storage) and :ref:`tut_multi_commodity` (devices on different commodities scheduled together). @@ -417,15 +418,15 @@ Some examples from practice (usually industry) could be: - A centrifuge's daily work of combing through sludge water. Depends on amount of sludge present. - Production processes with a target amount of output until the end of the current shift. The target usually comes out of production planning. -- Application of coating under hot temperature, with fixed number of times it needs to happen before some deadline. - +- Application of coating under hot temperature, with fixed number of times it needs to happen before some deadline. + .. list-table:: :header-rows: 1 :widths: 20 25 90 * - Field - Example value - - Description + - Description * - ``power`` - ``"15kW"`` - Nominal power of the load. @@ -436,7 +437,7 @@ Some examples from practice (usually industry) could be: - ``"MAX"`` - Objective of the scheduler, to maximize (``"MAX"``) or minimize (``"MIN"``). * - ``time_restrictions`` - - ``[{"start": "2015-01-02T08:00:00+01:00", "duration": "PT2H"}]`` + - ``[{"start": "2015-01-02T08:00:00+01:00", "duration": "PT2H"}]`` - Time periods in which the load cannot be scheduled to run. * - ``process_type`` - ``"INFLEXIBLE"``, ``"SHIFTABLE"`` or ``"BREAKABLE"`` @@ -686,5 +687,3 @@ Here are some thoughts on further innovation: This is ongoing architecture design work, and therefore happens in development settings, until we are happy with the outcomes. Thoughts welcome :) - Aggregating flexibility of a group of assets (e.g. a neighborhood) and optimizing its aggregated usage (e.g. for grid congestion support) is also an exciting direction for expansion. - - diff --git a/documentation/tut/flex-model-v2g.rst b/documentation/tut/flex-model-v2g.rst index 068fcc5e4b..ae528060c4 100644 --- a/documentation/tut/flex-model-v2g.rst +++ b/documentation/tut/flex-model-v2g.rst @@ -40,10 +40,11 @@ Constraining the cycling to occur within a static 25-85% SoC range can be modell A starting SoC below 15 kWh (25%) will lead to immediate charging to get within limits (as shown above). Likewise, a starting SoC above 51 kWh (85%) would lead to immediate discharging. -Setting a SoC target outside of the static range leads to an infeasible problem and will be rejected by the FlexMeasures API. +By default, these boundaries are soft constraints, so the scheduler may breach them when necessary and will penalize the breach. +To enforce them as hard limits, set ``"relax-soc-constraints": false`` in the flex-context. +An exact ``soc-targets`` value remains hard and can still make a schedule infeasible when it conflicts with hard limits. -The soc-min and soc-max settings are constant constraints. -To enable a temporary target SoC of more than 85% (for car reservations, see the next section), it is necessary to relax the ``soc-max`` field to 60 kWh (100%), and to instead use the ``soc-maxima`` field to convey the desired upper limit for regular cycling: +To enable a temporary target SoC of more than 85% (for car reservations, see the next section), this example keeps a global ``soc-max`` of 60 kWh and uses the legacy ``soc-maxima`` alias for the desired upper limit during regular cycling: .. code-block:: json @@ -61,7 +62,9 @@ To enable a temporary target SoC of more than 85% (for car reservations, see the } } -The maxima constraints should be relaxed—or withheld entirely—within some time window before any SoC target (as shown above). +The ``soc-maxima`` field is retained for backwards compatibility and is deprecated for new configurations. +When no separate global ``soc-max`` is needed, use a dynamic ``soc-max`` time series instead. +The dynamic maximum should be relaxed—or withheld entirely—within some time window before any SoC target (as shown above). This time window should be at least wide enough to allow the target to be reached in time, and can be made wider to allow the scheduler to take advantage of favourable market prices along the way. @@ -70,13 +73,13 @@ This time window should be at least wide enough to allow the target to be reache Car reservations ================ -Given a reservation for 8 AM on February 5th, constraint 2 can be modelled through the following (additional) ``soc-minima`` constraint: +Given a reservation for 8 AM on February 5th, constraint 2 can be modelled through the following dynamic ``soc-min`` constraint: .. code-block:: json { "flex-model": { - "soc-minima": [ + "soc-min": [ { "value": "57 kWh", "datetime": "2024-02-05T08:00:00+01:00" @@ -86,13 +89,13 @@ Given a reservation for 8 AM on February 5th, constraint 2 can be modelled throu } This constraint also signals that if the car is not plugged out of the Charge Point at 8 AM, the scheduler is in principle allowed to start discharging immediately afterwards. -To make sure the car remains at or above 95% SoC for some time, additional soc-minima constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM: +To make sure the car remains at or above 95% SoC for some time, additional dynamic ``soc-min`` constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM: .. code-block:: json { "flex-model": { - "soc-minima": [ + "soc-min": [ { "value": "57 kWh", "start": "2024-02-05T08:00:00+01:00", @@ -109,7 +112,7 @@ Alternatively, to keep the car from discharging altogether during that time, lim { "flex-model": { - "soc-minima": [ + "soc-min": [ { "value": "57 kWh", "datetime": "2024-02-05T08:00:00+01:00" @@ -125,7 +128,7 @@ Alternatively, to keep the car from discharging altogether during that time, lim } } -.. note:: In case the ``soc-minima`` field defines partially overlapping time periods, FlexMeasures automatically resolves this by selecting the maximum. Likewise, the minimum is selected for partially overlapping time periods in the ``soc-maxima``, ``power-capacity``, ``production-capacity`` and ``consumption-capacity`` flex-model fields, and also in the ``site-power-capacity``, ``site-production-capacity`` and ``site-consumption-capacity`` flex-context fields. +.. note:: In case the dynamic ``soc-min`` field defines partially overlapping time periods, FlexMeasures automatically resolves this by selecting the maximum. The legacy ``soc-minima`` and ``soc-maxima`` aliases remain supported. Likewise, the minimum is selected for partially overlapping time periods in the dynamic ``soc-max`` field, ``power-capacity``, ``production-capacity`` and ``consumption-capacity`` flex-model fields, and also in the ``site-power-capacity``, ``site-production-capacity`` and ``site-consumption-capacity`` flex-context fields. .. _earning_by_cycling: @@ -146,4 +149,5 @@ To provide an incentive for cycling the battery in response to market prices, th We hope this demonstration helped to illustrate the flex-model of the storage scheduler. Until now, optimizing a single storage device (like a battery) has been the sole focus of these tutorial series. -In :ref:`tut_multi_feed_storage`, we'll cover scheduling several devices that feed into one shared storage. \ No newline at end of file +In :ref:`tut_multi_feed_storage`, we'll cover scheduling several devices that feed into one shared storage. +After that, :ref:`tut_toy_schedule_process` turns to something different: the optimal timing of processes with fixed energy work and duration. diff --git a/documentation/tut/toy-example-from-scratch.rst b/documentation/tut/toy-example-from-scratch.rst index 2f77140060..060347cba0 100644 --- a/documentation/tut/toy-example-from-scratch.rst +++ b/documentation/tut/toy-example-from-scratch.rst @@ -64,6 +64,10 @@ There is more information being used by the scheduler, such as the battery's cap ], "soc-usage": [{"sensor": 73}] } + + The ``soc-maxima`` field in this combined fixed-plus-dynamic example is a + supported legacy alias. Prefer dynamic ``soc-max`` when no separate global + ``soc-max`` value is needed. $ flexmeasures add schedule \ --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py index 0668cde606..a6ffdc2f68 100644 --- a/flexmeasures/api/v3_0/assets.py +++ b/flexmeasures/api/v3_0/assets.py @@ -1676,7 +1676,7 @@ def trigger_schedule( power-capacity: 25 kW consumption-capacity: {sensor: 42} production-capacity: 30 kW - soc-minima: + soc-min: - {start: "2015-06-02T12:00:00+00:00", end: "2015-06-02T13:00:00+00:00", value: 10 kWh} - sensor: 932 consumption-capacity: 0 kW diff --git a/flexmeasures/api/v3_0/sensors.py b/flexmeasures/api/v3_0/sensors.py index a63f1e52fd..2c02378d98 100644 --- a/flexmeasures/api/v3_0/sensors.py +++ b/flexmeasures/api/v3_0/sensors.py @@ -964,9 +964,9 @@ def trigger_schedule( soc-targets: - value: "25 kWh" datetime: "2015-06-02T16:00:00+00:00" - soc-minima: + soc-min: sensor: 300 - soc-min: "10 kWh" + default: "10 kWh" soc-max: "25 kWh" charging-efficiency: "120%" discharging-efficiency: diff --git a/flexmeasures/api/v3_0/tests/test_jobs_api.py b/flexmeasures/api/v3_0/tests/test_jobs_api.py index ca8fea6cf8..8852baefde 100644 --- a/flexmeasures/api/v3_0/tests/test_jobs_api.py +++ b/flexmeasures/api/v3_0/tests/test_jobs_api.py @@ -398,6 +398,8 @@ def test_get_job_status_failed_infeasible_schedule_includes_exc_info( ): charging_station = add_charging_station_assets["Test charging station"].sensors[0] message = message_for_trigger_schedule(with_targets=True, realistic_targets=False) + message["flex-model"]["soc-targets"][0]["value"] = 250000 + message["flex-context"] = {"relax-soc-constraints": False} with app.test_client() as client: trigger_response = client.post( @@ -418,7 +420,7 @@ def test_get_job_status_failed_infeasible_schedule_includes_exc_info( assert response.status_code == 422 data = response.json assert data["status"] == "FAILED" - assert "infeasible problem" in data["message"].lower() + assert "infeasible" in data["message"].lower() assert ( "ValueError: The input data yields an infeasible problem." in data["exc-info"] ) diff --git a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py index 0eb7d95b6f..a2b37566d4 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py @@ -387,7 +387,8 @@ def test_get_schedule_unfinished_job_returns_202_when_sunset_active( @pytest.mark.parametrize( "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True ) -def test_get_schedule_fallback( +@pytest.mark.parametrize("fallback_redirect", [True, False]) +def test_get_schedule_infeasible_storage_job_without_fallback( app, add_battery_assets, add_market_prices, @@ -396,13 +397,15 @@ def test_get_schedule_fallback( keep_scheduling_queue_empty, requesting_user, db, + fallback_redirect, + monkeypatch, ): """ - Test if the fallback job is created after a failing StorageScheduler call. This test - is based on flexmeasures/data/models/planning/tests/test_solver.py + Test that a failing StorageScheduler call reports the failure without creating a fallback job. + + This test is based on flexmeasures/data/models/planning/tests/test_solver.py. """ - assert app.config["FLEXMEASURES_FALLBACK_REDIRECT"] is False - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = True + monkeypatch.setitem(app.config, "FLEXMEASURES_FALLBACK_REDIRECT", fallback_redirect) target_soc = 9 charging_station_name = "Test charging station" @@ -422,17 +425,12 @@ def test_get_schedule_fallback( assert capacity == 2 assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id} - # check that no Fallback schedule has been saved before - models = [ - source.model for source in charging_station.search_beliefs().sources.unique() - ] - assert "StorageFallbackScheduler" not in models - # create a scenario that yields an infeasible problem (unreachable target SOC at 2am) message = { "start": start, "duration": "PT24H", "resolution": "PT15M", # just schedule in the original sensor resolution + "force-new-job-creation": True, "flex-model": { "soc-at-start": 10, "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0), @@ -486,196 +484,21 @@ def test_get_schedule_fallback( # Make sure the resolution shows up in the job kwargs assert job.kwargs.get("resolution") == pd.Timedelta(message["resolution"]) - # the callback creates the fallback job which is still pending - assert len(app.queues["scheduling"]) == 1 - fallback_job_id = Job.fetch( - job_id, connection=app.queues["scheduling"].connection - ).meta.get("fallback_job_id") - - # check that the fallback_job_id is stored on the metadata of the original job - assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id - assert fallback_job_id != job_id + # no storage fallback job is created + assert len(app.queues["scheduling"]) == 0 + assert job.meta.get("fallback_job_id") is None get_schedule_response = client.get( url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), ) - # requesting the original job redirects to the fallback job - assert ( - get_schedule_response.status_code == 303 - ) # Status code for redirect ("See other") - assert ( + assert get_schedule_response.status_code == 400 + assert "Scheduling job failed with InfeasibleProblemException: infeasible." in ( get_schedule_response.json["message"] - == "Scheduling job failed with InfeasibleProblemException: infeasible. StorageScheduler was used." ) + assert "StorageScheduler was used." in get_schedule_response.json["message"] assert get_schedule_response.json["status"] == "UNKNOWN_SCHEDULE" assert get_schedule_response.json["result"] == "Rejected" - # check that the redirection location points to the fallback job - assert ( - get_schedule_response.headers["location"] - == f"http://localhost/api/v3_0/sensors/{charging_station.id}/schedules/{fallback_job_id}" - ) - - # run the fallback job - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - # check that the queue is empty - assert len(app.queues["scheduling"]) == 0 - - # get the fallback schedule - fallback_schedule = client.get( - get_schedule_response.headers["location"], - json={"duration": "PT24H"}, - ).json - - # check that the fallback schedule has the right status and start dates - assert fallback_schedule["status"] == "PROCESSED" - assert parse_datetime(fallback_schedule["start"]) == parse_datetime(start) - - models = [ - source.model - for source in charging_station.search_beliefs().sources.unique() - ] - assert "StorageFallbackScheduler" in models - - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False - - -@pytest.mark.parametrize( - "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True -) -def test_get_schedule_fallback_not_redirect( - app, - add_battery_assets, - add_market_prices, - battery_soc_sensor, - add_charging_station_assets, - keep_scheduling_queue_empty, - requesting_user, - db, -): - """ - Test if the fallback scheduler is returned directly after a failing StorageScheduler call. This test - is based on flexmeasures/data/models/planning/tests/test_solver.py - """ - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False - - target_soc = 9 - charging_station_name = "Test charging station" - - start = "2015-01-02T00:00:00+01:00" - epex_da = get_test_sensor(db) - charging_station = get_sensor_by_name( - add_charging_station_assets[charging_station_name], "power" - ) - - capacity = charging_station.get_attribute( - "capacity_in_mw", - ur.Quantity(charging_station.get_attribute("site-power-capacity")) - .to("MW") - .magnitude, - ) - assert capacity == 2 - assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id} - - # create a scenario that yields an infeasible problem (unreachable target SOC at 2am) - message = { - "start": start, - "duration": "PT24H", - "flex-model": { - "soc-at-start": 10, - "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0), - "soc-max": charging_station.get_attribute("max-soc-in-mwh", target_soc), - "roundtrip-efficiency": charging_station.get_attribute( - "roundtrip-efficiency", 1 - ), - "storage-efficiency": charging_station.get_attribute( - "storage-efficiency", 1 - ), - "soc-targets": [ - { - "value": target_soc, - "start": "2015-01-02T02:00:00+01:00", - "duration": "PT0H", - } - ], - }, - } - - with app.test_client() as client: - # trigger storage scheduler - trigger_schedule_response = client.post( - url_for("SensorAPI:trigger_schedule", id=charging_station.id), - json=message, - ) - - # check that the call is successful - assert trigger_schedule_response.status_code == 202 - job_id = trigger_schedule_response.json["schedule"] - - # look for scheduling jobs in queue - assert ( - len(app.queues["scheduling"]) == 1 - ) # only 1 schedule should be made for 1 asset - job = app.queues["scheduling"].jobs[0] - assert job.kwargs["asset_or_sensor"]["id"] == charging_station.id - assert job.kwargs["start"] == parse_datetime(message["start"]) - assert job.id == job_id - - # process only the job that runs the storage scheduler (max_jobs=1) - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - # check that the job is failing - job = Job.fetch(job_id, connection=app.queues["scheduling"].connection) - assert job.is_failed - - # Make sure that the db flex_context shows up in the job kwargs - assert "flex-context" not in message and job.kwargs.get("flex_context") - - # the callback creates the fallback job which is still pending - assert len(app.queues["scheduling"]) == 1 - - fallback_job_id = Job.fetch( - job_id, connection=app.queues["scheduling"].connection - ).meta.get("fallback_job_id") - - # check that the fallback_job_id is stored on the metadata of the original job - assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id - assert fallback_job_id != job_id - - get_schedule_response = client.get( - url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), - ) - - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - get_schedule_response = client.get( - url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), - ) - - assert get_schedule_response.status_code == 200 - - schedule = get_schedule_response.json - - # check that the fallback schedule has the right status and start dates - assert schedule["status"] == "PROCESSED" - assert parse_datetime(schedule["start"]) == parse_datetime(start) - assert schedule["scheduler_info"]["scheduler"] == "StorageFallbackScheduler" - - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False - @pytest.mark.parametrize( "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index c3ecc45bd2..a3adcda075 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -3,7 +3,6 @@ import re import copy from datetime import datetime, timedelta -from typing import Type import pandas as pd import numpy as np @@ -32,7 +31,6 @@ initialize_series, initialize_df, get_power_values, - fallback_charging_policy, get_continuous_series_sensor_or_quantity, ) from flexmeasures.data.models.planning.exceptions import InfeasibleProblemException @@ -1257,6 +1255,24 @@ def device_list_series( discharging_efficiency=discharging_efficiency[d], ) + # Fold a fixed soc-min into the relaxed minima path (after off-tick + # projection, which needs the scalar bound), so it is softened along + # with any dynamic minima instead of staying behind as a hard bound. + # When relaxation was auto-enabled purely for off-tick projection + # (the user explicitly opted out), the scalar bound stays hard. + if ( + self.flex_context.get("soc_minima_breach_price") is not None + and soc_at_start[d] is not None + and not getattr(self, "scope_soc_relaxation_to_off_tick_devices", False) + and self._soc_relaxation_applies_to(device_stock_key.get(d), sensor_d) + ): + soc_min[d], soc_minima[d] = self._relax_scalar_soc_minimum( + soc_min=soc_min[d], + soc_minima=soc_minima[d], + query_window=(start + resolution, end + resolution), + resolution=resolution, + beliefs_before=belief_time, + ) if ( self.flex_context.get("soc_minima_breach_price") is not None and soc_minima[d] is not None @@ -1324,6 +1340,24 @@ def device_list_series( # soc-minima will become a soft constraint (modelled as stock commitments), so remove hard constraint soc_minima[d] = None + # Fold a fixed soc-max into the relaxed maxima path (after off-tick + # projection, which needs the scalar bound), so it is softened along + # with any dynamic maxima instead of staying behind as a hard bound. + # When relaxation was auto-enabled purely for off-tick projection + # (the user explicitly opted out), the scalar bound stays hard. + if ( + self.flex_context.get("soc_maxima_breach_price") is not None + and soc_at_start[d] is not None + and not getattr(self, "scope_soc_relaxation_to_off_tick_devices", False) + and self._soc_relaxation_applies_to(device_stock_key.get(d), sensor_d) + ): + soc_max[d], soc_maxima[d] = self._relax_scalar_soc_maximum( + soc_max=soc_max[d], + soc_maxima=soc_maxima[d], + query_window=(start + resolution, end + resolution), + resolution=resolution, + beliefs_before=belief_time, + ) if ( self.flex_context.get("soc_maxima_breach_price") is not None and soc_maxima[d] is not None @@ -1776,6 +1810,62 @@ def convert_to_commitments( return commitments + @staticmethod + def _relax_scalar_soc_minimum( + soc_min: float | None, + soc_minima: ( + Sensor | SensorReference | list[dict] | ur.Quantity | pd.Series | None + ), + **timing_kwargs, + ) -> tuple[ + None, Sensor | SensorReference | list[dict] | ur.Quantity | pd.Series | None + ]: + """Move a fixed SoC minimum into the relaxed minima path. + + If legacy dynamic minima are also configured, the fixed minimum still tightens + them, but no longer stays behind as a hard constraint. + """ + if soc_min is None: + return None, soc_minima + if soc_minima is None: + return None, soc_min * ur.Quantity("MWh") + soc_minima = get_continuous_series_sensor_or_quantity( + variable_quantity=soc_minima, + unit="MWh", + as_instantaneous_events=True, + resolve_overlaps="max", + **timing_kwargs, + ) + return None, soc_minima.clip(lower=soc_min) + + @staticmethod + def _relax_scalar_soc_maximum( + soc_max: float | None, + soc_maxima: ( + Sensor | SensorReference | list[dict] | ur.Quantity | pd.Series | None + ), + **timing_kwargs, + ) -> tuple[ + None, Sensor | SensorReference | list[dict] | ur.Quantity | pd.Series | None + ]: + """Move a fixed SoC maximum into the relaxed maxima path. + + If legacy dynamic maxima are also configured, the fixed maximum still tightens + them, but no longer stays behind as a hard constraint. + """ + if soc_max is None: + return None, soc_maxima + if soc_maxima is None: + return None, soc_max * ur.Quantity("MWh") + soc_maxima = get_continuous_series_sensor_or_quantity( + variable_quantity=soc_maxima, + unit="MWh", + as_instantaneous_events=True, + resolve_overlaps="min", + **timing_kwargs, + ) + return None, soc_maxima.clip(upper=soc_max) + def persist_flex_model(self): """Store new soc info as GenericAsset attributes @@ -2138,9 +2228,11 @@ def _get_soc_capacity_for_percent_conversion( raise ValueError( "Cannot derive state of charge from a `state-of-charge` sensor with '%' unit without `soc-max`." ) - if isinstance(soc_max, (Sensor, SensorReference)): + # A dict or list can arrive from the raw DB flex-model (e.g. a stored + # sensor reference or time series spec), so guard against those as well. + if isinstance(soc_max, (Sensor, SensorReference, dict, list)): raise ValueError( - "Cannot derive state of charge from a `state-of-charge` sensor with '%' unit when `soc-max` is a sensor reference." + "Cannot derive state of charge from a `state-of-charge` sensor with '%' unit when `soc-max` is a sensor reference or time series." ) if isinstance(soc_max, (int, float)): return str(ur.Quantity(soc_max, soc_unit).to("MWh")) @@ -2609,83 +2701,10 @@ def _ensure_variable_quantity( return q -class StorageFallbackScheduler(MetaStorageScheduler): - __version__ = "3" - __author__ = "Seita" - - def compute(self, skip_validation: bool = False) -> SchedulerOutputType: - """Schedule a battery or Charge Point by just starting to charge, discharge, or do neither, - depending on the first target state of charge and the capabilities of the Charge Point. - For the resulting consumption schedule, consumption is defined as positive values. - - Note that this ignores any cause of the infeasibility. - - :param skip_validation: If True, skip validation of constraints specified in the data. - :returns: The computed schedule. - """ - - ( - sensors, - start, - end, - resolution, - soc_at_start, - device_constraints, - ems_constraints, - commitments, - ) = self._prepare(skip_validation=skip_validation) - - # Fallback policy if the problem was unsolvable - storage_schedule = { - sensor: fallback_charging_policy( - sensor, device_constraints[d], start, end, resolution - ) - for d, sensor in enumerate(sensors) - if sensor is not None - } - - # Convert each device schedule to the unit of the device's power sensor - storage_schedule = { - sensor: convert_units( - storage_schedule[sensor], - "MW", - sensor.unit, - event_resolution=sensor.event_resolution, - ) - for sensor in sensors - if sensor is not None - } - - # Round schedule - if self.round_to_decimals: - storage_schedule = { - sensor: storage_schedule[sensor].round(self.round_to_decimals) - for sensor in sensors - if sensor is not None - } - - if self.return_multiple: - # Iterate over the dict keys (not the sensors list, which may hold the - # same sensor for multiple devices), so no sensor is emitted twice. - return [ - { - "name": "storage_schedule", - "sensor": sensor, - "data": storage_schedule[sensor], - } - for sensor in storage_schedule.keys() - if sensor is not None - ] - else: - return storage_schedule[sensors[0]] - - class StorageScheduler(MetaStorageScheduler): __version__ = "8" __author__ = "Seita" - fallback_scheduler_class: Type[Scheduler] = StorageFallbackScheduler - @staticmethod def _build_soc_schedule( # noqa: C901 flex_model: list[dict], @@ -2713,9 +2732,9 @@ def _build_soc_schedule( # noqa: C901 For '%' sensors, the soc-max flex-model field is used as capacity. If soc-max is missing or zero for a '%' sensor, the schedule is skipped with a warning. - Note: soc-max is a QuantityField (not a VariableQuantityField), so it is always a float - after deserialization and cannot be a sensor reference. - The isinstance guard below is therefore a defensive check for forward-compatibility. + Note: dynamic soc-max values are routed to soc_maxima during deserialization, so + soc_max is still a fixed float after deserialization. The isinstance guard below + is therefore a defensive check for malformed data. :returns: Tuple of (soc_schedule keyed by SoC sensor in sensor unit, soc_schedule_mwh keyed by device index in MWh). @@ -2808,11 +2827,14 @@ def _build_soc_schedule( # noqa: C901 capacity = None if soc_unit == "%": soc_max = flex_model_d0.get("soc_max") - if isinstance(soc_max, (Sensor, SensorReference)): + # A dict or list can arrive from the raw DB flex-model (e.g. a + # stored sensor reference or time series spec), so guard against + # those as well. + if isinstance(soc_max, (Sensor, SensorReference, dict, list)): raise ValueError( f"Cannot convert state-of-charge schedule to '%' unit for sensor " - f"{state_of_charge_sensor.id}: soc-max as a sensor reference is " - "not supported for '%' unit conversion." + f"{state_of_charge_sensor.id}: soc-max as a sensor reference or " + "time series is not supported for '%' unit conversion." ) if not soc_max: raise ValueError( diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 57cf8840b3..b70736d32c 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -502,22 +502,18 @@ def test_charging_station_solver_day_2( (15, "Test charging station (bidirectional)"), ], ) -def test_fallback_to_unsolvable_problem( +def test_storage_scheduler_reports_unsolvable_problem_without_fallback( target_soc, charging_station_name, setup_planning_test_data, db ): """Starting with a state of charge 10 kWh, within 2 hours we should be able to reach any state of charge in the range [10, 14] kWh for a unidirectional station, or [6, 14] for a bidirectional station, given a charging capacity of 2 kW. - Here we test target states of charge outside that range, ones that we should be able - to get as close to as 1 kWh difference. - We want our scheduler to handle unsolvable problems like these with a sensible fallback policy. - - The StorageScheduler raises an Exception which triggers the creation of a new job to compute a fallback - schedule. + Here we test target states of charge outside that range. + The StorageScheduler should report this infeasible problem without hiding it behind + a fallback schedule. """ soc_at_start = 10 duration_until_target = timedelta(hours=2) - expected_gap = 1 epex_da = get_test_sensor(db) charging_station = setup_planning_test_data[charging_station_name].sensors[0] @@ -582,26 +578,8 @@ def test_fallback_to_unsolvable_problem( # calling the scheduler with an infeasible problem raises an Exception with pytest.raises(InfeasibleProblemException): - consumption_schedule = scheduler.compute(skip_validation=True) - - # check that the fallback scheduler provides a sensible fallback policy - fallback_scheduler = scheduler.fallback_scheduler_class(**kwargs) - fallback_scheduler.config_deserialized = True - consumption_schedule = fallback_scheduler.compute(skip_validation=True) - - soc_schedule = integrate_time_series( - consumption_schedule, soc_at_start, decimal_precision=6 - ) - - # Check if constraints were met - assert min(consumption_schedule.values) >= capacity * -1 - assert max(consumption_schedule.values) <= capacity - print(consumption_schedule.head(12)) - print(soc_schedule.head(12)) - assert ( - abs(abs(soc_schedule.loc[target_soc_datetime] - target_soc) - expected_gap) - < TOLERANCE - ) + scheduler.compute(skip_validation=True) + assert scheduler.fallback_scheduler_class is None @pytest.mark.parametrize( @@ -806,6 +784,7 @@ def compute_schedule(flex_model): end, resolution, flex_model=flex_model, + flex_context={"relax-soc-constraints": False}, ) schedule = scheduler.compute() @@ -1291,6 +1270,7 @@ def compute_schedule(flex_model): end, resolution, flex_model=flex_model, + flex_context={"relax-soc-constraints": False}, ) schedule = scheduler.compute() @@ -1372,6 +1352,7 @@ def test_numerical_errors(app_with_each_solver, setup_planning_test_data, db): ], "soc-unit": "MWh", }, + flex_context={"relax-soc-constraints": False}, ) ( @@ -2109,6 +2090,7 @@ def test_battery_stock_delta_sensor( end, resolution, flex_model=flex_model, + flex_context={"relax-soc-constraints": False}, ) if stock_delta_sensor == "delta fails": @@ -2466,6 +2448,7 @@ def compute_schedule(flex_model): "site-power-capacity": "100 MW", "production-price": {"sensor": epex_da.id}, "consumption-price": {"sensor": epex_da.id}, + "relax-soc-constraints": False, }, ) return scheduler.compute() @@ -2491,6 +2474,17 @@ def compute_schedule(flex_model): # because soc-maxima = soc-minima = soc-targets assert all(abs(soc[8:].values - expected_soc_schedule) < 1e-5) + # remove legacy soc-minima/soc-maxima and use dynamic canonical soc-min/soc-max + del flex_model["soc-minima"] + del flex_model["soc-maxima"] + flex_model["soc-min"] = {"sensor": soc_minima.id, "default": "0 MWh"} + flex_model["soc-max"] = {"sensor": soc_maxima.id, "default": "10 MWh"} + schedule = compute_schedule(flex_model) + + soc = check_constraints(power, schedule, soc_at_start) + + assert all(abs(soc[8:].values - expected_soc_schedule) < 1e-5) + @pytest.mark.parametrize("unit", [None, "MWh", "kWh"]) @pytest.mark.parametrize("soc_unit", ["kWh", "MWh"]) @@ -2576,7 +2570,7 @@ def test_battery_storage_different_units( @pytest.mark.parametrize( - "ts_field, ts_specs", + "ts_field, ts_specs, expected_charge, expected_discharge", [ # The battery only has time to charge up to 950 kWh halfway ( @@ -2588,6 +2582,8 @@ def test_battery_storage_different_units( "value": "850 kW", } ], + 0.85, + -0.85, ), # Same, but the event time is specified with a duration instead of an end time ( @@ -2599,6 +2595,8 @@ def test_battery_storage_different_units( "value": "850 kW", } ], + 0.85, + -0.85, ), # Can only charge up to 950 kWh halfway ( @@ -2609,6 +2607,20 @@ def test_battery_storage_different_units( "value": "950 kWh", } ], + 0.85, + -0.85, + ), + # Same dynamic maximum through the canonical soc-max field + ( + "soc-max", + [ + { + "datetime": "2015-01-02T16:00+01", + "value": "950 kWh", + } + ], + 0.85, + -0.85, ), # Must end up at a maximum of 200 kWh, for which it is cheapest to charge to 950 and then to discharge to 200 ( @@ -2620,6 +2632,47 @@ def test_battery_storage_different_units( "value": "200 kWh", } ], + 0.85, + -0.85, + ), + # Same dynamic maximum through the canonical soc-max field + ( + "soc-max", + [ + { + "start": "2015-01-02T16:45+01", + "duration": "PT15M", + "value": "200 kWh", + } + ], + 0.85, + -0.85, + ), + # Must end up at a minimum of 200 kWh, so it is cheapest to fill completely and then discharge to 200 + ( + "soc-minima", + [ + { + "start": "2015-01-02T16:45+01", + "duration": "PT15M", + "value": "200 kWh", + } + ], + 0.9, + -0.8, + ), + # Same dynamic minimum through the canonical soc-min field + ( + "soc-min", + [ + { + "start": "2015-01-02T16:45+01", + "duration": "PT15M", + "value": "200 kWh", + } + ], + 0.9, + -0.8, ), ], ) @@ -2628,6 +2681,8 @@ def test_battery_storage_with_time_series_in_flex_model( db, ts_field, ts_specs, + expected_charge, + expected_discharge, ): """ Test scheduling a 1 MWh battery for 2h with a low -> high price transition with @@ -2677,14 +2732,8 @@ def test_battery_storage_with_time_series_in_flex_model( soc_at_start = ur.Quantity(soc_at_start).to("MWh").magnitude check_constraints(battery, schedule, soc_at_start) - # charge 850 kWh in the cheap price period (100 kWh -> 950kWh) - assert schedule[:4].sum() * 0.25 == pytest.approx(0.85) - - # discharge fully or to what's needed in the expensive price period (950 kWh -> 100 or 200 kWh) - if ts_field == "soc-minima": - assert schedule[4:].sum() * 0.25 == pytest.approx(-0.75) - else: - assert schedule[4:].sum() * 0.25 == pytest.approx(-0.85) + assert schedule[:4].sum() * 0.25 == pytest.approx(expected_charge) + assert schedule[4:].sum() * 0.25 == pytest.approx(expected_discharge) def test_unavoidable_capacity_breach(): diff --git a/flexmeasures/data/models/planning/tests/test_storage.py b/flexmeasures/data/models/planning/tests/test_storage.py index 4bf74a73a1..32115b5fe7 100644 --- a/flexmeasures/data/models/planning/tests/test_storage.py +++ b/flexmeasures/data/models/planning/tests/test_storage.py @@ -22,6 +22,7 @@ get_sensors_from_db, series_to_ts_specs, ) +from flexmeasures.utils.unit_utils import ur from flexmeasures.data.services.utils import get_or_create_model from flexmeasures.data.services.scheduling_result import SchedulingJobResult @@ -198,7 +199,8 @@ def test_battery_relaxation(add_battery_assets, db): """Check that resolving SoC breaches is more important than resolving device power breaches. The battery is still charging with 25 kW between noon and 4 PM, when the consumption capacity is supposed to be 0. - It is still charging because resolving the still unmatched SoC minima takes precedence (via breach prices). + It is still charging because resolving the still unmatched SoC minima takes precedence + through the default SoC breach prices. """ _, battery = get_sensors_from_db( db, add_battery_assets, battery_name="Test battery" @@ -276,7 +278,6 @@ def test_battery_relaxation(add_battery_assets, db): "site-peak-production-price": series_to_ts_specs( pd.Series(260, production_prices.index), unit="EUR/MW" ), - "soc-minima-breach-price": "6000 EUR/kWh", # high breach price (to mimic a hard constraint) "consumption-breach-price": f"{device_power_breach_price} EUR/kW", # lower breach price (thus prioritizing minimizing soc breaches) "production-breach-price": f"{device_power_breach_price} EUR/kW", # lower breach price (thus prioritizing minimizing soc breaches) }, @@ -314,6 +315,56 @@ def test_battery_relaxation(add_battery_assets, db): ) # 100 EUR/(kW*h) * 0.025 MW * 1000 kW/MW * 4 hours +def test_percent_soc_capacity_rejects_stored_dynamic_soc_max(): + """A raw sensor reference dict (e.g. from the DB flex-model) raises a clean error.""" + scheduler = StorageScheduler.__new__(StorageScheduler) + scheduler.sensor = None + + with pytest.raises(ValueError, match="sensor reference or time series"): + scheduler._get_soc_capacity_for_percent_conversion( + flex_model={ + "soc-max": {"sensor": 51, "default": "100 kWh"}, + "soc-unit": "%", + } + ) + + +def test_scalar_soc_minimum_moves_to_relaxed_minimum(): + soc_min, soc_minima = StorageScheduler._relax_scalar_soc_minimum( + soc_min=0.2, + soc_minima=None, + ) + + assert soc_min is None + assert soc_minima == 0.2 * ur.Quantity("MWh") + + +def test_scalar_soc_bounds_tighten_legacy_dynamic_bounds(): + index = pd.date_range( + "2015-01-01T00:00:00+01:00", + periods=2, + freq="15min", + ) + timing_kwargs = { + "query_window": (index[0], index[-1] + timedelta(minutes=15)), + "resolution": timedelta(minutes=15), + } + + _, soc_minima = StorageScheduler._relax_scalar_soc_minimum( + soc_min=0.4, + soc_minima=pd.Series([0.1, 0.5], index=index), + **timing_kwargs, + ) + _, soc_maxima = StorageScheduler._relax_scalar_soc_maximum( + soc_max=0.8, + soc_maxima=pd.Series([0.7, 0.9], index=index), + **timing_kwargs, + ) + + assert list(soc_minima) == pytest.approx([0.4, 0.5]) + assert list(soc_maxima) == pytest.approx([0.7, 0.8]) + + def test_unresolved_targets_soc_minima(add_battery_assets, db): """Test that unresolved soc-minima targets are reported in the scheduling result. @@ -1445,10 +1496,8 @@ def test_off_tick_soc_minima_are_projected_into_soft_commitments( storage_constraints = device_constraints[0].tz_convert(tz) assert ( - storage_constraints["min"] == 0 - ).all(), ( - "with a breach price, only the global soc-min should remain a hard constraint" - ) + storage_constraints["min"].isna().all() + ), "with a breach price, the global soc-min is folded into the soft commitments instead of staying a hard constraint" soc_minima_commitments = [ c for c in commitments if getattr(c, "name", "") == "any soc minima" diff --git a/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py b/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py index d6d7fa3000..a4473409cb 100644 --- a/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py +++ b/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py @@ -10,6 +10,7 @@ from flexmeasures.data.schemas.sensors import SensorReference from flexmeasures.data.models.planning.storage import StorageScheduler from flexmeasures.data.models.planning.utils import get_series_from_quantity_or_sensor +from flexmeasures.utils.unit_utils import ur def test_get_series_from_sensor_reference_source_filter_integration(fresh_db): @@ -88,6 +89,48 @@ def test_get_series_from_sensor_reference_source_filter_integration(fresh_db): assert result_forecaster.iloc[0] == pytest.approx(200.0) +def test_get_series_from_sensor_reference_default_fills_missing_values(fresh_db): + """A SensorReference default fills query slots with no matching sensor belief.""" + query_window = ( + pd.Timestamp("2025-06-01 08:00:00+02:00"), + pd.Timestamp("2025-06-01 08:30:00+02:00"), + ) + source = DataSource(name="test-default-source", type="scheduler") + fresh_db.session.add(source) + asset_type = GenericAssetType(name="test-asset-type-default") + fresh_db.session.add(asset_type) + asset = GenericAsset(name="test-asset-default", generic_asset_type=asset_type) + fresh_db.session.add(asset) + sensor = Sensor( + name="test-sensor-default", + generic_asset=asset, + event_resolution=timedelta(minutes=15), + unit="MW", + ) + fresh_db.session.add(sensor) + fresh_db.session.flush() + fresh_db.session.add( + TimedBelief( + event_start=query_window[0], + belief_horizon=timedelta(0), + event_value=0.1, + source=source, + sensor=sensor, + ) + ) + fresh_db.session.commit() + + result = get_series_from_quantity_or_sensor( + variable_quantity=SensorReference(sensor=sensor, default=ur.Quantity("1 MW")), + query_window=query_window, + resolution=sensor.event_resolution, + unit="kW", + as_instantaneous_events=False, + ) + + assert list(result) == pytest.approx([100.0, 1000.0]) + + def test_get_series_from_sensor_reference_sources_filter_integration(fresh_db): """A :class:`SensorReference` with ``sources`` returns only beliefs from the specified source. diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index 09f59d8b4b..87fdfe4944 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -247,83 +247,6 @@ def get_power_values( return -series -def fallback_charging_policy( - sensor: Sensor, - device_constraints: pd.DataFrame, - start: datetime, - end: datetime, - resolution: timedelta, -) -> pd.Series: - """This fallback charging policy is to just start charging or discharging, or do neither, - depending on the first target state of charge and the capabilities of the Charge Point. - Note that this ignores any cause of the infeasibility and, - while probably a decent policy for Charge Points, - should not be considered a robust policy for other asset types. - """ - max_charge_capacity = ( - device_constraints[["derivative max", "derivative equals"]].min().min() - ) - max_discharge_capacity = ( - -device_constraints[["derivative min", "derivative equals"]].max().max() - ) - charge_power = max_charge_capacity if sensor.get_attribute("is_consumer") else 0 - discharge_power = ( - -max_discharge_capacity if sensor.get_attribute("is_producer") else 0 - ) - - charge_schedule = initialize_series(charge_power, start, end, resolution) - discharge_schedule = initialize_series(discharge_power, start, end, resolution) - idle_schedule = initialize_series(0, start, end, resolution) - if ( - device_constraints["equals"].first_valid_index() is not None - and device_constraints["equals"][ - device_constraints["equals"].first_valid_index() - ] - > 0 - ): - # start charging to get as close as possible to the next target - return idle_after_reaching_target(charge_schedule, device_constraints["equals"]) - if ( - device_constraints["equals"].first_valid_index() is not None - and device_constraints["equals"][ - device_constraints["equals"].first_valid_index() - ] - < 0 - ): - # start discharging to get as close as possible to the next target - return idle_after_reaching_target( - discharge_schedule, device_constraints["equals"] - ) - if ( - device_constraints["max"].first_valid_index() is not None - and device_constraints["max"][device_constraints["max"].first_valid_index()] < 0 - ): - # start discharging to try and bring back the soc below the next max constraint - return idle_after_reaching_target(discharge_schedule, device_constraints["max"]) - if ( - device_constraints["min"].first_valid_index() is not None - and device_constraints["min"][device_constraints["min"].first_valid_index()] > 0 - ): - # start charging to try and bring back the soc above the next min constraint - return idle_after_reaching_target(charge_schedule, device_constraints["min"]) - # stand idle - return idle_schedule - - -def idle_after_reaching_target( - schedule: pd.Series, - target: pd.Series, - initial_state: float = 0, -) -> pd.Series: - """Stop planned (dis)charging after target is reached (or constraint is met).""" - first_target = target[target.first_valid_index()] - if first_target > initial_state: - schedule[schedule.cumsum() > first_target] = 0 - else: - schedule[schedule.cumsum() < first_target] = 0 - return schedule - - def get_series_from_quantity_or_sensor( variable_quantity: Sensor | SensorReference | list[dict] | ur.Quantity, unit: ur.Quantity | str, @@ -403,6 +326,14 @@ def get_series_from_quantity_or_sensor( time_series = convert_units( time_series, variable_quantity.unit, unit, resolution ) + if variable_quantity.default is not None: + default_value = convert_units( + variable_quantity.default.magnitude, + str(variable_quantity.default.units), + unit, + resolution, + ) + time_series = time_series.fillna(default_value) elif isinstance(variable_quantity, Sensor): bdf: tb.BeliefsDataFrame = TimedBelief.search( variable_quantity, diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 7ed14ae0dc..46412d2379 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -341,7 +341,7 @@ class SharedSchema(Schema): ) relax_soc_constraints = fields.Bool( data_key="relax-soc-constraints", - load_default=False, + load_default=True, metadata=metadata.RELAX_SOC_CONSTRAINTS.to_dict(), ) relax_capacity_constraints = fields.Bool( @@ -1042,11 +1042,21 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): return data # Fill in default soc breach prices when asked to relax SoC constraints, unless already set explicitly. + # Both relax-soc-constraints and relax-constraints default to True, so an + # explicit relax-soc-constraints wins and otherwise the umbrella + # relax-constraints decides: setting either flag to False keeps + # SoC minima/maxima as hard constraints. Note that off-tick SoC constraint + # projection relies on this precedence: it injects an explicit + # relax-soc-constraints=true, which must win over an explicit + # relax-constraints=false. + if "relax-soc-constraints" in original_data: + relax_soc_constraints = data["relax_soc_constraints"] + else: + relax_soc_constraints = data["relax_constraints"] if ( - data["relax_soc_constraints"] - or data["relax_constraints"] - and not data.get("soc_minima_breach_price") - and not data.get("soc_maxima_breach_price") + relax_soc_constraints + and data.get("soc_minima_breach_price") is None + and data.get("soc_maxima_breach_price") is None ): self.set_default_breach_prices( data, @@ -1056,10 +1066,9 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): # Fill in default capacity breach prices when asked to relax capacity constraints, unless already set explicitly. if ( - data["relax_capacity_constraints"] - or data["relax_constraints"] - and not data.get("consumption_breach_price") - and not data.get("production_breach_price") + (data["relax_capacity_constraints"] or data["relax_constraints"]) + and data.get("consumption_breach_price") is None + and data.get("production_breach_price") is None ): self.set_default_breach_prices( data, @@ -1069,10 +1078,9 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): # Fill in default site capacity breach prices when asked to relax site capacity constraints, unless already set explicitly. if ( - data["relax_site_capacity_constraints"] - or data["relax_constraints"] - and not data.get("ems_consumption_breach_price") - and not data.get("ems_production_breach_price") + (data["relax_site_capacity_constraints"] or data["relax_constraints"]) + and data.get("ems_consumption_breach_price") is None + and data.get("ems_production_breach_price") is None ): self.set_default_breach_prices( data, @@ -1248,7 +1256,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "description": rst_to_openapi(metadata.SOC_MIN.description), "types": { "backend": "typeThree", - "ui": "One fixed value or a dynamic signal (via a sensor).", + "ui": "A fixed lower boundary or a dynamic lower boundary with an optional default fallback.", }, "example-units": EXAMPLE_UNIT_TYPES["energy"], }, @@ -1257,7 +1265,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "description": rst_to_openapi(metadata.SOC_MAX.description), "types": { "backend": "typeThree", - "ui": "One fixed value or a dynamic signal (via a sensor).", + "ui": "A fixed upper boundary or a dynamic upper boundary with an optional default fallback.", }, "example-units": EXAMPLE_UNIT_TYPES["energy"], }, @@ -1266,7 +1274,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "description": rst_to_openapi(metadata.SOC_MINIMA.description), "types": { "backend": "typeTwo", - "ui": "A sensor which records the state of charge.", + "ui": "Deprecated alias for dynamic soc-min values.", }, "example-units": EXAMPLE_UNIT_TYPES["energy"], }, @@ -1275,7 +1283,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "description": rst_to_openapi(metadata.SOC_MAXIMA.description), "types": { "backend": "typeTwo", - "ui": "A sensor which records the state of charge.", + "ui": "Deprecated alias for dynamic soc-max values.", }, "example-units": EXAMPLE_UNIT_TYPES["energy"], }, @@ -1491,6 +1499,38 @@ def _build_ui_flex_model_schema() -> Dict[str, Dict[str, Any]]: class DBFlexContextSchema(FlexContextSchema, NoTimeSeriesSpecs): + # The relaxation defaults are turned off here, so validating a stored asset + # flex-context does not fill in default breach prices. The API-side defaults + # (which are True) are applied at scheduling time instead, after the stored + # flex-context is merged with the one passed in the scheduling request. + relax_constraints = fields.Bool( + data_key="relax-constraints", + load_default=False, + metadata={ + **metadata.RELAX_CONSTRAINTS.to_dict(), + "description": ( + "Defaults to False when stored on an asset (unlike the True default " + "used when scheduling): a stored flex-context should not silently bake " + "in default breach prices. The scheduling-time default of True is " + "applied after this stored flex-context is merged with the one passed " + "in the scheduling request." + ), + }, + ) + relax_soc_constraints = fields.Bool( + data_key="relax-soc-constraints", + load_default=False, + metadata={ + **metadata.RELAX_SOC_CONSTRAINTS.to_dict(), + "description": ( + "Defaults to False when stored on an asset (unlike the True default " + "used when scheduling): a stored flex-context should not silently bake " + "in default breach prices. The scheduling-time default of True is " + "applied after this stored flex-context is merged with the one passed " + "in the scheduling request." + ), + }, + ) commitments = fields.Nested( DBCommitmentSchema, data_key="commitments", required=False, many=True diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index c4db606f3e..d81d285eda 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -170,20 +170,20 @@ def to_dict(self): example="260 EUR/MW", ) SOC_MINIMA_BREACH_PRICE = MetaData( - description="""This **penalty value** is used to discourage the violation of ``soc-minima`` constraints in the flex-model, which the scheduler will attempt to minimize. + description="""This **penalty value** is used to discourage the violation of dynamic lower SoC boundary constraints in the flex-model, which the scheduler will attempt to minimize. It must use the same currency as the other price settings and cannot be negative. While it's an internal nudge to steer the scheduler—and doesn't represent a real-life cost—it should still be chosen in proportion to the actual energy prices at your site. If it's too high, it will overly dominate other constraints; if it's too low, it will have no effect. -Without this value, the soc-minima become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed. [#penalty_field]_ [#breach_field]_ +Without this value, dynamic ``soc-min`` boundaries and legacy ``soc-minima`` boundaries become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed. [#penalty_field]_ [#breach_field]_ """, example="120 EUR/kWh", ) SOC_MAXIMA_BREACH_PRICE = MetaData( - description="""This **penalty value** is used to discourage the violation of ``soc-maxima`` constraints in the flex-model, which the scheduler will attempt to minimize. + description="""This **penalty value** is used to discourage the violation of dynamic upper SoC boundary constraints in the flex-model, which the scheduler will attempt to minimize. It must use the same currency as the other price settings and cannot be negative. While it's an **internal nudge** to steer the scheduler—and doesn't represent a real-life cost—it should still be chosen in proportion to the actual energy prices at your site. If it's too high, it will overly dominate other constraints; if it's too low, it will have no effect. -Without this value, the soc-maxima become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed. [#penalty_field]_ [#breach_field]_ +Without this value, dynamic ``soc-max`` boundaries and legacy ``soc-maxima`` boundaries become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed. [#penalty_field]_ [#breach_field]_ """, example="120 EUR/kWh", ) @@ -204,19 +204,20 @@ def to_dict(self): example="10 EUR/kW", ) RELAX_CONSTRAINTS = MetaData( - description="""If True (default is ``False``), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority: + description="""If True (default), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority: 1. Avoid breaching the site consumption/production capacity. 2. Avoid not meeting SoC minima/maxima. 3. Avoid breaching the desired device consumption/production capacity. -We recommend to set this field to ``True`` to enable the default prices and associated priorities as defined by FlexMeasures. +SoC minima/maxima are already relaxed by default through ``relax-soc-constraints``. +Set this field to ``True`` to also enable the default site and device capacity breach prices and associated priorities as defined by FlexMeasures. For tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have ``breach-price`` in their name). """, example=True, ) RELAX_SOC_CONSTRAINTS = MetaData( - description="If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", + description="If True (default), avoids not meeting SoC boundaries as relaxed constraints. Setting this field (or ``relax-constraints``) to False keeps SoC boundaries as hard constraints unless breach prices are supplied explicitly; an explicit ``relax-soc-constraints`` takes precedence over ``relax-constraints``.", example=True, ) RELAX_CAPACITY_CONSTRAINTS = MetaData( @@ -318,25 +319,31 @@ def to_dict(self): example="kWh", ) SOC_MIN = MetaData( - description="""A constant and non-negotiable lower boundary for all SoC values in the schedule. + description="""Lower boundary for all SoC values in the schedule. If omitted, no lower boundary is applied. -If used, this is regarded as an unsurpassable physical limitation. -To set softer boundaries, use the ``soc-minima`` flex-model field instead together with the ``soc-minima-breach-price`` field in the flex-context. [#quantity_field]_ +This boundary is soft in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-minima-breach-price``. +When passed as a sensor reference or time series, it defines dynamic lower boundaries. +Sensor references may include a ``default`` fallback quantity for missing sensor values, for example ``{"sensor": 50, "default": "0 kWh"}``. +Set ``relax-soc-constraints`` (or ``relax-constraints``) to ``False`` to keep lower boundaries as hard constraints unless ``soc-minima-breach-price`` is supplied explicitly. [#maximum_overlap]_ [#projecting_scheduling_constraints]_ """, - example="2.5 kWh", + example={"sensor": 50, "default": "0 kWh"}, ) SOC_MAX = MetaData( - description="""A constant and non-negotiable upper boundary for all values in the schedule (for storage devices, this defaults to max soc-target, if that is provided). + description="""Upper boundary for all SoC values in the schedule (for storage devices, this defaults to max soc-target, if that is provided). If omitted, no upper boundary is applied. -If used, this is regarded as an unsurpassable physical limitation. -To set softer boundaries, use the ``soc-maxima`` flex-model field instead together with the ``soc-maxima-breach-price`` field in the flex-context. [#quantity_field]_ +This boundary is soft in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-maxima-breach-price``. +When passed as a sensor reference or time series, it defines dynamic upper boundaries. +Sensor references may include a ``default`` fallback quantity for missing sensor values, for example ``{"sensor": 51, "default": "100 kWh"}``. +Set ``relax-soc-constraints`` (or ``relax-constraints``) to ``False`` to keep upper boundaries as hard constraints unless ``soc-maxima-breach-price`` is supplied explicitly. [#minimum_overlap]_ [#projecting_scheduling_constraints]_ """, - example="7 kWh", + example={"sensor": 51, "default": "100 kWh"}, ) SOC_MINIMA = MetaData( - description="""Set points that form lower boundaries, e.g. to target a full car battery in the morning. -If a ``soc-minima-breach-price`` is defined, the ``soc-minima`` become soft constraints in the optimization problem. -Otherwise, they become hard constraints. [#maximum_overlap]_. Both single points in time and ranges are possible, see example. [#projecting_scheduling_constraints]_""", + description="""[Deprecated field] Use dynamic ``soc-min`` values instead. +Set points that form lower boundaries, e.g. to target a full car battery in the morning. +The ``soc-minima`` legacy alias is soft in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-minima-breach-price``. +Set ``relax-soc-constraints`` (or ``relax-constraints``) to ``False`` to keep them as hard constraints unless ``soc-minima-breach-price`` is supplied explicitly [#maximum_overlap]_. +Both single points in time and ranges are possible, see example. [#projecting_scheduling_constraints]_""", example=[ {"datetime": "2024-02-05T08:00:00+01:00", "value": "8.2 kWh"}, { @@ -347,9 +354,10 @@ def to_dict(self): ], ) SOC_MAXIMA = MetaData( - description="""Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window. -If a ``soc-maxima-breach-price`` is defined, the ``soc-maxima`` become soft constraints in the optimization problem. -Otherwise, they become hard constraints. [#minimum_overlap]_ [#projecting_scheduling_constraints]_""", + description="""[Deprecated field] Use dynamic ``soc-max`` values instead. +Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window. +The ``soc-maxima`` legacy alias is soft in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-maxima-breach-price``. +Set ``relax-soc-constraints`` (or ``relax-constraints``) to ``False`` to keep them as hard constraints unless ``soc-maxima-breach-price`` is supplied explicitly. [#minimum_overlap]_ [#projecting_scheduling_constraints]_""", example=[ { "value": "51 kWh", diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index 770d13b7ad..e0baf82df8 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -343,19 +343,20 @@ class StorageFlexModelSchema(Schema): metadata=metadata.SOC_AT_START.to_dict(), ) - soc_min = QuantityField( - validate=validate.Range(min=ur.Quantity("0 MWh")), + soc_min = VariableQuantityField( to_unit="MWh", default_src_unit="dimensionless", # placeholder, overridden in __init__ - return_magnitude=False, + timezone="placeholder", data_key="soc-min", + value_validator=validate.Range(min=0), metadata=metadata.SOC_MIN.to_dict(), ) - soc_max = QuantityField( + soc_max = VariableQuantityField( to_unit="MWh", default_src_unit="dimensionless", # placeholder, overridden in __init__ - return_magnitude=False, + timezone="placeholder", data_key="soc-max", + value_validator=validate.Range(min=0), metadata=metadata.SOC_MAX.to_dict(), ) @@ -427,7 +428,8 @@ class StorageFlexModelSchema(Schema): default_src_unit="dimensionless", # placeholder, overridden in __init__ timezone="placeholder", data_key="soc-maxima", - metadata=metadata.SOC_MAXIMA.to_dict(), + value_validator=validate.Range(min=0), + metadata={**metadata.SOC_MAXIMA.to_dict(), "deprecated": True}, ) soc_minima = VariableQuantityField( @@ -436,7 +438,7 @@ class StorageFlexModelSchema(Schema): timezone="placeholder", data_key="soc-minima", value_validator=validate.Range(min=0), - metadata=metadata.SOC_MINIMA.to_dict(), + metadata={**metadata.SOC_MINIMA.to_dict(), "deprecated": True}, ) soc_targets = VariableQuantityField( @@ -712,11 +714,29 @@ def post_load_sequence(self, data: dict, **kwargs) -> dict: if data.get("soc_at_start") is not None: data["soc_at_start"] = (data["soc_at_start"] / ur.Quantity("MWh")).magnitude + # Canonical dynamic bounds reuse the scheduler's existing minima/maxima path internally. + dynamic_types = (Sensor, SensorReference, list) + if isinstance(data.get("soc_min"), dynamic_types): + if data.get("soc_minima") is not None: + raise ValidationError( + "Fields `soc-min` and `soc-minima` are mutually exclusive.", + field_name="soc-min", + ) + data["soc_minima"] = data.pop("soc_min") + + if isinstance(data.get("soc_max"), dynamic_types): + if data.get("soc_maxima") is not None: + raise ValidationError( + "Fields `soc-max` and `soc-maxima` are mutually exclusive.", + field_name="soc-max", + ) + data["soc_maxima"] = data.pop("soc_max") + # Convert soc_min to dimensionless - if data.get("soc_min") is not None: + if isinstance(data.get("soc_min"), ur.Quantity): data["soc_min"] = (data["soc_min"] / ur.Quantity("MWh")).magnitude # Convert soc_max to dimensionless - if data.get("soc_max") is not None: + if isinstance(data.get("soc_max"), ur.Quantity): data["soc_max"] = (data["soc_max"] / ur.Quantity("MWh")).magnitude return data @@ -771,6 +791,7 @@ class DBStorageFlexModelSchema(Schema): data_key="soc-minima", required=False, value_validator=validate.Range(min=0), + metadata={"deprecated": True}, ) soc_maxima = VariableQuantityField( @@ -778,6 +799,7 @@ class DBStorageFlexModelSchema(Schema): data_key="soc-maxima", required=False, value_validator=validate.Range(min=0), + metadata={"deprecated": True}, ) soc_targets = VariableQuantityField( diff --git a/flexmeasures/data/schemas/sensors.py b/flexmeasures/data/schemas/sensors.py index ecfde3e581..8607e4be3e 100644 --- a/flexmeasures/data/schemas/sensors.py +++ b/flexmeasures/data/schemas/sensors.py @@ -24,6 +24,7 @@ ) import marshmallow.validate as validate from pandas.api.types import is_numeric_dtype +from pint.errors import PintError import timely_beliefs as tb from werkzeug.datastructures import FileStorage from marshmallow.validate import Validator @@ -365,6 +366,11 @@ def _serialize(self, value: Sensor, attr, obj, **kwargs) -> int: class VariableQuantityField(MarshmallowClickMixin, fields.Field): + _UNSUPPORTED_VALUE_TYPE_MESSAGE = ( + "Unsupported value type. `{value_type}` was provided but only dict, list, " + "str, pint Quantity, tuple, and numeric values with a default source unit are supported." + ) + def __init__( self, to_unit, @@ -434,20 +440,48 @@ def __init__( @with_appcontext_if_needed() def _deserialize( - self, value: dict[str, int] | list[dict] | str, attr, data, **kwargs - ) -> Sensor | list[dict] | ur.Quantity: + self, + value: ( + dict[str, Any] + | list[dict] + | str + | ur.Quantity + | tuple[Any, ...] + | numbers.Real + ), + attr, + data, + **kwargs, + ) -> Sensor | SensorReference | list[dict] | ur.Quantity: if isinstance(value, dict): - return self._deserialize_dict(value) + return self._deserialize_dict(value, attr, data, **kwargs) elif isinstance(value, list): return self._deserialize_list(value) elif isinstance(value, str): return self._deserialize_str(value) + elif isinstance(value, ur.Quantity): + return value.to(self.to_unit) + elif isinstance(value, tuple): + try: + return ur.Quantity.from_tuple(value).to(self.to_unit) + except (PintError, TypeError, ValueError, AttributeError, IndexError): + if ( + len(value) == 1 + and isinstance(value[0], numbers.Real) + and self.default_src_unit is not None + ): + return self._deserialize_numeric(value[0], attr, data, **kwargs) + if len(value) == 2: + return self._deserialize_str(f"{value[0]} {value[1]}") + raise FMValidationError( + self._UNSUPPORTED_VALUE_TYPE_MESSAGE.format(value_type=type(value)) + ) elif isinstance(value, numbers.Real) and self.default_src_unit is not None: return self._deserialize_numeric(value, attr, data, **kwargs) else: raise FMValidationError( - f"Unsupported value type. `{type(value)}` was provided but only dict, list and str are supported." + self._UNSUPPORTED_VALUE_TYPE_MESSAGE.format(value_type=type(value)) ) _SOURCE_FILTER_KEYS = SENSOR_REFERENCE_SOURCE_FILTER_KEYS @@ -504,13 +538,15 @@ def _deserialize_source_filters(self, value: dict[str, Any]) -> tuple[ return source_types, exclude_source_types, sources, source_account - def _deserialize_dict(self, value: dict[str, Any]) -> Sensor | SensorReference: + def _deserialize_dict( + self, value: dict[str, Any], attr, data, **kwargs + ) -> Sensor | SensorReference: """Deserialize a sensor reference to a Sensor or SensorReference. - Returns a plain :class:`Sensor` when no source filter keys are present - (backward compatible), and a :class:`SensorReference` when any of - ``source-types``, ``exclude-source-types``, ``sources``, or - ``source-account`` are provided. + Returns a plain :class:`Sensor` when no source filter or default keys are + present (backward compatible), and a :class:`SensorReference` when any of + ``source-types``, ``exclude-source-types``, ``sources``, ``source-account`` + or ``default`` are provided. """ if "sensor" not in value: raise FMValidationError("Dictionary provided but `sensor` key not found.") @@ -530,8 +566,12 @@ def _deserialize_dict(self, value: dict[str, Any]) -> Sensor | SensorReference: unit=self.to_unit if not self.to_unit.startswith("/") else None ).deserialize(value["sensor"], None, None) - # If source filter keys are present, return a SensorReference instead of a plain Sensor. - if self._SOURCE_FILTER_KEYS.isdisjoint(value.keys()): + default = None + if "default" in value: + default = self._deserialize_default(value["default"], attr, data, **kwargs) + + # If no source filter or default keys are present, keep returning a plain Sensor. + if self._SOURCE_FILTER_KEYS.isdisjoint(value.keys()) and default is None: return sensor # backward compat: no filters → plain Sensor source_types, exclude_source_types, sources, source_account = ( @@ -543,8 +583,23 @@ def _deserialize_dict(self, value: dict[str, Any]) -> Sensor | SensorReference: exclude_source_types=exclude_source_types, sources=sources, source_account=source_account, + default=default, ) + def _deserialize_default(self, value, attr, data, **kwargs) -> ur.Quantity: + """Deserialize a sensor reference fallback value.""" + if isinstance(value, str): + default = self._deserialize_str(value) + elif isinstance(value, numbers.Real) and self.default_src_unit is not None: + default = self._deserialize_numeric(value, attr, data, **kwargs) + else: + raise FMValidationError( + "Sensor reference `default` must be a quantity string or a numeric value with a known default source unit." + ) + if self.value_validator is not None: + self.value_validator(default) + return default + def _deserialize_list(self, value: list[dict]) -> list[dict]: """Deserialize a time series to a list of timed events.""" if self.return_magnitude is True: @@ -596,6 +651,15 @@ def _serialize( sensor_reference["source-account"] = [ account.id for account in value.source_account ] + if value.default is not None: + # `default` was already resolved to a concrete, compatible unit at + # deserialization time (see _deserialize_default), so for a + # denominator-only to_unit (e.g. "/MWh", not a valid pint unit on + # its own) there is nothing left to convert to. + if self.to_unit.startswith("/"): + sensor_reference["default"] = str(value.default) + else: + sensor_reference["default"] = str(value.default.to(self.to_unit)) return sensor_reference elif isinstance(value, Sensor): return dict(sensor=value.id) @@ -963,13 +1027,13 @@ class QuantitySchema(Schema): @dataclass class SensorReference: - """A sensor reference that wraps a Sensor with optional source filters for belief queries. + """A sensor reference that wraps a Sensor with optional query settings. Exposes the same ``unit``, ``id``, and ``event_resolution`` properties as a plain :class:`~flexmeasures.data.models.time_series.Sensor`, so code that reads those - properties works without modification. The source filters are passed through to - :meth:`TimedBelief.search ` - in :func:`~flexmeasures.data.models.planning.utils.get_series_from_quantity_or_sensor`. + properties works without modification. The source filters and optional default + value are passed through to + :func:`~flexmeasures.data.models.planning.utils.get_series_from_quantity_or_sensor`. """ sensor: Sensor @@ -977,6 +1041,7 @@ class SensorReference: exclude_source_types: list[str] | None = field(default=None) sources: list[DataSource] | None = field(default=None) source_account: list[Account] | None = field(default=None) + default: ur.Quantity | None = field(default=None) @property def unit(self) -> str: @@ -1011,7 +1076,7 @@ class OutputSensorReferenceSchema(SharedSensorReferenceSchema): class SensorReferenceSchema(SharedSensorReferenceSchema): - """Sensor reference with optional source filters.""" + """Sensor reference with optional source filters and fallback value.""" class Meta: description = "Sensor reference from which to look up a variable quantity." @@ -1047,6 +1112,14 @@ class Meta: description="Only use beliefs from data sources linked to these account IDs.", ), ) + default = fields.String( + required=False, + allow_none=False, + metadata=dict( + description="Fallback quantity to use when the referenced sensor has missing values. Note that every time slot the sensor leaves empty is filled with this value, so a sparse setpoint sensor becomes densely constrained.", + example="0 kWh", + ), + ) class InflexibleDeviceSchema(SensorReferenceSchema): @@ -1127,9 +1200,21 @@ class TimeSeriesSchema(Schema): fields.Dict, required=True, metadata=dict( - description="Time series specification containing a list of segments that together describe a variable quantity.", + description=( + "Time series specification containing a list of segments that together " + "describe a variable quantity. Each segment may specify either " + "`datetime`, `start` and `end`, `start` and `duration`, or `end` and " + "`duration`." + ), example=[ - {"value": "23 kW", "start": "2025-11-20T15:15+01", "duration": "PT1H"} + {"value": "23 kW", "datetime": "2025-11-20T15:15+01"}, + { + "value": "24 kW", + "start": "2025-11-20T16:00+01", + "end": "2025-11-20T17:00+01", + }, + {"value": "25 kW", "start": "2025-11-20T17:00+01", "duration": "PT1H"}, + {"value": "26 kW", "end": "2025-11-20T19:00+01", "duration": "PT1H"}, ], ), ) diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 15dffa17b7..5fdecc34f8 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -611,6 +611,91 @@ def test_flex_context_schema( check_schema_loads_data(schema=schema, data=flex_context, fails=fails) +def test_flex_context_schema_relaxes_soc_constraints_by_default(): + loaded_flex_context = FlexContextSchema().load({"consumption-price": "1 EUR/MWh"}) + + assert loaded_flex_context["relax_constraints"] is True + assert loaded_flex_context["relax_soc_constraints"] is True + assert loaded_flex_context["soc_minima_breach_price"].to( + "EUR/MWh" + ).magnitude == pytest.approx(1_000_000) + assert loaded_flex_context["soc_maxima_breach_price"].to( + "EUR/MWh" + ).magnitude == pytest.approx(1_000_000) + + +def test_flex_context_schema_preserves_explicit_soc_breach_prices(): + loaded_flex_context = FlexContextSchema().load( + { + "consumption-price": "1 EUR/MWh", + "soc-minima-breach-price": "5 EUR/kWh", + "soc-maxima-breach-price": "7 EUR/kWh", + } + ) + + assert loaded_flex_context["relax_soc_constraints"] is True + assert loaded_flex_context["soc_minima_breach_price"].to( + "EUR/kWh" + ).magnitude == pytest.approx(5) + assert loaded_flex_context["soc_maxima_breach_price"].to( + "EUR/kWh" + ).magnitude == pytest.approx(7) + + +@pytest.mark.parametrize( + "disabled_field", + ["relax-soc-constraints", "relax-constraints"], +) +def test_flex_context_schema_disables_default_soc_breach_prices(disabled_field): + loaded_flex_context = FlexContextSchema().load( + { + "consumption-price": "1 EUR/MWh", + disabled_field: False, + } + ) + + assert "soc_minima_breach_price" not in loaded_flex_context + assert "soc_maxima_breach_price" not in loaded_flex_context + + +def test_flex_context_schema_explicit_soc_relaxation_overrides_umbrella_opt_out(): + """An explicit relax-soc-constraints wins over an explicit relax-constraints. + + Off-tick SoC constraint projection relies on this precedence: it injects an + explicit relax-soc-constraints=true next to whatever the user configured. + """ + loaded_flex_context = FlexContextSchema().load( + { + "consumption-price": "1 EUR/MWh", + "relax-constraints": False, + "relax-soc-constraints": True, + } + ) + + assert loaded_flex_context["soc_minima_breach_price"].to( + "EUR/MWh" + ).magnitude == pytest.approx(1_000_000) + + +def test_flex_context_schema_umbrella_opt_out_disables_capacity_relaxation(): + """Setting relax-constraints to False also skips default capacity breach prices.""" + loaded_flex_context = FlexContextSchema().load( + {"consumption-price": "1 EUR/MWh", "relax-constraints": False} + ) + + assert "consumption_breach_price" not in loaded_flex_context + assert "ems_consumption_breach_price" not in loaded_flex_context + + +def test_db_flex_context_schema_does_not_relax_soc_constraints_by_default(): + loaded_flex_context = DBFlexContextSchema().load({}) + + assert loaded_flex_context["relax_constraints"] is False + assert loaded_flex_context["relax_soc_constraints"] is False + assert "soc_minima_breach_price" not in loaded_flex_context + assert "soc_maxima_breach_price" not in loaded_flex_context + + def check_schema_loads_data(schema, data, fails): if fails: with pytest.raises(ValidationError) as e_info: @@ -671,7 +756,7 @@ def check_schema_loads_data(schema, data, fails): ( {"site-power-capacity": 100}, { - "site-power-capacity": f"Unsupported value type. `{type(100)}` was provided but only dict, list and str are supported." + "site-power-capacity": f"Unsupported value type. `{type(100)}` was provided but only dict, list, str, pint Quantity, tuple, and numeric values with a default source unit are supported." }, ), ( @@ -884,6 +969,84 @@ def test_flex_context_schema_rejects_filtered_aggregate_power( assert "cannot use source filters" in str(exc_info.value) +def test_storage_flex_model_schema_rejects_filtered_consumption( + setup_dummy_sensors, setup_sources, db +): + _, _, _, power_sensor = setup_dummy_sensors + seita_source = setup_sources["Seita"] + db.session.flush() + + for schema in [ + StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None), + DBStorageFlexModelSchema(), + ]: + with pytest.raises(ValidationError) as exc_info: + schema.load( + { + "consumption": { + "sensor": power_sensor.id, + "sources": [seita_source.id], + } + } + ) + assert exc_info.value.messages["consumption"]["sources"] == ["Unknown field."] + + +def test_storage_flex_model_schema_rejects_filtered_production( + setup_dummy_sensors, setup_sources, db +): + _, _, _, power_sensor = setup_dummy_sensors + seita_source = setup_sources["Seita"] + db.session.flush() + + for schema in [ + StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None), + DBStorageFlexModelSchema(), + ]: + with pytest.raises(ValidationError) as exc_info: + schema.load( + { + "production": { + "sensor": power_sensor.id, + "sources": [seita_source.id], + } + } + ) + assert exc_info.value.messages["production"]["sources"] == ["Unknown field."] + + +def test_soc_min_sensor_reference_with_default_loads_as_dynamic_minimum( + setup_dummy_sensors, +): + energy_sensor, _, _, _ = setup_dummy_sensors + schema = StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None) + + loaded_flex_model = schema.load( + {"soc-min": {"sensor": energy_sensor.id, "default": "0 kWh"}} + ) + + assert "soc_min" not in loaded_flex_model + assert isinstance(loaded_flex_model["soc_minima"], SensorReference) + assert loaded_flex_model["soc_minima"].sensor == energy_sensor + assert loaded_flex_model["soc_minima"].default == ur.Quantity("0 MWh") + + +def test_soc_max_sensor_reference_with_default_loads_as_dynamic_maximum( + setup_dummy_sensors, +): + energy_sensor, _, _, _ = setup_dummy_sensors + schema = StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None) + + loaded_flex_model = schema.load( + {"soc-max": {"sensor": energy_sensor.id, "default": "1000 kWh"}} + ) + + assert "soc_max" not in loaded_flex_model + assert isinstance(loaded_flex_model["soc_maxima"], SensorReference) + assert loaded_flex_model["soc_maxima"].sensor == energy_sensor + assert loaded_flex_model["soc_maxima"].default == ur.Quantity("1 MWh") + + @pytest.mark.parametrize( ["flex_model", "fails"], [ @@ -895,6 +1058,103 @@ def test_flex_context_schema_rejects_filtered_aggregate_power( {"soc-min": "3500 kWh"}, False, ), + ( + {"soc-max": "3500 kWh"}, + False, + ), + ( + {"soc-max": (1, "MWh")}, + False, + ), + ( + {"soc-max": (1,)}, + [ + False, + { + "soc-max": "Unsupported value type. `` was provided but only dict, list, str, pint Quantity, tuple, and numeric values with a default source unit are supported." + }, + ], + ), + ( + {"soc-max": ur.Quantity("1 MWh")}, + False, + ), + ( + {"soc-max": ur.Quantity("1 MWh").to_tuple()}, + False, + ), + ( + {"soc-min": {"sensor": "energy-sensor", "default": "0 kWh"}}, + False, + ), + ( + {"soc-max": {"sensor": "energy-sensor", "default": "1 MWh"}}, + False, + ), + ( + {"soc-min": {"sensor": "price-sensor", "default": "0 kWh"}}, + {"soc-min": "Cannot convert EUR/MWh to MWh"}, + ), + ( + {"soc-max": {"sensor": "price-sensor", "default": "1 MWh"}}, + {"soc-max": "Cannot convert EUR/MWh to MWh"}, + ), + ( + { + "soc-min": [ + { + "datetime": "2026-06-01T12:00:00+00:00", + "value": "1 MWh", + } + ] + }, + [ + False, + { + "soc-min": "A time series specification (listing segments) is not supported when storing flex-model fields." + }, + ], + ), + ( + { + "soc-max": [ + { + "datetime": "2026-06-01T12:00:00+00:00", + "value": "2 MWh", + } + ] + }, + [ + False, + { + "soc-max": "A time series specification (listing segments) is not supported when storing flex-model fields." + }, + ], + ), + ( + { + "soc-min": {"sensor": "energy-sensor", "default": "0 kWh"}, + "soc-minima": {"sensor": "energy-sensor"}, + }, + [ + { + "soc-min": "Fields `soc-min` and `soc-minima` are mutually exclusive." + }, + False, + ], + ), + ( + { + "soc-max": {"sensor": "energy-sensor", "default": "1 MWh"}, + "soc-maxima": {"sensor": "energy-sensor"}, + }, + [ + { + "soc-max": "Fields `soc-max` and `soc-maxima` are mutually exclusive." + }, + False, + ], + ), ( {"soc-minima": {"sensor": "energy-sensor"}}, False, diff --git a/flexmeasures/data/schemas/tests/test_sensor.py b/flexmeasures/data/schemas/tests/test_sensor.py index f41805cb9a..93df41616c 100644 --- a/flexmeasures/data/schemas/tests/test_sensor.py +++ b/flexmeasures/data/schemas/tests/test_sensor.py @@ -5,6 +5,7 @@ from flexmeasures.data.schemas.sensors import ( QuantityOrSensor, SensorReference, + SensorReferenceSchema, VariableQuantityField, floor_bdf_event_starts, ) @@ -220,6 +221,41 @@ def test_sensor_reference_backward_compatible(setup_dummy_sensors): assert result.id == sensor1.id +def test_sensor_reference_with_default(setup_dummy_sensors): + """``{"sensor": , "default": ...}`` deserializes to a SensorReference.""" + sensor1, _, _, _ = setup_dummy_sensors + field = VariableQuantityField(to_unit="MWh", return_magnitude=False) + + result = field.deserialize({"sensor": sensor1.id, "default": "500 kWh"}) + + assert isinstance(result, SensorReference) + assert result.sensor == sensor1 + assert result.default == ur.Quantity("0.5 MWh") + + +def test_sensor_reference_schema_rejects_null_default(setup_dummy_sensors): + sensor1, _, _, _ = setup_dummy_sensors + + with pytest.raises(ValidationError) as exc_info: + SensorReferenceSchema().load({"sensor": sensor1.id, "default": None}) + + assert "default" in exc_info.value.messages + + +def test_sensor_reference_field_rejects_null_default(setup_dummy_sensors): + """``default`` must be a concrete fallback quantity when provided.""" + sensor1, _, _, _ = setup_dummy_sensors + field = VariableQuantityField(to_unit="MWh", return_magnitude=False) + + with pytest.raises(ValidationError) as exc_info: + field.deserialize({"sensor": sensor1.id, "default": None}) + + assert ( + "Sensor reference `default` must be a quantity string or a numeric value with a known default source unit." + in str(exc_info.value) + ) + + def test_sensor_reference_with_source_types(setup_dummy_sensors): """``{"sensor": , "source-types": [...]}`` deserializes to a :class:`SensorReference`. @@ -342,6 +378,18 @@ def test_sensor_reference_serialization_preserves_source_filters( } +def test_sensor_reference_serialization_preserves_default(setup_dummy_sensors): + sensor1, _, _, _ = setup_dummy_sensors + field = VariableQuantityField(to_unit="MWh", return_magnitude=False) + source_reference = field.deserialize({"sensor": sensor1.id, "default": "500 kWh"}) + + assert isinstance(source_reference, SensorReference) + assert serialize_variable_quantity(source_reference) == { + "sensor": sensor1.id, + "default": "0.5 MWh", + } + + def test_sensor_reference_filters_are_kept_per_reference( setup_dummy_sensors, setup_sources, db ): diff --git a/flexmeasures/data/tests/conftest.py b/flexmeasures/data/tests/conftest.py index 73820df4fa..db897a94bc 100644 --- a/flexmeasures/data/tests/conftest.py +++ b/flexmeasures/data/tests/conftest.py @@ -413,6 +413,8 @@ def flex_description_sequential( ], "site-production-capacity": "2kW", "site-consumption-capacity": "5kW", + # These tests exercise the scheduling pipeline with hard physical SoC bounds. + "relax-soc-constraints": False, # Cheap commitments that are not expected to affect the resulting schedule # todo: CommitmentSchema should have a commodity field that defaults to electricity "commitments": [ diff --git a/flexmeasures/data/tests/test_scheduling_jobs.py b/flexmeasures/data/tests/test_scheduling_jobs.py index cb479c6f39..d0af0d2929 100644 --- a/flexmeasures/data/tests/test_scheduling_jobs.py +++ b/flexmeasures/data/tests/test_scheduling_jobs.py @@ -469,6 +469,8 @@ def test_save_state_of_charge_percent_sensor( "production-price": "0 EUR/MWh", "site-production-capacity": "1MW", "site-consumption-capacity": "1MW", + "relax-soc-constraints": False, + "relax-constraints": False, } create_scheduling_job( diff --git a/flexmeasures/data/tests/test_scheduling_sequential.py b/flexmeasures/data/tests/test_scheduling_sequential.py index aea1d5317f..f219c62b60 100644 --- a/flexmeasures/data/tests/test_scheduling_sequential.py +++ b/flexmeasures/data/tests/test_scheduling_sequential.py @@ -156,13 +156,13 @@ def test_create_sequential_jobs(db, app, flex_description_sequential, smart_buil # ) -def test_create_sequential_jobs_fallback( +def test_create_sequential_jobs_without_storage_fallback( db, app, flex_description_sequential, smart_building ): - """Test fallback scheduler in a chain of sequential scheduling (sub)jobs. + """Test an infeasible first subjob in a chain of sequential scheduling jobs. - Checks execution of a sequential scheduling job, where 1 of the subjobs is set up to fail and trigger its fallback. - The deferred subjobs should still succeed after the fallback succeeds, even though the first subjob fails. + Checks that no storage fallback job is created. The deferred subjobs should remain + deferred because the first subjob failed. """ assets, sensors, _ = smart_building queue = app.queues["scheduling"] @@ -181,56 +181,60 @@ def test_create_sequential_jobs_fallback( storage_module = "flexmeasures.data.models.planning.storage" with patch(f"{storage_module}.StorageScheduler.persist_flex_model"): - with patch(f"{storage_module}.StorageFallbackScheduler.persist_flex_model"): - with patch( - f"{storage_module}.StorageScheduler.compute", - side_effect=iter([InfeasibleProblemException(), [], []]), - ): - create_sequential_scheduling_job( - asset=assets["Test Site"], - scheduler_specs=scheduler_specs, - enqueue=True, - force_new_job_creation=True, # otherwise the cache might kick in due to sub-jobs already created in other tests - **flex_description_sequential, - ) - - # There should be 3 jobs: - # 2 jobs scheduling the 2 flexible devices in the flex-model, plus 1 'done job' to wrap things up - queued_jobs = app.queues["scheduling"].jobs - deferred_jobs = [ - Job.fetch(job_id, connection=queue.connection) - for job_id in app.queues[ - "scheduling" - ].deferred_job_registry.get_job_ids() - ] - # Sort deferred_jobs by their created_at attribute - deferred_jobs = sorted(deferred_jobs, key=lambda job: job.created_at) - assert ( - len(queued_jobs) == 1 - ), "Only the job for scheduling the first device sequentially should be queued." - assert ( - len(deferred_jobs) == 2 - ), "The job for scheduling the second device, and the wrap-up job, should be deferred." - - # Work on jobs - work_on_rq(queue, exc_handler=handle_scheduling_exception) - - # Refresh jobs so that the fallback_job_id (which should be set by now) can be read - for job in queued_jobs: - job.refresh() - - finished_jobs = queue.finished_job_registry.get_job_ids() - failed_jobs = queue.failed_job_registry.get_job_ids() - - # Original job failed - assert queued_jobs[0].id in failed_jobs - - # The fallback job ran successfully - assert queued_jobs[0].meta["fallback_job_id"] in finished_jobs - - # The deferred jobs ran successfully - assert deferred_jobs[0].id in finished_jobs - assert deferred_jobs[1].id in finished_jobs + with patch( + f"{storage_module}.StorageScheduler.compute", + side_effect=InfeasibleProblemException(), + ): + create_sequential_scheduling_job( + asset=assets["Test Site"], + scheduler_specs=scheduler_specs, + enqueue=True, + force_new_job_creation=True, # otherwise the cache might kick in due to sub-jobs already created in other tests + **flex_description_sequential, + ) + + # There should be 3 jobs: + # 2 jobs scheduling the 2 flexible devices in the flex-model, plus 1 'done job' to wrap things up + queued_jobs = app.queues["scheduling"].jobs + deferred_jobs = [ + Job.fetch(job_id, connection=queue.connection) + for job_id in app.queues[ + "scheduling" + ].deferred_job_registry.get_job_ids() + ] + # Sort deferred_jobs by their created_at attribute + deferred_jobs = sorted(deferred_jobs, key=lambda job: job.created_at) + assert ( + len(queued_jobs) == 1 + ), "Only the job for scheduling the first device sequentially should be queued." + assert ( + len(deferred_jobs) == 2 + ), "The job for scheduling the second device, and the wrap-up job, should be deferred." + + # Work on jobs + work_on_rq(queue, exc_handler=handle_scheduling_exception) + + for job in queued_jobs: + job.refresh() + for job in deferred_jobs: + job.refresh() + + finished_jobs = queue.finished_job_registry.get_job_ids() + failed_jobs = queue.failed_job_registry.get_job_ids() + + # Original job failed and no fallback job was created + assert queued_jobs[0].id in failed_jobs + assert queued_jobs[0].meta.get("fallback_job_id") is None + + # The deferred jobs should not run when their dependency fails without fallback + assert deferred_jobs[0].id not in finished_jobs + assert deferred_jobs[1].id not in finished_jobs + + # Without a fallback to unblock the chain, the deferred subjobs stay deferred + # for good, so clear them here rather than leaking them into the next test. + for deferred_job_id in queue.deferred_job_registry.get_job_ids(): + queue.deferred_job_registry.remove(deferred_job_id) + queue.empty() def test_create_sequential_jobs_with_sign_explicit_context( diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index ebe6f3a407..a7d417327b 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -7,7 +7,7 @@ }, "termsOfService": null, "title": "FlexMeasures", - "version": "1.0.0" + "version": "0.33.2" }, "externalDocs": { "description": "FlexMeasures runs on the open source FlexMeasures technology. Read the docs here.", @@ -1555,10 +1555,10 @@ "datetime": "2015-06-02T16:00:00+00:00" } ], - "soc-minima": { - "sensor": 300 + "soc-min": { + "sensor": 300, + "default": "10 kWh" }, - "soc-min": "10 kWh", "soc-max": "25 kWh", "charging-efficiency": "120%", "discharging-efficiency": { @@ -4334,7 +4334,7 @@ "sensor": 42 }, "production-capacity": "30 kW", - "soc-minima": [ + "soc-min": [ { "start": "2015-06-02T12:00:00+00:00", "end": "2015-06-02T13:00:00+00:00", @@ -5030,6 +5030,11 @@ "items": { "type": "integer" } + }, + "default": { + "type": "string", + "description": "Fallback quantity to use when the referenced sensor has missing values. Note that every time slot the sensor leaves empty is filled with this value, so a sparse setpoint sensor becomes densely constrained.", + "example": "0 kWh" } }, "required": [ @@ -5039,11 +5044,25 @@ }, "TimeSeries": { "type": "array", - "description": "Time series specification containing a list of segments that together describe a variable quantity.", + "description": "Time series specification containing a list of segments that together describe a variable quantity. Each segment may specify either `datetime`, `start` and `end`, `start` and `duration`, or `end` and `duration`.", "example": [ { "value": "23 kW", - "start": "2025-11-20T15:15+01", + "datetime": "2025-11-20T15:15+01" + }, + { + "value": "24 kW", + "start": "2025-11-20T16:00+01", + "end": "2025-11-20T17:00+01" + }, + { + "value": "25 kW", + "start": "2025-11-20T17:00+01", + "duration": "PT1H" + }, + { + "value": "26 kW", + "end": "2025-11-20T19:00+01", "duration": "PT1H" } ], @@ -5272,25 +5291,25 @@ "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-minima-breach-price": { - "description": "This penalty value is used to discourage the violation of soc-minima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-minima become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed.\n", + "description": "This penalty value is used to discourage the violation of dynamic lower SoC boundary constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, dynamic soc-min boundaries and legacy soc-minima boundaries become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed.\n", "example": "120 EUR/kWh", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-maxima-breach-price": { - "description": "This penalty value is used to discourage the violation of soc-maxima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-maxima become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed.\n", + "description": "This penalty value is used to discourage the violation of dynamic upper SoC boundary constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, dynamic soc-max boundaries and legacy soc-maxima boundaries become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed.\n", "example": "120 EUR/kWh", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "relax-constraints": { "type": "boolean", "default": true, - "description": "If True (default is False), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nWe recommend to set this field to True to enable the default prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", + "description": "If True (default), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nSoC minima/maxima are already relaxed by default through relax-soc-constraints.\nSet this field to True to also enable the default site and device capacity breach prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", "example": true }, "relax-soc-constraints": { "type": "boolean", - "default": false, - "description": "If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", + "default": true, + "description": "If True (default), avoids not meeting SoC boundaries as relaxed constraints. Setting this field (or relax-constraints) to False keeps SoC boundaries as hard constraints unless breach prices are supplied explicitly; an explicit relax-soc-constraints takes precedence over relax-constraints.", "example": true }, "relax-capacity-constraints": { @@ -6967,15 +6986,20 @@ "example": "3.1 kWh" }, "soc-min": { - "type": "string", - "x-minimum": "0 MWh", - "description": "A constant and non-negotiable lower boundary for all SoC values in the schedule.\nIf omitted, no lower boundary is applied.\nIf used, this is regarded as an unsurpassable physical limitation.\nTo set softer boundaries, use the soc-minima flex-model field instead together with the soc-minima-breach-price field in the flex-context.\n", - "example": "2.5 kWh" + "description": "Lower boundary for all SoC values in the schedule.\nIf omitted, no lower boundary is applied.\nThis boundary is soft in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-minima-breach-price.\nWhen passed as a sensor reference or time series, it defines dynamic lower boundaries.\nSensor references may include a default fallback quantity for missing sensor values, for example {\"sensor\": 50, \"default\": \"0 kWh\"}.\nSet relax-soc-constraints (or relax-constraints) to False to keep lower boundaries as hard constraints unless soc-minima-breach-price is supplied explicitly.\n", + "example": { + "sensor": 50, + "default": "0 kWh" + }, + "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-max": { - "type": "string", - "description": "A constant and non-negotiable upper boundary for all values in the schedule (for storage devices, this defaults to max soc-target, if that is provided).\nIf omitted, no upper boundary is applied.\nIf used, this is regarded as an unsurpassable physical limitation.\nTo set softer boundaries, use the soc-maxima flex-model field instead together with the soc-maxima-breach-price field in the flex-context.\n", - "example": "7 kWh" + "description": "Upper boundary for all SoC values in the schedule (for storage devices, this defaults to max soc-target, if that is provided).\nIf omitted, no upper boundary is applied.\nThis boundary is soft in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-maxima-breach-price.\nWhen passed as a sensor reference or time series, it defines dynamic upper boundaries.\nSensor references may include a default fallback quantity for missing sensor values, for example {\"sensor\": 51, \"default\": \"100 kWh\"}.\nSet relax-soc-constraints (or relax-constraints) to False to keep upper boundaries as hard constraints unless soc-maxima-breach-price is supplied explicitly.\n", + "example": { + "sensor": 51, + "default": "100 kWh" + }, + "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "power-capacity": { "description": "Symmetric device-level power constraint. How much power can be applied to this asset in either direction.\nIf omitted, the scheduler infers this limit from the greatest of consumption-capacity and production-capacity when either is configured, before falling back to site-power-capacity.\nWhen exactly one of consumption-capacity or production-capacity is configured to non-zero capacity, the missing opposite capacity defaults to zero.", @@ -7050,7 +7074,7 @@ "example": true }, "soc-maxima": { - "description": "Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window.\nIf a soc-maxima-breach-price is defined, the soc-maxima become soft constraints in the optimization problem.\nOtherwise, they become hard constraints.", + "description": "[Deprecated field] Use dynamic soc-max values instead.\nSet points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window.\nThe soc-maxima legacy alias is soft in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-maxima-breach-price.\nSet relax-soc-constraints (or relax-constraints) to False to keep them as hard constraints unless soc-maxima-breach-price is supplied explicitly.", "example": [ { "value": "51 kWh", @@ -7058,10 +7082,11 @@ "end": "2024-02-05T13:30:00+01:00" } ], + "deprecated": true, "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-minima": { - "description": "Set points that form lower boundaries, e.g. to target a full car battery in the morning.\nIf a soc-minima-breach-price is defined, the soc-minima become soft constraints in the optimization problem.\nOtherwise, they become hard constraints.. Both single points in time and ranges are possible, see example.", + "description": "[Deprecated field] Use dynamic soc-min values instead.\nSet points that form lower boundaries, e.g. to target a full car battery in the morning.\nThe soc-minima legacy alias is soft in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-minima-breach-price.\nSet relax-soc-constraints (or relax-constraints) to False to keep them as hard constraints unless soc-minima-breach-price is supplied explicitly.\nBoth single points in time and ranges are possible, see example.", "example": [ { "datetime": "2024-02-05T08:00:00+01:00", @@ -7073,6 +7098,7 @@ "end": "2024-02-05T13:30:00+01:00" } ], + "deprecated": true, "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-targets": { diff --git a/flexmeasures/ui/templates/assets/asset_properties.html b/flexmeasures/ui/templates/assets/asset_properties.html index 61a141eb82..7fd32d79cd 100644 --- a/flexmeasures/ui/templates/assets/asset_properties.html +++ b/flexmeasures/ui/templates/assets/asset_properties.html @@ -607,14 +607,18 @@
Edit {{ asset.name }}'s flex-model
const card = activeCard(); const name = card.id.replace('-control', ''); const flexModel = getFlexModel() + const sensorReference = { "sensor": sensorId }; + const defaultInput = document.getElementById('flexSensorDefaultInput'); + if (defaultInput && defaultInput.value.trim()) { + sensorReference.default = defaultInput.value.trim(); + } if (FlexModelFieldValidTypes[name].includes(Array)) { const valueIndex = selectedIndex() const fieldValues = flexModel[name]; - fieldValues[valueIndex] = { "sensor": sensorId }; + fieldValues[valueIndex] = sensorReference; flexModel[name] = fieldValues; } else { - const value = flexModel[name]; - flexModel[name] = { "sensor": sensorId }; + flexModel[name] = sensorReference; } setFlexModel(flexModel); @@ -1120,6 +1124,7 @@
Edit {{ asset.name }}'s flex-model
tabContent.querySelector('.flex-input-group').appendChild(boolTabContentSaveBtn); } else if (dataType == Object) { // Object/Senosr Tab ================= btn.textContent = 'A sensor'; + const supportsSensorDefault = FlexModelFieldValidTypes[name].includes(Object); tabContent.innerHTML = `
@@ -1154,8 +1159,54 @@
Edit {{ asset.name }}'s flex-model
+ ${supportsSensorDefault ? ` + + + ` : ''} `; + if (supportsSensorDefault) { + const defaultInput = tabContent.querySelector('#flexSensorDefaultInput'); + defaultInput.value = value && typeof value === 'object' && value.default + ? value.default + : ''; + + const saveDefaultButton = document.createElement('button'); + saveDefaultButton.className = 'btn btn-secondary btn-sm me-2 mt-2'; + saveDefaultButton.textContent = 'Use fallback'; + saveDefaultButton.onclick = function () { + const currentFlexModel = getFlexModel(); + let sensorReference = currentFlexModel[name]; + if (Array.isArray(sensorReference)) { + sensorReference = sensorReference[selectedIndex()]; + } + if (!sensorReference || typeof sensorReference !== 'object' || !sensorReference.sensor) { + showToast("Select a sensor before setting a fallback", "info"); + return; + } + + const fallback = defaultInput.value.trim(); + if (fallback) { + sensorReference.default = fallback; + } else { + delete sensorReference.default; + } + setFlexModel(currentFlexModel); + }; + tabContent.querySelector('.flex-input-group').appendChild(saveDefaultButton); + } } else if (dataType == String) { // String Tab ================= btn.textContent = 'Fixed value';