Schedulers as data generators, and a generator on every automation by Flix6x · Pull Request #2464 · FlexMeasures/flexmeasures

Flix6x · 2026-09-02T16:00:32Z

Description

A scheduler's data source recorded only its class, version and author, so a single source described
every schedule that scheduler ever made, whatever it computed. It now also records the flex config
the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a
schedule can be traced back to the configuration that produced it — and so a schedule automation has
a data generator to point at, like a forecast automation does.

Scheduler is a DataGenerator. Its config is the asset, plus the serialized flex-model and
flex-context. Timing stays out: start, end and resolution differ from run to run, which is
what DataGenerator._clean_parameters already says about parameters rather than config. The config
is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities
and time series which do not survive a round trip.

No scheduler changes behaviour. resolve_flex_config returns the config as it was passed, and
StorageScheduler overrides it to merge in what the asset tree stores first. A scheduler which does
not read the asset tree — ProcessScheduler, or a plugin's — therefore keeps describing exactly what
it was handed, and keeps computing exactly what it did before.

One request stays one source. create_sequential_scheduling_job resolves the request's source
once and hands it to every device job. Without that, each device job would resolve a source from its
own slice of the flex-model, and a schedule could no longer be retrieved per device from the
request's job.

Every automation has a generator. generator_id is now required, and the check constraint
requiring it only for forecasts is gone. A schedule automation's generator is derived rather than
chosen: the scheduler follows from the asset and the config from the asset tree, so the runner
resolves it again on every run and moves the automation when either has changed. Editing an asset's
flex-model is a configuration change, and reads as one.

A schedule automation's flex config describes the site, not a moment. Because the schedule is
recomputed on every run, a field fixing a moment — soc-at-start, or a soc-targets entry with a
datetime — is refused when the automation is created, and the error names the field:

Error: flex-model[0].soc-targets[0] fixes a moment in time, so it cannot configure a recurring
schedule automation, which computes a fresh schedule on every run. Refer to a sensor instead of a
fixed value, or leave the field out.

Added changelog item in documentation/changelog.rst

Consequence for hosts

Schedules computed under different flex configs are now recorded by different data sources, where
one source per scheduler version used to record them all. A sensor can therefore carry schedules from
several sources, as it already could for forecasts.

Values describing a single moment stay out of that config, so a soc-at-start, or a soc-targets
entry at a given datetime, does not make every trigger a new data source. What does is a change to
what the site and its devices can do, such as a device's power-capacity. After such a change, the
sensor holds the schedule computed under each configuration, where the newer schedule used to
supersede the older one, so a chart draws both and the asset's KPIs total both, as they report what
the chart draws. Selecting a data source shows one configuration's schedule.

A stable configuration is unaffected — same config, same source, supersession as before. This is
called out as a warning in the changelog.

How to test

pytest flexmeasures/data flexmeasures/api flexmeasures/ui flexmeasures/cli

For the reviewer: check where your database is stamped

generator_id is created NOT NULL by 8ecec35b799c, so #2293's migration relaxing it and this
PR's restoring it cancelled out exactly. Both are removed here, along with the merge revision that
only existed to rejoin the relaxing branch, so the automation feature arrives in one migration rather
than three.

Three revisions that existed on these branches are therefore gone: 5a9c0e3b7d21, c63896a97a8e and
b8f4d2617ac9. A database stamped at exactly one of those cannot resolve its own
alembic_version any more and needs a flexmeasures db stamp — likely if you ran the #2293 branch
recently. A database at 3e91c47b0a58 or later is fine, since Alembic only resolves the revision it
is stamped with and walks up from there. Check with flexmeasures db current before upgrading.

The resulting chain was exercised on a sandboxed Postgres: upgrade from an empty database to head,
downgrade, and upgrade again. The column ends NOT NULL, with no check constraint on
automation.

Discussion

This implements option 3.1 from the discussion on #2293, where the alternative of leaving
generator_id nullable was weighed. Two consequences were agreed there before the work started:
sources fragmenting product-wide rather than only inside automations, and automations being definable
on a strict subset of trigger messages.

Related items

Based on #2293 (this PR's base branch), which stays reviewable on its own. Part of the automations
story #2334.

Sign-off

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs

…ator A scheduler's data source recorded only its class, version and author, so one source described every schedule that scheduler ever made, whatever it computed. It now also records the flex config the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a schedule can be traced back to the configuration that produced it. `Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and time series which do not survive a round trip. `resolve_flex_config` returns the config as passed, and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which does not read the asset tree keeps describing exactly what it was given. One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the request's source once and hands it to each device job, so a schedule can still be retrieved per device from the request's job, rather than each device job resolving a source from its own slice of the flex-model. A schedule automation now points at such a source, so `generator_id` is required for every automation and the constraint requiring it only for forecasts is gone. That generator is derived rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the runner resolves it again on every run and moves the automation when either has changed. For the same reason, a schedule automation's flex config may only describe the site and its devices: a field fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when the automation is created, naming the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>

read-the-docs-community · 2026-09-02T16:05:59Z

Documentation build overview

📚 flexmeasures | 🛠️ Build #34380212 | 📁 Comparing 4e348cc against latest (8944a8b)

🔍 Preview build

19 files changed · + 1 added · ± 18 modified

+ Added

_autosummary/flexmeasures.data.schemas.scheduling.config.html

± Modified

Copilot

🟡 Changes recommended

There is a confirmed crash risk when an invalid data_source_id is provided to make_schedule, and several new/updated comments/docstrings violate the repo’s line-break-after-punctuation convention.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR makes scheduler provenance consistent with forecasters/reporters by treating Scheduler as a DataGenerator whose data source identity includes the serialized flex config it computed under, and by requiring every automation (including scheduling) to reference a generator data source.

Changes:

Promote Scheduler to a DataGenerator and persist scheduler data sources with a config snapshot (asset, flex-model, flex-context) so schedules can be traced to the configuration used.
Ensure sequential scheduling requests use a single resolved scheduler data source across all per-device jobs, and update tests to query schedules by the job’s source.
Make Automation.generator_id mandatory for all types, add migration enforcement, and derive scheduling automation generators on create/run (including rejecting “momentary” flex fields for recurring scheduling automations).

File summaries

File	Description
flexmeasures/data/tests/test_scheduling_simultaneous.py	Update schedule assertions to query beliefs by the job’s resolved scheduler data source
flexmeasures/data/tests/test_scheduling_jobs.py	Fetch scheduler source from job metadata instead of looking up by name
flexmeasures/data/tests/test_automations_fresh_db.py	Add scheduling-automation generator derivation helpers and new generator-following test
flexmeasures/data/services/scheduling.py	Thread `data_source_id` through scheduling jobs; ensure sequential child jobs share one request-level source
flexmeasures/data/services/automations.py	Resolve and persist scheduling automation generator per run; detect “momentary” flex-config fields
flexmeasures/data/schemas/scheduling/init.py	Export scheduler-config schema from scheduling schema package
flexmeasures/data/schemas/scheduler_config.py	New Marshmallow schema for scheduler config stored on scheduler data sources
flexmeasures/data/models/planning/storage.py	Override scheduler config resolution to include asset-tree flex config before recording
flexmeasures/data/models/planning/init.py	Make `Scheduler` a `DataGenerator`; implement config capture and config-hashed data sources
flexmeasures/data/models/automations.py	Remove forecast-only generator constraint and require `generator_id` for all automations
flexmeasures/data/migrations/versions/b8f4d2617ac9_require_a_generator_on_every_automation.py	New migration to enforce non-null generator and drop old forecast-only constraint
flexmeasures/cli/tests/test_automations.py	Add CLI test for rejecting “momentary” flex fields; assert scheduling automations now have a generator
flexmeasures/cli/data_add.py	Derive and validate scheduling automation generators at creation time
flexmeasures/api/v3_0/tests/test_automations_api_fresh_db.py	Ensure API fixtures create scheduling automations with generator_id
documentation/features/automations.rst	Document scheduling automation generator derivation and “momentary” flex-field rejection
documentation/cli/change_log.rst	CLI changelog entry for scheduling automation parameter validation behavior
documentation/changelog.rst	Add warning about scheduler-source fragmentation by flex config; add feature entry

Review details

Suppressed comments (1)

flexmeasures/data/services/scheduling.py:908

If an invalid data_source_id is passed, db.session.get(DataSource, ...) returns None and the next block will crash at data_source.id; fail fast with a clear error (or fall back to scheduler.data_source) to avoid an AttributeError and silent mis-attribution.

    if data_source_id is not None:
        data_source = db.session.get(DataSource, data_source_id)
    else:
        data_source = scheduler.data_source

Files reviewed: 17/17 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Keep values that describe one moment out of a scheduler's data source. A `soc-at-start`, or a `soc-targets` entry at a given datetime, differs on every trigger, so recording it would have made every schedule the work of a brand new data source. Only what the site and its devices can do now tells one scheduler source from another. The rule that already refused such fields on a schedule automation is the same one, so both now live next to the config schema they are about. That schema moves back in with the other scheduling schemas, into a submodule of its own. The circular import which had pushed it out is fixed at its root instead: three scheduling schema modules imported `Sensor` and `Asset` from the `flexmeasures` package root, which is still initialising while they load, so they now import from the modules that define them. The migration relaxing the generator constraint and the one restoring it cancel out, and both were unreleased, so they are gone, along with the merge revision that only existed to rejoin the relaxing one. `generator_id` keeps the NOT NULL it was created with, the type rename no longer has a constraint to recreate, and the automation feature arrives in one migration rather than three. Also, fail with the source's id when a job is told to record under a data source that no longer exists, rather than an AttributeError one frame later, and reflow the comments and docstrings which broke mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>

… sensor The warning read as if summing two schedules of one period were a thing anyone would want to do, which made it sound like a defect rather than a change in provenance. It now says what actually changes: the newer schedule used to supersede the older one, and now both are kept, so a chart draws both and the asset's KPIs total both, because they deliberately report what the chart draws. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>

Copilot

🟡 Changes recommended

Scheduler data-source identity is still vulnerable to unintended fragmentation due to unstable serialization of non-JSON flex-config values and incomplete dropping of momentary fields.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 24/24 changed files
Comments generated: 2
Review effort level: Lite

Two ways the recorded config could differ while describing the same configuration, both from the Copilot review. A caller which deserialized the flex config first hands over sensors and assets, which were recorded by how they print. A sensor prints as its name, so renaming one described a different configuration, and two sensors sharing a name described the same one. They are now recorded by their id, which is what the serialized flex config names them by. A single moment may be written as one mapping or as a list of them, and stripping the momentary values left a null behind in the first case and an empty list in the second, so the same configuration looked like three different ones depending on how it was written. A field that stripping empties is now left out altogether, which is what leaving it out of the trigger message does too. A field that arrived empty stays as it was given, and a list with static entries beside momentary ones keeps them. The snapshot no longer deep-copies. It goes straight through the JSON-safe encoder, which yields plain structures, where the copy used to carry sensors along and be read after a commit had expired them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>

Copilot

🟡 Changes recommended

Sequential scheduling can enqueue child jobs referencing a newly-created (but uncommitted) scheduler data source ID, which can race and fail when workers try to look it up.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

flexmeasures/data/models/planning/init.py:81

This class docstring still wraps mid-phrase (e.g. the sentence split between the lines ending with "when" and starting with "knowledge"). The repo convention is to break docstring/comment lines only after punctuation; please reflow these sentences so each physical line ends with punctuation.

Files reviewed: 25/25 changed files
Comments generated: 1
Review effort level: Lite

…e id FlexMeasures does not auto-commit the session of the request that enqueues a scheduling job (see `flexmeasures.data.transactional`), and the schedule trigger endpoint does not commit either. A data source resolved while enqueueing therefore lives in a transaction the workers never see, so handing its id to the device jobs of a sequential schedule would have failed to find it. The jobs now carry the request's configuration instead, and each worker resolves the data source from it and commits, as `make_schedule` already does for everything else it writes. The device jobs of one request still describe one configuration, so they still share one source. A test pins that enqueueing writes no data source of its own, and that the device jobs carry the same configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl>

Copilot

🟡 Changes recommended

There are at least two concrete issues to address (a docstring line-break convention violation and a new test that can leak queued jobs into later tests, making the suite order-dependent).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 26/26 changed files
Comments generated: 2
Review effort level: Lite

The new sequential test never runs what it queues, so it left the jobs, the queue's deferred registry and the job cache behind. The job cache was the one that bit: a scheduling job's id is derived from what it schedules, so the next test's identical request was skipped as already made, and it saw no jobs queued. Also reflow a docstring line that ended mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl>

Copilot

🔵 Needs a closer look

Two changed locations violate the repo’s “wrap only after punctuation” convention for docstrings/comments and should be reflowed before approval.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

flexmeasures/data/services/scheduling.py:909

This inline comment wraps mid-sentence (line ends with "row" and continues on the next line), violating the repo convention to break lines only after punctuation. Reflow so the line break occurs after punctuation (or keep it on one line).
flexmeasures/data/tests/test_scheduling_sequential.py:22
Docstring line break splits the sentence mid-phrase (line ends with "jobs" and the next line continues), violating the repo convention to wrap docstrings/comments only after punctuation. Reflow this sentence onto one line (or break after punctuation) so each physical line ends with punctuation.

Files reviewed: 26/26 changed files
Comments generated: 0 new
Review effort level: Lite

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x · 2026-09-03T19:07:01Z

Both reflowed in bcf7570 — the comment in scheduling.py that ended on "row", and the test docstring that ended on "jobs". Each physical line now ends at punctuation.

These were the two suppressed findings from the round-5 review; no other findings were raised on that round.

Copilot

🔵 Needs a closer look

It changes scheduler/source identity semantics and automation generator invariants across core scheduling, automations, and migrations, so it needs careful human validation of upgrade paths and operational impact.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

flexmeasures/data/tests/test_scheduling_sequential.py:50

device_jobs fetches each RQ job twice (once for the list element and once in the filter), which is unnecessary and can lead to inconsistent results if job state changes between fetches. Fetch each job once and then filter based on its kwargs.

Files reviewed: 26/26 changed files
Comments generated: 0 new
Review effort level: Lite

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl>

…ons-crud The base branch had moved 29 commits ahead, mostly through #2464 (schedulers as data generators) and #2460 (running one automation on demand). Endpoints, tests and page handlers from both sides simply coexist: this branch's POST/PATCH/DELETE on /assets/(id)/automations next to the base's POST .../automations/(automation_id)/trigger. Two resolutions carry a decision: - create_automation() stays the one place an automation is created, so the logic the base had grown in the CLI moved into it: a scheduling automation now resolves its scheduler's data source there, like a forecast automation resolves its forecaster's, and after the permission check for the same reason (a refused request should leave nothing behind). _check_schedule_automation_parameters is gone; a flex config that pins a moment now raises RecurringScheduleFixesAMoment, which the CLI still reports as a usage error. - Every automation has a generator again, so this branch's nullable generator_id and its 'forecast_generator' check constraint are obsolete. The base had already deleted that migration in #2464, and its deletion is taken here. Keeping it would have been worse than redundant: the constraint tested type != 'forecasts', a value the rename no longer stores, so it would have silently stopped enforcing.

* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on scheduling jobs how they were created The scheduling job creators accept an optional trigger dict (stored as job meta data), like the forecasting pipeline already does. The API trigger endpoint records origin API; the CLI and automations follow in the next commit. The status page's 'Created Via' column picks this up automatically. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: schedules as automations Automations now also support the 'schedules' type: - `flexmeasures add automation --type schedules` validates the parameters as a schedule trigger message (per the AssetTriggerSchema, as accepted by the API trigger endpoint, without the asset id). The schedule 'start' may be omitted, in which case each run schedules from the run time (floored to the message's resolution, if given) — a fixed start draws a warning. - The runner dispatches schedules automations to the same job creators as the API trigger endpoint (sequential or simultaneous), recording trigger meta data (origin automation) on the queued jobs; `flexmeasures add schedule --as-job` now records origin CLI. - Job stats for schedules automations are counted from the scheduling job cache (asset-level wrap-up jobs and per-sensor device jobs). - The UI automations page's Schedules tab is now enabled, with automations filtered by type per tab. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for schedule automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: allow schedule automations without generators Context: - Schedule automations do not use a data generator, but the reviewed forecast automation schema required one. Change: - Make the foreign key nullable while retaining a database check that forecasts always have a generator. - Add a forward migration without weakening data-source deletion semantics. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover schedule automation validation Context: - Review uncovered untested forecast-only options, invalid durations, and DST start calculation. Change: - Add CLI and trigger-preparation regressions while retaining forecast sensor validation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover schedule automation dispatch Context: - Persistence, inherited flex configuration, descendant statistics, and job counts lacked realistic coverage. Change: - Move automation service tests under the fresh-database fixture and add end-to-end schedule regressions. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover schedule job provenance Context: - API provenance was not asserted across asset and sensor schedule jobs. Change: - Verify API trigger metadata on sequential descendants, wrap-up jobs, and sensor jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation type tabs Context: - The asset automation page tests still targeted the former single table. Change: - Assert type-specific tables, error rendering, filters, and hidden-tab column adjustment. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: harden automation dispatch Context: - Minimal asset schedules lost stored defaults, invalid timing could execute, provenance was incomplete, and descendant jobs were miscounted. Change: - Validate and floor fixed durations safely, inherit stored flex configuration, preserve provenance, and count the jobs actually dispatched. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: reject forecast options for schedule automations Context: - Schedule automation creation silently accepted forecaster settings that could never affect scheduling. Change: - Detect supplied forecast-only options and return a user-facing usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: resize automation tables on tab changes Context: - DataTables initialized in the hidden schedule tab could render with stale column widths. Change: - Add tab accessibility state and adjust initialized table columns when a tab becomes visible. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: clarify schedule automation inputs Context: - The feature guide linked only to the API root and omitted fixed-start and duration constraints. Change: - Document canonical fields, runtime start behavior, timing validation, and generator-free schedule automations. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover malformed automation YAML Context: - Manual testing exposed raw parser exceptions for malformed automation files. Change: - Require a user-facing usage error for invalid YAML in both config and parameter files. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: report malformed automation YAML Context: - PyYAML parser errors escaped automation creation without a useful CLI message. Change: - Translate malformed config and parameter files into a normal Click usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover stored schedule flex configuration Context: - Manual execution showed minimal automations failing for a one-device asset tree. Change: - Exercise both simultaneous and sequential dispatch using realistic flex configuration stored on the child asset. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: load stored flex config for minimal triggers Context: - A one-device asset tree collapsed to sensor scheduling without a sensor, and sequential dispatch could not resolve its stored output. Change: - Preserve asset-triggered flex models as a list and resolve sequential device sensors from stored consumption or production outputs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/scheduling: describe trigger propagation Context: - Scheduling service docstrings did not distinguish single-job and sequential provenance behavior. Change: - Document where trigger metadata is stored and correct the simultaneous return description. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * data/migrations: rejoin the two automation migration branches Merging the timezone and catch-up work alongside the schedule automations left two alembic heads, one adding an automation's timezone and scheduling cursor and one allowing a schedule automation without a data generator, so flexmeasures db upgrade refused to run and the Docker image build failed. The two touch different columns, so the merge point has nothing of its own to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): resolve schedule automation sensors Prepare the scheduling configuration through the same scheduler collection path used before queueing, then derive declared input and output sensors so generator-free schedule automations expose their stored flex dependencies and participate in sensor relationships. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover schedule sensor resolution Exercise a minimal schedule that inherits its flex model and context from the asset tree, confirming that price inputs, schedule outputs, and the sensor-to-automation relationship are all reported from stored configuration. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): expose schedule dependency details Create a generator-free schedule with stored price and output sensors and assert that the details endpoint returns both dependency sets instead of reporting empty arrays. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: refuse forecaster options that were given, not merely ones that differ from their default The guard against combining forecaster options with --type schedules compared the forecaster against its default, so naming the default forecaster explicitly passed silently and the automation was created, leaving the impression that the option had applied to a schedule automation. Ask which options were actually given on the command line instead, the way the --source conflict check already does, and name the offending options in the error rather than listing every option it could have been. Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: document schedule automations in the automations chapter The scheduling chapter now points to the automations chapter, the way the forecasting chapter already does, and the details of what a schedule automation stores live next to the rest of the automations documentation. The changelog entry is merged with the one for forecasting automations, into a single entry on automations. That entry had stayed in the v1.0.0 section although PR #2290 was merged after v1.0.0 was tagged, so the merged entry moves to v1.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Name automation types after the task, like the rest of the codebase Queue names and job types call these tasks "forecasting" and "scheduling", so the automation types now do too: 'forecasts' becomes 'forecasting' and 'schedules' becomes 'scheduling'. An automation's type is now the name of the queue its jobs go to, so the runner no longer maps one to the other. Automations were merged after v1.0.0 was tagged, so no released CLI or API surface changes. A migration renames the values of existing rows, in both directions, and recreates the check constraint that requires a data generator for forecast automations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: move the data source inspection entry to v1.1.0 Like the automations entry it accompanied, it was written while v1.0.0 was the open section, but PR #2290 was merged on 2026-09-01, after the v1.0.0 tag of 2026-08-26. Neither the v1.0.0 nor the v1.0.0rc5 tag contains it, and it is the last entry in that section that postdates the tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Schedulers as data generators, and a generator on every automation (#2464) * Make the Scheduler a DataGenerator, and give every automation a generator A scheduler's data source recorded only its class, version and author, so one source described every schedule that scheduler ever made, whatever it computed. It now also records the flex config the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a schedule can be traced back to the configuration that produced it. `Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and time series which do not survive a round trip. `resolve_flex_config` returns the config as passed, and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which does not read the asset tree keeps describing exactly what it was given. One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the request's source once and hands it to each device job, so a schedule can still be retrieved per device from the request's job, rather than each device job resolving a source from its own slice of the flex-model. A schedule automation now points at such a source, so `generator_id` is required for every automation and the constraint requiring it only for forecasts is gone. That generator is derived rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the runner resolves it again on every run and moves the automation when either has changed. For the same reason, a schedule automation's flex config may only describe the site and its devices: a field fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when the automation is created, naming the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Address review of the scheduler data generator Keep values that describe one moment out of a scheduler's data source. A `soc-at-start`, or a `soc-targets` entry at a given datetime, differs on every trigger, so recording it would have made every schedule the work of a brand new data source. Only what the site and its devices can do now tells one scheduler source from another. The rule that already refused such fields on a schedule automation is the same one, so both now live next to the config schema they are about. That schema moves back in with the other scheduling schemas, into a submodule of its own. The circular import which had pushed it out is fixed at its root instead: three scheduling schema modules imported `Sensor` and `Asset` from the `flexmeasures` package root, which is still initialising while they load, so they now import from the modules that define them. The migration relaxing the generator constraint and the one restoring it cancel out, and both were unreleased, so they are gone, along with the merge revision that only existed to rejoin the relaxing one. `generator_id` keeps the NOT NULL it was created with, the type rename no longer has a constraint to recreate, and the automation feature arrives in one migration rather than three. Also, fail with the source's id when a job is told to record under a data source that no longer exists, rather than an AttributeError one frame later, and reflow the comments and docstrings which broke mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: say what a second schedule source means for reading a sensor The warning read as if summing two schedules of one period were a thing anyone would want to do, which made it sound like a defect rather than a change in provenance. It now says what actually changes: the newer schedule used to supersede the older one, and now both are kept, so a chart draws both and the asset's KPIs total both, because they deliberately report what the chart draws. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Keep a scheduler's data source identity stable Two ways the recorded config could differ while describing the same configuration, both from the Copilot review. A caller which deserialized the flex config first hands over sensors and assets, which were recorded by how they print. A sensor prints as its name, so renaming one described a different configuration, and two sensors sharing a name described the same one. They are now recorded by their id, which is what the serialized flex config names them by. A single moment may be written as one mapping or as a list of them, and stripping the momentary values left a null behind in the first case and an empty list in the second, so the same configuration looked like three different ones depending on how it was written. A field that stripping empties is now left out altogether, which is what leaving it out of the trigger message does too. A field that arrived empty stays as it was given, and a list with static entries beside momentary ones keeps them. The snapshot no longer deep-copies. It goes straight through the JSON-safe encoder, which yields plain structures, where the copy used to carry sensors along and be read after a commit had expired them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Send the request's configuration to the device jobs, not a data source id FlexMeasures does not auto-commit the session of the request that enqueues a scheduling job (see `flexmeasures.data.transactional`), and the schedule trigger endpoint does not commit either. A data source resolved while enqueueing therefore lives in a transaction the workers never see, so handing its id to the device jobs of a sequential schedule would have failed to find it. The jobs now carry the request's configuration instead, and each worker resolves the data source from it and commits, as `make_schedule` already does for everything else it writes. The device jobs of one request still describe one configuration, so they still share one source. A test pins that enqueueing writes no data source of its own, and that the device jobs carry the same configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: clear Redis after the test that only queues jobs The new sequential test never runs what it queues, so it left the jobs, the queue's deferred registry and the job cache behind. The job cache was the one that bit: a scheduling job's id is derived from what it schedules, so the next test's identical request was skipped as already made, and it saw no jobs queued. Also reflow a docstring line that ended mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs: reflow two lines that broke mid-phrase Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: fetch each queued job once Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> --------- Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>

Flix6x requested a lite review from Copilot September 2, 2026 16:40

Copilot started reviewing on behalf of Flix6x September 2, 2026 16:41 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread flexmeasures/data/models/automations.py Outdated

Comment thread flexmeasures/data/services/scheduling.py Outdated

Comment thread flexmeasures/data/tests/test_scheduling_simultaneous.py Outdated

Flix6x commented Sep 3, 2026

View reviewed changes

Flix6x and others added 2 commits September 3, 2026 13:09

Flix6x requested a lite review from Copilot September 3, 2026 12:07

Copilot started reviewing on behalf of Flix6x September 3, 2026 12:08 View session