Reports as jobs and as automations by Flix6x · Pull Request #2297 · FlexMeasures/flexmeasures

Flix6x · 2026-07-11T18:38:28Z

Description

Automations can now compute reports on a recurring basis, alongside forecasts and schedules. The
automation still only decides when work is due and queues it; the reporting worker computes the
report and stores it. Nothing about how a report is computed changes.

Running a report as a background job is no longer part of this PR. The reporting queue,
flexmeasures add report --as-job and the job runner itself landed on main in #2298, so what is left
here is the recurring side, and the rolling window that makes recurrence worth having.

A reporter is a data generator, like a forecaster. The reporter class and its configuration live
on a steady data source, pinned by the automation's generator_id, while the automation's
parameters hold what resolves afresh on each run. The creation field forecaster is therefore
generalised to generator, taking a forecaster or a reporter class, and config now describes both.
Schedules still assemble their generator per run and take neither.

Task names. reporting joins forecasting and scheduling in the supported types, which is
what the placeholder comment on SUPPORTED_TYPES had been holding open, and report joins the
result nouns used in messages about a single result.

The report window rolls. An automation that always computed the same period would be pointless,
so the start and end resolve on every run, in one of three ways. start-offset and end-offset hold
comma-separated Pandas offsets — "-1D,DB" for the start of the previous day — applied to the run
time, in the timezone of the first output sensor, which is exactly what
flexmeasures add report --start-offset already does, so the same window spelled the same way means
the same thing whether it is run once or on a schedule. Without any timing fields the window runs
from the end of the last window this automation successfully covered. On the first run, when no
covered window is known, it falls back to the last cron period. Absolute start and end still
validate, and still draw a warning, because every run would compute the same period.

Coverage is recorded only on success. The reporting job carries its automation_id and records
the end of its window once the report has been stored, so a failed run leaves no permanent gap: the
next run starts where the last successful one ended, rather than skipping the period the failure
lost. Coverage is kept in Redis and never moves backwards, so a late-arriving job cannot rewind an
automation into recomputing what it has already reported.

Where a report may be recorded. validate_forecast_output_scope becomes
validate_automation_output_scope, so a report's output sensors are held to the rule a forecast's
already are: they must sit within the asset's own subtree, and the creator must be allowed to record
on them. A report names sensors in its config as well as its parameters, so both are checked.

CLI. flexmeasures add automation --type reporting takes the reporter through --reporter, its
configuration through --config, and the report parameters through --parameters:

flexmeasures add automation --asset 3 --name "Daily self-consumption report" \
  --cron "0 1 * * *" --type reporting --reporter PandasReporter \
  --config reporter-config.yml --parameters report-parameters.yml

--source accepts a reporter's data source, reusing its class and configuration the way it already
does for a forecaster.

Unset options. Click passes an empty tuple, not None, for a multiple-value option that was
never given, so unset options were leaking into the stored config and parameters as empty tuples.
They are now dropped along with the other unset values.

UI. The Reporting tab on an asset's Automations page is now populated, completing the three
tabs #2290 put in place. The New automation modal offers the data generator and its configuration.

Added changelog item in documentation/changelog.rst

Look & Feel

The Reports tab, which #2290 and #2293 both showed as a disabled placeholder, is now populated. Here
a daily energy-costs report on a campus asset, recurring at 01:00 in Europe/Amsterdam, with the same
per-row actions the other two tabs offer:

Creating one asks for the reporter and its configuration through the same Data generator fields a
forecast automation uses, and takes the report parameters as JSON:

How to test

See the manual test walkthrough in the PR comments.

pytest \
  flexmeasures/cli/tests/test_automations.py \
  flexmeasures/data/tests/test_automations_fresh_db.py \
  flexmeasures/data/schemas/tests/test_reporting.py \
  flexmeasures/api/v3_0/tests/test_automations_api.py

Eight tests are added. Coverage includes the full roundtrip — create a report automation, let the
runner queue a reporting job with its provenance, let a worker compute and store it, then check the
stored values — and the window resolution on its own, a report automation whose config names
another organisation's sensor, one whose output falls outside the asset's subtree, the refusal to
create one without a generator, and the guarantee that recorded coverage cannot move backwards. Each
was verified to fail without the code it covers.

Further improvements

Coverage lives in Redis, so a flushed cache sends an automation back to its cron-period fallback
rather than to the true last covered window. Durable run records are tracked in Add durable automation-run records and safe retry semantics #2393.
Report parameters cannot be edited after creation, for the reason forecast parameters cannot: it
would decouple the automation from the access check its creator passed.
The recommended rolling window has to be written out by hand for every report automation. Shipping
ready-made report definitions that carry one follows in Prepared report templates #2300.

Related items

Part of the automations story #2334, and of #2288. Stacked on #2294, followed by #2300 and #2299.
#2298, which this PR was originally written on top of, has since merged to main.

Sign-off

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

- Activate the reporting queue (it was prepared but commented out), including worker help texts and queue cleanup. - Reporters accept as_job: a job is queued (with trigger meta data) that rebuilds the reporter from its data source, computes the report and saves the results to the database. - `flexmeasures add report --as-job` queues such a job; reporting jobs show up in the asset's jobs overview (status page and API). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

Automations can now compute reports on a recurring basis: - `flexmeasures add automation --type reports --reporter <class>` stores the reporter config on a data source (steady across runs, so all report results attribute to the same source) and validates the report parameters. - The report window resolves freshly on each run: 'start-offset'/'end-offset' fields (comma-separated Pandas offsets, applied to the run time in the first output sensor's timezone) express a rolling window, and without any timing fields the window defaults to the last cron period (from the previous cron fire time until the run time). Absolute start/end still work, but draw a warning. - The API creation field 'forecaster' is generalized to 'generator' (also accepting reporter classes), and the UI's New automation modal gains data generator and config fields; the Reports tab is now enabled. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

read-the-docs-community · 2026-07-11T18:43:13Z

Documentation build overview

📚 flexmeasures | 🛠️ Build #34509184 | 📁 Comparing 67a7e2b against latest (4a5892f)

🔍 Preview build

157 files changed · + 27 added · ± 130 modified

+ Added

± Modified

Each automation run is recorded in Redis; a report automation without timing fields then reports on the period since its actual last run, falling back to the last cron period when no last run is known (e.g. on the first run, or after a Redis flush). This gives gapless coverage even when runs are missed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

Consolidates the shared automations concept (model, lifecycle, runner deployment, provenance) into documentation/features/automations.rst, with the per-feature pages linking to it and keeping only their type-specific parameter semantics. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

- API automation creation now checks that the caller may read every sensor referenced in the parameters/config and record data on the sensors the automation writes to, closing a cross-account data read/write hole. - `add report --as-job` implies --save-config (the worker rebuilds the reporter from its data source, so jobs without stored config always crashed). - Report automation parameters are validated with the chosen reporter's own parameters schema, not the base schema. - Invalid start-offset/end-offset strings are rejected at creation instead of being silently skipped at run time (which yielded empty report windows). - run_report_job wipes the shared cached reporter's parameter state, like the automation runner already did, so consecutive jobs in one worker process don't pollute each other. - Default report windows now anchor to the end of the last *successfully* covered window, recorded by the reporting job upon success — failed jobs no longer create permanent reporting gaps, and the enqueue-time minute-rollover gap is gone (the recorded anchor is the window end itself). - The cron-period fallback window is computed in the platform timezone, matching how the runner decides when automations fire. - Job stats for schedules automations also scan flex-model device sensors (which may belong to child assets), so failed per-device jobs show up. - The trigger provenance kwarg is excluded from the job cache hash, so identical schedule requests from different origins dedupe again. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

Merge the current automation CRUD foundation into the report automation branch so report jobs share the established timezone, catch-up, permission, API, and UI behavior. The conflict resolution keeps report-specific queues, CLI options, views, documentation, and tests while retaining the newer scheduling and sensor-access invariants from the lower stack. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Make reporters declare the sensors they read from and write to so automation creation, detail rendering, and sensor links use one authoritative dependency model. Profit and loss reports now include price sensors stored in reporter configuration, preventing cross-organisation report automations from bypassing access checks, and the incomplete duplicate scanner in the API is removed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Require every report payload to declare at least one output and preserve the inherited required constraints when specialized profit and aggregation schemas narrow list lengths. Invalid automations now fail during API or CLI validation instead of being accepted and later crashing a reporting worker with missing method arguments. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Apply the existing automation subtree invariant to report outputs during both creation and execution, after access checks have established that sensor metadata may be discussed. Reporter parameters are prepared before dependency resolution so recurring reports with relative windows expose and validate their sensors consistently on API details and worker runs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Extend the database invariant that protects executable automations so report rows, like forecast rows, cannot exist without a data generator. The migration replaces the forecast-only check constraint while preserving generator-free schedule automations and provides a reversible downgrade to the previous rule. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Build recurring report windows from the canonical cron occurrence claimed by the runner and interpret prior occurrences in each automation's own IANA timezone. Delayed catch-up runs now produce stable boundaries, including across daylight-saving gaps, instead of using the platform timezone and the worker's later wall-clock time. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Update the Redis coverage anchor with an optimistic transaction that only accepts a later report end. Concurrent or out-of-order reporting workers can no longer let an older completion rewind the next default window and cause already reported periods to be processed again. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Assert that the asset automations page exposes the reports tab and its dedicated table alongside forecasts and schedules. This protects the report UI added by the stacked branch from disappearing during future template reconciliations. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Describe per-automation timezone selection and the successful-coverage model used by recurring reports. The documentation now distinguishes the claimed cron occurrence from delayed runner time and explains how first runs, daylight-saving transitions, and out-of-order worker completions determine report windows. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Commit the fixture-owned sensor setup after rejected report automation requests so the shared API test database does not retain an idle transaction during teardown. This keeps the complete automation API module deterministic while preserving assertions that no unauthorized automation was created. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Context: - PR #2300 is stacked on #2297, whose automation, timezone, permission, and CLI validation changes advanced after the template branch diverged. Change: - Merge the updated report-automation branch while preserving prepared-template commands and combining them with the current automation CLI validation and output-scope invariant. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

…ations Context: - Picking up the review changes from #2290, in which scheduling_cursor became cursor, automation "occurrences" became "runs", and the automations documentation was pulled out of the forecasting page. - This branch already has a features/automations.rst of its own, covering all three automation types, how to manage them and who may, which is the better page. Change: - Kept this branch's automations page and dropped the one #2290 added, folding in the part only that one had: an appendix on the cursor, catch-up and daylight-saving-time rules. - Replaced the claim that a Redis guard is what prevents queueing twice, which predates the durable cursor from #2396, with the at-most-once and catch-up behaviour. - Kept a forecast-specific section in features/forecasting.rst, since the automations page links back to it, now passing --type forecasts explicitly and introducing --forecaster and --config before saying --source makes them unnecessary. - Fixed two calls to _canonical_occurrence_time, renamed to _canonical_run_time upstream, which git merged without a conflict and which would have raised NameError. - Removed a leftover stub of _asset_and_ancestor_ids, replaced upstream by the shared helper in data/queries/generic_assets. - Merged the CLI changelog entries and regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl>

Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl>

… HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>

* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>

… HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>

…ations The base carries the merged PR #2298, which upstreamed the report job runner, and the rename of the automation types from their results to their tasks. Resolved by taking the base's `flexmeasures/data/services/reporting.py` (the version now on main) and re-applying only what report automations add to it: the automation id travels with the job, and a successful run records the end of its report window. Added "reporting" to `SUPPORTED_TYPES` and `RESULT_NOUNS`, which the base had left a placeholder for, and renamed this branch's own `"reports"` literals to `"reporting"` throughout, including the Automations page's tab and table ids, which are built from the type name. Dropped the two migrations this branch added for the generator check constraint: the base creates `automation.generator_id` as NOT NULL for every type, and deleted the constraint and the revision they were chained to. The alembic graph has a single head again. Extracted the forecasting branch of `create_automation` into `_prepare_forecast_automation`, which was already defined here but had no caller, to keep the function under the complexity limit now that it has a third branch. Dropped `_prepare_schedule_automation`, which had no caller either, as scheduling is validated inline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s-ui-polish Follows the automation type rename that came up the stack, in `_job_cache_refs`, `get_asset_automations_job_stats` and the Automations page. Kept this branch's refactor of the job-stats lookup, which already covers reporting, over the inline version it was extracted from. Kept the *Run now* button, which arrived from PR #2460 by way of the base, alongside this branch's batched job counts: the counts now come with the listing, so the cell is filled in directly rather than fetched per row. Dropped the changelog entries this branch carried for PRs #2290, #2293, #2294 and #2297, which the base states in consolidated form, including the API entry for the automation CRUD endpoints, whose permission wording PR #2294 has since settled. This branch's own entry no longer claims the UI edit action: PR #2294 has since shipped it, so what is left here is the batched job counts and loading an automation's details only when they are opened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on scheduling jobs how they were created The scheduling job creators accept an optional trigger dict (stored as job meta data), like the forecasting pipeline already does. The API trigger endpoint records origin API; the CLI and automations follow in the next commit. The status page's 'Created Via' column picks this up automatically. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: schedules as automations Automations now also support the 'schedules' type: - `flexmeasures add automation --type schedules` validates the parameters as a schedule trigger message (per the AssetTriggerSchema, as accepted by the API trigger endpoint, without the asset id). The schedule 'start' may be omitted, in which case each run schedules from the run time (floored to the message's resolution, if given) — a fixed start draws a warning. - The runner dispatches schedules automations to the same job creators as the API trigger endpoint (sequential or simultaneous), recording trigger meta data (origin automation) on the queued jobs; `flexmeasures add schedule --as-job` now records origin CLI. - Job stats for schedules automations are counted from the scheduling job cache (asset-level wrap-up jobs and per-sensor device jobs). - The UI automations page's Schedules tab is now enabled, with automations filtered by type per tab. Part of FlexMeasures/flexmeasures#2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for schedule automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: allow schedule automations without generators Context: - Schedule automations do not use a data generator, but the reviewed forecast automation schema required one. Change: - Make the foreign key nullable while retaining a database check that forecasts always have a generator. - Add a forward migration without weakening data-source deletion semantics. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover schedule automation validation Context: - Review uncovered untested forecast-only options, invalid durations, and DST start calculation. Change: - Add CLI and trigger-preparation regressions while retaining forecast sensor validation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover schedule automation dispatch Context: - Persistence, inherited flex configuration, descendant statistics, and job counts lacked realistic coverage. Change: - Move automation service tests under the fresh-database fixture and add end-to-end schedule regressions. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover schedule job provenance Context: - API provenance was not asserted across asset and sensor schedule jobs. Change: - Verify API trigger metadata on sequential descendants, wrap-up jobs, and sensor jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation type tabs Context: - The asset automation page tests still targeted the former single table. Change: - Assert type-specific tables, error rendering, filters, and hidden-tab column adjustment. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: harden automation dispatch Context: - Minimal asset schedules lost stored defaults, invalid timing could execute, provenance was incomplete, and descendant jobs were miscounted. Change: - Validate and floor fixed durations safely, inherit stored flex configuration, preserve provenance, and count the jobs actually dispatched. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: reject forecast options for schedule automations Context: - Schedule automation creation silently accepted forecaster settings that could never affect scheduling. Change: - Detect supplied forecast-only options and return a user-facing usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: resize automation tables on tab changes Context: - DataTables initialized in the hidden schedule tab could render with stale column widths. Change: - Add tab accessibility state and adjust initialized table columns when a tab becomes visible. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: clarify schedule automation inputs Context: - The feature guide linked only to the API root and omitted fixed-start and duration constraints. Change: - Document canonical fields, runtime start behavior, timing validation, and generator-free schedule automations. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover malformed automation YAML Context: - Manual testing exposed raw parser exceptions for malformed automation files. Change: - Require a user-facing usage error for invalid YAML in both config and parameter files. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: report malformed automation YAML Context: - PyYAML parser errors escaped automation creation without a useful CLI message. Change: - Translate malformed config and parameter files into a normal Click usage error. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/tests: cover stored schedule flex configuration Context: - Manual execution showed minimal automations failing for a one-device asset tree. Change: - Exercise both simultaneous and sequential dispatch using realistic flex configuration stored on the child asset. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * scheduling: load stored flex config for minimal triggers Context: - A one-device asset tree collapsed to sensor scheduling without a sensor, and sequential dispatch could not resolve its stored output. Change: - Preserve asset-triggered flex models as a list and resolve sequential device sensors from stored consumption or production outputs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/scheduling: describe trigger propagation Context: - Scheduling service docstrings did not distinguish single-job and sequential provenance behavior. Change: - Document where trigger metadata is stored and correct the simultaneous return description. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * data/migrations: rejoin the two automation migration branches Merging the timezone and catch-up work alongside the schedule automations left two alembic heads, one adding an automation's timezone and scheduling cursor and one allowing a schedule automation without a data generator, so flexmeasures db upgrade refused to run and the Docker image build failed. The two touch different columns, so the merge point has nothing of its own to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): resolve schedule automation sensors Prepare the scheduling configuration through the same scheduler collection path used before queueing, then derive declared input and output sensors so generator-free schedule automations expose their stored flex dependencies and participate in sensor relationships. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover schedule sensor resolution Exercise a minimal schedule that inherits its flex model and context from the asset tree, confirming that price inputs, schedule outputs, and the sensor-to-automation relationship are all reported from stored configuration. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): expose schedule dependency details Create a generator-free schedule with stored price and output sensors and assert that the details endpoint returns both dependency sets instead of reporting empty arrays. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: refuse forecaster options that were given, not merely ones that differ from their default The guard against combining forecaster options with --type schedules compared the forecaster against its default, so naming the default forecaster explicitly passed silently and the automation was created, leaving the impression that the option had applied to a schedule automation. Ask which options were actually given on the command line instead, the way the --source conflict check already does, and name the offending options in the error rather than listing every option it could have been. Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: document schedule automations in the automations chapter The scheduling chapter now points to the automations chapter, the way the forecasting chapter already does, and the details of what a schedule automation stores live next to the rest of the automations documentation. The changelog entry is merged with the one for forecasting automations, into a single entry on automations. That entry had stayed in the v1.0.0 section although PR #2290 was merged after v1.0.0 was tagged, so the merged entry moves to v1.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Name automation types after the task, like the rest of the codebase Queue names and job types call these tasks "forecasting" and "scheduling", so the automation types now do too: 'forecasts' becomes 'forecasting' and 'schedules' becomes 'scheduling'. An automation's type is now the name of the queue its jobs go to, so the runner no longer maps one to the other. Automations were merged after v1.0.0 was tagged, so no released CLI or API surface changes. A migration renames the values of existing rows, in both directions, and recreates the check constraint that requires a data generator for forecast automations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: move the data source inspection entry to v1.1.0 Like the automations entry it accompanied, it was written while v1.0.0 was the open section, but PR #2290 was merged on 2026-09-01, after the v1.0.0 tag of 2026-08-26. Neither the v1.0.0 nor the v1.0.0rc5 tag contains it, and it is the last entry in that section that postdates the tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Schedulers as data generators, and a generator on every automation (#2464) * Make the Scheduler a DataGenerator, and give every automation a generator A scheduler's data source recorded only its class, version and author, so one source described every schedule that scheduler ever made, whatever it computed. It now also records the flex config the scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a schedule can be traced back to the configuration that produced it. `Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and time series which do not survive a round trip. `resolve_flex_config` returns the config as passed, and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which does not read the asset tree keeps describing exactly what it was given. One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the request's source once and hands it to each device job, so a schedule can still be retrieved per device from the request's job, rather than each device job resolving a source from its own slice of the flex-model. A schedule automation now points at such a source, so `generator_id` is required for every automation and the constraint requiring it only for forecasts is gone. That generator is derived rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the runner resolves it again on every run and moves the automation when either has changed. For the same reason, a schedule automation's flex config may only describe the site and its devices: a field fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when the automation is created, naming the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Address review of the scheduler data generator Keep values that describe one moment out of a scheduler's data source. A `soc-at-start`, or a `soc-targets` entry at a given datetime, differs on every trigger, so recording it would have made every schedule the work of a brand new data source. Only what the site and its devices can do now tells one scheduler source from another. The rule that already refused such fields on a schedule automation is the same one, so both now live next to the config schema they are about. That schema moves back in with the other scheduling schemas, into a submodule of its own. The circular import which had pushed it out is fixed at its root instead: three scheduling schema modules imported `Sensor` and `Asset` from the `flexmeasures` package root, which is still initialising while they load, so they now import from the modules that define them. The migration relaxing the generator constraint and the one restoring it cancel out, and both were unreleased, so they are gone, along with the merge revision that only existed to rejoin the relaxing one. `generator_id` keeps the NOT NULL it was created with, the type rename no longer has a constraint to recreate, and the automation feature arrives in one migration rather than three. Also, fail with the source's id when a job is told to record under a data source that no longer exists, rather than an AttributeError one frame later, and reflow the comments and docstrings which broke mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs/changelog: say what a second schedule source means for reading a sensor The warning read as if summing two schedules of one period were a thing anyone would want to do, which made it sound like a defect rather than a change in provenance. It now says what actually changes: the newer schedule used to supersede the older one, and now both are kept, so a chart draws both and the asset's KPIs total both, because they deliberately report what the chart draws. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Keep a scheduler's data source identity stable Two ways the recorded config could differ while describing the same configuration, both from the Copilot review. A caller which deserialized the flex config first hands over sensors and assets, which were recorded by how they print. A sensor prints as its name, so renaming one described a different configuration, and two sensors sharing a name described the same one. They are now recorded by their id, which is what the serialized flex config names them by. A single moment may be written as one mapping or as a list of them, and stripping the momentary values left a null behind in the first case and an empty list in the second, so the same configuration looked like three different ones depending on how it was written. A field that stripping empties is now left out altogether, which is what leaving it out of the trigger message does too. A field that arrived empty stays as it was given, and a list with static entries beside momentary ones keeps them. The snapshot no longer deep-copies. It goes straight through the JSON-safe encoder, which yields plain structures, where the copy used to carry sensors along and be read after a commit had expired them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * Send the request's configuration to the device jobs, not a data source id FlexMeasures does not auto-commit the session of the request that enqueues a scheduling job (see `flexmeasures.data.transactional`), and the schedule trigger endpoint does not commit either. A data source resolved while enqueueing therefore lives in a transaction the workers never see, so handing its id to the device jobs of a sequential schedule would have failed to find it. The jobs now carry the request's configuration instead, and each worker resolves the data source from it and commits, as `make_schedule` already does for everything else it writes. The device jobs of one request still describe one configuration, so they still share one source. A test pins that enqueueing writes no data source of its own, and that the device jobs carry the same configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: clear Redis after the test that only queues jobs The new sequential test never runs what it queues, so it left the jobs, the queue's deferred registry and the job cache behind. The job cache was the one that bit: a scheduling job's id is derived from what it schedules, so the next test's identical request was skipped as already made, and it saw no jobs queued. Also reflow a docstring line that ended mid-phrase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * docs: reflow two lines that broke mid-phrase Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> * tests: fetch each queued job once Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtMZ49GH6LaiY5MdgtfNe2 Signed-off-by: F.N. Claessen <claessen@seita.nl> --------- Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>

…ations Takes the base's move of sensor access into `flexmeasures/data/services/data_generators.py`, so this branch resolves a report's sensors through `resolve_data_generator_sensors` and `check_sensor_access` rather than its own inline checks. Dropped the `Forbidden` and `check_access` imports this branch carried: the base raises from inside `check_sensor_access`, so neither is used here any more, and only a docstring still names `Forbidden`. Kept this branch's `apply_offset_chain` and `get_timezone`, which the rolling report window needs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

Flix6x and others added 3 commits July 11, 2026 20:37

docs: changelog entry for reports as jobs and automations

5eac468

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

Flix6x and others added 2 commits July 11, 2026 21:31

This was referenced Jul 11, 2026

Run reports via CLI and API #2298

Merged

Automations UI polish: edit action and batch job stats #2299

Open

BelhsanHmida linked an issue Aug 5, 2026 that may be closed by this pull request

Reports as automations #2371

Open

BelhsanHmida added 10 commits August 12, 2026 02:10

This was referenced Aug 12, 2026

Automations - first roundtrip for forecasts #2290

Merged

Schedules as automations #2293

Merged

Merge remote-tracking branch 'origin/feat/2288-automations-crud' into…

b8b8db5

… HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x and others added 2 commits September 1, 2026 17:41

Merge remote-tracking branch 'origin/feat/2288-automations-crud' into…

2701717

… HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>

bferanmi806-sketch mentioned this pull request Sep 9, 2026

A report is saved in two places, so the CLI and the job can drift apart #2485

Open

nhoening mentioned this pull request Sep 10, 2026

Aggregation reporter #2520

Open

BelhsanHmida marked this pull request as ready for review September 11, 2026 11:43

BelhsanHmida mentioned this pull request Sep 11, 2026

Prepared report templates #2300

Open

3 tasks

Ahmad-Wahid mentioned this pull request Sep 11, 2026

Aggregation reporter: select sensors by asset, name and unit #2525

Open

3 tasks

Navigation Menu

Uh oh!

Reports as jobs and as automations - #2297

Reports as jobs and as automations#2297
Flix6x wants to merge 21 commits into
feat/2288-automations-crudfrom
feat/2288-report-automations

Flix6x commented Jul 11, 2026 •

edited by BelhsanHmida

Loading

Uh oh!

read-the-docs-community Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Flix6x commented Jul 11, 2026 • edited by BelhsanHmida Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Look & Feel

How to test

Further improvements

Related items

Sign-off

Uh oh!

read-the-docs-community Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Flix6x commented Jul 11, 2026 •

edited by BelhsanHmida

Loading

read-the-docs-community Bot commented Jul 11, 2026 •

edited

Loading