Feat/2393 durable automation runs

documentation/api/change_log.rst

@@ -26,6 +26,7 @@ v3.0-33 | September 1, 2026
 - Added ``POST /api/v3_0/assets/<id>/reports/trigger`` to queue a one-off report as a background job. It returns ``202 Accepted`` with the canonical ``job`` and ``job-url`` fields, and shares the trigger rate limit with forecast and schedule endpoints.
 - Added ``GET /api/v3_0/assets/<id>/automations`` and ``GET /api/v3_0/assets/<id>/automations/<automation_id>`` for listing and inspecting forecast automations, including the sensors an automation reads from and writes to. Each automation shows the IANA ``timezone`` in which its cron expression is interpreted, and a ``cursor``: the offset-aware UTC time of the most recent run it committed to. The cursor advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded. Asset job entries now include ``created_via`` provenance; automation identity is included only when the caller may read that automation.
 - Added ``GET /api/v3_0/sources/<id>`` to show the full record of one data source, including the attributes in which data generators store their configuration.
+- Automation responses now also include ``schedule_revision``, which counts the execution-affecting edits made to the automation's schedule, and automation detail responses gained a ``run_stats`` object. It summarizes the durable runs of that automation and describes the most recent ones: their scheduled time, dispatch state (``pending``, ``claimed``, ``partially_queued``, ``queued`` or ``failed``), execution state (``pending``, ``running``, ``succeeded``, ``failed`` or ``canceled``), attempt count, intended and queued job counts, timestamps, last error, latest attempt, and the individual jobs they created. Both additions are backward compatible: no existing field changed.
 - ``GET /api/v3_0/sensors/<id>/stats`` now reports an ``All sources`` entry summarising every data source, whenever more than one recorded. Its mean divides by the values that were summed, not by ``Number of values``, which also counts rows holding NaN.
 
 v3.0-32 | August 11, 2026

documentation/changelog.rst

@@ -27,6 +27,7 @@ New features
 * A forecaster can now be told which data sources hold the truth about the sensor it forecasts, the way its regressors already could, so that a sensor several sources report on is trained on the ones you trust [see `PR #2542 <https://www.github.com/FlexMeasures/flexmeasures/pull/2542>`_]
 * Try out a forecast without recording it, using ``flexmeasures add forecasts --dry-run``, which computes the forecast in full and reports the sensor, data source, number of beliefs and event range it would have saved [see `PR #2483 <https://www.github.com/FlexMeasures/flexmeasures/pull/2483>`_]
 * Run one-off reports as background jobs from the CLI or the asset API, with sensor-level authorization and a dedicated reporting worker queue [see `PR #2298 <https://github.com/FlexMeasures/flexmeasures/pull/2298>`_]
+
 * The asset's status page now splits its sensor data and its jobs over two tabs, of which only the opened one loads its data, and it opens the tab you last looked at [see `PR #2470 <https://www.github.com/FlexMeasures/flexmeasures/pull/2470>`_]
 * Both tabs of an asset's status page now name the asset each row belongs to, and the jobs tab also lists the jobs of the asset's sub-assets, so a site asset shows what happened anywhere below it, which you can switch off per session [see `PR #2500 <https://www.github.com/FlexMeasures/flexmeasures/pull/2500>`_]
 * Changing the selected time range on an asset or sensor chart now only loads the data that is actually new, instead of reloading the whole range, which makes stepping through or extending a long period much faster; reloading the page, or leaving it open for five minutes, still fetches everything afresh [see `PR #2433 <https://www.github.com/FlexMeasures/flexmeasures/pull/2433>`_]
@@ -74,6 +75,7 @@ Automations arrived over several pull requests. This is what each of them contri
 * Reports as well as forecasts and schedules: a report automation stores report parameters, with its reporter and the reporter's configuration on a data source, and reports on a period resolved afresh on each run, either from ``start-offset`` and ``end-offset`` applied to the run time in the automation's timezone, or since the last successful report ended, while a fixed ``start`` or ``end`` is refused; a report job records only on the sensors the automation was checked against [see `PR #2297 <https://www.github.com/FlexMeasures/flexmeasures/pull/2297>`_]
 * Every automation times its runs the same way: a fixed ``start``, ``end`` or ``prior`` in its parameters is refused, as every run would share that moment, and two of ``start-offset``, ``end-offset`` and ``duration`` describe the period each run covers instead, with the offsets applied to the time the run was due on the automation's own clock, so that, for instance, a schedule automation can plan the whole of the next day [see `PR #2551 <https://www.github.com/FlexMeasures/flexmeasures/pull/2551>`_]
 * Look up automations from the command line with ``flexmeasures show automations``, which lists them all (inactive ones included) with the IDs that the edit, delete and run commands expect, and, with ``--id``, shows a single automation's recurrence, cursor, parameters and the sensors it reads from and writes to [see `PR #2533 <https://www.github.com/FlexMeasures/flexmeasures/pull/2533>`_]
+* Automations now keep a durable record of every scheduled run, so a forecast run which failed before queueing any work is simply picked up again, while one which failed halfway only queues the jobs it still owes; an automation's details show, per run, what it queued, how many attempts that took, and, for its forecast jobs, how they ended [see `PR #2457 <https://www.github.com/FlexMeasures/flexmeasures/pull/2457>`_]
 
 
 v1.0.1 | September 9, 2026

documentation/cli/change_log.rst

@@ -14,9 +14,11 @@ since v1.1.0 | September XX, 2026
 * Add ``flexmeasures add automation``, ``flexmeasures edit automation`` and ``flexmeasures delete automation`` to manage automations (recurring tasks on an asset, with ``--type forecasting``, ``--type scheduling`` or ``--type reporting`` saying which task to automate). Each automation carries its own IANA timezone (``--timezone``), in which its cron expression is interpreted.
 * ``flexmeasures add automation --type scheduling`` refuses a flex config field which fixes a moment in time, such as ``soc-at-start`` or a ``soc-targets`` entry with a ``datetime``, naming the field: a recurring schedule automation computes a fresh schedule on every run, so such a value would be stale on the next one.
 * ``flexmeasures add automation`` refuses a fixed ``start``, ``end`` or ``prior`` in the parameters of any automation, as every run would share that moment. The parameters take ``start-offset`` and ``end-offset`` instead, or either one with a ``duration``, applied to the time each run was due, on the automation's own clock, and so do the new ``--start-offset`` and ``--end-offset`` options, next to ``--duration``.
-* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. It makes at most one queueing attempt per automation per minute, including when an attempt fails after partially queueing jobs. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once.
+* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. Each scheduled run is claimed durably, so its jobs are queued exactly once even when several runners overlap. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once.
 * Add ``flexmeasures jobs run-automation --automation <id>`` to queue the jobs for a single run of one automation, now, on top of its recurring runs. This leaves the automation's cursor alone, so its next recurring run still happens as scheduled, and inactive automations can be run this way, too.
 * ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run.
+* ``flexmeasures jobs run-automations`` now records a durable run for each scheduled run it claims, and retries the forecast runs whose queueing did not finish. A run which failed before queueing anything is dispatched again in full, and one which queued only part of its jobs resumes from its stored plan, reusing the job IDs it already queued. Each attempt is recorded with its owner, outcome and error, and the command reports the run and attempt it is working on. A run is only picked up by another runner once the claim lease of the runner holding it has expired, which is how a runner that died mid-queueing hands its work over.
+* ``flexmeasures edit automation`` now counts up the automation's schedule revision whenever it rebases the cursor (on a changed cron string or timezone, or on reactivation), which keeps the durable runs of the old and the new schedule apart, even at the same scheduled UTC time.
 
 since v1.0.1 | September 9, 2026
 =================================

documentation/features/automations.rst

@@ -185,11 +185,37 @@ Each due automation then queues its jobs.
 If the runner misses runs, because it was down or overloaded, it catches up when it resumes: it queues only the latest missed run of each automation, rather than replaying stale ones.
 Timing parameters that default to the run time are resolved when that catch-up run is queued, so it produces a current forecast, schedule or report.
 
-Each scheduled run receives at most one automatic queueing attempt.
-If the process crashes, or queueing fails after creating some jobs, that run is not retried automatically, because a retry could duplicate partial work.
+Each scheduled run a runner picks up is recorded durably, so a queueing attempt which fails can be retried without duplicating the jobs it already created.
+See :ref:`automation_runs`.
 
 The jobs record how they were created, which is shown on the asset's status page (UI), where recent jobs are listed.
 
+.. _automation_runs:
+
+Runs and retries
+----------------
+
+Every scheduled run a runner picks up gets a record in the database, which outlives the jobs it creates (jobs in Redis expire).
+The runner claims the run before doing any work, and the database allows only one record per automation, scheduled time and schedule revision, so two runners started in the same minute cannot both execute it.
+A claim comes with a lease: while one runner holds a live lease on a run, no other runner touches it, and once that lease expires the run is up for grabs again, which is how a runner that died mid-queueing hands its work over.
+
+Before queueing anything, the runner writes down the plan for the run: the parameters it will use and the individual jobs it intends to create, each with its own logical name and a job ID derived from the run.
+This is what makes a retry safe.
+A run which failed before queueing anything is dispatched again in full.
+A run which queued only some of its jobs resumes from the same plan, recognizes the jobs already in Redis by their IDs, and queues only the ones still missing, so a retry never duplicates work, and never silently drops it either.
+Because the plan is stored, a retry hours later still uses the parameters the run was planned with, even if the automation has been edited since.
+Timings the automation left to the run time are not part of those parameters, so they are resolved afresh on each attempt: a resumed run's jobs can therefore cover a later window than the ones its first attempt queued.
+
+Retrying a failed dispatch this way is what a forecast run does.
+A schedule run is recorded, claimed and reported in just the same way, but is left where it failed rather than dispatched again, because its jobs get a fresh ID on every dispatch, so a retry could not tell an already queued schedule from a missing one.
+
+A run tracks two things separately: how far its *dispatch* got (``pending``, ``claimed``, ``partially_queued``, ``queued`` or ``failed``), and how its *execution* by the workers ended (``pending``, ``running``, ``succeeded``, ``failed`` or ``canceled``).
+Each attempt to dispatch a run is recorded too, with the runner which made it, what it queued, and why it failed if it did.
+This is what an operator needs to tell a run which failed before queueing anything, one which queued half its work, and one which queued everything but then failed while computing, apart from each other.
+
+Editing an automation's cron string or timezone, or reactivating it, counts up its schedule revision.
+Runs of the old and the new schedule therefore stay distinct, even when they fall on the same scheduled UTC time.
+
 Running one automation on demand
 --------------------------------
 
@@ -213,7 +239,7 @@ Viewing automations
 
 Automations defined on an asset can be viewed on the asset's *Automations* page in the UI, and listed with the API endpoint `[GET] /assets/(id)/automations <../api/v3_0.html#get--api-v3_0-assets-id-automations>`_.
 The page shows the next scheduled run for each automation (excluding any pending catch-up run).
-An automation's details show the sensors it reads from and writes to, linking to each sensor's page.
+An automation's details show the sensors it reads from and writes to, linking to each sensor's page, and summarize its recent runs and their outcomes.
 Conversely, a sensor's page lists the automations that write data to it.
 
 .. _automation_cursor:
@@ -230,7 +256,9 @@ Runs at or before the cursor are never queued again.
 Before queueing any jobs, the runner advances the cursor to the run it is about to queue, and saves it.
 The cursor therefore records that a run was claimed, not that queueing or the task itself succeeded.
 
-Keeping a single moving timestamp, rather than a record per run, is what makes the behaviour above fall out: a runner that has been down catches up by moving the cursor straight to the latest due run, and two runners started in the same minute cannot queue the same run twice, because the cursor is advanced with a conditional update that only one of them can win.
+Keeping a single moving timestamp is what makes the catch-up behaviour above fall out: a runner that has been down catches up by moving the cursor straight to the latest due run, rather than replaying every run it missed.
+The cursor also decides who may claim a newly due run, because it is advanced with a conditional update which only one of two runners started in the same minute can win.
+What happened to a run once it is claimed is kept in its own record instead (see :ref:`automation_runs`), which is why the cursor alone says nothing about whether queueing or the task succeeded.
 
 A new automation starts from its creation minute and does not replay runs from before it existed.
 Changing its cron expression or timezone, or reactivating it, restarts from the time of that change.

flexmeasures/api/v3_0/assets.py

@@ -59,6 +59,7 @@
     delete_automation as remove_automation,
     describe_cronstr,
     get_automation_job_stats,
+    get_automation_run_stats,
     resolve_automation_sensors,
     run_automation,
     update_automation,
@@ -1503,6 +1504,7 @@ def get_automations(self, id: int, asset: GenericAsset):
                             cursor: "2026-07-11T06:00:00+02:00"
                             next-run: "2026-07-12T06:00:00+02:00"
                             recurrence-description: "At 06:00"
+                            schedule-revision: 1
                             active: true
             401:
               description: UNAUTHORIZED
@@ -1544,8 +1546,8 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset):
             the automation's parameters (forecast parameters or a schedule trigger message),
             the data source it records under, as its `source` (null for schedule automations),
             the sensors it reads from and writes to,
-            and counts of recently created jobs, per job status.
-            Note that jobs in Redis have a limited TTL, so not all past jobs will be counted.
+            durable run status, and counts of recently created jobs, per job status.
+            Note that jobs in Redis have a limited TTL, so not all past jobs will be counted, while durable run status records queueing attempts and outcomes even after those jobs expire.
             The cursor is the time of the most recent run the automation committed to, in the automation's own timezone; runs at or before it are never queued again.
             It advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded.
           security:
@@ -1582,6 +1584,7 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset):
                         cursor: "2026-07-11T06:00:00+02:00"
                         next-run: "2026-07-12T06:00:00+02:00"
                         recurrence-description: "At 06:00"
+                        schedule-revision: 1
                         active: true
                         parameters:
                           sensor: 2092
@@ -1599,6 +1602,23 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset):
                         job-stats:
                           finished: 3
                           failed: 1
+                        run-stats:
+                          total: 1
+                          dispatch:
+                            queued: 1
+                          execution:
+                            succeeded: 1
+                          latest-run:
+                            id: 12
+                            scheduled-at: "2026-07-11T04:00:00+00:00"
+                            schedule-revision: 1
+                            dispatch-state: queued
+                            execution-state: succeeded
+                            attempt-count: 1
+                            intended-job-count: 2
+                            queued-job-count: 2
+                            last-error: null
+                          recent-runs: []
                         redis-connection-err: null
             401:
               description: UNAUTHORIZED
@@ -1652,6 +1672,7 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset):
         except NoRedisConfigured as e:
             automation_data["job-stats"] = {}
             redis_connection_err = e.args[0]
+        automation_data["run-stats"] = get_automation_run_stats(automation)
         automation_data["redis-connection-err"] = redis_connection_err
         return automation_data, 200