augur#12 follow-up: tighter EDH→augur trigger via repository_dispatch (currently polling)

[ducroq]

Context

The augur#12 cron→systemd migration deployed 2026-06-08 (commit `c3e91d3`+`8805dfc`+`aa9936c`) uses a polling trigger:

• `augur-daily.timer` fires at 16:30 UTC
• `ExecStartPre=wait_for_edh.sh` polls `origin/main:data/data_quality_report.json:timestamp` every 2 min until today's date appears (4h max-wait)
• Proceeds anyway on timeout — absent healthchecks ping becomes the alarm

This is the cheapest fit given current constraints. This issue tracks the upgrade path when polling becomes a problem (e.g., recurring late-EDH days, or wanting sub-minute coupling).

The cleanest upgrade: `repository_dispatch`

EDH's `collect-data.yml` workflow gets a new final step (only runs on success + `overall_status` ok):

```yaml

• name: Trigger augur daily-update
  if: success()
  run: |
  gh api repos/ducroq/augur/dispatches
  -f event_type=edh-published
  -f client_payload[timestamp]=${{ steps.report.outputs.timestamp }}
  env:
  GH_TOKEN: ${{ secrets.AUGUR_DISPATCH_PAT }}
  ```

Augur side: a new workflow `.github/workflows/augur-daily.yml` listens for `repository_dispatch: types: [edh-published]` and SSHs sadalsuud to run `systemctl --user start augur-daily.service` (which re-uses the existing service unit — the `wait_for_edh.sh` ExecStartPre returns immediately since EDH just published).

Tradeoffs vs polling

Aspect	Polling (current)	repository_dispatch
Latency from EDH publish → augur start	up to 2 min (poll interval)	seconds
Sadalsuud inbound network exposure	none	none (it's outbound SSH from GH)
New credentials	none	PAT in EDH repo secrets + SSH key for GH→sadalsuud
Failure modes	absent healthchecks ping (24h alarm)	EDH workflow success-only gate; missed-dispatch invisible until daily expected commit absent
Rollback	uncomment cron	disable workflow, polling timer still works

When to do this

• If polling timeouts recur (e.g., EDH publish drifts past 20:30 UTC repeatedly)
• If sub-minute coupling becomes valuable (e.g., real-time dashboard sharpening)
• Otherwise: shelve. Polling is good enough.

Other options considered, rejected

• Webhook receiver on sadalsuud: requires inbound network on home machine. Rejected — adds attack surface, no existing reverse-tunnel infra.
• Move EDH onto sadalsuud's systemd: tightest coupling via `After=edh-daily.service`, but doubles infra (secrets, QA gate, strategic-feed tripwire all re-implemented locally) and makes sadalsuud a single point of failure for the whole pipeline.
• GH-Actions cron tuning: doesn't actually solve drift — shared-runner queueing dominates.

🤖 Generated with Claude Code

[ducroq]

New evidence: polling cost us a vintage on 2026-08-24, by 90 seconds

This issue argued for repository_dispatch on architectural grounds. There is now a concrete failure to point at.

wait_for_edh.sh polled for "a data_quality_report.json whose timestamp is dated today". EDH skipped its 08-23 publish and posted a catch-up at 06:28 UTC on 08-24 carrying the missed collection. That satisfied the date test, so the gate released the run instantly at 16:30. Augur finished at 16:31. EDH's real 08-24 publish landed at 16:32:17 — ninety seconds later. Augur trained and forecast on data that predated the day's actual publish, t0 stalled, the vintage was overwritten by the dedup, and the knock-on cost the 08-25 vintage permanently.

Patched today (05b4d43) by adding a floor: the report must be stamped ≥12:00 UTC as well as dated today, since the NL day-ahead auction clears ~12:00 CET and a pre-noon stamp cannot hold today's prices. Verified against the live upstream state at the time of deploy — old gate: RELEASE, new gate: WAIT.

But that is a threshold, and this issue is about not needing one. Two residual weaknesses that only an event-driven trigger removes:

1. The floor is a guess about EDH's schedule. Observed publish window is 16:1x–16:5x UTC; 12:00 is chosen to sit clear of it while excluding overnight catch-ups. If EDH's schedule moves, the gate either admits stale data again or waits its full 4h cap every day. Nothing in Augur notices the assumption going stale.
2. The race narrows but doesn't close. If EDH's publish drifts to straddle Augur's 16:30 start, the gate can release on a previous qualifying publish before today's lands — the same failure at a different clock offset. Recorded as Alternative 2 in the docs/hypothesis-log.md [2026-08-28] entry, with a signal to watch for.

repository_dispatch removes both: the run starts because EDH published, so there is no window to be early in and no threshold to maintain.

Scale of what's being absorbed: over 2026-07-25..08-28 EDH published on 31 of 35 days — missing 08-03, 08-06, 08-23, 08-27, with catch-up double-publishes on 08-09 and 08-24. Roughly an 11% miss rate, so the gate's fail-open path is exercised about every other week rather than being a rare edge case. Filed upstream as ducroq/energydatahub#50.

Not urgent now that the failure is loud ([ALARM: t0 stale …] in the commit subject, augur#14), but the case is stronger than when this issue was opened.

[ducroq]

Correcting the premise: the 2026-08-31..09-04 outage is not an argument for this

I had recorded the 2026-08-24 90-second race as the concrete case for an event-driven trigger, and it still is. But the five-day outage that just happened is not additional support, and it is worth saying so before it gets cited that way.

An event-driven repository_dispatch fires when EDH publishes. Throughout 08-31..09-03 EDH published nothing — its workflow failed before the data commit — so there was no event to dispatch on. Polling and event-driven would have behaved identically: five runs against a frozen parquet. A trigger change addresses latency between publish and consumption, not absence of a publish, and conflating the two would have justified this work on evidence that does not bear on it.

Two things that are new and relevant:

1. EDH now has publish-failure alerting (ducroq/energydatahub PR #64, merged a5993e7): a failing run opens a tracking issue with the run URL, comments rather than duplicating, and self-closes on recovery. That reduces the value of tightening our trigger, because the "why is the vintage stale" question — the main practical cost of polling — now has an upstream answer.
2. That alerting is structurally blind to a run that never starts — it is a job inside collect-data.yml with needs: [collect-and-publish, deploy], and the workflow triggers only on schedule/workflow_dispatch. A dropped cron produces no run, no alert, and a clean issue tracker. Our own wait_for_edh.sh gate plus the heartbeat remain the only detector for that case, and a repository_dispatch trigger would be equally blind to it, for the same reason.

So the case for this issue is unchanged in strength and slightly weaker in payoff. Still worth doing eventually for the race it was filed for; not urgent, and explicitly not motivated by the outage.

[ducroq]

Partially addressed 2026-09-10, and the remaining case for this issue is now sharper.

wait_for_edh.sh can now give up early rather than polling blind to its 03:00 UTC deadline: it reads EDH's open publish-failure issues (unauthenticated api.github.com via python3 stdlib — not gh, which is not installed on sadalsuud) and abandons the wait when one was opened after this run started, emitting [ALARM: EDH publish FAILED upstream — repo#N]. That removes the specific waste this issue's polling design causes when upstream has definitively failed: on 2026-09-09 the issue opened at 19:13 UTC and the gate sat polling until 03:00, then retrained on an unchanged parquet and republished a byte-identical vintage over a still-evaluable one.

What this does not fix, i.e. why this issue stays open:

• It only shortens a wait that ends in failure. A successful publish is still discovered by polling, so the 90-second race that motivated this issue is untouched.
• EDH opens one issue per failure streak and closes it on recovery — #70 sat open from 09-09 with updatedAt == createdAt and zero comments. So on the second and later nights of a streak nothing is newer than our start time and the gate waits to the deadline exactly as before.
• It is deliberately secondary. Their alert job is a step inside the workflow it watches, so it is structurally blind to a run that never starts. Detection of an absent vintage remains ours (t0 hold-back + heartbeat); their issue only explains a run that fired and failed.

A repository_dispatch trigger would address the success path, which is the half still costing us. Note also that EDH published twice on 2026-09-10 (a 07:41 UTC dispatched recovery plus the scheduled ~18:00 run), which is the cadence an event-driven trigger would handle naturally and a fixed poll window handles badly.

[ducroq]

Stronger motivation from #34: polling at a fixed hour does not merely add latency — it has a stable failure mode that costs vintages silently.

wait_for_edh.sh fires at 16:30 UTC and EDH publishes ~18:00–18:56 UTC, so the gate is always started before the thing it waits for. Its readiness test is "strictly newer than the last consumed report", which is satisfied instantly by yesterday's publish whenever one was left unconsumed. Any EDH day with more than one publish leaves one — and from then on the pipeline runs a full day behind, permanently, with every guard reading green because t0 still advances exactly +1 day per run.

Seeded twice in seven days (2026-09-04, four publishes; 2026-09-10, two). Cleared only by accidents that delayed a run past ~18:30 UTC, and the second clearing cost the t0=2026-09-13 vintage outright.

A repository_dispatch trigger removes this class entirely rather than tuning around it. If the run is started by the publish, there is no window in which an older unconsumed publish can satisfy the gate, no fixed-hour-vs-publish-hour race, and no need for a freshness heuristic at all — the readiness question is answered by the event. It also makes the 03:00 UTC deadline and the publish-failure issue probe into pure fallbacks rather than load-bearing machinery.

Worth noting for whoever picks this up: the gate must stay fail-open. An event-driven trigger needs a timer-based backstop for the case where the dispatch itself is lost, or a dropped event becomes a silent no-run — the same shape as the blind spot recorded in memory/upstream-edh.md (EDH's alert job cannot fire for a run that never starts).