Raise an alert when a publish reports success but ships nothing

[iderex]

The failure this prevents is specific and has already happened to a neighbouring board: a release was created, the manifest update failed on its own step, and the successful publish shipped nothing installable. A green run is therefore not proof that anything shipped.

Done when

• A watcher raises a tracking issue when the publish or the freshness check fails, with the run linked.
• The alert distinguishes a failed publish from a successful publish with a stale manifest, because the repairs differ.
• The watcher is tested by deliberately failing a publish in a non-production channel, and the run and the resulting issue are linked here.
• The repair procedure is in the operator-facing release notes for maintainers, not only in the alert.

[iderex]

Not started. One thing should go into the issue before the watcher is designed, because it changes what the watcher has to tell apart.

This issue names two cases: a failed publish, and a successful publish with a stale manifest. This repository can currently produce a third, and it is the one hardest to see. A release run produces one artifact, for one of the two lines the manifest declares, because the packaging tool reads a single framework and ABI pair off the top of build.yaml and knows nothing about the targets list. That is written up with its evidence in #117.

So a green publish here can ship a real artifact, update a manifest that is not stale, and still leave one declared server line with nothing installable on it. Neither of the two cases named above catches that. A watcher comparing the artifacts a run attached against the lines the manifest declares would, and that comparison is cheap because both numbers are in the tree.

Whether that belongs to this issue or to #117 is a judgement. #117 stops the release producing the wrong set. This issue is about noticing when it did anyway, which is the same argument the issue already makes for why a green run is not proof that anything shipped. It is written here so the watcher is not built against a two-case model and then found to have a third.

The rest of this issue is unchanged and unstarted. It depends on a publication route, which is #119 and carries blocked-on-decision, and on the freshness check in #120, and neither has landed.

Nothing was landed against this issue.

[iderex]

The third condition has a place to run now, and that is the only thing here that has changed.

That condition asks for the watcher to be tested by deliberately failing a publish in a non-production channel. There was no second channel: the route triggered on -stable tags only and marked every release it made as not a pre-release. #198 is on the mainline as 6b15860 and both channels are there:

git show origin/master:.github/workflows/publish.yaml | grep -c 'prerelease"'
2
git show origin/master:.github/workflows/publish.yaml | grep -n 'prerelease: '
60:      prerelease: ${{ steps.metadata.outputs.prerelease }}
525:          prerelease: ${{ needs.gate.outputs.prerelease }}

What that buys this issue is narrower than it looks and the boundary is worth having written down. A -prerelease tag is a real release that stays, so a deliberate failure rehearsed on that channel spends a version number the same way a stable one does. The route refuses a version either channel has already published, so the rehearsal costs a number rather than costing nothing. That is the price the answer on #1 chose over a trial release that is deleted afterwards, and it is a price rather than an absence.

The last note here also names a third case the watcher has to tell apart, a green publish that ships a real artifact and leaves one declared server line with nothing installable. That is unchanged, and so is the rest of what this issue waits on: the freshness check in #120, and the manifest generator that #119 records as absent.

Nothing was landed against this issue.

[iderex]

One of the three things the note above names is no longer missing, and this is that one line
rather than a claim that anything here has started.

git grep -n 'generate-manifest.py' origin/master -- .github/workflows/publish.yaml
.github/workflows/publish.yaml:670:          generator=.github/generate-manifest.py
.github/workflows/publish.yaml:730:            python3 .github/generate-manifest.py \
.github/workflows/publish.yaml:748:            python3 .github/generate-manifest.py \

Read at f5daf5a, which is what the mainline is on, and run before this was written. The
generator is on #119 and its bounds are there rather than here.

What that changes for this issue is the second condition, which asks the alert to tell a failed
publish from a successful publish with a stale manifest, because the repairs differ. There is a
manifest for one to be stale about now. There is still nothing that compares a published
manifest against the newest release, which is #120, so the distinction has one of its two ends.

The third case the first note here names is unchanged and is the one to keep in view when the
watcher is designed: a green publish that ships a real artifact, updates a manifest that is not
stale, and leaves one declared server line with nothing installable. The generator does not
close that. It refuses a release carrying more than one archive rather than describing it,
which is #117's open decision left where it is, so a run that shipped one line still produces
an index nobody can tell from a complete one.

Nothing else here has moved and nothing in this tree was changed.

[iderex]

#342 is on the mainline as 646f59e, merged as 8488ddcc04f1354360a68a65a313f44db5c72335. The first condition is met for what exists to fail, the fourth is met, the second is met in shape and unmet in population, and the third is not met and is not mine to meet. This note is those four and one dry run, rather than a claim that this issue is done.

The first condition

.github/workflows/publish-failure-alert.yml sweeps every run on the default branch and every run on a tag, once every half hour, and files one issue labelled release-alert when any of them concluded other than success, with the run linked, the step that failed named and the end of that step's log carried. The body is rewritten on every tick to the current state, a comment marks a change in the set of red workflows and nothing else, and the watcher closes the issue itself once nothing is red.

What it sweeps is derived rather than enumerated, which is the half of this condition worth writing down. No workflow is named outside the file's comments, PublishFailureAlertTests.TheWatcherNamesNoOtherWorkflowOutsideItsComments refuses one that is, and the tag list is read from the API so the release tag pattern keeps its one home. So the freshness check is watched on the day #120 lands, without anybody registering it, and so is any workflow after it. The board this is modelled on watched six by name and missed the seventh for fifteen hours.

The first dry run against the runs API, started by hand after the merge and filing nothing:

gh workflow run publish-failure-alert.yml --ref master -f dry_run=true
https://github.com/Flowfin/jellyfin-plugin-watch-sync/actions/runs/33788026774

gh run view 33788026774 --log | grep -E 'Examined|Nothing red|No open alert'
Examined 300 run(s); 0 on master or on a tag concluded other than success since 2026-09-02T18:01:33Z.
Nothing red.
No open alert issue to close.

So the sweep parses, reads the runs API with the grants it declares, and answers the state the mainline is in, which the check runs on 8488ddc also show. What it did not exercise is the filing path, because nothing was red to file; that path is the sibling board's, read line by line rather than run here, and the first real failure is its first run.

One bound the reading shows and the file did not say. The sweep reads the 300 newest runs and then applies the window, so what it examines is the smaller of the two. Read before this was written:

cutoff=$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)
gh run list --limit 600 --json updatedAt --jq "map(select(.updatedAt >= \"$cutoff\")) | length"
192

A failure is seen by the tick after it, and only a half hour in which more than 300 runs land could push one past the list before that tick. On this board's rate that has not happened, and it is a bound rather than an impossibility, so it is written here.

The second condition

Met in shape. The body names the workflow that went red, and docs/RELEASING.md under ## When the watcher files an issue carries one repair per workflow the body can name: the publish route, where whether the release exists decides everything and ## Re-running is the repair, and the freshness check, where the publish succeeded and what is served is stale. The heading is declared once, in the workflow, and TheWatcherAndTheRunbookNameOneRepairHeading holds the runbook to carrying it and to naming the workflow by its path.

Unmet in population. The freshness check is #120 and is not in this tree, so nothing today produces the second of the two failures and every alert is a red workflow of the kinds this repository has. The runbook says so at that repair rather than leaving a reader to find out.

The third condition

Not met. The rehearsal is a publish deliberately failed on the pre-release channel, and it spends a version number and a tag, which is the price the notes above already record for that channel. A tag push on this board is yours, so the rehearsal and the two links this condition asks for wait on that act. What was exercised instead is the dry run above, which reads the runs API and files nothing, and the six hand-applied changes on #342, each of which reddened exactly one fact.

The fourth condition

Met. The repair procedure is in the release runbook, not only in the alert, and the alert points at it by a heading the suite holds both files to.

What is not covered

The filter is proven by the dry run and by nothing that runs in the suite. What the suite holds is that the sweep names no workflow and that the grants and the heading are what the runbook says; whether the jq selects what it describes is a property of a run.

A run that created a release, attached one archive and wrote the manifest has shipped one server line where build.yaml declares two. That run is green, the watcher cannot see it, the runbook says so, and the gap is #117's, as the first note here already records.

The issue the watcher files carries a label and no assignee and no milestone, because the workflow token is nobody's. Whoever reads it assigns it.

Nobody but me read the change. The evidence on #342 stands in place of a second reader rather than replacing one.

What this issue waits on

The rehearsal on the pre-release channel, which is a tag push and is yours, and #120 for the second subject of the second condition.

[iderex]

#344 is on the mainline as 1e59a7f, merged as d234e9c. It repairs a defect in what landed this morning rather than moving a condition, and this note is that defect and what is still where the note above leaves it.

What was wrong

That note records the first condition as met for what exists to fail. It was met for a day at a time. The sweep reported only runs updated within the last 24 hours, and every scheduled workflow in this repository but the watcher itself fires weekly or monthly:

grep -rn 'cron:' .github/workflows/
.github/workflows/code-scanning.yaml:25:    - cron: '24 2 * * 4'
.github/workflows/dependency-scan.yml:20:    - cron: '41 3 * * 2'
.github/workflows/fuzz.yaml:28:    - cron: '0 3 * * 1'
.github/workflows/mutation.yaml:23:    - cron: '17 4 * * 3'
.github/workflows/publish-failure-alert.yml:31:#     grep -rn 'cron:' .github/workflows/
.github/workflows/publish-failure-alert.yml:67:    - cron: "*/30 * * * *"
.github/workflows/scorecard.yml:31:    - cron: "27 4 * * 1"
.github/workflows/sync-labels.yaml:5:    - cron: '0 0 1 * *'

So the alert was filed on the day a run went red and closed again by the next day's tick, with a comment saying nothing was red any more, while the failure stood until that workflow next ran. Six days for the weekly ones and a month for the label sync. A publish that failed on a tag was the same case, because nothing pushes that tag again.

That is the silence this issue exists to end, arriving one step further along and with a closing comment on top of it.

How it was found, because it was not found by reading this file

I was reading what the watcher would do with the scheduled dependency scan, which is what #97's third condition asks for. The scan is weekly, the window was 24 hours, and the two numbers do not fit.

Why the window went rather than grew

A fixed window cannot serve both ends at once. Its own comment said what it was for: stopping a long-dead red run from reopening the issue after it was closed. To cover a weekly schedule it has to be longer than a week, and a window that long is exactly what lets a failure that was repaired go on being reported for a week afterwards. No number satisfies both.

The latest run satisfies both, because a repair produces a newer run and an unrepaired failure does not. The runs are reduced to the watched population, grouped by workflow, sorted newest first, and a workflow is red when the run at the head of its own list is. A run still going carries no conclusion and a cancelled run is somebody's own act, so both are dropped before the head is taken rather than being allowed to mask the run beneath them.

What it costs, stated because an operator will meet it

An alert can no longer be got rid of by waiting a day. A workflow whose latest run is red and which is not scheduled to run again keeps its alert until somebody runs it, and every workflow here carries workflow_dispatch, so that is a button rather than a wait. docs/RELEASING.md carries this under the heading the issue body already points at.

The guard

PublishFailureAlertTests gains a fourth property and refuses the reintroduction of a cutoff in either spelling, a sweep that stops grouping by workflow, and one that forgets to order each group newest first. Five changes were applied by hand and reverted one at a time; each reddens exactly one fact, and the table is in #344 rather than repeated here. The near-miss fixture is the code that was on the mainline this morning rather than one invented for it.

DOTNET_CLI_UI_LANGUAGE=en-US DOTNET_ROLL_FORWARD=Major dotnet test Jellyfin.Plugin.WatchSync.sln --nologo
Passed!  - Failed:     0, Passed:  1194, Skipped:     1, Total:  1195 (net10.0)
Passed!  - Failed:     0, Passed:  1189, Skipped:     1, Total:  1190 (net9.0)

Two facts added, from 1192 and 1187 at 8488ddc. DOTNET_ROLL_FORWARD is there because this machine carries the .NET 10 runtime and not the .NET 9 one the projects also target. The one skip is the POSIX file mode read in StoreFolderTests, which says which platform it did not read a mode on and was skipping before this.

What this does not move

The four conditions are where the note above leaves them. The third is still the rehearsal against a deliberately failed publish on the pre-release channel, and this change does not exercise the watcher against a real failure any more than the last one did. What it changes is what the watcher would say once something is red, not whether anything has been red yet.

The bound the test class states about itself is unchanged: it reads two files and cannot run the sweep, so what the filter selects is held by fixtures and by a dry run rather than by the suite.

A second landing that carried no code

#343 carries this same tree and one commit without a Signed-off-by trailer, which DCO sign-off refused. What a message is missing is repaired by a new commit rather than by a rewrite of one already pushed, so the correction went out under a new branch name and #343 is closed with the reason in its body. Nothing was lost and one branch name was spent.