v7.21.4 and v7.21.5 were tagged on 2026-09-17 and 2026-09-20 and neither reached a registry. npm and crates.io sat at 7.21.3 for three days. I re-ran both Publish runs by hand today and they succeeded unchanged, so nothing was wrong with the code being released.
This is the same failure shape as #718, and #719's fixes do not cover it.
What failed
| Tag |
Run |
Step |
Error |
| v7.21.4 |
35270645888 |
Upload assets to draft release |
HTTP 500: Error saving asset (https://uploads.github.com/.../assets?...name=ferrflow-darwin-x64.tar.gz.sigstore.json) |
| v7.21.5 |
35503666674 |
Install cosign |
curl: (35) Recv failure: Connection reset by peer fetching cosign v3.1.3 |
Both are transient network noise, seconds long, on someone else's infrastructure.
Why #719 did not catch it
#719 added sign_with_retry, 3 attempts with backoff, to both signing steps. That is the step that failed in #718 and it held. These two failures landed on steps either side of it, and neither has a retry:
Install cosign is sigstore/cosign-installer, which downloads the binary itself. It runs before the retried signing step, so a reset connection there kills the job with the retry logic still untouched further down.Upload assets to draft release is a bare loop, gh release upload "$TAG" "$file" --clobber, with no retry. It is already idempotent thanks to --clobber, so it is the cheapest possible thing to retry. v7.21.4 died on the 8th of 20 assets.
Install cosign for Docker signing in the docker job uses the same unretried action, so it has the same exposure.
Why the orphan sweep did not save it either
#719 also added "Publish any orphaned older draft releases", which is exactly the right idea. It sits inside the upload job, after the steps above. When that job dies, the sweep never runs, so it can only rescue an orphan on a later release whose own upload job survives every fragile step first.
Two consecutive releases failed in that one job, so the sweep never fired for either. It recovers from one bad run followed by a good one; it cannot recover from a run of bad luck, which is what happened.
Why nobody noticed for three days
Every publish job is needs: upload, so when that job fails, crates.io, npm, @ferrflow/wasm, @ferrflow/doc and Docker are all reported as skipped rather than failed. The run goes red on a tag, which blocks no pull request and appears on no dashboard anyone was watching. The releases looked cut: tags existed, GitHub releases existed as drafts, CHANGELOG.md was updated.
It surfaced only because ferrflow.com was still serving 7.21.3 docs, which is a long way downstream from the cause.
What would fix it
- Retry the cosign install, in both
upload and docker. The installer action takes no retry input, so either wrap the step or install the binary with the same 3-attempt loop sign_with_retry already uses.
- Retry
gh release upload per file. --clobber makes it safe to repeat, and a partial upload is exactly the state a retry should recover from.
- Move the orphan sweep somewhere that runs when
upload fails. A separate job with if: always(), or a scheduled sweep, so recovery does not depend on the job that just died. This is the one that turns a bad minute into a self-healing blip instead of a three-day outage.
- Say something when
Publish fails on a tag. A red run on a tag is currently indistinguishable from silence. Even a job that fails loudly with the list of skipped registries would have shortened this from three days to one.
Items 1 and 2 stop these two specific failures. Item 3 is what stops the next one nobody predicted, and item 4 is what stops it being measured in days.
v7.21.4 and v7.21.5 were tagged on 2026-09-17 and 2026-09-20 and neither reached a registry. npm and crates.io sat at 7.21.3 for three days. I re-ran both
Publishruns by hand today and they succeeded unchanged, so nothing was wrong with the code being released.This is the same failure shape as #718, and #719's fixes do not cover it.
What failed
HTTP 500: Error saving asset (https://uploads.github.com/.../assets?...name=ferrflow-darwin-x64.tar.gz.sigstore.json)curl: (35) Recv failure: Connection reset by peerfetching cosign v3.1.3Both are transient network noise, seconds long, on someone else's infrastructure.
Why #719 did not catch it
#719 added
sign_with_retry, 3 attempts with backoff, to both signing steps. That is the step that failed in #718 and it held. These two failures landed on steps either side of it, and neither has a retry:Install cosignissigstore/cosign-installer, which downloads the binary itself. It runs before the retried signing step, so a reset connection there kills the job with the retry logic still untouched further down.Upload assets to draft releaseis a bare loop,gh release upload "$TAG" "$file" --clobber, with no retry. It is already idempotent thanks to--clobber, so it is the cheapest possible thing to retry. v7.21.4 died on the 8th of 20 assets.Install cosign for Docker signingin thedockerjob uses the same unretried action, so it has the same exposure.Why the orphan sweep did not save it either
#719 also added "Publish any orphaned older draft releases", which is exactly the right idea. It sits inside the
uploadjob, after the steps above. When that job dies, the sweep never runs, so it can only rescue an orphan on a later release whose own upload job survives every fragile step first.Two consecutive releases failed in that one job, so the sweep never fired for either. It recovers from one bad run followed by a good one; it cannot recover from a run of bad luck, which is what happened.
Why nobody noticed for three days
Every publish job is
needs: upload, so when that job fails, crates.io, npm,@ferrflow/wasm,@ferrflow/docand Docker are all reported as skipped rather than failed. The run goes red on a tag, which blocks no pull request and appears on no dashboard anyone was watching. The releases looked cut: tags existed, GitHub releases existed as drafts,CHANGELOG.mdwas updated.It surfaced only because ferrflow.com was still serving 7.21.3 docs, which is a long way downstream from the cause.
What would fix it
uploadanddocker. The installer action takes no retry input, so either wrap the step or install the binary with the same 3-attempt loopsign_with_retryalready uses.gh release uploadper file.--clobbermakes it safe to repeat, and a partial upload is exactly the state a retry should recover from.uploadfails. A separate job withif: always(), or a scheduled sweep, so recovery does not depend on the job that just died. This is the one that turns a bad minute into a self-healing blip instead of a three-day outage.Publishfails on a tag. A red run on a tag is currently indistinguishable from silence. Even a job that fails loudly with the list of skipped registries would have shortened this from three days to one.Items 1 and 2 stop these two specific failures. Item 3 is what stops the next one nobody predicted, and item 4 is what stops it being measured in days.