fix(autodev): serialize concurrent runs to stop dev-branch rollback race - #490
Merged
Conversation
The autodev job rebuilds the target branch from scratch and force-pushes it, but has no concurrency control while triggering on both push and PR labeled/unlabeled/closed events. Merging a PR fires push and closed nearly simultaneously, letting two unserialized runs force-push over each other; whichever finishes last wins, not whichever reflects the newest base, producing a visible rollback until the next trigger corrects it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
flaxel
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Observed in
Staffbase/mopsaround the merge of PR #18031:devlabel added to the PR → Autodev rebuildsdev(base + PR) → chart bumped to0.8.0, rolled out to the dev cluster. ✅main.devcluster rolled back to the pre-merge chart version (0.7.6).devupdated again to0.8.0.A merge should just leave
devat the version it already had — the rollback-then-recovery blip shouldn't happen.Companion fix opened in the action this workflow calls: Staffbase/autodev-action#429. See "How this relates to autodev-action#429" below for why both are needed.
Root cause
The
autodevjob intemplate_autodev.ymlrebuilds the targetbranch(e.g.dev) from scratch —base+ every currently labeled PR merged together — and force-pushes the result. The job has:push(any push to a non-target branch) and onpull_request: labeled | unlabeled | closed.Merging a PR fires both a
pushevent (the merge commit landing onmain) and apull_request: closedevent, essentially simultaneously. Both satisfy the job'sif:condition, so twoautodevruns start concurrently with nothing serializing them. Callers that also run Flux Image Automation (likemops) make this worse: those bots commit tomainvery frequently, and each such push independently re-triggersautodevtoo — so there's almost always a run in flight, raising the odds one of them overlaps with a merge.Because the target branch is rebuilt-and-force-pushed rather than updated incrementally, whichever run's push lands last wins — not whichever run reflects the newest
base. A run that checked outbasebefore the merge commit was live can finish and force-push after a run that checked out the post-mergebase, silently revertingbranchto stale content until the next trigger (any subsequent push/label event) rebuilds it correctly again. That's the observed rollback-then-recovery blip.This repo already has precedent for the fix elsewhere in the org:
Staffbase/mops'supdate-cluster-base-branch-dev.ymldoes the same rebuild-and-force-push pattern and guards it with:template_autodev.ymlnever got the equivalent.Fix
Add a
concurrencygroup to theautodevjob, scoped perbase/branch, withcancel-in-progress: true:This serializes overlapping runs and cancels a stale in-flight run as soon as a newer trigger shows up, so a run built from an older
basesnapshot can no longer force-push over a newer one.How this relates to autodev-action#429
While digging into this, we found
Staffbase/autodev-action(pinned here asv2.9.0) already guards the destination ref correctly: it pushes with--force-with-leaseagainst a snapshot oforigin/${branch}taken at run start, so a concurrent run can never silently overwrite a fresher push — that race is already closed at the action level.What it didn't check is whether
baseitself was still current when the content being pushed was built. Staffbase/autodev-action#429 closes that gap: it snapshotsorigin/${base}too and re-verifies right before the push, skipping (with a warning) ifbasemoved during the run.These two fixes are complementary, not overlapping:
basewent stale would now catch itself before pushing bad content.Recommend merging both — neither fully subsumes the other. Note:
template_autodev.ymlpins autodev-action by exact SHA/tag, and this repo'sdependabot.ymlignores patch-version bumps for GitHub Actions, so the pin here won't update on its own unless autodev-action#429 ships as a minor release (or the pin is bumped by hand).Note / limitation: GitHub Actions cancellation isn't always instantaneous mid-step — if a stale run is already inside its push step when cancellation is requested, it may still complete before being killed. This change closes the race in the common case and matches the existing mitigation pattern used elsewhere in the org, but combined with autodev-action#429's freshness check, the two together are a much stronger guarantee than either alone.
Test plan
mops)main) result in only the later run's force-push surviving, with the earlier run showing as cancelled in the Actions run list