Skip to content

fix(autodev): serialize concurrent runs to stop dev-branch rollback race - #490

Merged
monotek merged 1 commit into
mainfrom
fix/autodev-concurrency-race
Aug 17, 2026
Merged

fix(autodev): serialize concurrent runs to stop dev-branch rollback race#490
monotek merged 1 commit into
mainfrom
fix/autodev-concurrency-race

Conversation

@monotek

@monotek monotek commented Aug 13, 2026

Copy link
Copy Markdown
Member

Background

Observed in Staffbase/mops around the merge of PR #18031:

  1. dev label added to the PR → Autodev rebuilds dev (base + PR) → chart bumped to 0.8.0, rolled out to the dev cluster. ✅
  2. PR merged into main.
  3. Moments later, the dev cluster rolled back to the pre-merge chart version (0.7.6).
  4. Shortly after that, dev updated again to 0.8.0.

A merge should just leave dev at the version it already had — the rollback-then-recovery blip shouldn't happen.

Companion fix opened in the action this workflow calls: Staffbase/autodev-action#429. See "How this relates to autodev-action#429" below for why both are needed.

Root cause

The autodev job in template_autodev.yml rebuilds the target branch (e.g. dev) from scratch — base + every currently labeled PR merged together — and force-pushes the result. The job has:

  • No concurrency control, and
  • Triggers on push (any push to a non-target branch) and on pull_request: labeled | unlabeled | closed.

Merging a PR fires both a push event (the merge commit landing on main) and a pull_request: closed event, essentially simultaneously. Both satisfy the job's if: condition, so two autodev runs start concurrently with nothing serializing them. Callers that also run Flux Image Automation (like mops) make this worse: those bots commit to main very frequently, and each such push independently re-triggers autodev too — so there's almost always a run in flight, raising the odds one of them overlaps with a merge.

Because the target branch is rebuilt-and-force-pushed rather than updated incrementally, whichever run's push lands last wins — not whichever run reflects the newest base. A run that checked out base before the merge commit was live can finish and force-push after a run that checked out the post-merge base, silently reverting branch to stale content until the next trigger (any subsequent push/label event) rebuilds it correctly again. That's the observed rollback-then-recovery blip.

This repo already has precedent for the fix elsewhere in the org: Staffbase/mops's update-cluster-base-branch-dev.yml does the same rebuild-and-force-push pattern and guards it with:

concurrency:
  group: update-cluster-base-branch-dev
  cancel-in-progress: true

template_autodev.yml never got the equivalent.

Fix

Add a concurrency group to the autodev job, scoped per base/branch, with cancel-in-progress: true:

concurrency:
  group: autodev-${{ inputs.base }}-${{ inputs.branch }}
  cancel-in-progress: true

This serializes overlapping runs and cancels a stale in-flight run as soon as a newer trigger shows up, so a run built from an older base snapshot can no longer force-push over a newer one.

How this relates to autodev-action#429

While digging into this, we found Staffbase/autodev-action (pinned here as v2.9.0) already guards the destination ref correctly: it pushes with --force-with-lease against a snapshot of origin/${branch} taken at run start, so a concurrent run can never silently overwrite a fresher push — that race is already closed at the action level.

What it didn't check is whether base itself was still current when the content being pushed was built. Staffbase/autodev-action#429 closes that gap: it snapshots origin/${base} too and re-verifies right before the push, skipping (with a warning) if base moved during the run.

These two fixes are complementary, not overlapping:

  • autodev-action#429 is the correctness backstop: even if this workflow-level concurrency fix weren't merged, a run whose base went stale would now catch itself before pushing bad content.
  • This PR reduces how many overlapping runs exist in the first place, which matters independently of that backstop: without it, every push/label/close event still spins up a full checkout-merge-push run even when a newer trigger immediately supersedes it, wasting CI time and spamming the Actions log with "skipped, base moved" warnings across every caller of this workflow. It also shrinks the residual window between autodev-action's freshness re-check and its actual push call, since fewer stale runs are ever racing to reach that point at all.

Recommend merging both — neither fully subsumes the other. Note: template_autodev.yml pins autodev-action by exact SHA/tag, and this repo's dependabot.yml ignores patch-version bumps for GitHub Actions, so the pin here won't update on its own unless autodev-action#429 ships as a minor release (or the pin is bumped by hand).

Note / limitation: GitHub Actions cancellation isn't always instantaneous mid-step — if a stale run is already inside its push step when cancellation is requested, it may still complete before being killed. This change closes the race in the common case and matches the existing mitigation pattern used elsewhere in the org, but combined with autodev-action#429's freshness check, the two together are a much stronger guarantee than either alone.

Test plan

  • Confirm the reusable workflow still runs correctly end-to-end for a labeled PR in a downstream repo (e.g. mops)
  • Confirm two rapid-fire triggers (e.g. label PR, then quickly push an unrelated commit to main) result in only the later run's force-push surviving, with the earlier run showing as cancelled in the Actions run list

The autodev job rebuilds the target branch from scratch and force-pushes it,
but has no concurrency control while triggering on both push and PR
labeled/unlabeled/closed events. Merging a PR fires push and closed nearly
simultaneously, letting two unserialized runs force-push over each other;
whichever finishes last wins, not whichever reflects the newest base,
producing a visible rollback until the next trigger corrects it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@monotek monotek closed this Aug 13, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 13, 2026
@monotek monotek reopened this Aug 13, 2026
@monotek
monotek requested a lite review from Copilot August 13, 2026 12:26
@monotek
monotek marked this pull request as ready for review August 14, 2026 14:55
@monotek
monotek requested a review from a team as a code owner August 14, 2026 14:55
@monotek
monotek requested review from axdotl and flaxel August 14, 2026 14:55
@monotek
monotek merged commit 7496385 into main Aug 17, 2026
10 checks passed
@monotek
monotek deleted the fix/autodev-concurrency-race branch August 17, 2026 07:52
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants