Skip to content

Mark a worker that exits 0 as Completed instead of recreating it forever#181

Merged
Shine-neko merged 1 commit into
mainfrom
fix/worker-exit-zero-completes
Jun 30, 2026
Merged

Mark a worker that exits 0 as Completed instead of recreating it forever#181
Shine-neko merged 1 commit into
mainfrom
fix/worker-exit-zero-completes

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Problem

A worker container that exits 0 (a successful, clean exit) was treated as a crash and recreated every reconcile tick, forever. classify_exit_code mapped 0 to Retry, so apply_unexpected_exits bumped restart_count and the scheduler recreated the container — which, under the default image_pull_policy: Always, re-pulled the image each time.

In production this manifested as a one-shot pg_dump-style container (declared as a worker) looping endlessly: pull image → run → exit 0 → recreate → pull → ... Each iteration did a full image pull plus the dump, saturating the reconcile cycle. Because all deployments of a namespace reconcile together, this starved every other deployment in the kemeter namespace (Grafana among them) of timely reconciliation.

Fix

Treat exit code 0 as terminal Completed, never retryable:

  • classify_exit_code(Some(0)) now returns Terminal(Completed).
  • apply_unexpected_exits short-circuits a clean exit to Completed without incrementing restart_count, so the container is never recreated.

A worker that finished successfully converges to Completed (which the scheduler excludes from reconciliation) instead of looping. Non-zero exits keep their existing retry/fast-fail behaviour (126/127 still fail fast; generic 1, signal kills, and unknown codes stay retryable).

Verification

Deployed to production and confirmed the loop stops:

before after
postgres image pulls / 2 min 13 0
containers recreated / 2 min continuous 0
deployments stuck in creating 2 0

Plus a unit test (clean_exit_completes_without_restart) reproducing the loop: a container exiting 0 lands on Completed with restart_count untouched. Full suite green, cargo fmt/clippy clean.

@Shine-neko Shine-neko merged commit d2c919d into main Jun 30, 2026
5 checks passed
@Shine-neko Shine-neko deleted the fix/worker-exit-zero-completes branch June 30, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant