Skip to content

Always list all container states so finished jobs are visible#182

Merged
Shine-neko merged 1 commit into
mainfrom
fix/list-instances-all-flag
Jun 30, 2026
Merged

Always list all container states so finished jobs are visible#182
Shine-neko merged 1 commit into
mainfrom
fix/list-instances-all-flag

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Problem

build_list_options set the Docker list_containers all flag conditionally:

let all = !matches!(status, "all");

For a status = "all" lookup this yields all = false, so Docker returns running containers only. Every exited container is invisible to that listing — the opposite of what the function's own comment promises ("Always list every container and filter by state client-side").

This breaks the job path. handle_job_deployment calls list_instances(deployment.id, "all") to find its container and converge to Completed. A pg_dump backup job (kind: job, replicas: 1) exits ~0.5s after starting; on the next scheduler tick there are zero running containers, so the inverted flag makes the listing empty even though the finished container is right there with a matching ring_deployment label. Ring concludes the job has no instance, recreates a new container, and sets Running — every tick, forever.

In production this looped ~30 backup deployments and piled up ~8900 Exited(0) postgres:18-alpine containers, saturating the reconcile cycle (which slowed everything else, including unrelated redeploys).

Regression from a Podman fix (605a25c, 2026-06-19) that correctly stopped a different loop but tied all to the filter.

Fix

Make the flag unconditional:

ListContainersOptionsBuilder::new().all(true).build()

Listing is exhaustive; matches_status already does per-state filtering client-side (including "all", "active", "exited", ...), so this restores the intended behaviour without reintroducing the Podman created-container regression (that fix lives in matches_status's active arm, untouched here).

Verification

  • New regression test list_options_always_request_all_states asserts all == true for every filter. Confirmed it fails when the buggy expression is restored and passes with the fix.
  • Deployed to production: container recreation stopped immediately (0 recreations post-deploy vs ~30/cycle before), and the Exited-container count stopped growing.
  • Full suite green, cargo fmt/clippy clean.

Follow-ups (out of scope): reap accumulated exited job containers; consider making list_instances("all").first() pick the most-recent container explicitly rather than relying on Docker list order.

@Shine-neko Shine-neko merged commit 96f9b6a into main Jun 30, 2026
5 checks passed
@Shine-neko Shine-neko deleted the fix/list-instances-all-flag branch June 30, 2026 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant