Always list all container states so finished jobs are visible#182
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
build_list_optionsset the Dockerlist_containersallflag conditionally:For a
status = "all"lookup this yieldsall = false, so Docker returns running containers only. Every exited container is invisible to that listing — the opposite of what the function's own comment promises ("Always list every container and filter by state client-side").This breaks the job path.
handle_job_deploymentcallslist_instances(deployment.id, "all")to find its container and converge toCompleted. Apg_dumpbackup job (kind: job,replicas: 1) exits ~0.5s after starting; on the next scheduler tick there are zero running containers, so the inverted flag makes the listing empty even though the finished container is right there with a matchingring_deploymentlabel. Ring concludes the job has no instance, recreates a new container, and setsRunning— every tick, forever.In production this looped ~30 backup deployments and piled up ~8900 Exited(0)
postgres:18-alpinecontainers, saturating the reconcile cycle (which slowed everything else, including unrelated redeploys).Regression from a Podman fix (
605a25c, 2026-06-19) that correctly stopped a different loop but tiedallto the filter.Fix
Make the flag unconditional:
Listing is exhaustive;
matches_statusalready does per-state filtering client-side (including"all","active","exited", ...), so this restores the intended behaviour without reintroducing the Podmancreated-container regression (that fix lives inmatches_status'sactivearm, untouched here).Verification
list_options_always_request_all_statesassertsall == truefor every filter. Confirmed it fails when the buggy expression is restored and passes with the fix.cargo fmt/clippyclean.Follow-ups (out of scope): reap accumulated exited job containers; consider making
list_instances("all").first()pick the most-recent container explicitly rather than relying on Docker list order.