Skip to content

fix: delete a garbage collected Job's Pods with it - #1104

Closed
frankvollenweider wants to merge 1 commit into
jenkins-x-plugins:mainfrom
frankvollenweider:fix-gc-jobs-orphaned-pods
Closed

frankvollenweider wants to merge 1 commit into
jenkins-x-plugins:mainfrom
frankvollenweider:fix-gc-jobs-orphaned-pods

Conversation

@frankvollenweider

Copy link
Copy Markdown

Fixes #1103

jx gitops gc jobs deleted Jobs with empty DeleteOptions. For batch/v1 Jobs, the API server then keeps the Job's Pods, and it warns about this on every delete:

Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning

The orphaned Pods have no ownerReferences, so nothing collects them afterwards. With the default jx-gcjobs CronJob (gc jobs --namespace jx-git-operator), every boot Job leaves one completed jx-boot-* Pod behind. jx-gcpods covers only jx, so the Pods build up until the nodes are replaced. On our cluster that was 74 orphans in six days.

Change

  • pkg/cmd/gc/jobs/jobs.go: delete with PropagationPolicy: Background, the same as kubectl delete job does. The garbage collector then removes the Pods together with the Job.
  • pkg/cmd/gc/jobs/jobs_test.go: new test, since the command had none. It checks that only the Job past --age is deleted and that the delete carries background propagation. The test fails without the fix.

go vet and go test ./pkg/cmd/gc/jobs/ pass with Go 1.26.

🤖 Generated with Claude Code

gc jobs deleted Jobs with empty DeleteOptions. For batch/v1 Jobs the API
server then orphans the Job's Pods, and says so in a warning on every
delete. The orphaned Pods have no ownerReferences, so nothing collects
them afterwards. With the default jx-gcjobs CronJob this leaves one
completed jx-boot Pod in jx-git-operator per boot Job, and they
accumulate until the nodes are replaced.

Delete with background propagation, as kubectl delete job does, so the
garbage collector removes the Pods together with the Job.

Fixes jenkins-x-plugins#1103

Signed-off-by: frankvollenweider <frank.vollenweider@wolkenreich.ch>
@jenkins-x-bot

Copy link
Copy Markdown
Contributor

Hi @frankvollenweider. Thanks for your PR.

I'm waiting for a jenkins-x-plugins member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

@jenkins-x-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
The changes made require 1 more approval(s).
To complete the pull request process, please assign mentlak0
You can assign the PR to them by writing /assign @mentlak0 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@frankvollenweider

Copy link
Copy Markdown
Author

Closing: as discussed in #1103, the boot Jobs already clean themselves up via ttlSecondsAfterFinished, so the better fix is to not run jx-gcjobs at all rather than change gc jobs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gc jobs: deleted Jobs leave their Pods behind (no propagationPolicy)

2 participants