feat(chartreuse): post-deployment Job restoring Deployments stranded at 0 replicas#885
Merged
Conversation
…at 0 replicas With upgradeBeforeDeployment, the migration Job intentionally skips start_pods() and relies on the deployment that follows to restore replicas. That works for Deployments whose chart sets spec.replicas, but HPA-managed ones omit it and an HPA whose target is at 0 replicas with minReplicas >= 1 is ScalingDisabled: they stayed stranded at 0 after every migration. Render a chartreuse-restore Job as an ArgoCD PostSync hook (argocd mode only, gated by the new restorePods value) that re-enables HPAs and scales the stopped Deployments back up once the new pod template is applied. Grant patch on deployments so chartreuse can track the Deployments it stopped via an annotation. Requires a chartreuse image >= 7.0 (appVersion bumped accordingly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Canary validated on pythie-cayzn-staging-demo: the job-restore.yaml PostSync hook rendered by this chart ran after a real upgradeBeforeDeployment migration and restored the HPA-managed workers that the migration had stopped (annotations cleared, HPAs re-armed, minReplicas enforced). Companion package PR: wiremind/chartreuse#55 (validated together). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
upgradeBeforeDeployment, the chartreuse migration Job skipsstart_pods()and relies on the deployment that follows to restore replicas. HPA-managed Deployments omitspec.replicasin their chart, so nothing ever restores them, and an HPA whose target is at 0 replicas withminReplicas >= 1isScalingDisabledforever: they stay stranded at 0 after every migration (hitpythie-cayzn-staging-demoon 2026-07-07).Fix
chartreuse-restoreJob rendered as an ArgoCD PostSync hook (argocd mode only, gated by the newrestorePodsvalue +stopPods+upgradeBeforeDeployment): runs once the new pod template is applied, so it only ever starts new-version pods. It re-enables HPAs and scales back up the Deployments that the migration stopped — tracked via thewiremind.io/stopped-byannotation, so deliberately-stopped Deployments are left alone.patchondeployments(needed for the annotation).-ephemeralresource set is deleted right after the migration hook, so the restore Job is not rendered there (documented in the template header); Helm-mode behavior is unchanged.Version coupling
appVersionbumped to 7.0.0: the restore entrypoint ships in chartreuse 7.0.0 (wiremind/chartreuse#55). App images must be rebuilt with it before picking up this chart version —ensure_safe_runaborts on a major.minor mismatch, and the restore Job's entrypoint doesn't exist in older images.Companion PR: wiremind/chartreuse#55
🤖 Generated with Claude Code