Skip to content

Reset restart_count for host-network workers stuck in Creating with a live container#185

Merged
Shine-neko merged 1 commit into
mainfrom
fix/healthy-window-creating-instances
Jul 4, 2026
Merged

Reset restart_count for host-network workers stuck in Creating with a live container#185
Shine-neko merged 1 commit into
mainfrom
fix/healthy-window-creating-instances

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Problem

A worker in network: host mode can never satisfy Ring's readiness checks (its container has no resolvable address), so it stays Creating for its entire life even though its container runs fine.

The anti-flap window that refills restart_count only reset the count when the deployment reached Running. A host-network worker therefore accumulated a monotonic restart_count: its retry backoff climbed to the 60s cap and never came back down. Every redeploy got progressively slower (observed: ~5 min to recreate a container after a drop).

Fix

Treat a deployment as healthy-running for the anti-flap window whenever it actually has a live container, not only when it reached Running: Running, or Creating with a non-empty instance list. Extracted into a small pure has_live_container() helper.

Impact

Measured in production after deploying this fix: container recreation after a drop dropped from ~5 min to ~20s.

Tests

Added 4 unit tests covering has_live_container (Running, Creating+instance, Creating without instance, other states). Full suite: 735 passed.

@Shine-neko Shine-neko force-pushed the fix/healthy-window-creating-instances branch from 5196d14 to d651dca Compare July 3, 2026 23:44
@Shine-neko Shine-neko force-pushed the fix/healthy-window-creating-instances branch from d651dca to 8a859ca Compare July 4, 2026 00:03
@Shine-neko Shine-neko merged commit b25b8a0 into main Jul 4, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant