Skip to content

Honor start_period grace before the readiness rollout deadline#183

Merged
Shine-neko merged 4 commits into
mainfrom
fix/readiness-deadline-honors-start-period
Jul 3, 2026
Merged

Honor start_period grace before the readiness rollout deadline#183
Shine-neko merged 4 commits into
mainfrom
fix/readiness-deadline-honors-start-period

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

What

Readiness health checks now accept an optional start_period grace window. Some containers legitimately fail readiness for their whole boot because they build their app after starting (e.g. bun run build behind Caddy). Without a grace, a short RING_ROLLOUT_DEADLINE fails a perfectly healthy slow-building app the instant its build outlasts the deadline.

With start_period, the rollout deadline only starts counting after the grace window. The effective budget becomes start_period + RING_ROLLOUT_DEADLINE. For command checks on Docker, the value is also forwarded to the native HEALTHCHECK start_period.

health_checks:
  - type: command
    command: test -f /var/run/kemeter/ready
    interval: 5s
    timeout: 2s
    threshold: 3
    on_failure: alert
    readiness: true
    start_period: "180s"     # 3 min of build/boot before failures count
  • Per-check; the scheduler takes the maximum across readiness checks.
  • A malformed value logs a warning and is ignored (grace falls back to zero rather than blocking a slow build).
  • Ignored when readiness: false. Same duration syntax as interval / timeout.

Bug caught by e2e

The apply client re-declares its own HealthCheck struct to parse YAML, and it was missing the new field — so start_period was silently dropped before ever reaching the API, leaving the feature inert. Cargo tests build the deployment in memory and never exercise the apply → API → DB round-trip, so they passed while the feature did nothing. The e2e (which drives the real binary) surfaced it: the stored health_checks JSON had no start_period, and a deployment with start_period: 40s still failed at the bare 10s deadline. Fixed by adding the field to the apply-side struct.

Tests

  • tests/e2e/docker/t37_readiness_start_period.sh (new): with a never-ready probe and a short deadline, proves a no-grace deployment fails at the deadline, a start_period: 40s deployment survives well past it, and the deferred safety valve still fires once the budget elapses.
  • 730 cargo tests pass; t22_readiness_rolling_update.sh still green (no regression on the readiness gate).

@Shine-neko Shine-neko merged commit 539553d into main Jul 3, 2026
5 checks passed
@Shine-neko Shine-neko deleted the fix/readiness-deadline-honors-start-period branch July 3, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant