Honor start_period grace before the readiness rollout deadline

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- Readiness health checks accept a `start_period` grace window: readiness failures during a slow build/boot no longer trip the rollout deadline (effective budget becomes `start_period + RING_ROLLOUT_DEADLINE`), and the value is forwarded to Docker's native `HEALTHCHECK start_period` (#183)
 - containerd runtime over native gRPC: index/entrypoint resolution for multi-arch images, CNI networking, logs, command health checks (#144)
 - Podman runtime, opt-in under `[server.runtime.podman]` (#139)
 - Firecracker microVM runtime (experimental): boot, networking, outbound NAT, restart reconciliation (#142, #146, #147)

documentation/how-to/configure-health-checks.md

@@ -153,6 +153,30 @@ For services that need longer than 10 s to be truly ready (JVM cold start, big i
 - When several readiness checks declare different values, the **maximum** wins, so the gate waits on the slowest probe.
 - A typo (unparseable value) logs a warning and falls back to the 10 s default; rollouts aren't blocked.
 
+### Services that build at boot: tune `start_period`
+
+Some containers build their app *after* they start (e.g. `bun run build` behind Caddy). Their readiness probe legitimately fails for the whole build, which would trip the rollout deadline (`RING_ROLLOUT_DEADLINE`, default `600s`) and fail a perfectly healthy deployment. Set `start_period` on the readiness check to grant a grace window before failures count:
+
+```yaml
+- type: command
+  command: test -f /var/run/kemeter/ready
+  interval: 5s
+  timeout: 2s
+  threshold: 3
+  on_failure: alert
+  readiness: true
+  start_period: "180s"     # 3 min of build/boot before failures count
+```
+
+- The rollout deadline only starts counting **after** the grace period. Effective budget = `start_period + RING_ROLLOUT_DEADLINE`.
+- Same duration syntax as `interval` / `timeout` (`"500ms"`, `"180s"`).
+- Only matters when `readiness: true`.
+- When several readiness checks declare different values, the **maximum** wins.
+- For `command` checks on Docker, the value is also forwarded to the native `HEALTHCHECK start_period`, so Docker won't mark the container `unhealthy` during the build.
+- A typo (unparseable value) logs a warning and is ignored; the grace falls back to zero rather than blocking a slow build.
+
+`start_period` differs from `min_healthy_time`: `start_period` is a grace *before* failures count (build/boot), while `min_healthy_time` is how long the check must stay green *after* it first succeeds (anti-flap).
+
 **Proxy bonus:** a `readiness: true` check of `type: command` is **also** translated into a native Docker `HEALTHCHECK`. [Sozune](https://sozune.kemeter.io) (the companion proxy) and other label-aware proxies gate on `Status: healthy`, so they won't route traffic to the new container while the readiness command fails. See [how-to: expose HTTP traffic](/documentation/how-to/expose-http-traffic) for the integration, or [Health checks design → proxy integration](/documentation/concepts/health-checks-design#proxy-integration) for the why.
 
 For HTTP readiness with proxy gating, wrap the probe in a shell command:

documentation/reference/manifest.md

@@ -391,6 +391,7 @@ health_checks:
 | `on_failure` | yes | `restart` (recreate the instance), `stop` (delete the deployment), or `alert` (log only). |
 | `readiness` | no (default `false`) | When `true`, this check gates rolling updates and (for `command` on Docker) is translated into a native `HEALTHCHECK`. See [health checks (design) → the readiness gate](/documentation/concepts/health-checks-design#the-readiness-gate). |
 | `min_healthy_time` | no (default `10s`) | Anti-flap window for the readiness gate: the check must be green for this long before the parent is drained. Per-check; the scheduler takes the maximum across readiness checks. Ignored when `readiness: false`. Same syntax as `interval` / `timeout`. |
+| `start_period` | no (default none) | Build/boot grace period during which readiness failures don't count. The rollout deadline (`RING_ROLLOUT_DEADLINE`) only starts *after* this window, and it is forwarded to Docker's native `HEALTHCHECK start_period`. Per-check; the scheduler takes the maximum across readiness checks. A malformed value logs a warning and is ignored. Ignored when `readiness: false`. Same syntax as `interval` / `timeout`. |
 
 ### Type-specific fields
 

src/commands/apply.rs

@@ -188,6 +188,8 @@ enum HealthCheck {
         readiness: bool,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
     Http {
         url: String,
@@ -200,6 +202,8 @@ enum HealthCheck {
         readiness: bool,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
     Command {
         command: String,
@@ -212,6 +216,8 @@ enum HealthCheck {
         readiness: bool,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
 }
 

src/models/health_check.rs

@@ -36,6 +36,17 @@ pub(crate) enum HealthCheck {
         /// to the scheduler's built-in window when unset.
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        /// Grace period before this check's failures count against the
+        /// deployment. A deployment that builds at boot (e.g. `bun run
+        /// build` behind Caddy) isn't ready for many seconds, and a
+        /// readiness probe running during that window would fail purely
+        /// because the app hasn't started serving yet. The scheduler's
+        /// readiness deadline (`rollout_deadline_exceeded`) starts counting
+        /// only after this period, and it is forwarded to Docker's native
+        /// `HEALTHCHECK start_period`. Parsed by `parse_duration`. Ignored
+        /// when `readiness = false`.
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
     #[serde(rename = "http")]
     Http {
@@ -49,6 +60,8 @@ pub(crate) enum HealthCheck {
         readiness: bool,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
     #[serde(rename = "command")]
     Command {
@@ -62,6 +75,8 @@ pub(crate) enum HealthCheck {
         readiness: bool,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         min_healthy_time: Option<String>,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        start_period: Option<String>,
     },
 }
 
@@ -174,6 +189,17 @@ impl HealthCheck {
             } => min_healthy_time.as_deref(),
         }
     }
+
+    /// Returns the configured build/boot grace period for this check, if any.
+    /// Only meaningful when `is_readiness() == true`; the scheduler takes the
+    /// maximum across readiness checks so the slowest-starting one wins.
+    pub(crate) fn start_period(&self) -> Option<&str> {
+        match self {
+            HealthCheck::Tcp { start_period, .. }
+            | HealthCheck::Http { start_period, .. }
+            | HealthCheck::Command { start_period, .. } => start_period.as_deref(),
+        }
+    }
 }
 
 impl Default for HealthCheck {
@@ -186,6 +212,7 @@ impl Default for HealthCheck {
             on_failure: FailureAction::Restart,
             readiness: false,
             min_healthy_time: None,
+            start_period: None,
         }
     }
 }

src/runtime/docker/container.rs

@@ -110,7 +110,11 @@ fn build_health_config(health_checks: &[HealthCheck]) -> Option<HealthConfig> {
         interval: Some(to_nanos(hc.interval())),
         timeout: Some(to_nanos(hc.timeout())),
         retries: Some(hc.threshold() as i64),
-        start_period: None,
+        // Grace period before Docker's native HEALTHCHECK counts a failure —
+        // covers a slow in-container build (e.g. `bun run build` behind Caddy)
+        // that isn't serving yet. Mirrors the scheduler-side readiness deadline
+        // grace (`start_period_for`). None (Docker's default) when unset.
+        start_period: hc.start_period().map(to_nanos),
         start_interval: None,
     })
 }
@@ -993,6 +997,7 @@ mod tests {
             on_failure: FailureAction::Alert,
             readiness: false,
             min_healthy_time: None,
+            start_period: None,
         }];
         assert!(build_health_config(&hcs).is_none());
     }
@@ -1009,6 +1014,7 @@ mod tests {
             on_failure: FailureAction::Alert,
             readiness: true,
             min_healthy_time: None,
+            start_period: None,
         }];
         assert!(build_health_config(&hcs).is_none());
     }
@@ -1023,6 +1029,7 @@ mod tests {
             on_failure: FailureAction::Alert,
             readiness: true,
             min_healthy_time: None,
+            start_period: None,
         }];
         assert!(build_health_config(&hcs).is_none());
     }
@@ -1037,6 +1044,7 @@ mod tests {
             on_failure: FailureAction::Alert,
             readiness: true,
             min_healthy_time: None,
+            start_period: None,
         }];
         let cfg = build_health_config(&hcs).expect("should translate");
         assert_eq!(
@@ -1049,6 +1057,25 @@ mod tests {
         assert_eq!(cfg.interval, Some(10_000_000_000));
         assert_eq!(cfg.timeout, Some(5_000_000_000));
         assert_eq!(cfg.retries, Some(3));
+        // No start_period declared → Docker's default (None).
+        assert_eq!(cfg.start_period, None);
+    }
+
+    #[test]
+    fn build_health_config_forwards_start_period() {
+        let hcs = vec![HealthCheck::Command {
+            command: "test -f /var/run/kemeter/ready".to_string(),
+            interval: "10s".to_string(),
+            timeout: "5s".to_string(),
+            threshold: 3,
+            on_failure: FailureAction::Alert,
+            readiness: true,
+            min_healthy_time: None,
+            start_period: Some("180s".to_string()),
+        }];
+        let cfg = build_health_config(&hcs).expect("should translate");
+        // The grace period reaches Docker's native HEALTHCHECK in nanoseconds.
+        assert_eq!(cfg.start_period, Some(180_000_000_000));
     }
 
     #[test]
@@ -1062,6 +1089,7 @@ mod tests {
                 on_failure: FailureAction::Alert,
                 readiness: true,
                 min_healthy_time: None,
+                start_period: None,
             },
             HealthCheck::Command {
                 command: "/usr/local/bin/ready.sh".to_string(),
@@ -1071,6 +1099,7 @@ mod tests {
                 on_failure: FailureAction::Alert,
                 readiness: true,
                 min_healthy_time: None,
+                start_period: None,
             },
         ];
         let cfg = build_health_config(&hcs).expect("should translate the command HC");

src/scheduler/health_checker.rs

@@ -414,6 +414,7 @@ mod tests {
                 on_failure: FailureAction::Restart,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -447,6 +448,7 @@ mod tests {
                 on_failure: FailureAction::Restart,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -491,6 +493,7 @@ mod tests {
                 on_failure: FailureAction::Restart,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -523,6 +526,7 @@ mod tests {
                 on_failure: FailureAction::Restart,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -555,6 +559,7 @@ mod tests {
                 on_failure: FailureAction::Restart,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -602,6 +607,7 @@ mod tests {
                 on_failure: FailureAction::Stop,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -636,6 +642,7 @@ mod tests {
                 on_failure: FailureAction::Alert,
                 readiness: false,
                 min_healthy_time: None,
+                start_period: None,
             }],
         );
 
@@ -697,6 +704,7 @@ mod tests {
             on_failure: FailureAction::Restart,
             readiness: true,
             min_healthy_time: None,
+            start_period: None,
         }
     }
 
@@ -709,6 +717,7 @@ mod tests {
             on_failure: FailureAction::Restart,
             readiness: false,
             min_healthy_time: None,
+            start_period: None,
         }
     }