fix(kourier): extend gateway drain to match axon request timeout (3605s)

[jonoirwinrsa]

Supersedes CerebriumAI/kserving#215 — please close that one. It edited third_party/kourier-latest/kourier.yaml, which is knative/serving's nightly kourier-latest pin used by the e2e harness (version label 20250418-96095402). Production gateways carry app.kubernetes.io/version: 1.18.0 and are created by the operator from this kodata manifest, so #215 would never have reached a cluster.

What

• DRAIN_TIME_SECONDS: 15 → 3605
• terminationGracePeriodSeconds: 30 → 3665

Kourier fronts axon, which runs timeoutSeconds: 3605. The 15s drain cut long-running requests off on rollout. Deployment strategy is already maxUnavailable: 0 / maxSurge: 100%, so replacements are Ready before old pods start draining and capacity is preserved.

Sizing (measured, 100.2M requests / 14d)

response time	count
> 54m (3240s)	13,015
> 60m (3600s)	4,150
> 3665s	3,497
max observed	43,201s

3665s is chosen as the knee, not as full coverage. 8,865 requests sit in the dense 54–60m band against axon's timeout, so a grace period set under an hour would truncate most of them. Above 3665s there are still 3,497 requests (1,089 over 6h) — max-revision-timeout-seconds is 43200, so user revisions can legitimately outlive any practical drain window. Contrary to #215's description, this does not guarantee zero dropped requests; it moves truncation from 0.013% to 0.0035%. Extending to 2h would recover only ~2,300 more while making every node operation take two hours.

Measured on a live gateway

Validated on cortex-dev-virginia. 200 concurrent requests, 45s each, gateway pod deleted 20s in. Identical test either side; only the drain config differs.

	grace / drain	result	avg latency
before	30 / 15	0% success — 200/200 Code -1	35,215ms (truncated at 20s elapsed + 15s drain)
after	3665 / 3605	100% success — 200/200 Code 200	45,021ms (full 45s, no truncation)

Sockets used: 200 (for perfect keepalive, would be 200) in both, so the difference is the drain window, not connection behaviour.

⚠️ Known wart — the drain is a fixed sleep, not a poll

The preStop hook is curl .../drain_listeners?graceful; sleep $DRAIN_TIME_SECONDS. That sleep is unconditional — it does not wait on connections, it just sleeps. Verified on an idle pod with zero traffic: termination took 3,605s exactly.

So every gateway pod termination now takes a fixed ~60 minutes regardless of load — every rollout, node drain and scale-down. maxUnavailable: 0 / maxSurge: 100% means capacity is preserved throughout, so this is cost and slowness rather than risk, but a routine gateway rollout goes from ~1 minute to ~1 hour.

Worth making the hook poll Envoy for downstream_cx_active == 0 and exit early, with DRAIN_TIME_SECONDS as a ceiling rather than a floor. Not done here to keep the change to the two values.

⚠️ Prerequisite — do not roll this out on its own

The Karpenter core-network NodePool (which hosts every gateway) has terminationGracePeriod: 3600s — less than the pod's 3665s. Karpenter then deletes the pod at the start of disruption and grants it only the time remaining to the node deadline, so the drain is truncated to the node's budget rather than the pod's.

Observed live on cortex-dev-virginia while testing this change: pod spec.terminationGracePeriodSeconds: 3665 but metadata.deletionGracePeriodSeconds: 3599. The gateway PDB also sits at disruptionsAllowed=0 (minAvailable: 80%), and Karpenter logged DisruptionBlocked — Pdb prevents pod evictions, confirming eviction depends entirely on that deadline path.

Existing NodeClaims keep the terminationGracePeriod they were created with — raising the NodePool only affects newly created nodes, so long-lived gateway nodes stay mismatched until they are recycled.

Prerequisite PR: CerebriumAI/argocd-manifests#1092 raises core-network to 2h (matching the user-* pools), leaving 3535s of headroom. Merge and sync that first.

Verification

YAML parses (14 docs), grace > drain invariant asserted, diff limited to the two values. go test / go vet ./pkg/reconciler/knativeserving/ingress/... both exit 0. Only the 1.18 manifest is touched — 1.15–1.17 are unused, all clusters run 1.18.0.

🤖 Generated with Claude Code