feat(api): gate apply start on confirmed sibling PR check holds

[aparajon]

Why this matters

Holding sibling PR checks is only a guardrail if it happens before the apply changes anything. A multi-hour copy/cutover started from the CLI must not race the flip: if the holds land late, a sibling PR can merge on a green check the apply is about to invalidate. This PR makes the stored holds a hard precondition of the apply itself — and only the stored holds. The gate waits on storage-only writes, never on the code-host rendering of them, so a code-host outage can never block an apply — least of all the CLI apply mitigating an incident. It fails closed on any storage uncertainty. Stack 6/7, on top of #940.

What it does

• Preflight gate (gateApplyStartOnCheckPreflight), run when a driver claims an apply, before engine work:
  • No merge gate consumer registered (no code-host runtime) → skip; nothing to hold.
  • Apply has no tasks (a plan with no diff) → skip; an apply with no diff changes nothing, so there is nothing to hold against.
  • Otherwise: record a durable preflight request, kick the processor, and wait for the stored holds — holds_recorded_at, or a completed request for preflights coalesced into a same-target sibling's fan-out. The code-host rendering (Check Run update, hold comment) retries separately and never blocks the start. A terminally failed request is re-armed with ReopenForRetry and re-kicked.
  • Timeout or storage error → the drive attempt is abandoned and the apply stays claimable; the gate never converts uncertainty into a started apply. Because the hold phase is storage-only, a sustained timeout points at storage or the processor — never at the code host.
• Settle on every terminal state: completed applies always record a settle; failed/cancelled/errored applies record one when a preflight exists, so holds are always released by a re-plan against the real schema — never by cleanup alone.
• Gate outcome metric (schemabot.merge_gate.preflight_gate_total with passed / passed_render_pending / timeout / error): passed_render_pending means the apply started on stored holds while the code-host rendering is still retrying — expected and healthy during a code-host outage; a sustained timeout rate means the processor is not draining or storage is failing.
• The core stays code-host neutral. The gate keys off the registered merge gate consumer and durable request state — never a GitHub type. Core-layer (pkg/api, pkg/storage, pkg/metrics) comments, logs, and metric docs describe that contract; GitHub vocabulary lives only in the adapter (pkg/webhook, pkg/github).
• Outage proof end to end: an integration test drives a real apply to terminal success while every GitHub call returns 503 — the sibling's green stored check still flips action-required before engine work, the apply completes, and the render request stays retryable for when GitHub recovers.
• Test harness now mirrors production: the default webhook integration handler starts the merge gate processor (the gate requires one), with an explicit no-processor constructor for tests that drive the drain lifecycle manually.

 driver claims apply
        │
        ▼
 consumer registered? ──no──► start engine work (no code-host runtime)
        │yes
 apply has tasks? ────no────► start engine work (no diff, nothing held)
        │yes
 record preflight ──► kick processor ──► wait for STORED holds
        │                                    │ (storage-only; never
        │ storage error / timeout            │  the code-host render)
        ▼                                    ▼
 abandon drive attempt                 start engine work
 (apply stays claimable,               (sibling stored checks held;
  fail closed)                          Check Run + comment render
                                        retries independently)

Closing the loop: a commit pushed to a sibling PR while the apply is mid-flight would re-plan against the pre-apply schema and could mint a fresh green check — #942 closes that by storing such checks born held.

How it moves us toward the northstar

An apply's first observable effect is now telling every affected PR "this target is changing" — before a single row moves. Merge decisions and schema changes stop being able to race each other, and the dependency points the safe direction: the code host depends on SchemaBot's stored truth, never the other way around.

The chain: #867 (storage) → #868 (drive-tail recording) → #866 (settle re-plan processor) → #939 (request kinds + hold storage) → #940 (preflight hold fan-out) → #941 (apply-start gate) → #942 (plan-time holds). Merges bottom-up; each PR retargets to main as its base merges.

🤖 Generated with Claude Code

[Kiran01bm]

🤖 Review findings - created by Kiran's code review agent - for pull/941, 44166f4.

Verdict: 9 findings — 2 blocking (stop-then-restart bypasses the gate with holds released; stop reconciliation gated on GitHub), 3 non-blocking, 4 suggestions.

Blocking

1. A stopped-then-restarted apply passes the completed-preflight fast path with its sibling holds already released, and the completion settle is then swallowed as a duplicate — siblings are never re-planned against the changed schema. The fast path at operator.go#L1160 — if req != nil && req.State == storage.MergeGateCompleted { — treats a completed preflight as permanently valid, but this PR's own drive-tail settle breaks that invariant. Chain: (a) stopped is a terminal apply state (metadata.go#L86-L89), so stopping a preflighted apply mid-copy records a settle via the new drive-tail branch (operator.go#L1064-L1088); (b) the settle fan-out does not defer, because HasActivePreflightedApplyOnTarget counts only non-terminal applies (merge_gate_requests.go#L375, used at merge_gate.go#L522) — sibling checks are re-planned to live green verdicts, holds released; (c) the stopped apply is re-claimable on the same row via FindNextApply's stopped+pending-start arm (applies.go#L1270-L1277, args at applies.go#L1234-L1236; terminal-failed applies via ReapplyFailed, storage.go#L591-L597); (d) the resumed drive re-enters the gate at operator.go#L1433 and passes on the spent preflight without re-holding anything — copy/cutover runs while siblings are live and mergeable; (e) on completion the settle Record hits the (apply_id, kind) duplicate-key no-op (merge_gate_requests.go#L62), the operator logs "already recorded" and never kicks, and neither backstop sweep fires — both require a MISSING settle row (r.id IS NULL at merge_gate_requests.go#L310, settle.id IS NULL at merge_gate_requests.go#L350) and the row exists. The alternate timing (restart before the settle fan-out drains) instead leaves sibling checks held forever. No test covers stop-then-restart. Fix direction: treat a completed preflight as spent once a settle exists for the apply, or re-arm/delete the preflight when a settle records for a non-completed terminal apply.

2. Stop reconciliation for a never-started apply now runs through the preflight gate: it holds every sibling PR's check just to execute a stop, and during a code-host outage the acknowledged stop cannot land. The gate at operator.go#L1433 runs unconditionally in resumeClaimedApplyWithOptions, so the stop-reconciliation drive at operator.go#L866 passes through it. A pending apply with a pending stop is deliberately claimable, its tasks exist from creation, so taskCount > 0 and the gate records a fresh preflight (operator.go#L1181) — the processor holds every sibling PR's check and posts hold comments for an apply whose only remaining action is to stop (the drive consumes the stop before any engine work, local_control_resume.go#L1423), and the stopped settle then re-plans them all back: pure sibling-PR churn per stop. Worse, during a GitHub/processor outage each attempt parks a driver 90s (operator.go#L1246-L1249) and fails closed (operator.go#L867-L874), stranding the acknowledged stop — and stop reconciliation runs before other claims each tick (operator.go#L242). Pre-PR the stop drove immediately with no GitHub dependency. Skip the gate when the drive is servicing a pending stop.

Non-blocking

1. checkPreflightGateTimeout (90s) exceeds ApplyLeaseStaleAfter (60s) and the gate wait never heartbeats the applies-row lease, so on the non-heartbeated claim paths two drivers can pass the gate and call ResumeApply concurrently. operator.go#L1123 checkPreflightGateTimeout = 90 * time.Second vs storage.go#L25 const ApplyLeaseStaleAfter = time.Minute; waitForCheckPreflight (operator.go#L1215-L1261) only polls and sleeps. On the legacy whole-apply path (operator.go#L280, when operator_claim_operations=false) and stop reconciliation, no operation heartbeat runs, so one transient fan-out failure (retry deferred by mergeGateRetryDelay = time.Minute, merge_gate.go#L46) makes the row lease-stale at t=60s (applies.go#L1260), a peer reclaims it, and both drivers pass within one 1s poll window — duplicated engine work until the displaced driver's first lease-guarded write fails. The operation-claim paths are safe (heartbeat spans the wait, operator.go#L470, #L646). Cheap fix: heartbeat the apply lease in the poll loop, or cap the gate timeout below 60s.

2. The taskCount==0 preflight skip exempts VSchema-only applies that drive real target changes; the justifying comment is wrong. operator.go#L1176 returns nil (no preflight, no holds) justified by "its drive fails closed on the no-tasks claim gate" — but a VSchema-only apply carries an operation row and no tasks and is claimable (applies.go#L1250-L1251), and the no-tasks drive path exempts it (local_control_resume.go#L1447 if len(tasks) == 0 && !isTasklessVSchemaOnlyPlan(tasks, plan)). So a resharding/routing change starts with no holds while siblings can merge on verdicts planned against the pre-apply VSchema — and the completion settle then re-plans them, confirming the verdicts were stale.

3. The deadline-expiry "timeout" outcome — the fail-closed path the PR body headlines — is untestable and untested. waitForCheckPreflight uses time.Now()/time.After (operator.go#L1217, #L1246, #L1255) instead of the Service's injected clock used elsewhere (operator.go#L1387), and the timeout is a hardcoded 90s const, so exercising the branch needs 90 real seconds. TestGateApplyStartOnCheckPreflight (merge_gate_record_test.go#L220) covers only ctx-cancel fail-closed; the req==nil "disappeared" branch (operator.go#L1224-L1227) is also uncovered. Using s.clock and a Service-field timeout makes it a ~1s unit test — and proves the outcome=timeout metric the doc tells operators to alert on.

General suggestions

• preflight_gate_total outcome accounting diverges from its doc: "error" (documented as storage failure, metrics.go#L1611-L1612) is also emitted on routine drive-context cancellation (operator.go#L1253) and the request-disappeared case (operator.go#L1225), while "passed" is never emitted on the completed-preflight fast path (operator.go#L1160-L1166) — a rolling deploy reads as storage failures and resume-heavy applies skew the pass ratio pessimistic. Emit a distinct "cancelled" outcome and count fast-path passes (or document both exclusions).
• newE2EHandler now auto-starts a merge gate processor for every pre-existing test (webhook_integration_test.go#L285); its immediate startup pass (merge_gate.go#L156) claims residual requests from the package-shared merge_gate_requests table — cleaned only by merge-gate tests (merge_gate_integration_test.go#L47) — and fans them out through the current test's mock GitHub server: a cross-test coupling channel that can flake tests asserting on GitHub call patterns. Clear merge gate state in shared harness setup or default non-apply-driving suites to the no-processor constructor.
• Assertion message at merge_gate_integration_test.go#L120 says the kick is registered "at construction", but registration happens in StartMergeGateProcessor (merge_gate.go#L89), which explicitly documents living "here rather than at handler construction" — a reader trusting the message could construct without starting the processor and silently un-gate applies.
• The 9-field MergeGateRequest-from-Apply construction now appears at 4 production sites differing only in Kind (operator.go#L1181 — this PR's addition — plus operator.go#L1087, merge_gate.go#L196, merge_gate.go#L243), past AGENTS.md's 3+ threshold, and feat(github): store born-held checks while a preflighted apply changes the target #942 adds more. A newMergeGateRequestForApply(apply, kind) helper single-sources the mapping before an attribution field drifts.

The one thing that could have broken, verified

The completed-preflight fast path at operator.go#L1160: the gate's entire safety story rests on the invariant that a completed preflight implies its sibling-check holds remain in force whenever engine work resumes. I tried to prove that invariant and instead disproved it — it is unsafe (Blocking #1). Every link was verified in the worktree: stopped is Terminal: true (metadata.go#L86-L89), so the new drive-tail branch records a settle for a stopped preflighted apply; the settle fan-out releases the holds because HasActivePreflightedApplyOnTarget joins only non-terminal applies (merge_gate_requests.go#L375); yet the stopped apply resumes on the same row (stopped+pending-start claim arm, applies.go#L1270) and sails through the fast path with zero holds in place; its completion settle is swallowed by the duplicate-key no-op (merge_gate_requests.go#L62) and both backstop sweeps require a missing settle row, which exists. Proving it safe would require the gate to also check that no settle exists for the apply (or a re-arm of the preflight on restart); neither exists in this PR or, visibly, in #942.

Verified correct

• CI is effectively green at this head: the latest run passes all 32 checks; the 3 failing rows are stale entries from a superseded duplicate run.
• Gate coverage is complete: every engine-drive entry point (legacy whole-apply operator.go#L280, single-operation, multi-op/cutover, stop reconciliation operator.go#L866) funnels through resumeClaimedApplyWithOptions, and the gate runs before RoutingTernClient and all panicsafe ResumeApply calls.
• Fail-closed contract holds on every gate error path: storage read/record errors, request-disappeared, timeout, and ctx cancellation all return an error; the sole caller returns (false, err) leaving the apply claimable (operator.go#L1433-L1441), and no gate failure terminalizes an operation.
• Preflight recording is correctly double-gated and idempotent: recorded only with a consumer registered and taskCount > 0; the (apply_id, kind) unique key dedups concurrent gate attempts from sibling operation drives, and the processor kick fires even when recorded == false (operator.go#L1205), covering a racing recorder that died before kicking.
• Re-arm semantics are exact: the Failed && RetryAfter == nil condition (operator.go#L1234) matches precisely the two terminal-failure producers (MarkFailed past the attempt cap; TerminateStuckProcessing); ReopenForRetry is CAS-guarded so concurrent re-arms are safe, and retry-scheduled failures are correctly left to ClaimNext.
• Gate timeout vs processor cadence: 90s exceeds the 30s processor poll interval (merge_gate.go#L35), so a wake-up kick lost across pods still completes within one wait.
• Drive-tail settle change fails safe: a preflight-lookup storage error for a non-success terminal apply logs, counts a record failure, and returns without recording (operator.go#L1072-L1078), deferring to the release sweep — no path converts lookup uncertainty into a released or missing hold; completed applies fall through to Record exactly as before.
• Startup ordering leaves no ungated-apply boot window: serve.go starts the merge gate processor (which sets OnMergeGateRecorded synchronously) strictly before StartOperator (serve.go#L557).
• Operation-lease heartbeat runs for the full gate wait on both operation-claim paths (operator.go#L470, #L646), so a 90s gate wait cannot lose the operation lease to a peer on the default claim mode.
• Metric plumbing is correct: RecordCheckPreflightGateOutcome attribute order matches its addCounter (metrics.go#L1613-L1620); the check_preflight_gate resume-failure reason (operator.go#L1440) matches the signature and the established reason-arg pattern.
• The gate-recorded preflight carries the same attribution fields as the drive-tail settle, and CLI applies (PullRequest 0) get an empty ChangeKey meaning exclude-nothing.
• Test-harness rewiring is sound: newE2EHandler starts/stops the processor with t.Context()/cleanup matching production; all manual-lifecycle merge-gate tests that drive sweeps by hand were migrated to newE2EHandlerWithoutMergeGateProcessor, so no manual pass races a background one; the drive-tail E2E chains the handler's original kick rather than replacing it, so the gated apply still completes.
• The removed assert.Equal(storage.MergeGatePending, ...) assertion was necessary, not lost coverage: with a live processor the settle may drain to completed before the read; the recorded-as-pending invariant is still enforced by the store and asserted in pkg/api/merge_gate_record_test.go via the capturing store.
• Test-side dedup is a genuine improvement: newMergeGateTestService (merge_gate_record_test.go#L112) replaces per-test closures, is reused across all three test functions, and the capturing store's mutex is race-safe under -race.
• Repo conventions hold across the delta: driver/drive vocabulary, code-host-neutral wording in pkg/api / pkg/storage / pkg/metrics, no internal numeric row IDs in new logs, errors wrapped with context, every early gate return logs its reason.
• Performance is bounded: the settle path's extra GetByApplyAndKind runs only for non-completed terminal applies (one query per terminal apply, no N+1), and the gate's steady-state cost on resumes/cutovers is a single storage read via the completed fast path.

This review was generated by Claude Code (claude-fable-5).