feat(github): hold sibling PR checks and comment when an apply preflights a target

[aparajon]

Why this matters

A green check on a sibling PR is a promise that its plan still matches the target schema. The moment an apply starts changing that schema, the promise is stale — but today nothing on the PR says so, and the merge gate processor only knows how to re-plan after the apply finishes. This PR teaches the processor to consume preflight requests: hold every sibling PR's check action-required and explain why with a PR comment, before the apply's engine work begins. The fan-out runs in two phases so the hold itself never depends on the code host being reachable. Stack 5/7, on top of #939.

What it does

• Preflight fan-out in two phases (fanOutCheckPreflight):
  • Hold (storage-only): for each open sibling PR with stored check state on the apply's target, conditionally flips the stored check to blocked (apply_in_flight_on_target), then stamps holds_recorded_at — the signal the operator gate (feat(api): gate apply start on confirmed sibling PR check holds #941) starts the apply on. No code-host call in this phase, so the holds land even when the code host is fully down.
  • Render (code host, retries independently): updates the aggregate Check Run and posts one explanatory PR comment per sibling. A render failure keeps the request retryable without re-blocking the apply — the failure direction is a stale rendering, never a missing hold.
  • Checks owned by an in-progress apply and closed PRs are skipped with logged outcomes. The flip is optimistic on the head SHA, so a PR that moves mid-fan-out is never stomped.
• Hold comment (RenderCheckHold): tells the PR's author what is changing, on which target, by whom, and what to do next (wait for the apply to settle; checks re-plan automatically). Idempotent via a hidden per-apply marker, so webhook redeliveries and retries never double-post.
• Render re-arm sweep: each processor pass re-arms terminally failed preflight renders whose apply is still active (ReopenTerminalPreflightsForActiveApplies, counted by schemabot.merge_gate.preflight_renders_rearmed_total), so a hold's Check Run and comment keep retrying until the code host recovers — nothing else would retry them once the apply has started.
• Settle deferral: a settle fan-out completes without re-planning while a later preflighted apply is still active on the same target — the re-plan would mint fresh green checks against a schema mid-change.
• Release sweep (sweepPreflightedAppliesMissingSettle): settles are now the release valve for holds, so any apply that reaches a terminal state with a preflight but no settle (e.g. cancelled while queued, where no drive tail runs) gets a settle backfilled. A hold can never outlive its apply.
• Consumer registration moves from handler construction to StartMergeGateProcessor: a registered consumer now means "a processor is running and will drain requests", which the operator gate in the follow-up PR relies on.

 preflight request ──► HOLD (storage-only)          ──► RENDER (code host)
                        flip sibling stored checks       aggregate Check Run
                        stamp holds_recorded_at          + PR comment (once)
                        │ skip: in-progress             │ failure: request
                        │ apply-owned, moved head       │ stays retryable;
                        │                               │ re-armed while the
                        │                               │ apply is active
 settle request    ──► later preflighted apply active on target?
                        ├─ yes: complete without re-plan (defer)
                        └─ no:  re-plan siblings, releasing holds
 release sweep     ──► terminal apply w/ preflight, no settle
                        └─ backfill settle (holds never leak)

How it moves us toward the northstar

The PR's checks become an honest, live rendering of the target's state: "an apply is changing this schema right now" is visible where merges are decided, not buried in an operator log. The final PR in the stack turns this into a hard gate on apply start.

The chain: #867 (storage) → #868 (drive-tail recording) → #866 (settle re-plan processor) → #939 (request kinds + hold storage) → #940 (preflight hold fan-out) → #941 (apply-start gate) → #942 (plan-time holds). Merges bottom-up; each PR retargets to main as its base merges.

🤖 Generated with Claude Code

[Kiran01bm]

🤖 Review findings - created by Kiran's code review agent - for pull/940, d30dc67.

Verdict: 9 findings — 2 blocking (fail-open aggregate publish, broken comment dedup), 4 non-blocking, 3 suggestions.

Blocking

1. Preflight fan-out fire-and-forgets the visible aggregate Check Run update, so the request completes "held" even when the GitHub Checks write failed and the sibling's merge button stays green. merge_gate.go:714 h.updateAggregateCheck(prCtx, client, repo, pr, actionable[0].HeadSHA) — the wrapper discards the fold error (check_publisher.go:111 followUp, _ := h.updateAggregateCheckOnce(...)), and a failed aggregate publish is exactly the errored disposition the fold contract tells durable-retry owners to act on. Failure: MarkBlockedForApplyInFlight succeeds, CreateCheckRun gets a 500, fanOutCheckPreflight returns nil, the request is MarkCompleted (merge_gate.go:407), the feat(api): gate apply start on confirmed sibling PR check holds #941 gate confirms and starts the apply — while the only recovery is the in-memory, budget-bounded refold timer, lost on pod restart. The sibling's visible check stays green for the whole apply: fail-open on the exact surface the hold exists to block, violating the repo's tier-0 "GitHub API uncertainty must never become a passing check" rule. The comment failure two lines below correctly fails the request to stay retryable (merge_gate.go:716) — the merge-blocking surface deserves at least the treatment its explanation gets. Fix: propagate the aggregate publish error into the fan-out result so the request retries.

2. HasIssueCommentWithMarker's dedup silently fails on any sibling PR with >100 issue comments, so retries post duplicate hold comments. client.go:651 passes Sort: new("created") / Direction: new("desc") to the per-issue ListComments endpoint, which accepts only since/per_page/page and always returns ascending-ID order (go-github v86 honors Sort/Direction only on the repo-level endpoint). So the single page inspected is the 100 oldest comments, inverting the doc claim at client.go:644 ("Only the newest page of comments is inspected"). Failure: a sibling PR accumulates >100 bot comments (plan/apply progress makes this realistic); the fan-out posts the hold comment (comment Feature: automigrate (CI based schema change execution) #101+, beyond page 1), then any retry — lease handover, crash before MarkCompleted, or errors.Join failure on a different sibling (merge_gate.go:622) re-running every PR — finds no marker and double-posts, once per retry, directly contradicting the stated retries-converge-without-duplicates contract. The integration test can't catch it: its fake serves all comments in one unpaginated page. Fix: pass Since with a lookback, or paginate to resp.LastPage, or persist a posted flag with the request row.

Non-blocking

3. TOCTOU: the settle fan-out checks HasActivePreflightedApplyOnTarget once at start, and re-plan writes are head-SHA-guarded only, so a concurrent preflight hold can be overwritten green — latent until feat(api): gate apply start on confirmed sibling PR check holds #941, closed at write time by feat(github): store born-held checks while a preflighted apply changes the target #942. merge_gate.go:522 is the only hold check before the slow per-PR re-plans; the write path guards only head-SHA currency (check_records.go:177), and the hold row sets apply_id = NULL (checks.go:567) so it never matches UpsertPlanResult's sole preservation guard (checks.go:163 AND NOT (status = ? AND apply_id IS NOT NULL)). ClaimNext is FOR UPDATE SKIP LOCKED per-row with no per-target exclusion (merge_gate_requests.go:151), so pod-1 mid-settle-fan-out and pod-2 draining a fresh preflight on the same target can interleave: pod-2 flips PR-X held, feat(api): gate apply start on confirmed sibling PR check holds #941's gate starts apply B, then pod-1's same-head re-plan overwrites the hold green and publishes it. Not blocking here because at this head nothing in production records preflight requests (the recorder lands in feat(api): gate apply start on confirmed sibling PR check holds #941), and feat(github): store born-held checks while a preflighted apply changes the target #942's diff adds exactly the write-time re-check (born-held plan writes). Actionable item: ensure feat(api): gate apply start on confirmed sibling PR check holds #941 never ships/deploys without feat(github): store born-held checks while a preflighted apply changes the target #942, or pull that write-time guard forward.

4. The settle deferral trusts any recorded preflight row regardless of its own state and permanently consumes the settle, so a wedged apply with a terminally-failed preflight suppresses re-plans while an unheld sibling stays green-stale. merge_gate.go:522's query has no predicate on the preflight's state — only the apply's non-terminality — so a preflight that MarkFailed at the attempt cap (merge_gate.go:373) before flipping any sibling still triggers the deferral, and the nil return reaches MarkCompleted so the settle is consumed, never retried. Failure: apply B's preflight fails terminally on persistent GitHub 500s (siblings never held), B sits queued behind the feat(api): gate apply start on confirmed sibling PR check holds #941 gate; apply A on the same target completes and its settle defers to B — sibling PR-X keeps a green verdict computed against the pre-A schema, mergeable, until an operator terminalizes B and the release sweep backfills. Note the naive fix (filter deferral to non-terminal preflights) is insufficient once feat(api): gate apply start on confirmed sibling PR check holds #941's re-arm keeps the row pending; the real defect is treating the preflight row's existence as proof its holds landed while consuming the settle instead of leaving it retryable.

5. Stale nil-callback contract docs: this PR moved OnMergeGateRecorded registration to StartMergeGateProcessor, but three doc sites still say nil means "GitHub is not configured". operator.go:1024 ("registers OnMergeGateRecorded at construction, so a nil callback means…"), the Debug log at operator.go:1049, and the field doc at service.go:181-192. Since registration now happens at merge_gate.go:89 behind the storage-nil early return at merge_gate.go:80, nil also means "processor not started" — and feat(api): gate apply start on confirmed sibling PR check holds #941 gates apply start on this consumer's output, so an engineer triaging silently-skipped recordings will be pointed at the wrong cause.

6. The preflight fan-out's two skip guards are untested at any layer. The in-flight apply-owned skip at merge_gate.go:635 and the closed-PR skip at merge_gate.go:669 are reached only via preflight-kind requests, and every preflight drain in the suite seeds a green completed check on an open PR. The bigger exposure is the closed-PR skip: closed PRs can't be held, so a regression there fails the fan-out on every retry and — via the feat(api): gate apply start on confirmed sibling PR check holds #941 gate — delays the apply's start indefinitely. (The store-side AND NOT (status = ? AND apply_id IS NOT NULL) guard at checks.go:577 independently protects apply-owned rows and is store-tested, softening the first gap; the hold_superseded/!flipped branch is covered by the retry half of TestE2ECheckPreflightHoldsSiblingChecksAndComments.)

General suggestions

• sanitizeInlineCode (check_hold.go:49) duplicates the package's existing markdownInlineCode (errors.go:312) with subtly different behavior (replace-backtick-with-space vs strip); each "`%s`"+sanitize call site in RenderCheckHold reduces to one markdownInlineCode call. RenderCheckHold also lacks the package's conventional sanitization test (cf. TestRenderRollbackRejectedSanitizesReason).
• The byPR grouping in the preflight fan-out (merge_gate.go:595) handles multi-check-per-PR-per-target groups that the checks unique key (checks.sql:20) and GetByTarget's pinned target make impossible — every group is a provable singleton. holdPRChecksForPreflight could take one *storage.Check like refreshPRPlanForTarget does, deleting prTargetKey, the map, and the arbitrary-looking actionable[0].HeadSHA.
• Incomplete rename sweep: merge_gate.go:1060 still emits return fmt.Sprintf("%s/%d/check-refresh", hostname, os.Getpid()) into merge_gate_requests.lease_owner and logs — the last non-test "check-refresh" string in the codebase, and this PR's own rename commit renamed the test lease-owner constant but missed the production one.

The one thing that could have broken, verified

The preflight-hold contract itself — "when the preflight request is marked completed, every sibling PR is un-mergeable" — because the #941 gate will trust that completion before starting an apply. Verified it is fail-open on both of its surfaces and sound on every other axis. Visible surface: the aggregate Check Run write's error is discarded at merge_gate.go:714 (finding 1), so completion does not imply the merge button is blocked. Stored surface: the hold has no write-path guard protecting it — it writes apply_id = NULL (checks.go:567) so UpsertPlanResult's only preservation clause never matches, and a concurrent same-head settle re-plan on another pod overwrites it green (finding 3; inert until #941, closed by #942's born-held write). Everything else about the mechanism proved sound: markBlockedConditional's three guards (head-SHA currency, apply-owned exclusion, already-held skip at checks.go:577) make the flip race-safe and retry-convergent; the settle-deferral convergence chain guarantees every deferred-to apply its own settle (settles recorded only for terminal applies, deferral matches only non-terminal ones, release sweep covers all terminal states); and the integration tests pin both.

Verified correct

• CI effectively green on head d30dc67: latest run passes all 32 checks; the 3 failing rows are stale entries from a superseded duplicate run.
• Kind dispatch fails closed on unknown request kinds (error carries kind + apply identifier), so a newer writer's kinds terminalize via the attempt cap instead of silently no-opping.
• Release sweep correctness and idempotency: FindTerminalAppliesWithPreflightMissingSettle (merge_gate_requests.go:335) inner-joins preflight existence, left-joins for missing settles, and covers ALL terminal states within lookback; Record is idempotent per (apply_id, kind), and TestE2ECheckReleaseSweepSettlesFailedPreflightedApply asserts a second sweep returns the same settle row and the drained settle clears blocking_reason via a real re-plan.
• Hold flip idempotency and race safety: the conditional write flips only on matching head SHA, never touches in-progress apply-owned rows, and reports flipped=false on retries — a racing synchronize on a newer head wins; verified end-to-end by the reset-and-redrain phase of TestE2ECheckPreflightHoldsSiblingChecksAndComments.
• In-flight apply-owned sibling rows are protected symmetrically on both read (skips in hold and settle paths) and write (store conditional) sides, keeping the started apply authoritative per AGENTS.md.
• Preflight comment failure keeps the request retryable and the apply blocked (error flows to MarkFailed with retryAfter) — fail-closed per the repo's tier-0 check-state bar.
• Aggregate fold blocks the merge button on holds: any action_required check folds to an action_required aggregate, and the fan-out recomputes the aggregate unconditionally so retries converge; a stale head SHA is deferred to a leader re-fold, never stomped.
• Caller-influenced text is sanitized on every rendered surface (RequestedBy/ApplyIdentifier/Database/Environment through sanitizeInlineCode; stored ChangeSummary through clampDriftSummary; the dedup marker built from the server-generated ApplyIdentifier only).
• Consumer registration ordering is sound: Server.Start starts the merge gate processor — which registers OnMergeGateRecorded — before StartOperator in both webhook runtimes, so production drive tails never observe the nil-consumer skip; sweeps backfill any recording missed during downtime.
• Settle deferral convergence: a settle can never defer to its own apply (settles recorded only for terminal applies vs a non-terminal-only deferral predicate), the deferred-to apply is always guaranteed its own settle (drive tail or release sweep), and coalescing cannot swallow it (PendingForTarget captured before the fan-out); TestE2ECheckSettleDefersToActivePreflightedApply proves holds survive the earlier apply's settle.
• isOriginatingChange remains correct for both new kinds: CLI/gRPC applies have an empty ChangeKey which matches nothing, so CLI-originated preflights hold every PR on the target and never mis-skip siblings.
• The settle fan-out preserves base re-plan behavior verbatim (matches base fanOutMergeGate line-for-line with only the deferral prepended — verified against pr-939-review-tmp); no logging, metrics, originating-PR skip, or error-contract behavior dropped by the split.
• fanOutCheckPreflight reuses the settle path's exact bounded-concurrency shape (semaphore of 3 + WaitGroup + mutex-guarded errs + errors.Join); no new concurrency idiom.
• HasIssueCommentWithMarker wraps its call in the existing retryGitHubUnavailableRead + classifyGitHubAPIError machinery, consistent with every other InstallationClient read; no prior marker-search helper existed to reuse.
• New metrics follow existing conventions (MergeGateSourceReleaseSweep parallels existing sources; held/hold_superseded extend the existing outcome label set).
• No N+1 at the storage layer: one GetByTarget per request, one conditional flip per (PR, target), one deferral query per settle; GitHub round trips are per-sibling-PR under the concurrency-3 semaphore.
• Test coverage of the new paths is real and at the right layer (integration-tagged, per AGENTS.md): preflight hold+comment+retry-convergence, release-sweep backfill idempotency, settle deferral leaving blocking_reason intact, and the kick-registration assertion relocated to pin the new registration site.
• Helper reuse is otherwise sound (environmentTitleSuffix, clampDriftSummary, mergeGateSweepLookback, resolveRepoWebhookInstallation, commandBootstrap, FetchPullRequestNoCache, updateAggregateCheck all reused rather than re-implemented), and both commits use the AGENTS.md-mandated feat(github)/refactor(github) scopes with consistent post-rename terminology (one lease-owner string excepted, reported above).

This review was generated by Claude Code (claude-fable-5).