From e884bd48fb0ee2fb91c64ec557949e7b204f364a Mon Sep 17 00:00:00 2001 From: Evan Baker Date: Fri, 4 Sep 2026 23:09:22 -0500 Subject: [PATCH] docs: define clean-room frontier PRD --- docs/frontier-prd.md | 206 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 docs/frontier-prd.md diff --git a/docs/frontier-prd.md b/docs/frontier-prd.md new file mode 100644 index 0000000..69af6b8 --- /dev/null +++ b/docs/frontier-prd.md @@ -0,0 +1,206 @@ +# Frontier merge-queue model + +**Status:** Proposed +**Audience:** Shunt maintainers and contributors +**Scope:** The queue engine and its durable state model + +## Summary + +Shunt needs a merge-queue model that can test an ordered group of pull +requests, isolate failures, and keep making progress without losing the +ordering users requested. The model must remain correct across restarts, +source-head updates, and base-branch movement. + +This document is the product and behavior contract for that work. It is a +clean-room specification for Shunt: implementation details are deliberately +not prescribed, and existing prototypes are not the source of truth. The +implementation may be discarded and rebuilt from these requirements. + +## Problem + +A merge queue normally learns only whether a staged group passed or failed. +When a group fails, testing every pull request individually is slow; testing +many speculative groups concurrently is faster but can produce results that no +longer describe the queue after an earlier group is accepted or rejected. + +Shunt needs to preserve three properties at the same time: + +1. **Safety:** never report success for a pull request using evidence that does + not include its current head and the current base. +2. **Order:** never release a later pull request before an earlier queued pull + request is resolved. +3. **Progress:** a failed group should be reduced to useful smaller tests rather + than blocking all later work indefinitely. + +## Goals + +- Represent unresolved work as an ordered frontier of candidate tests. +- Reuse valid test results while making invalid or superseded results harmless. +- Isolate a failing pull request with fewer tests than one test per pull request. +- Resume safely after a process restart. +- Keep Bolt/bbolt the primary local durable store with no new runtime + dependency. +- Make behavior observable through existing statuses, comments, logs, and + metrics. +- Keep the design usable by an independent Shunt installation against + Forgejo-compatible APIs. + +## Non-goals + +- Replacing Forgejo/Gitea's merge worker or branch protection. +- Directly pushing the base branch or bypassing forge merge requirements. +- Multi-replica coordination in the first implementation. +- Making Postgres a prerequisite or changing its schema as part of the Bolt + implementation. Postgres compatibility is a separate follow-up and must not + weaken the Bolt-first path. +- Speculative fan-out before the serial frontier behavior is correct and + measurable. +- A new public API solely for this feature. + +## Product behavior + +### Queue admission + +- A pull request enters the queue only if it is open, still eligible for + auto-merge, and passes the repository's configured admission rules. +- Admission records the source head SHA and the base identity used for the + decision. +- A source-head update removes the old evidence from consideration and causes + the current pull request to be tested again. + +### Candidate tests + +A candidate is an ordered list of pull requests plus the immutable inputs that +make its result meaningful: + +- repository and base reference; +- base revision or generation; +- ordered pull-request numbers and source head SHAs; +- merge method and relevant queue configuration; +- a unique staging attempt identity. + +A candidate is never considered reusable merely because it has the same pull +request numbers. Its recorded inputs must match the inputs being evaluated. + +Each candidate has one of three externally useful states: + +- **unknown:** not yet tested or no longer trustworthy; +- **passing:** its staged result passed the configured gate; +- **failing:** its staged result failed or could not be produced. + +Infrastructure errors are not silently converted into a passing or failing +candidate. They remain retryable or become an explicit terminal error through +the existing queue outcome path. + +### Frontier reduction + +The engine maintains ordered unresolved work. At each step it may: + +1. test the earliest unresolved candidate; +2. accept a passing candidate when all earlier work is resolved; +3. split a failing candidate into smaller ordered candidates; or +4. bounce a candidate that cannot be staged because of a deterministic conflict. + +A split must preserve queue order and must not drop, duplicate, or reorder pull +requests. A result for a later candidate may be retained as evidence, but it +cannot release or mark success for a later pull request while an earlier +candidate remains unresolved. + +The initial implementation should use a deterministic serial frontier. Any +concurrent fan-out is a later optimization with explicit invalidation rules +and measurements showing that it improves throughput without changing the +serial behavior. + +### Base and source-head changes + +Evidence is invalid when any of its inputs change, including: + +- the base branch advances in a way that changes the staging result; +- a pull-request head SHA changes; +- the pull request closes, merges, or loses queue eligibility; +- merge method or other result-affecting queue configuration changes. + +Invalid evidence must not release a pull request. The engine deletes or +abandons the obsolete staging attempt, re-resolves the current queue entries, +and creates fresh work as needed. + +### Landing + +- Only the earliest resolved pull request is released at a time. +- A passing queue status is written only after the candidate's evidence is + valid and all earlier work is resolved. +- Shunt waits for the forge to report the pull request merged before releasing + the next one. +- Existing cancellation, timeout, rejection, and outcome-comment behavior + remains the single path for terminal outcomes. + +## Durable state + +The checkpoint must be sufficient to recover without trusting process memory: + +- pending ordered candidates; +- active candidate inputs and staging identity; +- candidate outcome and phase timestamps; +- base identity/generation; +- queue sequence or attempt identity needed to avoid branch collisions; +- any frontier evidence required to continue reduction. + +A restart must either resume a candidate whose recorded inputs still match or +safely discard it and requeue current work. It must never resume a gate result +for an old source head or old base. + +Bolt/bbolt stores one JSON snapshot per queue in the existing queue bucket. +Adding fields to that JSON does not require a Bolt schema migration. If the +logical checkpoint format changes incompatibly, the loader must reject unsafe +in-flight state explicitly and rederive or requeue it rather than guessing. + +## Acceptance criteria + +The implementation is ready for a follow-up implementation PR when these +scenarios pass as engine tests: + +1. **All pass:** an ordered batch is staged once and lands in queue order. +2. **Single failure:** a failed multi-PR candidate is reduced and the failing + pull request is isolated without losing the remaining entries. +3. **Prefix conflict:** a staging conflict bounces the conflicting entry and + preserves the order of entries after it. +4. **Head update:** changing one source head invalidates its old evidence and + prevents the old result from releasing it. +5. **Base update:** advancing the base invalidates affected evidence and causes + fresh staging. +6. **Restart:** a checkpoint resumes only matching work; unsafe or incompatible + in-flight state is not treated as passing. +7. **Ordering:** a later passing candidate cannot release while an earlier + candidate is unresolved. +8. **Closure:** closed or merged entries are removed without leaving an + untracked staging branch. +9. **Persistence:** a Bolt save/load round trip preserves every field needed + for the frontier and active-candidate recovery. +10. **No queue:** an idle queue removes its checkpoint using the existing + lifecycle behavior. + +## Operational requirements + +- Emit enough structured logging to identify queue, candidate, staging attempt, + and invalidation reason without logging credentials or private payloads. +- Expose metrics for active candidates, pending candidates, staging attempts, + reductions, invalidations, bounces, and time spent waiting on gates. +- Document configuration only after behavior is implemented and tested. +- Keep the public deployment path dependency-light and compatible with the + existing `make lint test build` validation. + +## Implementation sequence + +1. Land this PRD as the behavior contract. +2. Implement the neutral checkpoint/domain model with Bolt round-trip tests. +3. Implement deterministic serial frontier reduction in the engine. +4. Add restart, head-change, base-change, ordering, and failure-isolation + tests before considering concurrency. +5. Measure serial behavior in representative queues. +6. Evaluate optional fan-out as a separate proposal and implementation. +7. Evaluate Postgres persistence and multi-replica coordination separately, + with compatibility tests against the same domain model. + +The PRD is complete when the implementation can be reviewed against the +acceptance criteria above without relying on a historical implementation or +private deployment assumptions.