From 8e6a334ec23fb432e450c59d8b0cdfd41826b21d Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 9 Sep 2026 22:26:00 -0400 Subject: [PATCH] feat(gitea-mq): raise the batch cap to twenty for flake-update waves The five-entry cap matched Mic92/dotfiles, SBEE-Lab/infra and mulatta/dots. Two conditions those deployments do not share justify diverging: our update-flake-inputs lane produces waves of twenty to forty simultaneously ready PRs twice weekly, and bisectMaxSteps = 0 makes bisection unlimited, so Engine.HandleFail isolates a failing entry in about log2(N) builds and the eject-the-whole-batch path is unreachable here. Accepted trade: a large batch holds the queue for one build cycle, and one bad entry delays the other nineteen while bisection runs. The cap is a ceiling, not a target. FormBatch is greedy, taking whatever is queued when a poll runs, and a pull_request event triggers that poll immediately, so batch size stays emergent from arrival rate against build time. Observed here: five PRs landed in two batches of two and three. No speedup beyond that observation is claimed. design.md records the reversal with both reasons; brainstorm.md keeps the earlier zero-to-five decision untouched as the historical record. Change-Id: Ifc76b011fd3b7a8361469c693da847db6a6a6964 --- .../adr-substitution-first-rollup-landing.md | 32 ++++++++++++++----- modules/nixos/gitea-mq.nix | 6 ++-- .../stand-up-gitea-mq-on-magnetite/design.md | 22 +++++++++++-- .../stand-up-gitea-mq-on-magnetite/plan.md | 20 ++++++------ .../proposal.md | 17 ++++++++-- .../specs/merge-queue-interface/spec.md | 6 ++-- .../specs/merge-queue-service/spec.md | 4 +-- .../stand-up-gitea-mq-on-magnetite/tasks.md | 8 ++--- 8 files changed, 80 insertions(+), 35 deletions(-) diff --git a/docs/notes/development/version-control/adr-substitution-first-rollup-landing.md b/docs/notes/development/version-control/adr-substitution-first-rollup-landing.md index 57cd21ccc..d3c5e9521 100644 --- a/docs/notes/development/version-control/adr-substitution-first-rollup-landing.md +++ b/docs/notes/development/version-control/adr-substitution-first-rollup-landing.md @@ -89,7 +89,7 @@ gitea-mq, GitHub backend: Neither path reads `Depends-On:`; native registration is required for the stack-aware label path (`internal/github/forge.go::ResolveStack`). - A labelled stack forms one queue entry; lower members land as ancestors of the selected head (`internal/poller/poller.go::enqueuePR`, `labeledTargetBranch`; `internal/batch/batch.go::Engine.Build`). Enqueue checks the selected head, not every lower member (`internal/poller/poller.go::prCheckResult`). -- With `batchMax = 5`, entries become `testing` and successful `Engine.HandlePass` deletes them through `internal/queue/batch.go::Service.SaveBatch`. +- With `batchMax = 20`, entries become `testing` and successful `Engine.HandlePass` deletes them through `internal/queue/batch.go::Service.SaveBatch`. Batch finalization posts completion and calls `Engine.ensureMergedOrClose` only for actual queue entries: for a labelled stack, the selected top PR alone. That helper polls for merged-or-closed state for about ten seconds and, if still unresolved, comments with the landing SHA and closes the entry PR. It never calls the forge merge endpoint (`internal/batch/batch.go::Engine.HandlePass`, `Engine.ensureMergedOrClose`). @@ -98,7 +98,7 @@ gitea-mq, GitHub backend: The hint goes on unlabelled, unqueued branch-chain members, deduplicated by head SHA, with “Stack detected — add the 'merge-queue' label to the topmost PR you want to merge”. Their merged-or-closed state after the fast-forward is GitHub's behaviour alone, unobserved by the queue. `finalizeLabeledMerge` and its “Merge queue passed (stack)” completion status are legacy/non-batch only, reachable here solely for persisted non-batch entries surviving a configuration change. -- The batch engine is active when `batchMax != 1`; zero means unlimited, while five limits queue entries, not commits (`internal/batch/batch.go::Engine.Enabled`; `internal/queue/batch.go::Service.FormBatch`). +- The batch engine is active when `batchMax != 1`; zero means unlimited, while twenty limits queue entries, not commits (`internal/batch/batch.go::Engine.Enabled`; `internal/queue/batch.go::Service.FormBatch`). A single up-to-date entry with skipping enabled can land its original head without a batch ref or extra branch CI run (`internal/batch/batch.go::Engine.headIfUpToDate`, `Engine.Build`). Existing head checks are still evaluated (`internal/poller/poller.go::pollMergeBranchChecks`; `internal/batch/monitor.go::Engine.HandleCheck`). - A head behind the target, an `IsUpToDate` error, or two or more entries takes `CreateMergeBranch`/`MergeInto` through `Engine.stack` (`internal/batch/batch.go::Engine.Build`, `Engine.headIfUpToDate`; `internal/github/forge.go::mergeHead`). @@ -130,6 +130,7 @@ mergify-cli: Four consumers were surveyed; none uses `batchMax = 0`, and none showed a linear-history mandate. The three GitHub deployments set `batchMax = 5`: Mic92/dotfiles (`machines/eve/modules/gitea-mq.nix::services.gitea-mq`), SBEE-Lab/infra (`modules/gitea-mq/default.nix::services.gitea-mq`), and mulatta/dots (`machines/cask/modules/gitea-mq.nix::services.gitea-mq`). +Our prior choice of five matched these deployments; the Decision below supersedes that choice with twenty for flake-update waves and unlimited bisection. clan-lol/clan-infra uses the Gitea backend at the default one (`modules/web01/gitea-mq.nix::services.gitea-mq`); its live Gitea protection was not checked. Mic92/dotfiles' `.github/settings.yml::branches.protection.required_linear_history` is false, with “Disabled for bors to work”; this bors-era declaration is vestigial against the live GitHub state. @@ -169,8 +170,8 @@ These observations distinguish original-head landing from batched merge history; Do not combine it with gitea-mq (mulatta/dots `home/bin/gh-bootstrap::check_gitea_mq`). 7. gitea-mq for one labelled stack at a time, after an orchestrator assembles and prebuilds a linear rollup on `staging`: rejected in this revision. Queue-created merge commits are tested before landing, so assembly and serialization add no substitution guarantee and prevent multi-entry batching. -8. gitea-mq alone for queue formation and landing, with `batchMax = 5`, author cache warming, and shape-specific authorization signals: chosen. - This follows all three surveyed GitHub deployments and preserves the tested-tree property without a bespoke rollup service. +8. gitea-mq alone for queue formation and landing, with `batchMax = 20`, author cache warming, and shape-specific authorization signals: chosen. + This preserves the tested-tree property without a bespoke rollup service; the Decision records why the cap now differs from the three surveyed GitHub deployments' five. ## Decision @@ -189,7 +190,22 @@ We will separate authorization into two orthogonal axes: For a registered stack, we will label its topmost intended PR `merge-queue` after verifying registration and head ancestry. We will never enable auto-merge on any stack member, including the bottom member; auto-merge enqueue does not resolve stacks and wins over label enqueue. -We will deploy `services.gitea-mq.batchMax = 5` and `skipQueueIfUpToDate = true`, keeping the default `merge-queue` label. +We will deploy `services.gitea-mq.batchMax = 20` and `skipQueueIfUpToDate = true`, keeping the default `merge-queue` label. +This supersedes our prior `batchMax = 5` decision, which matched Mic92/dotfiles, SBEE-Lab/infra, and mulatta/dots (Context, Deployment survey and live rulesets). +Our flake-update lane produces waves of 20–40 simultaneously-ready PRs, scheduled twice weekly (`.github/workflows/update-flake-inputs.yaml::on.schedule`); the reference deployments do not have this lane. +The cap of twenty can cover a twenty-entry wave; a forty-entry wave still needs multiple batches. +Our `bisectMaxSteps = 0` means unlimited bisection (`nix/module.nix::services.gitea-mq.bisectMaxSteps`), so `internal/batch/batch.go::Engine.HandleFail` isolates a failing entry in roughly log2(N) builds. +Its whole-batch ejection guard, `BisectMaxSteps > 0 && Builds >= BisectMaxSteps`, which comments “batch bisection reached the configured limit”, is unreachable here. +Unlimited bisection makes recovery from twenty entries affordable; not every reference deployment shares that configuration. +We accept that a large batch holds the queue for one build cycle, and one bad entry delays the other nineteen while bisection runs. + +The empirical basis is one local observation: five PRs landed in two batches, first two and then three, using two batch builds. +`internal/queue/batch.go::Service.FormBatch` greedily takes up to `BatchMax` entries at the instant a poll runs, with no accumulation window. +`internal/webhook/github.go::maybeTriggerPoll` requests an immediate poll on green checks; `GithubHandler`'s `PullRequestEvent` path does so when auto-merge is enabled (`prTriggerActions`). +These triggers and greedy selection explain the observed split: batch size emerges from arrival rate versus build time. +Raising the cap raises the ceiling; it does not force larger batches. +Any speedup from twenty is a projection, not a measured result. + We will enable repository `allow_auto_merge` and maintain two default-branch rulesets: ours requiring deletion protection, non-fast-forward protection, `nixbot/nix-eval`, and `nixbot/nix-build`; the App's requiring `gitea-mq`. We will permit the queue App's ruleset bypass and use neither `required_linear_history` nor classic branch protection. We will derive the required CI set from our ruleset; an environment fallback will not substitute for that ruleset. @@ -206,7 +222,7 @@ Business justification: - Time to market: we will make landing asynchronous after authorization, allowing a person or agent to start the next task while the queue tests and lands work. Required review and CI remain prerequisites; there is no fixed author-wait bound. - User satisfaction: we will provide the same discoverable signals to people and agents, with pending stack hints and queue status, instead of requiring a human to wait for orchestrator assembly. -- Strategic positioning: we will follow the deployed Nix ecosystem configuration and maintain source-filtering and cache integration rather than a second queue implementation. +- Strategic positioning: we will retain the deployed Nix ecosystem's queue integration, with a larger batch cap for our flake-update lane, and maintain source-filtering and cache integration rather than a second queue implementation. ## Consequences @@ -250,7 +266,7 @@ Automated checks to implement in the related OpenSpec changes: - Retain `filter-check-sources-for-substitution`'s `structure-check-source-isolation` check, negative control, and transitive `check-source-audit` probe against its declared whole-tree allow-list (`openspec/changes/filter-check-sources-for-substitution/proposal.md`, “Assertion of the property”). Unrelated source changes must leave unaffected filtered checks' derivation hashes stable; the deliberately whole-tree `gitleaks` scan is the declared exception. -- Add a module assertion pinning `services.gitea-mq.batchMax = 5` and `skipQueueIfUpToDate = true`; assert no merge-label override, because `nix/module.nix::services.gitea-mq` exposes batch size but not the label. +- Add a module assertion pinning `services.gitea-mq.batchMax = 20` and `skipQueueIfUpToDate = true`; assert no merge-label override, because `nix/module.nix::services.gitea-mq` exposes batch size but not the label. - Add a read-only GitHub ruleset check requiring both `nixbot/nix-eval` and `nixbot/nix-build` in our default-branch ruleset. Also verify the separate App-owned `gitea-mq` gate, bypass actor, absence of linear-history and classic protection, and `allow_auto_merge = true`. Checking only `GITEA_MQ_REQUIRED_CHECKS` is insufficient because `internal/monitor/monitor.go::ResolveRequiredChecks` prefers the forge-derived set. @@ -320,7 +336,7 @@ Source filtering and cache warming are the core; process policies and upstream w - R8. Keep unconditional PR builds without orchestrator-controlled publication timing. - R9. Required external contexts are `nixbot/nix-eval` and `nixbot/nix-build`. - R10. Scope PR previews with `effects_on_pull_requests` and non-default branch effects with `effects_branches`, preserving the previews identified in V10. -- R11. Serve `mq.scientistexperience.net` with `batchMax = 5`, `skipQueueIfUpToDate = true`, and the default `merge-queue` label; provision its database and reverse proxy. +- R11. Serve `mq.scientistexperience.net` with `batchMax = 20`, `skipQueueIfUpToDate = true`, and the default `merge-queue` label; provision its database and reverse proxy. - R12. Publish stacks with `--github-native` and verify registration and selected-head ancestry before enqueue; `Depends-On:` alone does not enable queue stack resolution. - R13. Use a separate GitHub App from nixbot's, with the permissions in gitea-mq `README.md`, “GitHub setup”: Checks read/write, Commit statuses read, Contents read/write, Pull requests read/write, Administration read/write, Metadata read. The required subscribable event set is exactly `check_run`, `pull_request`, `status`, and our registered App `sciexp-gitea-mq` (id 4875422) carries exactly those (`GET /apps/sciexp-gitea-mq`: `events: ["check_run", "pull_request", "status"]`). diff --git a/modules/nixos/gitea-mq.nix b/modules/nixos/gitea-mq.nix index b5a767dac..9d9631449 100644 --- a/modules/nixos/gitea-mq.nix +++ b/modules/nixos/gitea-mq.nix @@ -67,7 +67,7 @@ webhookSecretFile = gen.gitea-mq-github-webhook-secret.files."secret".path; repos = [ "cameronraysmith/vanixiets" ]; }; - batchMax = 5; + batchMax = 20; skipQueueIfUpToDate = true; requiredChecks = [ "nixbot/nix-eval" @@ -98,8 +98,8 @@ in [ { - assertion = cfg.batchMax == 5; - message = "services.gitea-mq.batchMax must be 5 (bors-style batching; landing fast-forwards the target to the exact tested batch SHA) per ${adr}"; + assertion = cfg.batchMax == 20; + message = "services.gitea-mq.batchMax must be 20 (flake-update waves with unlimited bisection; landing fast-forwards the target to the exact tested batch SHA) per ${adr}"; } { assertion = cfg.skipQueueIfUpToDate == true; diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/design.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/design.md index 1657e4dde..e51a2ae71 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/design.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/design.md @@ -4,7 +4,7 @@ Magnetite is the Hetzner CX53 host that carries this fleet's server-side service Its composition is the repository's deferred-module pattern: `flake.nix` hands `modules/` to `import-tree`, each aspect file assigns a deferred module into `flake.modules.nixos.`, `modules/machines/nixos/magnetite/default.nix` imports upstream modules and names the aspects the host takes, and `modules/clan/inventory/machines.nix` binds the result into `clan.machines.magnetite`, which `clan machines update magnetite` deploys. nixbot sits in that pattern as `inputs.nixbot.nixosModules.nixbot` imported at the host plus the aspect `flake.modules.nixos.nixbot` in `modules/nixos/nixbot.nix`, and serves `cameronraysmith/vanixiets` and `sciexp/ironstar` under the App `sciexp-nixbot` (id `4743700`). -The ADR `docs/notes/development/version-control/adr-substitution-first-rollup-landing.md` decides that changes land through gitea-mq's own bors-style batching: up to five queue entries tested together, checks read from nixbot's two contexts, and `main` fast-forwarded by non-force `UpdateRef` to the exact commit CI tested. +The ADR `docs/notes/development/version-control/adr-substitution-first-rollup-landing.md` decides that changes land through gitea-mq's own bors-style batching: up to twenty queue entries tested together, checks read from nixbot's two contexts, and `main` fast-forwarded by non-force `UpdateRef` to the exact commit CI tested. Revision 2 of that ADR retired the earlier orchestrator rollup onto `staging`, because `internal/batch/batch.go::Engine.HandlePass` fast-forwards the target to the tested batch SHA, so a separately assembled linear candidate adds no substitution guarantee. The ADR assigns this change R11 (the service and its four settings), R13 (a separate GitHub App), R14 (the two rulesets, App bypass, `allow_auto_merge`, no linear history, no classic protection), the second automated Compliance item (pinning the settings), and the world-assumption entries about gitea-mq and GitHub. R15 and R16, the human-and-agent authorization procedure, and R1 and R2, source filtering and cache warming, belong to sibling changes; this change references them where a dependency needs stating and does not absorb them. @@ -50,8 +50,24 @@ Any upstream filing to gitea-mq. ### D2: The four settings are module options, the label is a default, and an assertion pins all four -- **Choice**: `batchMax = 5`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`; no value for the merge label; `assertions` in the aspect that read `config.services.gitea-mq.batchMax`, `.skipQueueIfUpToDate`, and `.requiredChecks` from the merged configuration and compare them to those values, plus one asserting `!(config.systemd.services.gitea-mq.environment ? GITEA_MQ_MERGE_LABEL)`. -- **Reversal**: the earlier design chose unlimited `batchMax = 0` while relying on orchestrator serialization to supply one rollup entry at a time; zero itself never guaranteed a singleton (`internal/queue/batch.go::Service.FormBatch`). The rollup rested on a false premise about queue-created merge commits. `internal/batch/batch.go::Engine.HandlePass` calls `internal/github/forge.go::FastForward` with the exact tested batch SHA, preserving nixbot's tested-tree identity (`gitrepo.py::WorkTree.tree_hash`) even with merge history. Five matches the three surveyed GitHub deployments (`Mic92/dotfiles machines/eve/modules/gitea-mq.nix::services.gitea-mq`, `SBEE-Lab/infra modules/gitea-mq/default.nix::services.gitea-mq`, `mulatta/dots machines/cask/modules/gitea-mq.nix::services.gitea-mq`). +- **Choice**: `batchMax = 20`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`; no value for the merge label; `assertions` in the aspect that read `config.services.gitea-mq.batchMax`, `.skipQueueIfUpToDate`, and `.requiredChecks` from the merged configuration and compare them to those values, plus one asserting `!(config.systemd.services.gitea-mq.environment ? GITEA_MQ_MERGE_LABEL)`. +- **Reversal, zero to five**: the earlier design chose unlimited `batchMax = 0` while relying on orchestrator serialization to supply one rollup entry at a time; zero itself never guaranteed a singleton (`internal/queue/batch.go::Service.FormBatch`). + The rollup rested on a false premise about queue-created merge commits. + `internal/batch/batch.go::Engine.HandlePass` calls `internal/github/forge.go::FastForward` with the exact tested batch SHA, preserving nixbot's tested-tree identity (`gitrepo.py::WorkTree.tree_hash`) even with merge history. + Revision 2 chose five to match the three surveyed GitHub deployments (`Mic92/dotfiles machines/eve/modules/gitea-mq.nix::services.gitea-mq`, `SBEE-Lab/infra modules/gitea-mq/default.nix::services.gitea-mq`, `mulatta/dots machines/cask/modules/gitea-mq.nix::services.gitea-mq`). +- **Reversal, five to twenty**: `batchMax = 20` supersedes revision 2's `batchMax = 5` and its fleet-matching rationale, retained in `brainstorm.md::Q3`. + Our flake-update lane produces waves of 20–40 simultaneously-ready PRs, scheduled twice weekly (`.github/workflows/update-flake-inputs.yaml::on.schedule`); the reference deployments do not have this lane. + Twenty can cover a twenty-entry wave; a forty-entry wave still needs multiple batches. + Our `bisectMaxSteps = 0` means unlimited bisection (`nix/module.nix::services.gitea-mq.bisectMaxSteps`), so `internal/batch/batch.go::Engine.HandleFail` isolates a failing entry in roughly log2(N) builds. + Its whole-batch ejection guard, `BisectMaxSteps > 0 && Builds >= BisectMaxSteps`, which comments “batch bisection reached the configured limit”, is unreachable here. + Unlimited bisection makes recovery from twenty entries affordable; not every reference deployment shares that configuration. + We accept that a large batch holds the queue for one build cycle, and one bad entry delays the other nineteen while bisection runs. + The empirical basis is one local observation: five PRs landed in two batches, first two and then three, using two batch builds. + `internal/queue/batch.go::Service.FormBatch` greedily takes up to `BatchMax` entries at the instant a poll runs, with no accumulation window. + `internal/webhook/github.go::maybeTriggerPoll` requests an immediate poll on green checks; `GithubHandler`'s `PullRequestEvent` path does so when auto-merge is enabled (`prTriggerActions`). + These triggers and greedy selection explain the observed split: batch size emerges from arrival rate versus build time. + Raising the cap raises the ceiling; it does not force larger batches. + Any speedup from twenty is a projection, not a measured result. - **Rationale**: the module maps the three options straight to `GITEA_MQ_BATCH_MAX`, `GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE`, and `GITEA_MQ_REQUIRED_CHECKS`; `GITEA_MQ_MERGE_LABEL` is not an option, the module's `environment` set is closed, and `config.go::Load` defaults the label to `merge-queue`, so the only drift possible is an override from another module, which is what the fourth assertion detects. The assertions read the merged configuration, so an `lib.mkForce` elsewhere or a one-sided edit fails `checks.x86_64-linux.nixos-magnetite` at evaluation. The ADR's Compliance item allows a flake check or a module assertion; the assertion needs no new check attribute. - **Coupling, not two settings**: `requiredChecks` is a fallback the queue consults only when the forge names no required check (`internal/monitor/monitor.go::ResolveRequiredChecks`), and D7 keeps both nixbot contexts in our ruleset, so the forge list is non-empty and this fallback never fires. The assertion pins the fallback; D7 pins what is actually operative. - **Alternatives considered**: setting `GITEA_MQ_MERGE_LABEL` on the unit directly, rejected because it duplicates a default that already holds and creates the override the assertion exists to catch. A structure check under `modules/checks/structure/` evaluating the magnetite configuration, rejected as a second place to keep the same four values. diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/plan.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/plan.md index 7cdff85e2..4a66f04fe 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/plan.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/plan.md @@ -9,7 +9,7 @@ path. It decomposes tasks.md group by group with the same numbering, so task 4.2 Task 4 Step 2 here where the mapping is one to one and is cross-referenced where it is not. --> -**Goal:** Stand up gitea-mq on magnetite at `mq.scientistexperience.net` with `batchMax = 5`, the other pinned landing settings, a dedicated GitHub App, database/proxy provisioning (R11/R13), and two default-branch rulesets (R14). +**Goal:** Stand up gitea-mq on magnetite at `mq.scientistexperience.net` with `batchMax = 20`, the other pinned landing settings, a dedicated GitHub App, database/proxy provisioning (R11/R13), and two default-branch rulesets (R14). **Architecture:** A new first-party aspect `flake.modules.nixos.gitea-mq` at `modules/nixos/gitea-mq.nix` carries the service configuration, the PostgreSQL provisioning, the nginx vhost, two clan vars generators, and the pinning assertions; the upstream `inputs.gitea-mq.nixosModules.default` is imported at the host beside `inputs.nixbot.nixosModules.nixbot`. The unit runs as a dynamic user named for the unit, authenticates to PostgreSQL by peer identity, listens on loopback port 8092 behind the host's nginx, and reads both GitHub secrets as systemd credentials from root-owned clan vars files. Two operator gates stop the plan: App registration (Task 1) and the ruleset diff (Task 8). @@ -25,7 +25,7 @@ Its positional system parameter adds `--remote magnetite.zt --no-download` (`jus ## Global constraints - Nothing under `modules/nixos/nixbot.nix`, `modules/nixos/buildbot.nix`, their generators, vhosts, or databases is edited; `sciexp-nixbot` (id `4743700`) is not touched. Verified per task by `git diff --stat`. -- The four landing settings are `batchMax = 5`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and no merge-label override; the assertions in the aspect are the only place `GITEA_MQ_MERGE_LABEL` appears under `modules/`. +- The four landing settings are `batchMax = 20`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and no merge-label override; the assertions in the aspect are the only place `GITEA_MQ_MERGE_LABEL` appears under `modules/`. - Both generator files stay at the default owner `root`; the module reads them through `LoadCredential` and no static `gitea-mq` user exists. - `listenAddr` is `127.0.0.1:8092`; `:8080` is bound by the LiveKit JWT service (`modules/nixos/matrix.nix`). - `hideRefFromClients = false`; the default would inject an `ExecStartPre` into `systemd.services.gitea` on this host. @@ -203,7 +203,7 @@ The generator declaration commits with Task 4; the vars entries commit as `clan webhookSecretFile = gen.gitea-mq-github-webhook-secret.files."secret".path; repos = [ "cameronraysmith/vanixiets" ]; }; - batchMax = 5; + batchMax = 20; skipQueueIfUpToDate = true; requiredChecks = [ "nixbot/nix-eval" @@ -237,7 +237,7 @@ Verify: the three `nix eval` commands in tasks.md 4.1. - [ ] **Step 2: Confirm the environment (tasks.md 4.2)** Run: `nix eval .#nixosConfigurations.magnetite.config.systemd.services.gitea-mq.environment --apply 'e: { inherit (e) GITEA_MQ_BATCH_MAX GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE GITEA_MQ_REQUIRED_CHECKS; label = e ? GITEA_MQ_MERGE_LABEL; }' --json` -Expected: `{"GITEA_MQ_BATCH_MAX":"5","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval,nixbot/nix-build","GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE":"true","label":false}`. +Expected: `{"GITEA_MQ_BATCH_MAX":"20","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval,nixbot/nix-build","GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE":"true","label":false}`. - [ ] **Step 3: Assertions (tasks.md 4.3)** @@ -249,8 +249,8 @@ Expected: `{"GITEA_MQ_BATCH_MAX":"5","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval in [ { - assertion = cfg.batchMax == 5; - message = "services.gitea-mq.batchMax must be 5 (bors-style batching; landing fast-forwards the target to the exact tested batch SHA) per ${adr}"; + assertion = cfg.batchMax == 20; + message = "services.gitea-mq.batchMax must be 20 (flake-update waves with unlimited bisection; landing fast-forwards the target to the exact tested batch SHA) per ${adr}"; } { assertion = cfg.skipQueueIfUpToDate == true; @@ -272,8 +272,8 @@ Expected: `{"GITEA_MQ_BATCH_MAX":"5","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval ``` Verify, positive: `nix eval .#checks.x86_64-linux.nixos-magnetite.drvPath` succeeds. -Verify, negative control in a separate authorized scratch workspace: add `services.gitea-mq.batchMax = lib.mkForce 1;` to the host, run the same command, and expect failure with the message requiring 5; replace with `systemd.services.gitea-mq.environment.GITEA_MQ_MERGE_LABEL = "x";` and expect the label assertion's failure. -Capture both failures and revert only the scratch edits; task 4.3 is reopened because its earlier completion predates the five-entry assertion. +Verify, negative control in a separate authorized scratch workspace: add `services.gitea-mq.batchMax = lib.mkForce 1;` to the host, run the same command, and expect failure with the message requiring 20; replace with `systemd.services.gitea-mq.environment.GITEA_MQ_MERGE_LABEL = "x";` and expect the label assertion's failure. +Capture both failures and revert only the scratch edits; task 4.3 remains reopened because its earlier completion predates the twenty-entry assertion. Do not run a negative-control edit in this shared correction session. - [ ] **Step 4: Header (tasks.md 4.4)** @@ -468,7 +468,7 @@ Record every observation in verify.md with the `[operator]` and `[verified here] - [ ] **Step 3: Database and role for the dynamic user (11.3)** - [ ] **Step 4: Two rulesets, both nixbot contexts, App bypass, no linear-history/classic protection, `allow_auto_merge` true (11.4)** - [ ] **Step 5: V3 check-run names and the forge-derived required-check pair; fallback inactive (11.5)** -- [ ] **Step 6: Four settings in the running unit's environment, including batch maximum 5 (11.6)** -- [ ] **Step 7: V2 singleton original-head shortcut under `batchMax = 5`, ordinary auto-merge (11.7)** +- [ ] **Step 6: Four settings in the running unit's environment, including batch maximum 20 (11.6)** +- [ ] **Step 7: V2 singleton original-head shortcut under `batchMax = 20`, ordinary auto-merge (11.7)** - [ ] **Step 8: Re-confirm discharged V1 at the first live stacked landing; retired V6 ref probe removed (11.8)** - [ ] **Step 9: Rollback instantiation in an authorized scratch workspace; G2 reverse diff and queue-gate disabling recorded separately (11.9)** diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/proposal.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/proposal.md index 5c761b91a..6ba6312d1 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/proposal.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/proposal.md @@ -45,8 +45,21 @@ The 2026-09-09 ADR revision retires the orchestrator rollup onto `staging` becau **Four settings fixed by the ADR, and pinned** - From: nothing. -- To: `batchMax = 5`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and the merge label left at gitea-mq's default `merge-queue`; a NixOS assertion in the aspect reads the merged configuration back and fails evaluation of the host if any of the three options drifts or if any module sets `GITEA_MQ_MERGE_LABEL` on the unit. -- Reason: gitea-mq's batch engine fast-forwards the target to the exact commit CI tested (`internal/batch/batch.go::Engine.HandlePass` calling `internal/github/forge.go::FastForward`), so testing up to five entries together preserves the tested tree the substitution argument depends on; five is the value all three surveyed GitHub deployments use. The up-to-date shortcut lets a single ready entry land its own head without a further build. The configured checks are a fallback the queue consults only when the forge names none, and the merge label is not a module option, so the only drift possible there is an override, which is what the fourth assertion catches. +- To: `batchMax = 20`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and the merge label left at gitea-mq's default `merge-queue`; a NixOS assertion in the aspect reads the merged configuration back and fails evaluation of the host if any of the three options drifts or if any module sets `GITEA_MQ_MERGE_LABEL` on the unit. +- Reason: gitea-mq's batch engine fast-forwards the target to the exact commit CI tested (`internal/batch/batch.go::Engine.HandlePass` calling `internal/github/forge.go::FastForward`), so testing up to twenty entries together preserves the tested tree the substitution argument depends on. + Twenty supersedes the prior five-entry cap chosen to match Mic92/dotfiles, SBEE-Lab/infra, and mulatta/dots; `design.md::D2` preserves both decisions and their reasons. + Our twice-weekly flake-update lane produces waves of 20–40 simultaneously-ready PRs (`.github/workflows/update-flake-inputs.yaml::on.schedule`) absent from those reference deployments. + Twenty can cover a twenty-entry wave; a forty-entry wave still needs multiple batches. + Our unlimited `bisectMaxSteps = 0` makes twenty affordable to recover from: `internal/batch/batch.go::Engine.HandleFail` isolates a failing entry in roughly log2(N) builds. + Its whole-batch ejection guard `BisectMaxSteps > 0 && Builds >= BisectMaxSteps`, which comments “batch bisection reached the configured limit”, is unreachable here; not every reference deployment shares this configuration. + We accept that a large batch holds the queue for one build cycle, and one bad entry delays the other nineteen while bisection runs. + The empirical basis is one local observation: five PRs landed in two batches, first two and then three, using two batch builds. + `internal/queue/batch.go::Service.FormBatch` greedily takes up to `BatchMax` entries at the instant a poll runs, with no accumulation window. + `internal/webhook/github.go::maybeTriggerPoll` requests an immediate poll on green checks; `GithubHandler`'s `PullRequestEvent` path does so when auto-merge is enabled (`prTriggerActions`). + These triggers and greedy selection explain the observed split: batch size emerges from arrival rate versus build time. + Raising the cap raises the ceiling; it does not force larger batches, and any speedup from twenty remains a projection rather than a measured result. + The up-to-date shortcut lets a single ready entry land its own head without a further build. + The configured checks are a fallback the queue consults only when the forge names none, and the merge label is not a module option, so the only drift possible there is an override, which is what the fourth assertion catches. - Impact: any later edit to one of these values, in this aspect or by `lib.mkForce` elsewhere, fails `checks.x86_64-linux.nixos-magnetite`. **A dedicated GitHub App, separate from nixbot's** diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-interface/spec.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-interface/spec.md index 5df30da67..9d6a12442 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-interface/spec.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-interface/spec.md @@ -16,13 +16,13 @@ The machine SHALL serve the merge queue at one hostname of its own, distinct fro ### Requirement: The four landing settings are evaluated values guarded by an assertion -The machine's evaluated configuration SHALL carry the queue's batch maximum as `5`, its up-to-date shortcut as enabled, its configured fallback required checks as exactly `nixbot/nix-eval` and `nixbot/nix-build`, and no override of the merge label on the queue's unit, and SHALL refuse to evaluate when the merged configuration differs from those values. +The machine's evaluated configuration SHALL carry the queue's batch maximum as `20`, its up-to-date shortcut as enabled, its configured fallback required checks as exactly `nixbot/nix-eval` and `nixbot/nix-build`, and no override of the merge label on the queue's unit, and SHALL refuse to evaluate when the merged configuration differs from those values. The configured fallback is not the operative required set: the queue consults it only when the forge names no required check, and the ruleset requirement below keeps the forge's set non-empty, so the two settings are one invariant rather than two independent ones. #### Scenario: The evaluated options are read - **WHEN** the queue's options are read from the host's evaluated configuration -- **THEN** the batch maximum is `5`, the up-to-date shortcut is enabled, the configured fallback required checks are exactly the two build-service contexts, and the unit's environment carries no merge-label attribute +- **THEN** the batch maximum is `20`, the up-to-date shortcut is enabled, the configured fallback required checks are exactly the two build-service contexts, and the unit's environment carries no merge-label attribute #### Scenario: Another module forces a different value @@ -32,7 +32,7 @@ The configured fallback is not the operative required set: the queue consults it #### Scenario: The unit's environment is read on the host - **WHEN** the queue's unit environment is read on the host after activation -- **THEN** it carries the batch maximum `5`, the up-to-date shortcut `true`, the two contexts as the configured required-checks list, and no merge-label variable, so the queue's own default of `merge-queue` is in force +- **THEN** it carries the batch maximum `20`, the up-to-date shortcut `true`, the two contexts as the configured required-checks list, and no merge-label variable, so the queue's own default of `merge-queue` is in force ### Requirement: A database and role exist for the unit's dynamic user diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-service/spec.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-service/spec.md index 3255cad88..a02f3651c 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-service/spec.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/specs/merge-queue-service/spec.md @@ -18,14 +18,14 @@ The fleet SHALL provide a merge queue on the host that runs its build service, r ### Requirement: Landing advances the default branch to a tested commit -The merge queue SHALL test queued changes together, up to five queue entries at a time, and SHALL advance the default branch only to a commit whose content the build service has already reported on, by an ancestry-checked update that creates no commit the build service has not seen. +The merge queue SHALL test queued changes together, up to twenty queue entries at a time, and SHALL advance the default branch only to a commit whose content the build service has already reported on, by an ancestry-checked update that creates no commit the build service has not seen. A single queued change whose head already contains the default branch's tip MAY land that head without a further build. **Discharged by**: `merge-queue-interface` requirement `The four landing settings are evaluated values guarded by an assertion`, resting on world assumptions `A22 — gitea-mq resolves stacks only through GitHub's Stacks API`, `A23 — GitHub marks a fast-forwarded stack member merged`, and `A26 — the batch engine advances the target to the exact tested commit`. #### Scenario: Several ready changes are tested together -- **WHEN** between two and five ready queue entries targeting the same branch are selected together for a batch +- **WHEN** between two and twenty ready queue entries targeting the same branch are selected together for a batch - **THEN** the queue tests them as one unit and, on a pass, advances the default branch to the commit that was tested, so that what lands is what was tested even when the unit contains merge commits #### Scenario: An authorized publisher labels the intended top of a stack diff --git a/openspec/changes/stand-up-gitea-mq-on-magnetite/tasks.md b/openspec/changes/stand-up-gitea-mq-on-magnetite/tasks.md index 672d9070c..b1873c883 100644 --- a/openspec/changes/stand-up-gitea-mq-on-magnetite/tasks.md +++ b/openspec/changes/stand-up-gitea-mq-on-magnetite/tasks.md @@ -23,8 +23,8 @@ R1/R2 source filtering and cache warming and R15/R16 authorization procedure rem ## 4. First-party aspect - [x] 4.1 Write `modules/nixos/gitea-mq.nix` defining `flake.modules.nixos.gitea-mq` with `services.gitea-mq` enabled, `github.appId` from task 1.2, `github.privateKeyFile` and `github.webhookSecretFile` from the two generators, `github.repos = [ "cameronraysmith/vanixiets" ]`, `externalUrl = "https://mq.scientistexperience.net"`, `listenAddr = "127.0.0.1:8092"`, `hideRefFromClients = false`, `databaseUrl` at its default, `services.postgresql.ensureDatabases = [ "gitea-mq" ]` and `ensureUsers = [ { name = "gitea-mq"; ensureDBOwnership = true; } ]`, and `services.nginx.virtualHosts."mq.scientistexperience.net"` with `forceSSL`, `enableACME`, and `locations."/".proxyPass = "http://127.0.0.1:8092"` — verify: `nix eval .#nixosConfigurations.magnetite.config.services.gitea-mq --apply 's: { inherit (s) externalUrl listenAddr hideRefFromClients databaseUrl; repos = s.github.repos; appId = s.github.appId; }' --json` returns those values, `nix eval .#nixosConfigurations.magnetite.config.services.postgresql.ensureDatabases --json` contains `gitea-mq`, and `nix eval '.#nixosConfigurations.magnetite.config.services.nginx.virtualHosts."mq.scientistexperience.net".locations."/".proxyPass'` returns the loopback address -- [ ] 4.2 Set the four landing settings — `batchMax = 5`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and no assignment to the merge label anywhere — verify: `nix eval .#nixosConfigurations.magnetite.config.systemd.services.gitea-mq.environment --apply 'e: { inherit (e) GITEA_MQ_BATCH_MAX GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE GITEA_MQ_REQUIRED_CHECKS; label = e ? GITEA_MQ_MERGE_LABEL; }' --json` returns `{"GITEA_MQ_BATCH_MAX":"5","GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE":"true","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval,nixbot/nix-build","label":false}`, and `rg GITEA_MQ_MERGE_LABEL modules/` finds only the assertion -- [ ] 4.3 Add `assertions` to the aspect that read `config.services.gitea-mq.batchMax == 5`, `.skipQueueIfUpToDate == true`, `.requiredChecks == [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and `!(config.systemd.services.gitea-mq.environment ? GITEA_MQ_MERGE_LABEL)`, each with a message naming the setting and ADR R11 — verify: `nix eval .#checks.x86_64-linux.nixos-magnetite.drvPath` succeeds as-is; then, in a separate authorized scratch workspace, adding `services.gitea-mq.batchMax = lib.mkForce 1;` to the host makes the same command fail with the assertion's message requiring 5, and adding `systemd.services.gitea-mq.environment.GITEA_MQ_MERGE_LABEL = "x";` fails with the label assertion's message; both failures captured and scratch edits reverted. The previous completion predates the five-entry assertion, so this negative-control verification is reopened; do not perform it in the shared correction session. +- [ ] 4.2 Set the four landing settings — `batchMax = 20`, `skipQueueIfUpToDate = true`, `requiredChecks = [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and no assignment to the merge label anywhere — verify: `nix eval .#nixosConfigurations.magnetite.config.systemd.services.gitea-mq.environment --apply 'e: { inherit (e) GITEA_MQ_BATCH_MAX GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE GITEA_MQ_REQUIRED_CHECKS; label = e ? GITEA_MQ_MERGE_LABEL; }' --json` returns `{"GITEA_MQ_BATCH_MAX":"20","GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE":"true","GITEA_MQ_REQUIRED_CHECKS":"nixbot/nix-eval,nixbot/nix-build","label":false}`, and `rg GITEA_MQ_MERGE_LABEL modules/` finds only the assertion +- [ ] 4.3 Add `assertions` to the aspect that read `config.services.gitea-mq.batchMax == 20`, `.skipQueueIfUpToDate == true`, `.requiredChecks == [ "nixbot/nix-eval" "nixbot/nix-build" ]`, and `!(config.systemd.services.gitea-mq.environment ? GITEA_MQ_MERGE_LABEL)`, each with a message naming the setting and ADR R11 — verify: `nix eval .#checks.x86_64-linux.nixos-magnetite.drvPath` succeeds as-is; then, in a separate authorized scratch workspace, adding `services.gitea-mq.batchMax = lib.mkForce 1;` to the host makes the same command fail with the assertion's message requiring 20, and adding `systemd.services.gitea-mq.environment.GITEA_MQ_MERGE_LABEL = "x";` fails with the label assertion's message; both failures captured and scratch edits reverted. The previous completion predates the twenty-entry assertion, so this negative-control verification remains reopened; do not perform it in the shared correction session. - [ ] 4.4 Give the aspect file a header documenting its two generators, the peer-authentication coupling between the unit name and the role name, the loopback port and why 8080 is not used, `hideRefFromClients` and why, and the two-ruleset/forge-derived-check invariant — verify: the header names each item, including G2's single added check and setup's creation of a second ruleset, and `just lint` passes inside the dev shell ## 5. Host composition @@ -64,7 +64,7 @@ R1/R2 source filtering and cache warming and R15/R16 authorization procedure rem - [ ] 11.3 Verify the database and role exist for the dynamic user — verify: on the host `sudo -u postgres psql -c '\l'` lists `gitea-mq` owned by `gitea-mq`, `psql -c '\du'` shows the role without superuser or createdb, and `systemctl show gitea-mq.service -p User -p DynamicUser` reports `User=gitea-mq DynamicUser=yes` - [ ] 11.4 Verify setup's two-ruleset result — verify: read `gh api /repos/cameronraysmith/vanixiets/rulesets` and each ruleset body; ours retains `deletion`, `non_fast_forward`, both nixbot contexts pinned to `4743700`, and App integration bypass; the separate active `gitea-mq` ruleset requires only `gitea-mq` pinned to the queue App, with App/admin bypass. No `required_linear_history` exists, `gh api /repos/cameronraysmith/vanixiets/branches/main/protection` returns 404, and `gh api /repos/cameronraysmith/vanixiets --jq .allow_auto_merge` is `true`. Setup may create its own ruleset and add bypass to ours; require no setup permission warnings. If a disabled queue ruleset was pre-created, verify its approved activation explicitly. - [ ] 11.5 Verify V3 and the forge-derived required-check invariant — verify: on a pull request whose tip nixbot has built, `gh api /repos/cameronraysmith/vanixiets/commits//check-runs --jq '.check_runs[].name'` includes the exact names `nixbot/nix-eval` and `nixbot/nix-build`; read the ruleset bodies and classic-protection response to establish that `internal/github/forge.go::GetRequiredChecks` returns exactly that pair after excluding queue-owned contexts. `internal/monitor/monitor.go::ResolveRequiredChecks` prefers this non-empty forge list, so `GITEA_MQ_REQUIRED_CHECKS` does not fire. Corroborate through runtime logs if available; log/dashboard visibility of the resolved list is unverified and must not be assumed. -- [ ] 11.6 Verify the four landing settings in the running unit's environment — verify: `systemctl show gitea-mq.service -p Environment` on the host carries `GITEA_MQ_BATCH_MAX=5`, `GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE=true`, `GITEA_MQ_REQUIRED_CHECKS=nixbot/nix-eval,nixbot/nix-build`, and no `GITEA_MQ_MERGE_LABEL` -- [ ] 11.7 Verify V2 and the singleton shortcut end to end under `batchMax = 5` — verify: choose one ordinary trunk PR with an up-to-date head and both nixbot checks green before authorization, apply the sibling E1 policy, then enable native auto-merge with no other queued entry. Record head SHA, tested SHA, and landing SHA equality, GitHub `merged_at`, queue success status, no batch ref created, and elapsed signal-to-landing time. Do not infer no merge commits from parent count: the original head may already contain them. If target movement or comparison failure causes a batch-branch rebuild, record that path rather than claiming the shortcut (`internal/batch/batch.go::{Engine.headIfUpToDate,Engine.HandlePass}`). +- [ ] 11.6 Verify the four landing settings in the running unit's environment — verify: `systemctl show gitea-mq.service -p Environment` on the host carries `GITEA_MQ_BATCH_MAX=20`, `GITEA_MQ_SKIP_QUEUE_IF_UP_TO_DATE=true`, `GITEA_MQ_REQUIRED_CHECKS=nixbot/nix-eval,nixbot/nix-build`, and no `GITEA_MQ_MERGE_LABEL` +- [ ] 11.7 Verify V2 and the singleton shortcut end to end under `batchMax = 20` — verify: choose one ordinary trunk PR with an up-to-date head and both nixbot checks green before authorization, apply the sibling E1 policy, then enable native auto-merge with no other queued entry. Record head SHA, tested SHA, and landing SHA equality, GitHub `merged_at`, queue success status, no batch ref created, and elapsed signal-to-landing time. Do not infer no merge commits from parent count: the original head may already contain them. If target movement or comparison failure causes a batch-branch rebuild, record that path rather than claiming the shortcut (`internal/batch/batch.go::{Engine.headIfUpToDate,Engine.HandlePass}`). - [ ] 11.8 Re-confirm discharged V1 at the first live stacked landing, replacing the retired V6 landing-ref probe — verify: after native registration and head-ancestry checks and authorization under the sibling R15/R16 procedure, label only the topmost intended PR `merge-queue` and ensure every member has auto-merge disabled. Retain each member's head SHA and `merged_at`, observe GitHub retargeting/marking, and check subsequent mergify-cli sync recognizes every landed member as merged. V1 is discharged by operator confirmation of this mechanism and empirical non-stack Mic92/dotfiles #5887–#5890; this is re-confirmation, not a promotion blocker. The queue posts completion and runs `Engine.ensureMergedOrClose` only for actual entries, not lower members (`internal/batch/batch.go::Engine.HandlePass`). - [ ] 11.9 Verify the rollback path is available — verify: in a separate authorized scratch workspace, removing `gitea-mq` from magnetite's aspect list makes `nix eval .#checks.x86_64-linux.nixos-magnetite.drvPath` succeed with a different derivation path, then revert the scratch edit. Record the G2 reverse diff separately from the operator-approved disabling of the App-owned queue gate when stopping the service; preserve our nixbot build check and protection rules. No rollback mutation is performed by this correction session.