diff --git a/.agents/skills/fleet-queue/SKILL.md b/.agents/skills/fleet-queue/SKILL.md index 8b385a0..f705c83 100644 --- a/.agents/skills/fleet-queue/SKILL.md +++ b/.agents/skills/fleet-queue/SKILL.md @@ -474,7 +474,7 @@ Three answers, and the third is not the second: | it is not, or not from this branch | **leaves the task OPEN** and says so, loudly | | could not run | closes the task, and says the check could not run | -"Could not run" is `gh` absent, no network, a pull request it cannot read, or a +"Could not run" is the forge CLI absent, no network, a pull request it cannot read, or a base branch this machine cannot see. That must never read as a pass or a fail — CI and an offline laptop both still have to collect. `queue.sh show ` prints the method and the verdict, so both survive the scrollback. @@ -509,7 +509,7 @@ So a task gets a state AFTER `done`: | `abandoned` | the pull request was closed unmerged | released; the work is NOT on main | | `stuck` / `failed` | the worker gave up | **kept** — that session is the evidence, and you decide | -`landed` comes from asking `gh`, never from a worker claiming it, so it works +`landed` comes from asking the forge, never from a worker claiming it, so it works long after the session is gone. **Blockers clear on `landed`**, not on `done` — a dependent task waits for the code to actually be on `main`, which is the same bug in its other form: a task collected `shipped` once released its @@ -689,11 +689,11 @@ opened it. first pull request its worker reported. #25 was a *second* pull request from a task whose artifact still pointed at the already-merged #23, so a shepherd reading artifacts could not see it and the unattended pass would never have -merged it; a PR opened outside the queue was invisible the same way. So it runs -`gh pr list --state open` against every repo the queue's tasks name, and each -open pull request gets exactly one of these: +merged it; a PR opened outside the queue was invisible the same way. So it asks +the forge for every open change request against every repo the queue's tasks +name, and each open pull request gets exactly one of these: -| What `gh` says | What happens | +| What the forge says | What happens | |---|---| | the head branch is in someone else's fork | reported, never merged, **never given an agent** | | `mergeable: CONFLICTING` | a fixer is dispatched to rebase | @@ -736,15 +736,18 @@ Three things it will not do, and they are what make it safe to run: `working` or `blocked` is left alone. So is one whose state is merely *observed* — `running`, `uncovered`, `unreported` are not the agent saying it is at rest (`thurbox-session` §4a). -- **It will not guess.** No `gh`, no network, no thurbox: it says what it could - not determine and carries on. A PR it could not read is never called broken +- **It will not guess.** No forge, no network, no thurbox: it says what it + could not determine and carries on. A PR it could not read is never called broken and never called ready. **On merging, which is the part that runs unattended.** `Thurbeen/fleet` is public and has a fork, so "merge every open PR on a timer" has to survive a stranger opening one. Fleet merges only in the repos on `AUTO_MERGE_REPOS` in -`scripts/lib/queue.py` — `Thurbeen/fleet` — and only when **all** of these -hold: +`scripts/lib/queue.py` — `github.com/Thurbeen/fleet` and +`github.com/Thurbeen/thurview` — and only when **all** of these hold. Entries +there are HOST-QUALIFIED and one that names no forge is refused rather than +matched: `Thurbeen/fleet` on github.com and `Thurbeen/fleet` on a self-hosted +instance are not the same repository. - **The head branch is in that repository**, not a fork. A stranger cannot create a branch here, so this is the one claim about a pull request that diff --git a/AGENTS.md b/AGENTS.md index 1eb1261..a92150f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,6 +47,17 @@ names every path and the reason for each. directory your shell is in**, so a second clone of this repo cannot silently fork it: `topic add` and `add` refuse there, everything else warns, and `queue.sh root` names the directory in use. +- `scripts/lib/forge.py` — the FORGE seam. Everything fleet knows about a + change request — a pull request on GitHub, a merge request on GitLab — it + asks this module for; `scripts/lib/queue.py` runs no forge CLI itself and + builds no forge URL. GitHub, through `gh`, is the one implementation shipped, + and it is a CONFIGURATION and not an assumption. The file's own header owns + the interface and how to add another. Two things follow: a repository is + identified by HOST plus path (`github.com/Thurbeen/fleet`), because a bare + `owner/repo` names two different repositories once two forges exist; and + `queue-selftest.sh` drives `collect`, the landing check and `shepherd` + through a second, fake forge with no network and no `gh` behind it, which is + what keeps the seam honest rather than merely asserted. - `orchestration/reconcile/` — the reconciler's runtime state: its supervisor's pid, the heartbeat proving its loop is ticking, its log, the advisory `nudge` flag and the `down` flag. Written by `./scripts/reconcile.sh` and created on @@ -141,8 +152,10 @@ The loop, driven by `./scripts/queue.sh`: the queue's tasks name, not the tasks' recorded artifacts. A PR is linked back by artifact or head branch; an unlinked one is still classified and merged, it just has no session to fix it. It merges - only in the repos `AUTO_MERGE_REPOS` names in `scripts/lib/queue.py`, and - only for a PR whose head branch is in that repo, opened by someone who can + only in the repos `AUTO_MERGE_REPOS` names in `scripts/lib/queue.py` — each + entry host-qualified, and one that names no forge refused rather than + matched — and only for a PR whose head branch is in that repo, opened by + someone who can push there, carrying a `no-mistakes` attestation for its **current** head. That attestation gate is the one thing the declared publish method moves: a task that was declared `no-mistakes` and carries none gets a diff --git a/FLEET.md b/FLEET.md index 2012841..0caf236 100644 --- a/FLEET.md +++ b/FLEET.md @@ -90,7 +90,9 @@ YAML by hand. Nothing to push — the map is gitignored. recorded artifacts, dispatches a fixer for one that conflicts, fails a check, was reviewed with changes requested, or was declared `no-mistakes` and carries no attestation for its current head, and squash-merges one that - clears every gate in the repos `AUTO_MERGE_REPOS` allows. It writes down + clears every gate in the repos `AUTO_MERGE_REPOS` allows — entries there + name their forge (`github.com/Thurbeen/fleet`), because a bare `owner/repo` + is two different repositories once two forges are configured. It writes down what it saw either way, so a task's record says `checks-running` or `unattested` and not just `shipped`. 7. Review the PRs; the operator merges every one `shepherd` did not. Sessions @@ -240,7 +242,8 @@ The register never costs a fact. Where the two pull against each other, the fact wins: - **Uncertainty is a state word, not a hedge.** `waiting`, `not listed`, - `unavailable — gh not found`. Never "probably", never "should be" — and + `unavailable — gh not found` (that word is the GitHub adapter's own; another + forge names its own tool). Never "probably", never "should be" — and never a register word standing in for one. `holding, awaiting telemetry` in place of `unavailable — gh not found` has broken this section, not styled it. diff --git a/orchestration/queue/POLICY.md b/orchestration/queue/POLICY.md index 57547cb..4883fca 100644 --- a/orchestration/queue/POLICY.md +++ b/orchestration/queue/POLICY.md @@ -90,8 +90,9 @@ The default for every task here is the frontmatter at the top of this file. request title becomes the commit on `main`. Write the title accordingly. - **You do not merge.** Opening it is where your work ends. `queue.sh shepherd` may later merge it for you in the repos its `AUTO_MERGE_REPOS` - allowlist names, but only once your pull request clears its gates — never - merge it yourself in the meantime. + allowlist names (host-qualified, as in `github.com/owner/repo`), but only + once your pull request clears its gates — never merge it yourself in the + meantime. ## Reporting back — write a file, do not send mail diff --git a/orchestration/queue/README.md b/orchestration/queue/README.md index fad9649..245c5c7 100644 --- a/orchestration/queue/README.md +++ b/orchestration/queue/README.md @@ -92,7 +92,7 @@ branch (carrying a `no-mistakes` attestation for its head, for that method), git for a commit that reached the base branch. A task whose artifact is not there is reported and left OPEN, because "use the pipeline" is an instruction about a method and a method leaves no trace anyone can read. A check that could -not run — no `gh`, no network, a base branch this machine cannot see — says +not run — no forge CLI, no network, a base branch this machine cannot see — says exactly that and is never counted as either verdict. `collect --allow-unverified` closes a flagged task once you have read that artifact yourself. `queue.sh show` prints the method, the verdict and the publish state diff --git a/scripts/lib/fleet_status.py b/scripts/lib/fleet_status.py index 59878de..623ac5f 100644 --- a/scripts/lib/fleet_status.py +++ b/scripts/lib/fleet_status.py @@ -5,12 +5,12 @@ # WHY THIS EXISTS. Answering "where are we?" used to cost three to five # commands spread over three checkouts and four tools: a `git status` and a # `git log` per checkout, `queue.sh list`, `queue.sh plan`, `thurbox-cli -# session list`, `gh pr list`. Most of a long session's tool +# session list`, the forge's own list. Most of a long session's tool # calls were situational awareness rather than work, and every one of them cost # a round trip and a piece of the context window. This is those calls, folded # into one screen the lead can afford to run reflexively. # -# THE ONE RULE: DEGRADE, NEVER FAIL. No network, no `gh`, no thurbox, no +# THE ONE RULE: DEGRADE, NEVER FAIL. No network, no forge CLI, no thurbox, no # queue — each of those costs exactly its own section, which then # says what it could not determine and why. Every probe funnels through run(), # which converts every way a subprocess can go wrong into a reason string, and @@ -55,6 +55,11 @@ def _load_queue(): fleetqueue = _load_queue() +# The forge seam, which queue.py has already loaded and keyed in sys.modules — +# so this is the SAME module object and therefore the same registry, not a +# second opinion about which forges are configured. +forge = fleetqueue.forge + # The checkout this file ships in, found from the file rather than from the # working directory — the lead may run this from anywhere. REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -242,57 +247,40 @@ def probe_sessions(tasks: list) -> dict: return sec -# --- pull requests ----------------------------------------------------------- +# --- change requests --------------------------------------------------------- -CHECK_FAIL = {"FAILURE", "TIMED_OUT", "CANCELLED", "ACTION_REQUIRED", "STARTUP_FAILURE", "ERROR"} -CHECK_PASS = {"SUCCESS", "NEUTRAL", "SKIPPED"} +def rollup(checks) -> str: + """One word for a change request's checks: passing, failing, pending, or none. -def rollup(entries) -> str: - """One word for a PR's checks: passing, failing, pending, or none.""" - if not isinstance(entries, list) or not entries: + `cancelled` counts as failing here — this line's own long-standing + reading, separate from the shepherd's, which is why the forge hands back + `cancelled` as its own verdict rather than pre-deciding for either. + """ + if not checks: return "none" - failing = pending = 0 - for e in entries: - if not isinstance(e, dict): - continue - verdict = (e.get("conclusion") or e.get("state") or "").upper() - if verdict in CHECK_FAIL: - failing += 1 - elif verdict in CHECK_PASS: - continue - else: - pending += 1 - if failing: + if any(c.verdict in ("failed", "cancelled") for c in checks): return "failing" - return "pending" if pending else "passing" - - -def pr_slug(url: str) -> str: - """`Thurbeen/fleet#13` out of the URL gh already handed back. - - Deliberately not a second `gh repo view`: the identity is in the artifact, - and a status command should not spend an API call to pretty-print a name. - """ - parts = [p for p in str(url).split("/") if p] - if len(parts) >= 4 and parts[-2] == "pull": - return f"{parts[-4]}/{parts[-3]}#{parts[-1]}" - return str(url) + return "pending" if any(c.verdict == "pending" for c in checks) else "passing" def probe_prs(tasks: list) -> dict: - """Open PRs in the repos this queue is working in, matched back to tasks. + """Open change requests in the repos this queue works in, matched to tasks. - Matched by recorded artifact first, then by branch — so a PR a worker - opened and has not reported yet still shows up, which is exactly the gap - between "the worker should have opened a PR" and the artifact itself. + Matched by recorded artifact first, then by branch — so one a worker opened + and has not reported yet still shows up, which is exactly the gap between + "the worker should have opened a pull request" and the artifact itself. - One `gh pr list` per distinct repo, not one per task. + One list call per distinct repo, not one per task. Asked of the CHECKOUT + and not of a repository id: this command has a path on disk and no identity + for it, and a directory that is not a git repository at all still has to + produce a sentence rather than an empty list that reads as "nothing is + open". `scripts/lib/forge.py` decides which forge answers. """ sec: dict = {"unavailable": None, "prs": [], "errors": []} - # A remote task's `repo` is a path on its host, so `gh -C` here would ask - # the wrong filesystem and report "no such directory" about a checkout that - # exists. Skipped and SAID, rather than turned into an error that reads as + # A remote task's `repo` is a path on its host, so asking a forge CLI here + # would ask the wrong filesystem and report "no such directory" about a + # checkout that exists. Skipped and SAID, rather than turned into an error that reads as # a broken record. live = [t for t in tasks if t.get("repo") and t.get("state") != "queued"] # `kind` is what the headline counts. `unread` is a repo this sweep tried @@ -300,7 +288,7 @@ def probe_prs(tasks: list) -> dict: # swept, which is a different sentence and must not read as a failure. remote = [ {"repo": f"(on host {h})", "kind": "skipped", - "reason": "runs on a remote host; its pull requests are read by " + "reason": "runs on a remote host; its change requests are read by " "`queue.sh shepherd`, which asks the forge and not a checkout"} for h in sorted({t["host"] for t in live if t.get("host")}) ] @@ -319,41 +307,33 @@ def probe_prs(tasks: list) -> dict: sec["errors"].append({"repo": repo, "kind": "unread", "reason": "no such directory"}) continue - doc, why = run_json( - ["gh", "pr", "list", "--state", "open", "--limit", "50", "--json", - "number,url,title,headRefName,state,statusCheckRollup"], - cwd=repo, - timeout=20, - ) + crs, why = forge.open_change_requests_in_checkout(repo) if why: reasons.append(why) sec["errors"].append({"repo": repo, "kind": "unread", "reason": why}) continue - for pr in doc if isinstance(doc, list) else []: - if not isinstance(pr, dict): - continue - url = str(pr.get("url") or "") - head = pr.get("headRefName") + for cr in crs: + url, head = cr.url, cr.head_branch owner = next( (t for t in owners if t.get("artifact") and str(t["artifact"]).rstrip("/") == url.rstrip("/")), None, ) or next((t for t in owners if head and t.get("branch") == head), None) if owner is None: - continue # somebody else's PR in the same repo + continue # somebody else's change request in the same repo sec["prs"].append( { "ref": owner["ref"], "repo": repo, - "slug": pr_slug(url), - "number": pr.get("number"), + "slug": cr.name, + "number": cr.number, "url": url, - "title": pr.get("title"), + "title": cr.title, "branch": head, - "checks": rollup(pr.get("statusCheckRollup")), + "checks": rollup(cr.checks), } ) - # Every repo failed the same way — `gh` absent, most likely — so that is + # Every repo failed the same way — no forge CLI, most likely — so that is # the section's story rather than a list of identical per-repo errors. if reasons and len(sec["errors"]) == len(repos) and len(set(reasons)) == 1: sec["unavailable"] = reasons[0] diff --git a/scripts/lib/forge.py b/scripts/lib/forge.py new file mode 100644 index 0000000..91b4f08 --- /dev/null +++ b/scripts/lib/forge.py @@ -0,0 +1,714 @@ +#!/usr/bin/env python3 +"""The FORGE seam: what fleet needs to know about a proposed change, and +nothing about which forge answers it. + +WHY THIS EXISTS. Fleet's queue used to run `gh` inline in six places and build +`https://github.com/...` in a seventh, so "the forge" and "GitHub" were the +same word. They are not: the operator's fleet runs on GitHub *today*, and that +is a configuration, not a fact about the model. This module is the line between +the two. Above it, `queue.py` asks questions; below it, one adapter per forge +answers them with whatever CLI or API that forge has. + +THE WORD. A GitHub *pull request* and a GitLab *merge request* are the same +thing to this code, so the code says CHANGE REQUEST and never picks a side. +Prose that is genuinely about GitHub still says "pull request", because there +it is describing GitHub. `ChangeRequest` below is the whole vocabulary. + +THE QUESTIONS, and they are the entire interface. Each one is something fleet +actually decides on, and nothing here exists because a forge happens to offer +it: + + parse_change_url is this URL a change request, and which one + repo_from_remote which repository is this checkout's `origin` + get one change request in full: what `collect` needs to + check a publish claim — the body carrying the + no-mistakes attestation, the head commit it must + name, the branch it is open from, and the commits + that grew the head since + state open / merged / closed, for the landing check. A + second, narrower question than `get` on purpose: + `reap` sweeps every concluded task on a timer and + must not pay for a body it will not read + open_change_requests every open change request on a repository + open_change_requests_in_checkout + the same, asked of a local checkout rather than of + a repository id — what `fleet-status.sh` needs and + the only caller that has a path but no identity + can_push may this account push to this repository — the last + gate before an unattended merge + merge merge it, by a named method + +REPOSITORY IDENTITY CARRIES A HOST. `Thurbeen/fleet` names two different +repositories if two forges are configured, and self-hosted instances are the +NORMAL case for everything that is not github.com. So a repository is a +`RepoId(host, path)` and `AUTO_MERGE_REPOS` in queue.py is spelled +`github.com/Thurbeen/fleet` — an entry that names no host is refused rather +than guessed at. + +EVERY ANSWER CAN BE "I COULD NOT TELL". Each method returns its answer beside a +non-empty string saying why it could not be had, and no caller is allowed to +collapse that string into a verdict. A timeout must never be able to +manufacture a merge. + +ADDING A FORGE. Write a class with the methods below and register it: either +in `BUILTIN` here, or — for a test, or a forge that is not fleet's business to +ship — through `FLEET_FORGE_PLUGINS`, a colon-separated list of Python files +each exporting `forges()`. `scripts/queue-selftest.sh` drives the whole queue +through a plugin with no network and no `gh` behind it, which is how this seam +is proved rather than asserted. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import re +import shutil +import subprocess +import sys +from dataclasses import dataclass, field + +# --- identity ---------------------------------------------------------------- + + +@dataclass(frozen=True) +class RepoId: + """One repository, on one forge. The host is half of the name. + + `path` is whatever that forge puts after the host — `owner/repo` on GitHub + and Gitea, `group/subgroup/project` on GitLab. Nothing here parses it apart + from `owner`, which is the first segment on every forge fleet has met. + """ + + host: str + path: str + + def __str__(self) -> str: + # Reads as a sentence in a report: "... on many-owner/many-repo on + # github.com". `qualified` is the form for config and for JSON. + return f"{self.path} on {self.host}" + + @property + def qualified(self) -> str: + """`github.com/Thurbeen/fleet` — the form config files are written in.""" + return f"{self.host}/{self.path}" + + @property + def owner(self) -> str: + return self.path.split("/", 1)[0] + + @classmethod + def parse(cls, text: str) -> RepoId | None: + """A host-qualified repository, or None for anything else. + + A bare `owner/repo` is REFUSED and not guessed at. It is ambiguous the + moment a second forge exists, and the config that used to be written + that way is the auto-merge allowlist — the one place where guessing + wrong means acting on somebody else's repository. + """ + parts = [p for p in str(text or "").strip().split("/") if p] + if len(parts) < 3 or "." not in parts[0]: + return None + return cls(parts[0].lower(), "/".join(parts[1:])) + + +@dataclass(frozen=True) +class ChangeRef: + """A change request, named rather than fetched: which repository, and which number.""" + + repo: RepoId + number: int + url: str + + def __str__(self) -> str: + return self.url + + +@dataclass(frozen=True) +class Commit: + """One commit on a change request's head branch, oldest first. + + Read for ONE thing: telling the pipeline's own follow-up push apart from + somebody else pushing over it, which is the same refusal with two different + remedies. A forge that cannot enumerate them answers with an empty list, + and an empty list must never become a claim about who pushed what. + """ + + sha: str + headline: str + + +@dataclass(frozen=True) +class Check: + """One CI check, in fleet's own words rather than each forge's vocabulary. + + `pending` is never `failed`: reading a check that has not finished as a + broken one is how a shepherd spawns fixers for healthy change requests, and + reading it as passed is how it merges one whose CI never ran. `cancelled` + is its own word rather than folded into either: a check somebody called + off is not a passing one, but the shepherd and fleet-status have always + disagreed about whether it blocks a merge — see `GH_CHECK_FAILED` below — + and a shared verdict must let both keep their own answer. + """ + + name: str + verdict: str # "passed" | "failed" | "pending" | "cancelled" + + +@dataclass +class ChangeRequest: + """One proposed change, normalised. Every field is fleet's word, not a forge's. + + An empty string means THE FORGE DID NOT SAY, everywhere. Callers that act + on a field check it rather than defaulting it — `head_is_ours` is `None` + for "could not tell" for exactly that reason, and `classify` treats that as + undetermined rather than as a stranger or as one of ours. + """ + + ref: ChangeRef + title: str = "" + state: str = "" # "open" | "merged" | "closed" | "" (not said) + draft: bool = False + body: str = "" + head_branch: str = "" + base_branch: str = "" + head_sha: str = "" + author: str = "" + author_is_bot: bool = False + mergeable: str = "" # "mergeable" | "conflicting" | "" (not said / still computing) + review_decision: str = "" # "changes-requested" | "approved" | "" (not said) + checks: list = field(default_factory=list) + commits: list = field(default_factory=list) + # Is the head branch inside the target repository? The one claim about a + # change request that whoever opened it cannot write for themselves, and + # the only forge-specific judgement fleet delegates rather than derives: + # a fork, a mirror and a same-repo branch are told apart differently on + # every forge. `None` means the forge did not say. + head_is_ours: bool | None = None + head_location: str = "" # a phrase naming where the head branch lives + + @property + def repo(self) -> RepoId: + return self.ref.repo + + @property + def number(self) -> int: + return self.ref.number + + @property + def url(self) -> str: + return self.ref.url + + @property + def name(self) -> str: + """`Thurbeen/fleet#13` — short enough for a status line.""" + return f"{self.repo.path}#{self.number}" + + +# The shape of a change request URL on any forge fleet has met: GitHub and +# Gitea end in `/pull/`, GitLab in `/-/merge_requests/`. This answers +# only "is that artifact a change request at all" — which forge OWNS it, and +# whether that forge is configured, are separate questions with separate +# answers, so that an artifact on a forge nobody configured reads as "could +# not be checked" rather than as "the worker shipped nothing". +CHANGE_URL_RE = re.compile( + r"^(https?://[^/\s]+/[^/\s]+(?:/[^/\s]+)+?/(?:pull|merge_requests)/\d+)" + r"(?:[/?#].*)?$" +) + + +def change_url(url) -> str: + """The canonical change-request URL inside `url`, or '' if it is not one. + + Group 1 rather than the whole string, so a link someone pasted with + `/files` or a `#comment` on the end still resolves to what it names. + """ + m = CHANGE_URL_RE.match((url or "").strip()) + return m.group(1) if m else "" + + +# --- the interface ----------------------------------------------------------- + + +class Forge: + """One forge. Subclass, implement, register. + + The base class answers "I cannot" to everything, so a partial adapter + degrades into "could not be determined" — which every caller already + handles — instead of raising into the middle of a shepherd pass. + """ + + name = "forge" + hosts: tuple = () + # The merge methods this forge can actually perform. A caller asking for + # one that is not here is told so BEFORE anything is merged: a GitLab + # project can forbid squash, and "the forge refused this merge method" has + # to be a sentence fleet can say. + merge_methods: tuple = () + + def owns_host(self, host: str) -> bool: + return (host or "").lower() in self.hosts + + def parse_change_url(self, url: str) -> ChangeRef | None: + return None + + def repo_from_remote(self, remote_url: str) -> RepoId | None: + return None + + def get(self, ref: ChangeRef) -> tuple: + return None, f"{self.name} cannot read a change request" + + def state(self, ref: ChangeRef) -> tuple: + return None, f"{self.name} cannot read a change request state" + + def open_change_requests(self, repo: RepoId) -> tuple: + return [], f"{self.name} cannot list change requests" + + def open_change_requests_in_checkout(self, path: str) -> tuple: + return [], f"{self.name} cannot list change requests" + + def can_push(self, repo: RepoId, login: str) -> tuple: + return False, f"{self.name} cannot say who may push to {repo}" + + def describe_merge(self, method: str, delete_branch: bool) -> str: + """What this forge would run, for a dry run to print.""" + return f"{self.name}: merge by {method}" + + def merge(self, cr: ChangeRequest, method: str, delete_branch: bool) -> tuple: + return False, f"{self.name} cannot merge" + + +# --- GitHub, the first implementation ---------------------------------------- + + +# `gh pr list --limit` is a request cap, not a page size — gh paginates the +# GraphQL calls itself to reach it. Set high enough that hitting it means the +# repository genuinely has that many open pull requests, which the caller then +# treats as unreadable rather than silently returning a truncated list. +GH_LIST_LIMIT = 1000 + +# One `gh pr list` answers every question the shepherd asks, so a pass costs +# one call per repository rather than one per pull request. The last four are +# the safety fields: `headRefOid` is what an attestation has to name, and the +# other three are how a fork's pull request is told from ours. +GH_FIELDS = ( + "number,state,url,title,isDraft,mergeable,reviewDecision," + "statusCheckRollup,body,headRefName,baseRefName,headRefOid," + "author,headRepositoryOwner,isCrossRepository" +) + +# The narrower set `fleet-status.sh` needs: it prints a line per pull request +# and decides nothing, so it does not pay for the safety fields. +GH_STATUS_FIELDS = "number,url,title,headRefName,state,statusCheckRollup" + +# What ONE change request costs when `collect` checks a publish claim. The head +# branch is in there because it is the one claim about a pull request a worker +# cannot write into its own result.md, and it arrives free with the body. +GH_ONE_FIELDS = "body,headRefOid,headRefName,state,commits" + +# A check that FAILED. Anything still running is NOT a failure. `CANCELLED` is +# its own conclusion, mapped to the `cancelled` verdict rather than in here: +# it is absent from the shepherd's list and present in fleet-status's, both +# already so before this module existed, and this keeps that difference alive +# instead of erasing it onto whichever caller happened to read second. +GH_CHECK_FAILED = {"FAILURE", "TIMED_OUT", "STARTUP_FAILURE", "ACTION_REQUIRED", "ERROR"} +GH_CHECK_PASSED = {"SUCCESS", "NEUTRAL", "SKIPPED"} +GH_CHECK_CANCELLED = {"CANCELLED"} + +GH_URL_RE = re.compile( + r"^https?://([^/\s]+)/([^/\s]+/[^/\s]+?)(?:\.git)?/pull/(\d+)(?:[/?#].*)?$" +) +GH_REMOTE_RE = re.compile(r"^(?:[^@/\s]+@)?([^:/\s]+)[:/]([^/\s]+/[^/\s]+?)(?:\.git)?/?$") + +# GitHub's own state words, mapped onto fleet's three. Anything else is not +# translated into a guess: the caller is told the forge said something this +# adapter does not know, which is `unknown` and never `open`. +GH_STATES = {"OPEN": "open", "MERGED": "merged", "CLOSED": "closed"} + +GH_PUSH_PERMISSIONS = {"admin", "maintain", "write"} + + +class GitHubForge(Forge): + """GitHub, through the `gh` CLI. Every `gh` invocation fleet makes is here. + + `gh` rather than the REST API directly because it already holds the + operator's credentials, and because a fleet that needed its own token would + need one per machine a worker runs on. + """ + + name = "github" + merge_methods = ("squash", "merge", "rebase") + + def __init__(self, hosts=None): + # `GH_HOST` is gh's own variable for a GitHub Enterprise instance, so a + # self-hosted GitHub round-trips through this adapter the same way a + # self-hosted GitLab will have to through its own. + extra = [h.strip().lower() for h in (hosts or []) if h and h.strip()] + enterprise = os.environ.get("GH_HOST", "").strip().lower() + if enterprise: + extra.append(enterprise) + self.hosts = tuple(dict.fromkeys(["github.com", "www.github.com"] + extra)) + # One answer per (repo, login) per process. The question does not + # change inside a run and every open pull request would ask it again. + self._push: dict = {} + + # --- naming --- + + def parse_change_url(self, url: str) -> ChangeRef | None: + m = GH_URL_RE.match((url or "").strip()) + if not m or not self.owns_host(m.group(1)): + return None + host = m.group(1).lower() + return ChangeRef( + RepoId(host, m.group(2)), + int(m.group(3)), + f"https://{host}/{m.group(2)}/pull/{m.group(3)}", + ) + + def repo_from_remote(self, remote_url: str) -> RepoId | None: + m = GH_REMOTE_RE.match((remote_url or "").strip()) + if not m: + return None + host = m.group(1).lower() + # `git@github.com:owner/repo` has no scheme, so the host is whatever + # came before the colon; a URL that named none of our hosts is not ours. + if not self.owns_host(host): + return None + return RepoId(host, m.group(2)) + + # --- running gh --- + + def _run(self, argv: list, cwd: str | None = None, timeout: int = 60) -> tuple: + """(stdout, why-not). A non-empty second value is never a verdict.""" + if not shutil.which("gh"): + return None, "gh not found on PATH" + try: + out = subprocess.run( + ["gh"] + argv, capture_output=True, text=True, cwd=cwd, timeout=timeout + ) + except (OSError, subprocess.SubprocessError) as exc: + return None, f"gh could not be run: {exc}" + if out.returncode != 0: + detail = ((out.stderr or "") + (out.stdout or "")).strip().splitlines() + return None, (detail[0] if detail else f"gh exited {out.returncode}") + return out.stdout, "" + + def _json(self, argv: list, cwd: str | None = None, timeout: int = 60) -> tuple: + out, why = self._run(argv, cwd=cwd, timeout=timeout) + if why: + return None, why + try: + return json.loads(out), "" + except ValueError: + return None, "gh returned output that is not JSON" + + # --- the questions --- + + def get(self, ref: ChangeRef) -> tuple: + doc, why = self._json( + ["pr", "view", ref.url, "--json", GH_ONE_FIELDS], timeout=30 + ) + if why: + return None, f"gh pr view failed: {why}" + if not isinstance(doc, dict): + return None, "gh pr view did not answer with an object" + doc["url"] = ref.url + cr = self._change_request(doc, ref.repo) + if cr is None: + return None, "gh pr view did not answer about a pull request" + return cr, "" + + def state(self, ref: ChangeRef) -> tuple: + out, why = self._run( + ["pr", "view", ref.url, "--json", "state", "-q", ".state"], timeout=30 + ) + if why: + return None, f"gh pr view could not read the state: {why}" + said = (out or "").strip().upper() + state = GH_STATES.get(said) + if not state: + return None, f"gh answered an unrecognised pull request state: {said!r}" + return state, "" + + def open_change_requests(self, repo: RepoId) -> tuple: + docs, why = self._json( + ["pr", "list", "--repo", repo.path, "--state", "open", + "--limit", str(GH_LIST_LIMIT), "--json", GH_FIELDS] + ) + return self._listed(docs, why, repo, GH_LIST_LIMIT) + + def open_change_requests_in_checkout(self, path: str) -> tuple: + docs, why = self._json( + ["pr", "list", "--state", "open", "--limit", "50", + "--json", GH_STATUS_FIELDS], + cwd=path, + timeout=20, + ) + return self._listed(docs, why, None, 50) + + def _listed(self, docs, why: str, repo: RepoId | None, limit: int) -> tuple: + if why: + return [], why + if not isinstance(docs, list): + return [], "gh returned something that is not a list of pull requests" + docs = [d for d in docs if isinstance(d, dict)] + if len(docs) >= limit: + return [], ( + f"the repository has at least {limit} open pull requests; gh's " + "result may be truncated, so treating it as unreadable rather " + "than silently dropping some" + ) + out = [] + for d in docs: + cr = self._change_request(d, repo) + if cr: + out.append(cr) + return out, "" + + def _change_request(self, d: dict, repo: RepoId | None) -> ChangeRequest | None: + url = str(d.get("url") or "") + ref = self.parse_change_url(url) + if ref is None: + if repo is None or not d.get("number"): + return None + # A repository was asked for by name and gh answered with a pull + # request whose url it did not give: rebuild it rather than drop it. + n = int(d["number"]) + ref = ChangeRef(repo, n, url or f"https://{repo.host}/{repo.path}/pull/{n}") + + owner = str((d.get("headRepositoryOwner") or {}).get("login") or "") + if not owner: + ours, where = None, "" + elif d.get("isCrossRepository"): + ours, where = False, f"{owner}'s fork" + else: + ours = owner.lower() == ref.repo.owner.lower() + where = f"{owner}'s repository" + + author = d.get("author") or {} + return ChangeRequest( + ref=ref, + title=str(d.get("title") or ""), + state=GH_STATES.get(str(d.get("state") or "").upper(), ""), + draft=bool(d.get("isDraft")), + body=str(d.get("body") or ""), + head_branch=str(d.get("headRefName") or ""), + base_branch=str(d.get("baseRefName") or ""), + head_sha=str(d.get("headRefOid") or ""), + author=str(author.get("login") or ""), + author_is_bot=bool(author.get("is_bot")), + mergeable={"MERGEABLE": "mergeable", "CONFLICTING": "conflicting"}.get( + str(d.get("mergeable") or "").upper(), "" + ), + review_decision={"CHANGES_REQUESTED": "changes-requested", + "APPROVED": "approved"}.get( + str(d.get("reviewDecision") or "").upper(), "" + ), + checks=self._checks(d.get("statusCheckRollup")), + commits=[ + Commit(str(c.get("oid") or ""), str(c.get("messageHeadline") or "")) + for c in (d.get("commits") or []) + if isinstance(c, dict) + ], + head_is_ours=ours, + head_location=where, + ) + + @staticmethod + def _checks(rollup) -> list: + out = [] + for c in rollup or []: + if not isinstance(c, dict): + continue + name = c.get("name") or c.get("context") or "a required check" + if "state" in c and "conclusion" not in c: + # A StatusContext: one word, and PENDING is not a failure. + said = str(c.get("state") or "").upper() + elif str(c.get("status") or "").upper() != "COMPLETED": + out.append(Check(name, "pending")) + continue + else: + said = str(c.get("conclusion") or "").upper() + if said in GH_CHECK_FAILED: + out.append(Check(name, "failed")) + elif said in GH_CHECK_PASSED: + out.append(Check(name, "passed")) + elif said in GH_CHECK_CANCELLED: + out.append(Check(name, "cancelled")) + else: + out.append(Check(name, "pending")) + return out + + def can_push(self, repo: RepoId, login: str) -> tuple: + """Asked as "may this login push here" rather than "is this the owner". + + The owner of `Thurbeen/fleet` is an organisation and no pull request is + ever authored by one. `gh` has no `authorAssociation` field in every + version; the collaborator permission endpoint is in all of them. + """ + key = (repo.qualified, login) + if key in self._push: + return self._push[key] + doc, why = self._json( + ["api", f"repos/{repo.path}/collaborators/{login}/permission"] + ) + if why or not isinstance(doc, dict): + answer = ( + False, + f"could not check whether {login} can push to {repo}: " + f"{why or 'unexpected output'}", + ) + else: + perm = str(doc.get("permission") or "").lower() + # GitHub spells "no access" as the literal string `none`. + said = "no" if perm in ("", "none") else perm + answer = (perm in GH_PUSH_PERMISSIONS, f"{login} has {said} access to {repo}") + self._push[key] = answer + return answer + + def describe_merge(self, method: str, delete_branch: bool) -> str: + return "gh pr merge --" + method + (" --delete-branch" if delete_branch else "") + + def merge(self, cr: ChangeRequest, method: str, delete_branch: bool) -> tuple: + if method not in self.merge_methods: + return False, f"github cannot merge by {method}" + argv = ["pr", "merge", cr.url, f"--{method}"] + if delete_branch: + argv.append("--delete-branch") + _, why = self._run(argv, timeout=120) + if why: + return False, why + return True, f"{method}-merged" + (", branch deleted" if delete_branch else "") + + +# --- the registry ------------------------------------------------------------ + + +BUILTIN = (GitHubForge,) + +# A colon-separated list of Python files, each exporting `forges()`. This is +# how the selftest drives the whole queue through a forge that has no network +# and no `gh` behind it, and how a forge fleet does not ship can be tried out +# without editing this file. +PLUGIN_ENV = "FLEET_FORGE_PLUGINS" + +_REGISTRY: list | None = None + +# So a plugin file can `import fleet_forge` and get THIS module whichever name +# it was loaded under — `forge` off sys.path, `fleet_forge` through queue.py's +# importlib loader. One module object means one registry. +sys.modules.setdefault("fleet_forge", sys.modules[__name__]) + + +def forges() -> list: + """Every configured forge, built-ins first. Cached for the process.""" + global _REGISTRY + if _REGISTRY is None: + _REGISTRY = [cls() for cls in BUILTIN] + for path in os.environ.get(PLUGIN_ENV, "").split(":"): + path = path.strip() + if path: + _REGISTRY.extend(_load_plugin(path)) + return _REGISTRY + + +def reset() -> None: + """Forget the cached registry. For tests inside one process.""" + global _REGISTRY + _REGISTRY = None + + +def _load_plugin(path: str) -> list: + """Import one plugin file and take the forges it exports. + + A plugin that cannot be loaded is reported on stderr and skipped, never + raised: a bad entry in an environment variable must not take down a + shepherd pass that had nothing to do with it. + """ + try: + spec = importlib.util.spec_from_file_location( + "fleet_forge_plugin_" + re.sub(r"\W", "_", path), path + ) + if spec is None or spec.loader is None: + raise ImportError(f"{path} is not an importable Python file") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + got = mod.forges() + return [f for f in got if isinstance(f, Forge)] + except Exception as exc: # noqa: BLE001 - a plugin may fail any way it likes + print(f"forge: could not load {PLUGIN_ENV} entry {path}: {exc}", file=sys.stderr) + return [] + + +def for_url(url: str) -> tuple: + """(forge, ref) for a change request URL, or (None, why-not). + + "No configured forge owns that host" is a REASON and not a verdict: an + artifact on a forge nobody configured is one fleet could not ask about, + which is exactly what `collect` and the landing check call `unknown`. + """ + canonical = change_url(url) + if not canonical: + return None, "not a change request URL" + for f in forges(): + ref = f.parse_change_url(canonical) + if ref is not None: + return f, ref + host = canonical.split("/")[2] if "://" in canonical else canonical + return None, f"no configured forge owns {host}" + + +def for_repo(repo: RepoId) -> tuple: + """(forge, '') for a repository, or (None, why-not).""" + for f in forges(): + if f.owns_host(repo.host): + return f, "" + return None, f"no configured forge owns {repo.host}" + + +def repo_from_remote(remote_url: str) -> RepoId | None: + """The repository a git remote URL names, asked of every configured forge.""" + for f in forges(): + repo = f.repo_from_remote(remote_url) + if repo is not None: + return repo + return None + + +def open_change_requests_in_checkout(path: str) -> tuple: + """Every open change request in a local checkout, without naming its repository. + + The one question asked of a PATH rather than of a `RepoId`, because + `fleet-status.sh` has a checkout on disk and no identity for it — and a + directory that is not a git repository at all still has to produce a + sentence rather than an empty list that reads as "nothing is open". + """ + forge = None + remote = _git_remote(path) + if remote: + for f in forges(): + if f.repo_from_remote(remote) is not None: + forge = f + break + candidates = [forge] if forge else list(forges()) + why = "no forge is configured" + for f in candidates: + crs, err = f.open_change_requests_in_checkout(path) + if not err: + return crs, "" + why = err + return [], why + + +def _git_remote(path: str) -> str: + if not path or not os.path.isdir(path): + return "" + try: + out = subprocess.run( + ["git", "-C", path, "remote", "get-url", "origin"], + capture_output=True, text=True, timeout=10, + ) + except (OSError, subprocess.SubprocessError): + return "" + return out.stdout.strip() if out.returncode == 0 else "" diff --git a/scripts/lib/queue.py b/scripts/lib/queue.py index 36afd93..284641d 100644 --- a/scripts/lib/queue.py +++ b/scripts/lib/queue.py @@ -92,6 +92,34 @@ import yaml + +def _load_forge(): + """Load scripts/lib/forge.py beside this file, under a name of its own. + + Not a plain `import forge`: this file is loaded three ways — as `queue` off + `scripts/lib` on sys.path, and by `fleet_status.py` and the pane harness + through importlib with no path change at all — and only the first of those + would find a sibling module. Keyed in sys.modules so every one of them + shares ONE registry, and therefore one answer about which forges are + configured. + """ + if "fleet_forge" in sys.modules: + return sys.modules["fleet_forge"] + path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "forge.py") + spec = importlib.util.spec_from_file_location("fleet_forge", path) + mod = importlib.util.module_from_spec(spec) + sys.modules["fleet_forge"] = mod + spec.loader.exec_module(mod) + return mod + + +# WHICH FORGE. Everything fleet knows about a change request — a pull request +# on GitHub, a merge request on GitLab — it asks this module for. Nothing in +# this file runs `gh` or builds a github.com URL; the one exception is +# FORGE_PROBE's ssh check below, which is a different coupling (git hosting, +# not the forge API) and says so where it lives. +forge = _load_forge() + # The four conditions that justify making one task wait for another. They are # firstmate's, and they are a closed set on purpose: "these edit the same file" # is not among them and cannot be spelled here. @@ -332,12 +360,17 @@ def brief_shortfall(path: str) -> str: }, } -# A pull request URL, and nothing else — group 1 is the canonical form, so a -# link someone pasted with `/files` or a `#comment` on the end still resolves -# to the pull request it names. `not-applicable` and `stuck` produce no -# artifact at all, and an artifact that is not a PR (an issue, a doc, a commit) -# is not a pipeline claim — neither is a failure, and neither is checked. -PR_URL_RE = re.compile(r"^(https?://[^/\s]+/[^/\s]+/[^/\s]+/pull/\d+)(?:[/?#].*)?$") +# A pull request URL, and nothing else — `forge.change_url` returns the +# canonical form, so a link someone pasted with `/files` or a `#comment` on the +# end still resolves to the change request it names, and a `/-/merge_requests/` +# one resolves as readily as a `/pull/` one. `not-applicable` and `stuck` +# produce no artifact at all, and an artifact that is not a change request (an +# issue, a doc, a commit) is not a pipeline claim — neither is a failure, and +# neither is checked. +# +# Which forge OWNS a URL, and whether that forge is configured here, are +# separate questions with separate answers — so an artifact on a forge nobody +# configured reads as "could not be checked" and never as "shipped nothing". # The `push` method's artifact, in the same forge-agnostic shape. Group 1 is # the canonical link and group 2 the sha, which is the half git is asked about. @@ -1908,6 +1941,11 @@ def ssh_run(entry: dict, script: str, stdin: str | None = None): # HTTPS with a `gh` token has no such key and is perfectly able to push, so # testing only the key would refuse a working host. Either credential passes; # neither is read, moved, or reported beyond the word that says which was found. +# +# STILL GITHUB-SHAPED, and knowingly. This is GIT HOSTING — can this machine +# clone, fetch and push — and not the forge API that `scripts/lib/forge.py` +# covers. They are different couplings, and this one is left whole for its own +# task rather than half-done here. FORGE_PROBE = """\ if ssh -o BatchMode=yes -T git@github.com 2>&1 | grep -q 'successfully authenticated'; then printf 'an ssh key' @@ -1944,15 +1982,15 @@ def probe_host(entry: dict, repo: str) -> list: return out out.append({"check": "reachable", "ok": True, "detail": "answers ssh, POSIX shell"}) - forge = ssh_run(entry, FORGE_PROBE) - if forge.returncode != 0: + creds = ssh_run(entry, FORGE_PROBE) + if creds.returncode != 0: out.append({"check": "forge", "ok": False, "detail": ( "the host has no GitHub credentials of its own — neither an ssh key " "nor a `gh` login. It cannot clone, fetch or push. Give that MACHINE " "its own credentials; nothing here sends yours.")}) return out out.append({"check": "forge", "ok": True, - "detail": f"reaches GitHub with {forge.stdout.strip() or 'a credential'}"}) + "detail": f"reaches GitHub with {creds.stdout.strip() or 'a credential'}"}) quoted = shlex.quote(repo) check = ( @@ -2721,12 +2759,13 @@ def publish_verdict(task: Task, outcome, url) -> tuple[str, str]: passed the forge, or git, says the artifact this task's method names is there, from this task's branch, in the state claimed. missing `shipped` with no such artifact, or one that does not hold up. - unknown the check could not run — no `gh`, no network, no such pull - request, a base branch this machine cannot read. + unknown the check could not run — no forge configured for that + host, no network, no such change request, a base branch this + machine cannot read. `unknown` is a fourth word on purpose and never collapses into `passed` or - `missing`. An offline machine and a CI runner with no `gh` must both still - be able to collect, and "could not check" must never be reported as either + `missing`. An offline machine and a CI runner with no forge CLI must both + still be able to collect, and "could not check" must never be reported as either verdict — that is how a trusted claim gets manufactured out of a timeout. Every git call below goes through `git_out`, whose empty answer IS a failure, so `push` keeps that rule as strictly as the two forge methods do. @@ -2746,7 +2785,7 @@ def pull_request_verdict(task: Task, method: str, outcome, url) -> tuple[str, st """The forge as witness, for the two methods that end in a pull request. THE HEAD BRANCH IS CHECKED FOR BOTH, and it costs nothing — the field - arrives in the same `gh pr view`. It closes the one hole no body check ever + arrives in the same `forge.get`. It closes the one hole no body check ever closed: a worker pasting somebody ELSE's good pull request. "This pull request is from this task's branch" is the one claim about it that a worker cannot write into its own result.md. @@ -2761,50 +2800,37 @@ def pull_request_verdict(task: Task, method: str, outcome, url) -> tuple[str, st refusal — `pipeline_moved_the_head` tells the one stale attestation the pipeline caused itself apart from every other. It cannot change a verdict. """ - match = PR_URL_RE.match((url or "").strip()) - if not match: + if not forge.change_url(url): if outcome == "shipped": return "missing", "shipped with no pull request to check" return "skipped", "no pull request to check" - url = match.group(1) - if not shutil.which("gh"): - return "unknown", "gh not found on PATH" - try: - proc = subprocess.run( - ["gh", "pr", "view", url, "--json", - "body,headRefOid,headRefName,state,commits"], - capture_output=True, text=True, timeout=30, - ) - except (OSError, subprocess.SubprocessError) as exc: - return "unknown", f"gh pr view could not run: {exc}" - if proc.returncode != 0: - detail = (proc.stderr or proc.stdout).strip().splitlines() - return "unknown", "gh pr view failed: " + (detail[-1] if detail else "no output") - try: - pr = json.loads(proc.stdout) - except ValueError: - return "unknown", "gh pr view did not answer with JSON" + which, ref = forge.for_url(url) + if which is None: + return "unknown", ref + cr, why = which.get(ref) + if why: + return "unknown", why branch = str(task.doc.get("branch") or "") - head = str(pr.get("headRefName") or "") - if not head: - return "unknown", "GitHub did not say which branch this pull request is from" - if head != branch: + if not cr.head_branch: + return "unknown", "the forge did not say which branch this pull request is from" + if cr.head_branch != branch: return "missing", ( - f"the pull request is from branch {head}, and this task's is {branch}" + f"the pull request is from branch {cr.head_branch}, and this task's " + f"is {branch}" ) if method == "no-mistakes": - body = pr.get("body") - attested, why = attestation_verdict(body, pr.get("headRefOid") or "") + attested, why = attestation_verdict(cr.body, cr.head_sha) if not attested: - why += pipeline_moved_the_head(pr, body) + why += pipeline_moved_the_head(cr) return ("passed" if attested else "missing"), why - state = str(pr.get("state") or "").upper() - if state not in ("OPEN", "MERGED"): - return "missing", f"the pull request is {state.lower() or 'in no state gh named'}" - return "passed", f"the pull request is {state.lower()} and is from {branch}" + if cr.state not in ("open", "merged"): + return "missing", ( + f"the pull request is {cr.state or 'in no state the forge named'}" + ) + return "passed", f"the pull request is {cr.state} and is from {branch}" # The pipeline's own commits, which are the ONE way a `no-mistakes` branch @@ -2820,19 +2846,19 @@ def pull_request_verdict(task: Task, method: str, outcome, url) -> tuple[str, st PIPELINE_COMMIT_RE = re.compile(r"^(?:chore:\s*)?no-mistakes[:\s]", re.I) -def pipeline_moved_the_head(pr: dict, body: str) -> str: +def pipeline_moved_the_head(cr: forge.ChangeRequest) -> str: """The clause naming the pipeline's own commits, or "" for every other case. It only ever ADDS to `attestation_verdict`'s line. The verdict itself is not consulted and cannot be changed from here: this says why a refusal happened, never whether it should have. - "" is also what a case that cannot be TOLD APART reads as — `gh` answering - with no commit list, or with one whose last commit is not the head. An + "" is also what a case that cannot be TOLD APART reads as — the forge + answering with no commit list, or with one whose last commit is not the head. An absent list is silence, and silence must not become a claim about who pushed what. """ - m = ATTESTATION_RE.search(body or "") + m = ATTESTATION_RE.search(cr.body or "") if not m: return "" try: @@ -2840,24 +2866,18 @@ def pipeline_moved_the_head(pr: dict, body: str) -> str: except ValueError: return "" attested = str(doc.get("head_sha") or "").lower() if isinstance(doc, dict) else "" - head = str(pr.get("headRefOid") or "").lower() - commits = pr.get("commits") - if not attested or not head or not isinstance(commits, list) or not commits: + head = (cr.head_sha or "").lower() + commits = cr.commits + if not attested or not head or not commits: return "" - if not all(isinstance(c, dict) for c in commits): - return "" - oids = [str(c.get("oid") or "").lower() for c in commits] + oids = [c.sha.lower() for c in commits] if oids[-1] != head or attested not in oids: return "" after = commits[oids.index(attested) + 1:] - headlines = [str(c.get("messageHeadline") or "") for c in after] - if not after or not all(PIPELINE_COMMIT_RE.match(h) for h in headlines): + if not after or not all(PIPELINE_COMMIT_RE.match(c.headline) for c in after): return "" - named = ", ".join( - f"{str(c.get('oid'))[:8]} “{h}”" - for c, h in zip(after[:2], headlines[:2]) - ) + named = ", ".join(f"{c.sha[:8]} “{c.headline}”" for c in after[:2]) return ( f"; the pipeline pushed that head itself ({named}) after it attested, so " "nothing else has moved this branch — re-run `/no-mistakes --yes` and it " @@ -3090,7 +3110,7 @@ def cmd_collect(args) -> int: # # `landed` is a state the record keeps and everything reads: reaping, blocker # clearing, and `list`. Nothing re-opens — this is a LATER transition out of -# `done`, discovered by asking `gh`, never by a worker claiming it. +# `done`, discovered by asking the forge, never by a worker claiming it. # The only session states a reap will act on. `working` and `blocked` are the # ones that must never be touched, but they are not the whole exclusion: @@ -3131,43 +3151,32 @@ def artifact_landing(artifact) -> tuple[str, str]: branch, and the work did not land either. open the normal state of work awaiting review. Not an error and not a warning. - unknown the forge could not be asked — no `gh`, no network, no such - pull request. It never collapses into any of the others, for + unknown the forge could not be asked — none is configured for that + host, no network, no such change request. It never collapses into any of the others, for the same reason `collect`'s pipeline check has a fourth word: a timeout must not be able to manufacture a merge, and a merge is what authorises a deletion. """ - match = PR_URL_RE.match((artifact or "").strip()) - if not match: + url = forge.change_url(artifact) + if not url: return "none", "no pull request to wait for" - url = match.group(1) - if not shutil.which("gh"): - return "unknown", "gh not found on PATH" - try: - proc = subprocess.run( - ["gh", "pr", "view", url, "--json", "state", "-q", ".state"], - capture_output=True, text=True, timeout=30, - ) - except (OSError, subprocess.SubprocessError) as exc: - return "unknown", f"gh pr view could not run: {exc}" - if proc.returncode != 0: - detail = (proc.stderr or proc.stdout).strip().splitlines() - return "unknown", "gh pr view failed: " + (detail[-1] if detail else "no output") - - state = proc.stdout.strip().upper() - if state == "MERGED": + which, ref = forge.for_url(url) + if which is None: + return "unknown", ref + state, why = which.state(ref) + if why: + return "unknown", why + if state == "merged": return "merged", f"{url} is merged" - if state == "CLOSED": + if state == "closed": return "closed", f"{url} was closed without merging" - if state == "OPEN": - return "open", f"{url} is still open — work awaiting review" - return "unknown", f"gh answered an unrecognised pull request state: {state!r}" + return "open", f"{url} is still open — work awaiting review" def sweep_landings(q: Queue, dry: bool) -> dict: """Ask the forge about every `done` task and promote the ones that landed. - Works from the RECORD and `gh` alone. That is a requirement, not an + Works from the RECORD and the forge alone. That is a requirement, not an accident: a merge normally happens after the session that produced it is gone, so nothing here may depend on a worker being alive to say so. @@ -3950,7 +3959,8 @@ def cmd_refuel(args) -> int: # WHY NOT INSIDE `collect`. `collect` reads local files, closes tasks, and # works with the network down; shepherding calls out to GitHub and spawns # sessions. Folding a session-spawning side effect into the command whose whole -# contract is "read a file, close a task" makes `collect` fail when gh is down, +# contract is "read a file, close a task" makes `collect` fail when the forge +# is unreachable, # for a reason unrelated to what it was asked to do. It is a sibling — and # because a command nobody remembers to run reproduces the bug this fixes, # `collect` ends by naming it whenever it closed a task carrying an artifact, @@ -3962,7 +3972,7 @@ def cmd_refuel(args) -> int: # with the condition it went out for. A second pass over the # same still-broken PR sees work in flight, not a second job. # Never guess. "Could not check" is its own outcome and is never `broken`. -# No gh, no network, no thurbox: say what could not be +# No forge, no network, no thurbox: say what could not be # determined and carry on. Spawning a fixer for a healthy PR is # the one failure that costs more than the bug. # Never touch A pull request whose head branch lives in someone else's @@ -3971,36 +3981,6 @@ def cmd_refuel(args) -> int: # this repository, so that is the one claim a pull request # cannot make for itself. -# GitHub-specific and deliberately NOT the forge-agnostic PR_URL_RE above: this -# one exists to name `owner/repo`, which is what AUTO_MERGE_REPOS is checked -# against. Defining a second `PR_URL_RE` here shadowed that one and broke the -# pipeline check; the two answer different questions and keep different names. -GH_PR_URL_RE = re.compile(r"^https://github\.com/([^/\s]+/[^/\s]+)/pull/(\d+)/?$") - - -# One `gh pr list` answers everything below, so a pass costs one call per -# repository rather than one per pull request. The last four are the safety -# fields: `headRefOid` is what an attestation has to name, and the other three -# are how a fork's pull request is told from ours. -GH_PR_FIELDS = ( - "number,state,url,title,isDraft,mergeable,reviewDecision," - "statusCheckRollup,body,headRefName,baseRefName,headRefOid," - "author,headRepositoryOwner,isCrossRepository" -) - -# `gh pr list --limit` is a request cap, not a page size — gh paginates the -# GraphQL calls itself to reach it. Set high enough that hitting it means the -# repository genuinely has that many open pull requests, which open_prs then -# treats as unreadable rather than silently returning a truncated list. -GH_PR_LIST_LIMIT = 1000 - -# A check that FAILED. Anything still running is NOT a failure — reading a -# pending check as a broken one is how a shepherd spawns fixers for PRs whose -# CI simply has not finished, and how it would merge one whose CI has not -# either. Both directions of that mistake are covered by `checks-pending`. -CHECK_FAILED = {"FAILURE", "TIMED_OUT", "STARTUP_FAILURE", "ACTION_REQUIRED", "ERROR"} -CHECK_PASSED = {"SUCCESS", "NEUTRAL", "SKIPPED"} - # §4a of .agents/skills/thurbox-session/SKILL.md, as code. These two groups are # the AGENT SPEAKING about itself. Every other word in that table is an # observation — `running`, `uncovered`, `unreported` — and an observation is @@ -4022,11 +4002,14 @@ def cmd_refuel(args) -> int: # the body carries a `no-mistakes` attestation naming the pull request's # CURRENT head commit (so a stale attestation from an earlier push can never # authorise the push that replaced it), every check has CONCLUDED and passed, -# GitHub itself says MERGEABLE, and whoever opened it can push to this repo -# (`author_can_push` — the last thing checked, because it is the one claim the -# pull request body cannot make for itself). `--squash --delete-branch` -# because squash is the only method the remote allows; CONTRIBUTING.md owns -# that. +# the forge itself calls it mergeable, and whoever opened it can push to this +# repo (`author_can_push` — the last thing checked, because it is the one claim +# the pull request body cannot make for itself). +# +# HOST-QUALIFIED, and an entry that names no host is refused rather than +# guessed at (`forge.RepoId.parse`). `Thurbeen/fleet` is a different repository +# on github.com and on a self-hosted instance, and this is the one list where +# matching the wrong one means acting on somebody else's code. # # `Thurbeen/thurview` was added on the operator's standing instruction, "merge # everything when ready". It qualifies on the same terms rather than looser @@ -4035,37 +4018,53 @@ def cmd_refuel(args) -> int: # those would be a repo where every gate but `author_can_push` is vacuous — # that is the test for adding the next one, not whether the fleet happens to # have work there. -AUTO_MERGE_REPOS = {"Thurbeen/fleet", "Thurbeen/thurview"} - - -def pr_ref(artifact: str) -> tuple[str, int] | None: - """('owner/repo', number) for a GitHub PR URL, or None for anything else.""" - m = GH_PR_URL_RE.match((artifact or "").strip()) - return (m.group(1), int(m.group(2))) if m else None - +AUTO_MERGE_REPOS = {"github.com/Thurbeen/fleet", "github.com/Thurbeen/thurview"} + +# The one way to say it somewhere other than here, and it REPLACES the set +# rather than adding to it: a fleet driving somebody else's repositories is a +# different fleet, not this one plus an extra. Every entry is parsed, and one +# that names no host is dropped with a line on stderr rather than matched +# against a bare slug — which is the mistake this whole seam exists to make +# impossible. `queue-selftest.sh` is the second fleet it was written for. +AUTO_MERGE_ENV = "FLEET_AUTO_MERGE_REPOS" + +# Squash because it is the only method fleet's own remotes allow, so the pull +# request title becomes the commit on `main`; CONTRIBUTING.md owns that. A +# forge that cannot perform it says so BEFORE anything is merged rather than +# after something was merged another way — `Forge.merge_methods`. +MERGE_METHOD = "squash" +DELETE_MERGED_BRANCH = True + + +def auto_merge_repos() -> set: + """The repositories fleet may merge in, host-qualified, every time.""" + raw = os.environ.get(AUTO_MERGE_ENV, "").strip() + if not raw: + return AUTO_MERGE_REPOS + out = set() + for entry in re.split(r"[,\s]+", raw): + if not entry: + continue + repo = forge.RepoId.parse(entry) + if repo is None: + print( + f"{AUTO_MERGE_ENV}: ignoring {entry!r} — an auto-merge entry must " + "name its forge, as in github.com/owner/repo", + file=sys.stderr, + ) + continue + out.add(repo.qualified) + return out -def gh_json(argv: list) -> tuple[object, str]: - """Run gh and parse its JSON. A non-empty second value is why it could not. - Every caller treats that string as UNDETERMINED and never as a verdict. - """ - if not shutil.which("gh"): - return None, "gh is not installed" - try: - out = subprocess.run(["gh"] + argv, capture_output=True, text=True, timeout=60) - except (OSError, subprocess.SubprocessError) as exc: - return None, f"gh could not be run: {exc}" - if out.returncode != 0: - first = (out.stderr or "").strip().splitlines() - return None, first[0] if first else f"gh exited {out.returncode}" - try: - return json.loads(out.stdout), "" - except ValueError: - return None, "gh returned output that is not JSON" +def pr_ref(artifact: str) -> forge.ChangeRef | None: + """The change request an artifact names, or None for anything else.""" + which, ref = forge.for_url(artifact) + return ref if which is not None else None -def open_prs(slug: str) -> tuple[list, str]: - """Every OPEN pull request on one repository, straight from the forge. +def open_prs(repo: forge.RepoId) -> tuple[list, str]: + """Every OPEN change request on one repository, straight from the forge. THE BUG THIS FIXES. A task records ONE `artifact` — the first pull request its worker reported — so a shepherd that enumerated artifacts saw exactly @@ -4078,44 +4077,24 @@ def open_prs(slug: str) -> tuple[list, str]: A non-empty second value is why it could not be read, and a repository that could not be read contributes nothing rather than an empty answer. """ - docs, err = gh_json( - ["pr", "list", "--repo", slug, "--state", "open", "--limit", - str(GH_PR_LIST_LIMIT), "--json", GH_PR_FIELDS] - ) - if err: - return [], err - if not isinstance(docs, list): - return [], "gh returned something that is not a list of pull requests" - docs = [d for d in docs if isinstance(d, dict)] - if len(docs) >= GH_PR_LIST_LIMIT: - return [], ( - f"{slug} has at least {GH_PR_LIST_LIMIT} open pull requests; " - "gh's result may be truncated, so treating it as unreadable " - "rather than silently dropping some" - ) - return docs, "" + which, why = forge.for_repo(repo) + if which is None: + return [], why + return which.open_change_requests(repo) -def check_verdicts(rollup) -> tuple[list, list]: - """(names that failed, names still running). Everything else passed.""" - failed, pending = [], [] - for c in rollup or []: - if not isinstance(c, dict): - continue - name = c.get("name") or c.get("context") or "a required check" - if "state" in c and "conclusion" not in c: - # A StatusContext: one word, and PENDING is not a failure. - verdict = str(c.get("state") or "").upper() - elif str(c.get("status") or "").upper() != "COMPLETED": - pending.append(name) - continue - else: - verdict = str(c.get("conclusion") or "").upper() - if verdict in CHECK_FAILED: - failed.append(name) - elif verdict not in CHECK_PASSED: - pending.append(name) - return failed, pending +def check_verdicts(cr: forge.ChangeRequest) -> tuple[list, list]: + """(names that failed, names still running). Everything else passed. + + A `cancelled` check counts as still running here, not as a failure: that + is the shepherd's own long-standing reading (`checks-pending`, not + `checks-failed`), separate from fleet-status's, which is why the forge + hands back `cancelled` as its own word instead of pre-deciding for either. + """ + return ( + [c.name for c in cr.checks if c.verdict == "failed"], + [c.name for c in cr.checks if c.verdict in ("pending", "cancelled")], + ) # WHY NOT THE FIVE HEADINGS. `collect` looks for `## Intent` and its four @@ -4196,17 +4175,12 @@ def attestation_verdict(body: str, head_sha: str) -> tuple[bool, str]: return True, f"the pipeline attests {attested[:8]}, which is this head" -def head_owner(pr: dict) -> str: - """The login owning the repository the head branch lives in, or ''.""" - return str((pr.get("headRepositoryOwner") or {}).get("login") or "") - - -def classify(pr: dict, slug: str, method: str | None) -> tuple[str, str]: +def classify(cr: forge.ChangeRequest, method: str | None) -> tuple[str, str]: """(condition, one line saying why). Four conditions get a fixer, in the order FIXABLE lists them. `ready` - means the merge gates GitHub can answer hold. `foreign` is a pull request - that is not ours, which is neither merged nor handed to an agent. + means the merge gates the forge can answer hold. `foreign` is a pull + request that is not ours, which is neither merged nor handed to an agent. `undetermined` means the answer is not knowable yet and is never treated as any of the others. @@ -4219,9 +4193,11 @@ def classify(pr: dict, slug: str, method: str | None) -> tuple[str, str]: unlinked pull request is one fleet knows nothing about and the attestation is still the only thing that would ever authorise merging it. """ - if str(pr.get("state") or "").upper() != "OPEN": - return "closed", f"the pull request is {str(pr.get('state')).lower()}" - if pr.get("isDraft"): + if cr.state != "open": + return "closed", ( + f"the pull request is {cr.state or 'in no state the forge named'}" + ) + if cr.draft: return "undetermined", "still a draft" # NOT OURS, ASKED BEFORE ANYTHING ELSE. A stranger cannot create a branch @@ -4230,29 +4206,31 @@ def classify(pr: dict, slug: str, method: str | None) -> tuple[str, str]: # It gates the fixer as hard as it gates the merge: sending an agent to # "fix" a stranger's branch is worse than merging one, because it happens # without even the pretence of a gate. - owner = slug.split("/", 1)[0] - where = head_owner(pr) - if not where: - return "undetermined", "GitHub did not say which repository the head branch is in" - if pr.get("isCrossRepository") or where.lower() != owner.lower(): + # + # WHICH FORGE ANSWERS IT. A fork, a mirror and a same-repo branch are told + # apart differently on every forge, so the adapter decides and this reads + # its answer. `None` is "could not tell" and is never "not ours". + if cr.head_is_ours is None: + return "undetermined", "the forge did not say which repository the head branch is in" + if not cr.head_is_ours: return "foreign", ( - f"its head branch is in {where}'s repository, not {slug} — " - "fleet neither merges nor sends an agent at a pull request that is not ours" + f"its head branch is in {cr.head_location or 'another repository'}, " + f"not {cr.repo} — fleet neither merges nor sends an agent at a pull " + "request that is not ours" ) - mergeable = str(pr.get("mergeable") or "").upper() - base = pr.get("baseRefName") or "its base branch" - failed, pending = check_verdicts(pr.get("statusCheckRollup")) - attested, attest_why = attestation_verdict(pr.get("body"), pr.get("headRefOid") or "") + base = cr.base_branch or "its base branch" + failed, pending = check_verdicts(cr) + attested, attest_why = attestation_verdict(cr.body, cr.head_sha) fixable = { "conflicting": ( - mergeable == "CONFLICTING", + cr.mergeable == "conflicting", f"conflicts with {base} and cannot be merged as it stands", ), "checks-failed": (bool(failed), "failed checks: " + ", ".join(failed[:4])), "changes-requested": ( - str(pr.get("reviewDecision") or "").upper() == "CHANGES_REQUESTED", + cr.review_decision == "changes-requested", "a reviewer requested changes", ), "policy": (not attested and method in (None, "no-mistakes"), attest_why), @@ -4264,15 +4242,17 @@ def classify(pr: dict, slug: str, method: str | None) -> tuple[str, str]: if pending: return "undetermined", "checks still running: " + ", ".join(pending[:4]) - if not pr.get("statusCheckRollup"): - # An EMPTY rollup is not a pass. CI here only fires on pull requests, + if not cr.checks: + # NO CHECK AT ALL is not a pass. CI here only fires on pull requests, # so a PR whose checks have not been created yet reads exactly like a # PR with nothing to run — and merging the first one merges code CI # never saw. "No check has reported" is its own answer. return "undetermined", "no check has reported yet" - if mergeable != "MERGEABLE": - # UNKNOWN is GitHub still computing the merge, not a verdict. - return "undetermined", f"mergeable is {mergeable or 'absent'}; ask again shortly" + if cr.mergeable != "mergeable": + # An unsaid answer is the forge still computing the merge, not a verdict. + return "undetermined", ( + f"mergeable is {cr.mergeable or 'absent'}; ask again shortly" + ) if attested: return "ready", f"{attest_why}; checks green, mergeable, and the branch is ours" # Only a method that was never asked for an attestation reaches here — @@ -4284,7 +4264,7 @@ def classify(pr: dict, slug: str, method: str | None) -> tuple[str, str]: ) -def publish_word(pr: dict, condition: str, method: str | None) -> str: +def publish_word(cr: forge.ChangeRequest, condition: str, method: str | None) -> str: """`classify`'s condition as one word of the `publish.state` vocabulary. `classify` answers "what should fleet DO about this pull request"; the @@ -4315,20 +4295,21 @@ def publish_word(pr: dict, condition: str, method: str | None) -> str: fact, and both the operator and this code learned to read one as the other. The moment a `pr`-method task exists they come apart, and the only things standing between them are this word, its colour in the pane, and - `shepherd_pr`'s refusal to hand a `green` one to `gh_merge`. One word is + `shepherd_pr`'s refusal to hand a `green` one to `merge_change_request`. + One word is evidence; the other is trust. Merging them merges on the worker's choice of tool, for every repo at once, silently. """ if condition == "undetermined": # In `classify`'s own order: a draft is a draft whatever its checks say. - if pr.get("isDraft"): + if cr.draft: return "draft" - _failed, pending = check_verdicts(pr.get("statusCheckRollup")) - if pending or not pr.get("statusCheckRollup"): + _failed, pending = check_verdicts(cr) + if pending or not cr.checks: return "checks-running" return "undetermined" if condition == "ready": - attested, _why = attestation_verdict(pr.get("body"), pr.get("headRefOid") or "") + attested, _why = attestation_verdict(cr.body, cr.head_sha) return "ready" if attested else "green" if condition == "policy": return "unattested" @@ -4427,9 +4408,10 @@ def base_drift(repo: str, base: str, branch: str) -> str: } -def fixer_brief(task: Task, pr: dict, condition: str, detail: str, drift: str) -> str: - n = pr.get("number") - base = pr.get("baseRefName") or task.doc.get("base") or "main" +def fixer_brief(task: Task, cr: forge.ChangeRequest, condition: str, + detail: str, drift: str) -> str: + n = cr.number + base = cr.base_branch or task.doc.get("base") or "main" work = FIXER_WORK[condition].format(base=base) parts = [ f"# {FIXER_TITLES[condition].format(n=n, base=base)}", @@ -4438,9 +4420,9 @@ def fixer_brief(task: Task, pr: dict, condition: str, detail: str, drift: str) - f"the whole instruction set; you share no context with whoever wrote the " f"branch.", "", - f"- **Pull request.** {pr.get('url')} — {pr.get('title') or ''}".rstrip(" —"), + f"- **Pull request.** {cr.url} — {cr.title}".rstrip(" —"), f"- **Repo.** `{task.doc['repo']}`", - f"- **Branch.** `{pr.get('headRefName') or task.doc['branch']}` " + f"- **Branch.** `{cr.head_branch or task.doc['branch']}` " f"onto `{base}`. It already exists and you are already on it.", f"- **What is wrong.** {detail}.", "", @@ -4463,7 +4445,7 @@ def fixer_brief(task: Task, pr: dict, condition: str, detail: str, drift: str) - "", "## Done means", "", - f"- {pr.get('url')} is open, updated in place, and the condition above is" + f"- {cr.url} is open, updated in place, and the condition above is" " gone.", "- Nothing else about the pull request changed.", "", @@ -4622,13 +4604,7 @@ def spawn_fixer(task: Task, name: str, brief_path: str, branch: str) -> tuple[st return session, f"{note}; session {session}" -# One answer per (repo, login) per pass. The question does not change inside -# a run and every open pull request would otherwise ask it again. -_PUSH_ACCESS: dict[tuple[str, str], tuple[bool, str]] = {} -PUSH_PERMISSIONS = {"admin", "maintain", "write"} - - -def author_can_push(slug: str, pr: dict) -> tuple[bool, str]: +def author_can_push(cr: forge.ChangeRequest) -> tuple[bool, str]: """Was this pull request opened by someone who owns the repository? `classify`'s `foreign` check already proves the CODE is ours: a stranger @@ -4637,54 +4613,39 @@ def author_can_push(slug: str, pr: dict) -> tuple[bool, str]: body carrying the attestation would then be theirs to write — so the last thing checked before an unattended merge is who opened it. - Asked as "may this login push here" rather than "is this login the owner" - because the owner of `Thurbeen/fleet` is an organisation and no pull - request is ever authored by one. `gh` has no `authorAssociation` field in - every version; the collaborator permission endpoint is in all of them. + "May this account push here" and not "is this account the owner": the owner + of `Thurbeen/fleet` is an organisation and no pull request is ever authored + by one. How a forge answers that is the adapter's business, and so is + caching it for the pass — the answer does not change inside a run and every + open pull request would otherwise ask it again. """ - author = pr.get("author") or {} - login = str(author.get("login") or "") - if not login: - return False, "GitHub did not say who opened it" - if author.get("is_bot"): - return False, f"{login} is a bot" - key = (slug, login) - if key not in _PUSH_ACCESS: - doc, err = gh_json(["api", f"repos/{slug}/collaborators/{login}/permission"]) - if err or not isinstance(doc, dict): - _PUSH_ACCESS[key] = ( - False, - f"could not check whether {login} can push to {slug}: " - f"{err or 'unexpected output'}", - ) - else: - perm = str(doc.get("permission") or "").lower() - # GitHub spells "no access" as the literal string `none`. - said = "no" if perm in ("", "none") else perm - _PUSH_ACCESS[key] = ( - perm in PUSH_PERMISSIONS, - f"{login} has {said} access to {slug}", - ) - return _PUSH_ACCESS[key] + if not cr.author: + return False, "the forge did not say who opened it" + if cr.author_is_bot: + return False, f"{cr.author} is a bot" + which, why = forge.for_repo(cr.repo) + if which is None: + return False, why + return which.can_push(cr.repo, cr.author) -def gh_merge(url: str) -> tuple[bool, str]: - """Squash-merge, which is the only method this remote allows.""" - if not shutil.which("gh"): - return False, "gh is not installed" - try: - out = subprocess.run( - ["gh", "pr", "merge", url, "--squash", "--delete-branch"], - capture_output=True, - text=True, - timeout=120, +def merge_change_request(cr: forge.ChangeRequest) -> tuple[bool, str]: + """Merge, by the one method fleet's remotes allow. + + A forge that cannot perform that method is a refusal fleet can say out + loud, and it is said BEFORE the merge rather than reported after one that + silently used a different method. A GitLab project can forbid squash; this + is where that shows up as a sentence rather than as a surprise on `main`. + """ + which, why = forge.for_repo(cr.repo) + if which is None: + return False, why + if MERGE_METHOD not in which.merge_methods: + return False, ( + f"{which.name} cannot merge by {MERGE_METHOD}, which is the only " + "method fleet merges by" ) - except (OSError, subprocess.SubprocessError) as exc: - return False, f"gh could not be run: {exc}" - if out.returncode != 0: - first = ((out.stderr or "") + (out.stdout or "")).strip().splitlines() - return False, first[0] if first else f"gh exited {out.returncode}" - return True, "squash-merged, branch deleted" + return which.merge(cr, MERGE_METHOD, DELETE_MERGED_BRANCH) # --- the pass itself --------------------------------------------------------- @@ -4717,18 +4678,18 @@ def rec_for(task, url: str) -> dict: return rec if rec.get("pr", url) == url else {} -def shepherd_pr(pr: dict, slug: str, task, args) -> dict: - """Inspect one open pull request and do the one thing it calls for. +def shepherd_pr(cr: forge.ChangeRequest, task, args) -> dict: + """Inspect one open change request and do the one thing it calls for. `task` is the task it belongs to, or None: the forge is the source of the list now, so a pull request nobody recorded is shepherded like any other. It simply has no session to send a fixer into, which is said out loud. """ - url = pr.get("url") or f"https://github.com/{slug}/pull/{pr.get('number')}" + url = cr.url row = { "task": task.ref if task else "", "pr": url, - "repo": slug, + "repo": cr.repo.qualified, "condition": "undetermined", "detail": "", "action": "none", @@ -4736,11 +4697,11 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: } method = task_publish(task)[0] if task else None - condition, detail = classify(pr, slug, method) + condition, detail = classify(cr, method) row["condition"], row["detail"] = condition, detail - word = publish_word(pr, condition, method) + word = publish_word(cr, condition, method) # What the pass SAW, written down for every linked task and whatever the - # condition — the facts all arrived in the one `gh pr list` above, and a + # condition — the facts all arrived in the one forge listing above, and a # record that keeps them is the difference between "shipped" and "its # checks failed forty minutes ago". A dry run writes nothing, here as # everywhere else below. @@ -4768,7 +4729,7 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: if word == "green": # Asked BEFORE the allowlist, because this is the deeper reason: # not "fleet does not merge here" but "nothing vetted the head - # that would land". Never passed to `gh_merge` — see `publish_word` + # that would land". Never merged — see `publish_word` # for why this is not the same state as `ready`. row["action"] = "ready" row["note"] = ( @@ -4777,9 +4738,9 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: "ever ran on the commit that would land" ) return row - if slug not in AUTO_MERGE_REPOS: + if cr.repo.qualified not in auto_merge_repos(): row["action"] = "ready" - row["note"] = f"fleet does not merge in {slug}; this one is yours" + row["note"] = f"fleet does not merge in {cr.repo}; this one is yours" return row if args.no_merge: row["action"] = "ready" @@ -4788,7 +4749,7 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: # The last gate, and the one a pull request body cannot write for # itself. Asked before --dry-run answers, so a dry run is honest # about what it would actually merge. - allowed, why = author_can_push(slug, pr) + allowed, why = author_can_push(cr) if not allowed: row["action"] = "not-merged" row["note"] = ( @@ -4797,10 +4758,16 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: ) return row if args.dry_run: + which, _ = forge.for_repo(cr.repo) + how = ( + which.describe_merge(MERGE_METHOD, DELETE_MERGED_BRANCH) + if which + else f"merge by {MERGE_METHOD}" + ) row["action"] = "would-merge" - row["note"] = f"gh pr merge --squash --delete-branch ({why})" + row["note"] = f"{how} ({why})" return row - ok, note = gh_merge(url) + ok, note = merge_change_request(cr) row["action"], row["note"] = ("merged" if ok else "merge-failed"), note if ok and task: record_shepherd(task, {"condition": "merged", "detail": note, @@ -4880,9 +4847,9 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: ) return row - base = pr.get("baseRefName") or task.doc.get("base") or "main" - branch = pr.get("headRefName") or task.doc["branch"] - title = FIXER_TITLES[condition].format(n=pr.get("number"), base=base) + base = cr.base_branch or task.doc.get("base") or "main" + branch = cr.head_branch or task.doc["branch"] + title = FIXER_TITLES[condition].format(n=cr.number, base=base) if args.dry_run: row["action"] = "would-dispatch" @@ -4893,7 +4860,7 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: drift = base_drift(task.doc["repo"], base, branch) if condition == "conflicting" else "" path = next_fix_file(task, condition) with open(path, "w") as fh: - fh.write(fixer_brief(task, pr, condition, detail, drift)) + fh.write(fixer_brief(task, cr, condition, detail, drift)) if reuse: ok, report = trust_and_send( @@ -4924,69 +4891,67 @@ def shepherd_pr(pr: dict, slug: str, task, args) -> dict: return row -GH_REMOTE_RE = re.compile(r"github\.com[:/]([^/\s]+/[^/\s]+?)(?:\.git)?/?$") - - -def slug_from_remote(repo_path: str) -> str: - """`owner/repo` from a checkout's `origin`, or ''. Local, and no network. +def repo_from_checkout(repo_path: str) -> forge.RepoId | None: + """The repository a checkout's `origin` names, or None. Local, and no network. The last resort, and the one that makes this work on a topic whose tasks have not shipped anything yet: before the first artifact is reported, the only thing naming the repository is the checkout the tasks were given. + Which forge a remote URL belongs to is each adapter's own question. """ if not repo_path or not os.path.isdir(repo_path): - return "" - m = GH_REMOTE_RE.search(git_out(repo_path, ["remote", "get-url", "origin"]).strip()) - return m.group(1) if m else "" + return None + return forge.repo_from_remote( + git_out(repo_path, ["remote", "get-url", "origin"]).strip() + ) def shepherd_targets(tasks: list) -> dict: - """task.ref -> `owner/repo`, derived and never hardcoded. - - The queue's tasks name their repositories: an artifact URL gives - `owner/repo` outright, and a task that has not reported one yet inherits - the slug of the other tasks sharing its local checkout. Merging stays - limited to AUTO_MERGE_REPOS whatever comes out of here — knowing about a - repository and being allowed to merge in it are different questions. + """task.ref -> RepoId, derived and never hardcoded. + + The queue's tasks name their repositories: an artifact URL gives the + repository — host and path — outright, and a task that has not reported one + yet inherits the repository of the other tasks sharing its local checkout. + Merging stays limited to AUTO_MERGE_REPOS whatever comes out of here: + knowing about a repository and being allowed to merge in it are different + questions. """ - slug_of: dict[str, str] = {} - by_path: dict[str, str] = {} + repo_of: dict[str, forge.RepoId] = {} + by_path: dict[str, forge.RepoId] = {} for task in tasks: ref = pr_ref(task.doc.get("artifact")) if ref: - slug_of[task.ref] = ref[0] - by_path.setdefault(str(task.doc.get("repo") or ""), ref[0]) + repo_of[task.ref] = ref.repo + by_path.setdefault(str(task.doc.get("repo") or ""), ref.repo) for task in tasks: - if task.ref in slug_of: + if task.ref in repo_of: continue path = str(task.doc.get("repo") or "") - slug = by_path.get(path) - if not slug: - slug = slug_from_remote(path) - if slug: - by_path[path] = slug - if slug: - slug_of[task.ref] = slug - return slug_of + repo = by_path.get(path) + if repo is None: + repo = repo_from_checkout(path) + if repo is not None: + by_path[path] = repo + if repo is not None: + repo_of[task.ref] = repo + return repo_of -def link_task(pr: dict, tasks: list) -> object: - """The task this pull request belongs to, or None. +def link_task(cr: forge.ChangeRequest, tasks: list) -> object: + """The task this change request belongs to, or None. Two ways, and the second is the one that matters. The artifact is what a worker reported once. The HEAD BRANCH is what the pull request is actually open from, and it is what connects a task's second pull request back to it after its first one merged and its artifact stopped being current. """ - number = pr.get("number") for task in tasks: ref = pr_ref(task.doc.get("artifact")) - if ref and ref[1] == number: + if ref and ref.number == cr.number and ref.repo == cr.repo: return task - head = str(pr.get("headRefName") or "") - if head: + if cr.head_branch: for task in tasks: - if str(task.doc.get("branch") or "") == head: + if str(task.doc.get("branch") or "") == cr.head_branch: return task return None @@ -5011,44 +4976,45 @@ def cmd_shepherd(args) -> int: continue tasks.append(task) - slug_of = shepherd_targets(tasks) - slugs = sorted(set(slug_of.values())) + repo_of = shepherd_targets(tasks) + repos = sorted(set(repo_of.values()), key=lambda r: r.qualified) + named = [str(r) for r in repos] rows, unreadable = [], [] - for slug in slugs: - here = [t for t in tasks if slug_of.get(t.ref) == slug] - prs, err = open_prs(slug) + for repo in repos: + here = [t for t in tasks if repo_of.get(t.ref) == repo] + crs, err = open_prs(repo) if err: # A repository that could not be listed contributes nothing. An # empty answer and an unreadable one are not the same claim, and # only one of them means "nothing is open". - unreadable.append({"repo": slug, "detail": err}) + unreadable.append({"repo": repo.qualified, "named": str(repo), "detail": err}) continue - for pr in sorted(prs, key=lambda d: d.get("number") or 0): - rows.append(shepherd_pr(pr, slug, link_task(pr, here), args)) + for cr in sorted(crs, key=lambda c: c.number): + rows.append(shepherd_pr(cr, link_task(cr, here), args)) if args.json: print(json.dumps({ "queue": os.path.abspath(queue_root()), - "repos": slugs, + "repos": [r.qualified for r in repos], "unreadable": unreadable, "prs": rows, }, indent=2)) return 0 - if not slugs: - print("shepherd: no task names a GitHub repository yet") + if not repos: + print("shepherd: no task names a repository on a configured forge yet") return 0 for bad in unreadable: - print(f"shepherd: could not read the pull requests on {bad['repo']}: " + print(f"shepherd: could not read the pull requests on {bad['named']}: " f"{bad['detail']} — nothing there was touched") if not rows: if not unreadable: - print("shepherd: no open pull requests on " + ", ".join(slugs)) + print("shepherd: no open pull requests on " + ", ".join(named)) return 0 verb = "would do" if args.dry_run else "did" - print(f"shepherd: {len(rows)} open pull request(s) on {', '.join(slugs)} " + print(f"shepherd: {len(rows)} open pull request(s) on {', '.join(named)} " f"— what it {verb}:\n") for r in rows: print(f" {r['task'] or '(no task records it)'} {r['pr']}") @@ -5073,12 +5039,12 @@ def cmd_shepherd(args) -> int: if not args.dry_run and any(r["action"] == "dispatched" for r in rows): print(" Fixers are working in place on the existing branches; nothing forked.") print( - " Merging is limited to " + ", ".join(sorted(AUTO_MERGE_REPOS)) + " Merging is limited to " + ", ".join(sorted(auto_merge_repos())) + ", and only for a pull request whose\n" " head branch is in that repo, that someone who can push there " "opened, that\n" " carries a no-mistakes attestation for its CURRENT head, whose " - "checks passed,\n and that GitHub calls MERGEABLE." + "checks passed,\n and that the forge itself calls mergeable." ) if not args.dry_run: refresh_run_logs(q) diff --git a/scripts/queue-selftest.sh b/scripts/queue-selftest.sh index 69d9300..bd195a9 100755 --- a/scripts/queue-selftest.sh +++ b/scripts/queue-selftest.sh @@ -1378,7 +1378,9 @@ expect "and the record says gh was never there to ask" "gh not found" "$state" bare="$tmp/bare-clone" mkdir -p "$bare/scripts/lib" "$bare/orchestration/queue" cp scripts/queue.sh "$bare/scripts/queue.sh" -ln -s "$PWD/scripts/lib/queue.py" "$bare/scripts/lib/queue.py" +for lib in queue.py forge.py; do + ln -s "$PWD/scripts/lib/$lib" "$bare/scripts/lib/$lib" +done cat >"$bare/orchestration/queue/POLICY.md" <<'EOF' # Standing policy for fleet workers @@ -1663,16 +1665,18 @@ fi # # The whole thing is exercised against a THROWAWAY CLONE rather than this one, # so the test says the same thing on a machine with thurbox and on CI without -# it: a directory holding `scripts/queue.sh`, a symlink to the real -# `scripts/lib/queue.py` (the anchor is the script's own path, so a symlink is -# a whole clone for this purpose) and a rendered `extension.toml` that decides +# it: a directory holding `scripts/queue.sh`, symlinks to the real +# `scripts/lib/*.py` (the anchor is the script's own path, so a symlink is a +# whole clone for this purpose) and a rendered `extension.toml` that decides # whether that clone IS the control plane. clonetmp="$(mktemp -d)" fake="$clonetmp/second-clone" mkdir -p "$fake/scripts/lib" "$fake/deep/sub/dir" cp scripts/queue.sh "$fake/scripts/queue.sh" -ln -s "$PWD/scripts/lib/queue.py" "$fake/scripts/lib/queue.py" +for lib in queue.py forge.py; do + ln -s "$PWD/scripts/lib/$lib" "$fake/scripts/lib/$lib" +done FAKEQ="$fake/scripts/queue.sh" # Render the manifest install-extension.sh would have written, naming whichever @@ -4094,6 +4098,370 @@ else pass "the tracked template carries no path and no session id" fi +# --- 13. THE SEAM: the whole queue driven by a forge that is not GitHub ------ +# +# A seam with one implementation is a claim. This is the second implementation: +# a forge with no network, no `gh` and no GitHub anywhere in it, that `collect`, +# `reap`'s landing check and `shepherd` are driven all the way through. +# +# It is also the regression test. `gh` on this section's PATH is a TRIPWIRE, not +# a stub — it logs the call and fails — so any code that reaches around +# `scripts/lib/forge.py` and runs `gh` directly again shows up here by name +# instead of quietly working on the operator's machine and nowhere else. +# +# What it proves, beyond "the calls go through the interface": +# +# a self-hosted host with a PORT round-trips — identity is host + path, and +# `forge.test:8443/acme/widgets` is not `github.com/acme/widgets` +# a `/-/merge_requests/` URL is a change request, the same as `/pull/` +# AUTO_MERGE_REPOS is matched host-qualified, and an entry naming no forge +# is refused rather than matched against a bare slug +# a repository is discovered from a checkout's `origin` through the same seam +# a forge that cannot perform fleet's merge method SAYS SO, and nothing is +# merged by some other method instead + +fk="$tmp/fake-forge" +mkdir -p "$fk/crs" "$fk/push" "$fk/bin" +export FAKE_FORGE_DIR="$fk" +: >"$fk/gh-calls.log" +: >"$fk/merged.log" +printf '["squash"]\n' >"$fk/merge-methods.json" + +# A TRIPWIRE. Nothing in this section may reach GitHub, so `gh` records who +# tried and then fails the way an unreachable API fails. +cat >"$fk/bin/gh" <<'SH' +#!/bin/sh +echo "gh $*" >>"$FAKE_FORGE_DIR/gh-calls.log" +echo "gh: nothing in the fake-forge section may reach GitHub" >&2 +exit 1 +SH +chmod +x "$fk/bin/gh" + +# The second implementation. It answers the questions in scripts/lib/forge.py's +# header and knows nothing else — if it had to grow a field to keep the queue +# working, the seam would be in the wrong place and the fix would be to move +# the seam rather than to widen this. +cat >"$fk/forge_plugin.py" <<'PY' +"""A forge that is not GitHub: files on disk, no network, no CLI. + +Deliberately shaped like the forge fleet does NOT run on. It is self-hosted +with a port, it spells a change request `/-/merge_requests/`, and it can be +told it cannot squash — three of the ways a second adapter is expected to +differ. +""" + +import json +import os +import re + +import fleet_forge as fg + +HOST = "forge.test:8443" +URL_RE = re.compile(r"^https://" + re.escape(HOST) + r"/(.+?)/-/merge_requests/(\d+)$") +REMOTE_RE = re.compile(r"^https://" + re.escape(HOST) + r"/(.+?)(?:\.git)?/?$") + + +def _dir(): + return os.environ["FAKE_FORGE_DIR"] + + +def _down(): + return os.path.exists(os.path.join(_dir(), "down")) + + +def _docs(): + out = [] + crs = os.path.join(_dir(), "crs") + for name in sorted(os.listdir(crs)): + if name.endswith(".json"): + with open(os.path.join(crs, name)) as fh: + out.append(json.load(fh)) + return out + + +class FakeForge(fg.Forge): + name = "fake" + hosts = (HOST,) + + @property + def merge_methods(self): + with open(os.path.join(_dir(), "merge-methods.json")) as fh: + return tuple(json.load(fh)) + + def parse_change_url(self, url): + m = URL_RE.match((url or "").strip()) + if not m: + return None + return fg.ChangeRef(fg.RepoId(HOST, m.group(1)), int(m.group(2)), m.group(0)) + + def repo_from_remote(self, remote_url): + m = REMOTE_RE.match((remote_url or "").strip()) + return fg.RepoId(HOST, m.group(1)) if m else None + + def _find(self, ref): + if _down(): + return None, "the fake forge is unreachable" + for d in _docs(): + if d["number"] == ref.number and d["repo"] == ref.repo.path: + return d, "" + return None, f"no change request {ref.number} on {ref.repo}" + + def get(self, ref): + d, why = self._find(ref) + return (None, why) if why else (self._change_request(d, ref.repo), "") + + def state(self, ref): + d, why = self._find(ref) + return (None, why) if why else (d.get("state", "open"), "") + + def open_change_requests(self, repo): + if _down(): + return [], "the fake forge is unreachable" + return [ + self._change_request(d, repo) + for d in _docs() + if d["repo"] == repo.path and d.get("state", "open") == "open" + ], "" + + def open_change_requests_in_checkout(self, path): + repo = self.repo_from_remote(fg._git_remote(path)) + if repo is None: + return [], "not a checkout of a fake-forge repository" + return self.open_change_requests(repo) + + def _change_request(self, d, repo): + n = d["number"] + return fg.ChangeRequest( + ref=fg.ChangeRef(repo, n, f"https://{HOST}/{repo.path}/-/merge_requests/{n}"), + title=d.get("title", ""), + state=d.get("state", "open"), + body=d.get("body", ""), + head_branch=d.get("head_branch", ""), + base_branch=d.get("base_branch", "main"), + head_sha=d.get("head_sha", ""), + author=d.get("author", ""), + mergeable=d.get("mergeable", "mergeable"), + checks=[fg.Check(c[0], c[1]) for c in d.get("checks", [])], + commits=[fg.Commit(c[0], c[1]) for c in d.get("commits", [])], + head_is_ours=d.get("head_is_ours", True), + head_location=d.get("head_location", ""), + ) + + def can_push(self, repo, login): + if _down(): + return False, "the fake forge is unreachable" + ok = os.path.exists(os.path.join(_dir(), "push", login)) + return ok, f"{login} {'may' if ok else 'may not'} push to {repo}" + + def describe_merge(self, method, delete_branch): + return f"fake forge: {method}" + (" and delete the branch" if delete_branch else "") + + def merge(self, cr, method, delete_branch): + if method not in self.merge_methods: + return False, f"this project forbids {method} merges" + with open(os.path.join(_dir(), "merged.log"), "a") as fh: + fh.write(f"{cr.number} {method}\n") + return True, f"{method}-merged on the fake forge" + + +def forges(): + return [FakeForge()] +PY + +# One change request, written the way the fake forge stores them. The body it +# gets is what the pipeline would have written: the attestation naming THIS +# head, which is what both `collect` and the merge gate demand. +fake_cr() { + python3 - "$fk/crs" "$@" <<'PY' +import json +import sys + +out, n = sys.argv[1], int(sys.argv[2]) +sha = f"{n:040d}" +steps = [{"step": s, "status": "completed"} for s in + ("intent", "rebase", "review", "test", "document", "lint", "push")] +steps += [{"step": "pr", "status": "running"}, {"step": "ci", "status": "pending"}] +payload = json.dumps({"head_sha": sha, "steps": steps}) +body = f"\n\n" + "\n".join( + f"## {h}\nx\n" for h in + ("Intent", "What Changed", "Risk Assessment", "Testing", "Pipeline") +) +doc = {"number": n, "repo": "acme/widgets", "state": "open", "body": body, + "title": f"change {n}", "base_branch": "main", "head_sha": sha, + "author": "letur", "mergeable": "mergeable", "checks": [["gate", "passed"]]} +for pair in sys.argv[3:]: + key, _, value = pair.partition("=") + doc[key] = json.loads(value) +json.dump(doc, open(f"{out}/{n}.json", "w")) +PY +} + +frepo="$fk/repo" +mkdir -p "$frepo" +git -C "$frepo" init -q -b main +git -C "$frepo" -c user.email=t@t -c user.name=t commit -q --allow-empty -m base +# The repository this checkout belongs to, discovered through the seam rather +# than recorded anywhere: no task below has to name it. +git -C "$frepo" remote add origin "https://forge.test:8443/acme/widgets.git" + +touch "$fk/push/letur" # can push; `stranger` has no file here, so cannot + +FPATH="$fk/bin:$tbxbin:$sshbin:$base_path" +fq() { + env PATH="$FPATH" FLEET_QUEUE_DIR="$tmp/queue-fake" \ + FLEET_FORGE_PLUGINS="$fk/forge_plugin.py" \ + FLEET_AUTO_MERGE_REPOS="forge.test:8443/acme/widgets" \ + "$QUEUE" "$@" +} + +ftopic="$(fq topic add on-another-forge --title 'Work on a forge that is not GitHub' \ + --prompt 'fleet must not assume GitHub')" + +for spec in 01:landed:201 02:conflicting:202 03:green:203 04:foreign:204 06:cancelled-check:206; do + IFS=: read -r n slug num <<<"$spec" + fq add "$ftopic" "$slug" --title "A change that is $slug" --repo "$frepo" \ + --branch "fix/$slug" --number "$n" >/dev/null + cat >"$tmp/queue-fake/$ftopic/$n-$slug/result.md" </dev/null + +out="$(fq collect 2>&1)" +expect "collect verifies a publish claim on a forge that is not GitHub" \ + "01-landed" "$out" +expect "and it read a /-/merge_requests/ URL as a change request" \ + "merge_requests/201" "$out" +refute "its pull request is open, so nothing was reaped" "reaped" "$out" + +# A change request nobody can read is `unknown`, never `missing` — the fourth +# word has to survive the seam, or an unreachable forge starts holding tasks +# open on evidence nobody has. +fq add "$ftopic" unreadable --title 'One the forge cannot answer for' \ + --repo "$frepo" --branch fix/unreadable --number 05 >/dev/null +cat >"$tmp/queue-fake/$ftopic/05-unreadable/result.md" <<'EOF' +--- +outcome: shipped +artifact: https://forge.test:8443/acme/widgets/-/merge_requests/999 +--- +Shipped it; the forge cannot be asked about it from here. +EOF +out="$(fq collect 2>&1)" +expect "a change request the forge cannot answer for degrades to unknown" \ + "unchecked" "$out" +expect "and says what the forge said" "no change request 999" "$out" + +# --- 13b. the landing check asks the same seam ------------------------------- + +out="$(fq reap --dry-run 2>&1)" +refute "an open change request lands nothing" "would be landed" "$out" + +fake_cr 201 'head_branch="fix/landed"' 'state="merged"' +out="$(fq reap 2>&1)" +expect "a merge on the fake forge lands the task" "landed" "$out" +expect "and releases the session that produced it" "reaped" "$out" +expect "and it is the session the record held" \ + "aaaaaaaa-0000-0000-0000-000000000001" "$(cat "$deletions")" + +# --- 13c. shepherd, all the way through -------------------------------------- + +out="$(fq shepherd --topic "$ftopic" --dry-run 2>&1)" +expect "shepherd names the self-hosted repository, port and all" \ + "acme/widgets on forge.test:8443" "$out" +expect "and says what it would merge, in the fake forge's own words" \ + "fake forge: squash" "$out" +if [ -s "$fk/merged.log" ]; then + fail "a dry run merges nothing on the fake forge" "$(cat "$fk/merged.log")" +else + pass "a dry run merges nothing on the fake forge" +fi + +out="$(fq shepherd --topic "$ftopic" 2>&1)" +if grep -q '^203 squash$' "$fk/merged.log"; then + pass "a green, attested change request is merged through the seam" +else + fail "a green, attested change request is merged through the seam" \ + "$out$nl$(cat "$fk/merged.log")" +fi +expect "a conflicting one gets a fixer, in the base branch's own terms" \ + "conflicts with main" "$out" +expect "and the fixer is dispatched" "dispatched:" "$out" +expect "one whose head is not ours is left alone" "left-alone" "$out" +expect "and named as where the forge said it lives" "a stranger's fork" "$out" +refute "and a change request that is not ours is never merged" \ + "204 " "$(cat "$fk/merged.log")" + +# A CANCELLED check is not a FAILED one: the shepherd's own long-standing +# reading, kept alive through the seam rather than collapsed into +# fleet-status's stricter one now that both read the same `checks` field. +expect "a cancelled check reads as still running, not as a failed one" \ + "undetermined: checks still running: gate" "$out" +refute "so it never gets a fixer for a failed check" \ + "206" "$(cat "$fk/merged.log")" + +# --- 13d. a forge that cannot do fleet's merge method says so ---------------- +# +# THE MISMATCH WORTH CATCHING BEFORE A SECOND ADAPTER EXISTS. Fleet merges by +# squash because that is the only method its own remotes allow, and a project +# on another forge can forbid exactly that. "The forge refused this merge +# method" has to be a sentence the interface can say, and it has to be said +# BEFORE the merge rather than after one that quietly used another method. + +printf '["merge"]\n' >"$fk/merge-methods.json" +fake_cr 205 'head_branch="fix/green"' +before="$(wc -l <"$fk/merged.log")" +out="$(fq shepherd --topic "$ftopic" 2>&1)" +expect "a forge that cannot squash says so rather than merging some other way" \ + "cannot merge by squash" "$out" +count_is "and nothing is merged while it cannot" "$(wc -l <"$fk/merged.log")" \ + "$before" "$out" +printf '["squash"]\n' >"$fk/merge-methods.json" + +# --- 13e. the allowlist is host-qualified, and a bare slug is not a match ---- + +out="$(env PATH="$FPATH" FLEET_QUEUE_DIR="$tmp/queue-fake" \ + FLEET_FORGE_PLUGINS="$fk/forge_plugin.py" \ + FLEET_AUTO_MERGE_REPOS="acme/widgets" \ + "$QUEUE" shepherd --topic "$ftopic" --dry-run 2>&1)" +expect "an auto-merge entry that names no forge is refused, not matched" \ + "must name its forge" "$out" +refute "and nothing under it would be merged" "would-merge" "$out" + +# --- 13f. the regression test: nobody reached around the seam ---------------- + +if [ -s "$fk/gh-calls.log" ]; then + fail "no code path ran \`gh\` while a different forge was configured" \ + "$(cat "$fk/gh-calls.log")" +else + pass "no code path ran \`gh\` while a different forge was configured" +fi + +# The fixer above got a real worktree; take it back off the test repo so the +# temp directory can be removed without leaving a stale registration. +git -C "$frepo" worktree remove --force \ + "$tmp/queue-fake/.worktrees/${ftopic}__02-conflicting" 2>/dev/null + echo if [ "$failed" -eq 0 ]; then printf '\033[32mqueue-selftest: every claim holds\033[0m\n' diff --git a/scripts/queue.sh b/scripts/queue.sh index fefb3c7..3e3d8dc 100755 --- a/scripts/queue.sh +++ b/scripts/queue.sh @@ -50,7 +50,7 @@ # commit that would merge), or asks git whether a `push` task's # commit reached the base branch. An artifact that is not there is # reported and the task is left OPEN; a check that could not run -# (no `gh`, no network, a base branch this machine cannot read) +# (no forge CLI, no network, a base branch this machine cannot read) # says so and is never read as either verdict. The TOOL is # `--how`: free text rendered into the brief and never parsed, # which is what lets a task name a publisher fleet has never heard @@ -139,11 +139,12 @@ # IDEMPOTENT the fixer it sent is recorded on the task; a second pass sees # work in flight rather than a still-broken PR. `--force` to mean # it anyway. -# NEVER GUESS no gh, no network, no thurbox — it says what it could not +# NEVER GUESS no forge, no network, no thurbox — it says what it could not # determine and carries on. A PR it could not read is never # called broken, and never called ready. # NEVER TOUCH only artifacts recorded on this queue's own tasks, and it -# A STRANGER merges only in the repos AUTO_MERGE_REPOS names. +# A STRANGER merges only in the repos AUTO_MERGE_REPOS names, each of +# which names its forge (`github.com/owner/repo`). # # Usage: # scripts/queue.sh topic add --title T --prompt 'the ask' # or --prompt-file F|- @@ -283,7 +284,8 @@ # # Requires: python3 (with PyYAML) — the same dependency the rest of the gate # has. `dispatch`, `watch`, `reap` and `refuel` additionally need thurbox-cli, -# and `collect`, `reap` and `shepherd` ask `gh` about a pull request, and +# and `collect`, `reap` and `shepherd` ask the FORGE about a pull request — +# whichever `scripts/lib/forge.py` has configured, `gh` for GitHub — and # `shepherd` needs git as well. `refuel` reads the account's quota window with # `quota-axi` (https://github.com/kunchenguid/quota-axi), which fleet neither # installs nor sends any credential to. A task that names a `--host`