Skip to content

Mirror fleet: 27% of estate CI failures — destinations unprovisioned, advisory doctrine not implemented, 3 live caller refs #950

Description

@hyperpolymath

Finding

The mirror fleet is the single largest source of estate CI redness — 61 of 223
sampled failing check-runs (27%). Three distinct faults compound, and the
underlying push failure is not a credentials problem.

1. The keys work. The destinations do not exist.

A failing mirror job ends at:

fatal: Could not read from remote repository.
Please make sure you have the correct access rights
##[error]Process completed with exit code 128.

That reads like an auth failure, and it is not. In the same run the agent loads the
key cleanly:

Identity added: (stdin) (hyperpolymath-bitbucket-mirror)

And the same shared secrets mirror successfully elsewhere —
hyperpolymath/chapeliser and hyperpolymath/tangle both have recent
Mirror to Git Forges runs with conclusion success.

So the failure is per-repository destination provisioning: the destination repo
does not exist at the forge. In hyperpolymath/nexia-list all four destinations
(bitbucket, codeberg, disroot, gitea) fail together — the signature of a
repo that was never provisioned, not of a key that was revoked.

mirror-reusable.yml never probes whether the destination exists. Its sibling
github-backup-mirror-reusable.yml does this exactly right and is the model:
gh api -i "repos/$DEST_SLUG", an explicit 200/404 split, a refusal when the status
cannot be parsed ("Refusing to guess whether it exists"), and autocreate behind
vars.BACKUP_MIRROR_AUTOCREATE so it cannot silently spawn dozens of repos. The
seven-forge mirror has none of that, so an absent destination surfaces as a bare
exit 128.

2. The advisory doctrine is stated but not implemented

github-backup-mirror-reusable.yml cites the estate pattern in its own header:

Advisory behaviour comes from job-level continue-on-error, as in mirror-reusable.

Measured by parsing the YAML, not by reading it:

Version Advisory jobs Blocking jobs
main today mirror-gitlab only 6 of 7
d5fe075a (what 007-lang runs) none 7 of 7
d135b05b (what typefix-zero runs) none 7 of 7

The two continue-on-error occurrences in the deployed versions are step-level,
not job-level
, so they do not make the job advisory. The claim is true of one
seventh of the file it names on main, and of none of it in the field.

Observed consequence — every forge red on both callers:

failure  mirror / mirror-bitbucket   failure  mirror / mirror-gitea
failure  mirror / mirror-codeberg    failure  mirror / mirror-gitlab
failure  mirror / mirror-disroot     failure  mirror / mirror-radicle
                                     failure  mirror / mirror-sourcehut

3. Caller SHA drift — the estate is not running main

Three live refs of the same reusable, none of them each other:

Caller Pinned ref Dated
standards 571cc734 2026-08-29
007-lang d5fe075a 2026-08-25
typefix-zero d135b05b 2026-06-24

A fix to mirror-reusable.yml on main therefore reaches no caller until each
pin moves. Any remediation has to account for that or it will appear to do nothing.

⚠ Some repos (bitfuckit, branch-newspaper) do not call the reusable at all —
their check names are bare (mirror-gitlab, Push to GitLab Mirror) rather than
prefixed (mirror / mirror-gitlab). They carry their own copies.

Explicitly NOT established

Whether job-level continue-on-error suppresses a failing check-run conclusion
(and therefore stops it reddening statusCheckRollup), as opposed to only stopping
the workflow run failing. Every caller sampled runs a version with no job-level
continue-on-error at all, so the estate provides no controlled observation of it.
This needs one deliberate test before anyone relies on it as the cure — a
::warning:: cannot fail a job, but a red check run can still redden a branch.

Acceptance criteria

  • One controlled observation settling the question above: a caller pinned to a
    version with job-level continue-on-error, and the resulting check-run
    conclusion
    recorded — not just the workflow-run conclusion.
  • mirror-reusable.yml gains a destination-existence probe modelled on
    github-backup-mirror-reusable.yml: gh api -i, explicit 200/404, refuse
    rather than guess on an unparseable status, and a named reason in the step
    summary instead of exit 128.
  • A decision on whether a missing destination is a refusal or an autocreate,
    behind a repository variable as the backup mirror already does.
  • The header claim in github-backup-mirror-reusable.yml is corrected, or
    mirror-reusable.yml is changed to match it. Today they disagree.
  • Caller pins converge on one ref, with the count of distinct live refs recorded
    before and after.
  • Re-sample the failing-check census and record the new mirror/* share
    (baseline: 61 of 223, 27%).

Related: #949 (estate health, 87% red), #948. Decision surface: #787.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ji1bq3TypfycfUPAR7hSxR

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew capability or improvement to existing behavioursecuritySecurity posture, secrets, scanning, advisories, supply chain

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions