Skip to content

docs(ci): the red test-unity legs are fork PRs with no secrets, not a regression - #974

Merged
IvanMurzak merged 2 commits into
mainfrom
worktree-p0-unity-ci-regreen
Sep 3, 2026
Merged

docs(ci): the red test-unity legs are fork PRs with no secrets, not a regression#974
IvanMurzak merged 2 commits into
mainfrom
worktree-p0-unity-ci-regreen

Conversation

@IvanMurzak

@IvanMurzak IvanMurzak commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • The 12 red test-unity-* legs are not a CI regression. Every test-pull-request run
    between 2026-08-25 and 2026-08-29 came from a fork, and GitHub withholds repository
    secrets from a pull_request run whose head repository is a fork — so secrets: inherit
    hands test_unity_plugin.yml an empty UNITY_LICENSE/UNITY_EMAIL/UNITY_PASSWORD and
    game-ci/unity-test-runner aborts before it pulls an image. The licensed path is healthy and
    no secret needs re-issuing.
  • Documents the mechanism in docs/claude/ci-unity-license.md — including the duration tell
    that separates it from a bad .ulf and that survives log expiry, because job metadata
    outlives logs — and points at it from a comment on test_unity_plugin.yml's secrets: block.
  • No behavioural change. 68 insertions, 0 deletions, both files documentation-only.

Cause class

Not (a) license/activation, not (b) action/runtime, not (c) test-project content, not (d)
Editor/ product code. It is a fifth class the diagnosis had to add: fork PRs receive no
secrets.
It presents license-shaped — game-ci's error names UNITY_LICENSE — but is not
credential-shaped: the stored secrets are valid and untouched, so the owner gate G-P0 is not
exercised
and Editor/ was never involved.

Evidence line (verbatim, run 33758511822, job 100658764301)

##[error]Missing Unity License File and no Serial was found. If this
is a personal license, make sure to follow the activation
steps and set the UNITY_LICENSE GitHub secret or enter a Unity
serial number inside the UNITY_SERIAL GitHub secret.

Its env: block in that same log reads UNITY_LICENSE:, UNITY_EMAIL: and UNITY_PASSWORD:
with nothing after the colon.

Why the premise was wrong

Head repository of every run in the window (gh api repos/IvanMurzak/Unity-MCP/actions/runs/<id>):

run created head repository conclusion
32788655839 2026-08-24T23:17Z IvanMurzak/Unity-MCP (same repo) success
32822206166 2026-08-25T07:33Z Nghaiz/Unity-MCP (fork) failure
32989259890 2026-08-26T16:35Z akimaleo/Unity-MCP (fork) failure
32992648441 2026-08-26T17:11Z zorionarrillaga/Unity-MCP (fork) failure
33250121859 2026-08-29T11:26Z zorionarrillaga/Unity-MCP (fork) action_required

The last green run is the last same-repository run, and there was no same-repository PR in
between — so the red sample contains fork PRs and nothing else. Read as a time series it looks
exactly like a regression; it is a sampling artefact.

Two independent checks rule the alternatives out. Nothing in the repo changed: the only commit
touching .github/ between the last green and the first red is 91e2472a (release 0.90.0),
whose entire .github/ diff is three lines of .github/nuget-gate.lock, and
test_unity_plugin.yml was last modified 2026-07-20 (06ca3e1a). Nothing upstream changed
either: the action is SHA-pinned, and the pinned editor images ubuntu-2022.3.62f3-base-3,
ubuntu-2022.3.62f3-windows-mono-3 and ubuntu-6000.3.1f1-base-3 were last pushed 2026-04-13,
2026-04-13 and 2026-03-19 respectively — four months before the break.

The original logs are expired (HTTP 410), but job metadata is not. In 32992648441 every Unity
leg's Run game-ci/unity-test-runner@08fd329f… step started and completed in the same
second
(04:57:53Z -> 04:57:53Z) with Free disk space and actions/cache succeeding before
it. An action that rejects its inputs immediately never reached an activation attempt.

Discriminator — both halves, one variable, same commit

Both dispatches ran test_pull_request_manual.yml against 91e2472a, the commit the fork PRs
branched from. The only difference between them is whether the Unity job received the secrets.

half run result
secrets present — dispatch on main, workflow unmodified 33757559794 green
secrets withheld — same dispatch from a throwaway branch whose test_pull_request_manual.yml kept one Unity job and dropped its secrets: inherit line 33758511822 red, with the error line above

The red half reproduces the historical fingerprint rather than merely failing: step 7
13:01:37Z -> 13:01:38Z, upload-artifact failing straight after it on
Input required and not supplied: path (nothing was produced to upload), and step numbering
jumping 8 → 14 — the same shape 32992648441 recorded. The throwaway branch has been deleted;
its harness was one deletion of secrets: inherit, so it reproduces from this description.

Caveat stated plainly: the 2026-08-25..27 logs are gone, so the error text cannot be
compared byte-for-byte against them. What is compared is the structural fingerprint, which the
API still serves for both.

What this PR deliberately does not do

Making a fork's Unity checks pass would satisfy the required-status-check ruleset with jobs
that compiled nothing — a leg that stops failing because it stopped running is worse than a red
one, and it trades away the only mechanical guarantee that a fork's code builds. That is a
policy decision for the owner, open in PR #971 — issue #543, which first asked for it, was closed as completed in
2026-03 — and is out of scope here.

Test plan

  • python .github/scripts/check_nuget_gate.py locally → NuGet gate OK: 15 pins, generation UNITY_MCP_DEPS_3, propagation consistent. (exit 0)
  • test_unity_plugin.yml re-parses as YAML with its three secrets: keys intact (the edit is comment-only)
  • No Unity or CLI suite applies — the diff touches nothing under Unity-MCP-Plugin/** or cli/**
  • Diff grepped for <?xml, LicenseVersion, UNITY_PASSWORD= → 0 hits; no secret value appears in this PR, its commits, or its logs
  • This PR's own test_pull_request.yml run green on all jobs — nuget gate, test-cli (20 + 22) and all 12 test-unity-* legs actually running and passing: run
    33758936539, 17/17 checks
    green, the twelve Unity legs 3m43s—15m44s — the band of the last green run, not the
    1.3—3.2 min fail-fast band.

Closes #973

… regression

Every `test-pull-request` run between 2026-08-25 and 2026-08-29 showed all 12
`test-unity-*` legs red at `game-ci/unity-test-runner`, which reads as a week-long
CI outage. Every pull request in that window came from a fork, and GitHub withholds
repository secrets from a `pull_request` run whose head repository is a fork, so
`secrets: inherit` passes empty strings and game-ci aborts before it does any work:

  ##[error]Missing Unity License File and no Serial was found. If this
  is a personal license, make sure to follow the activation
  steps and set the UNITY_LICENSE GitHub secret or enter a Unity
  serial number inside the UNITY_SERIAL GitHub secret.

The licensed path is healthy and no secret needs re-issuing. Discriminator, one
variable, same commit 91e2472 the fork PRs branched from: a dispatch of
test_pull_request_manual.yml on main with the secrets present is green (run
33757559794), and the same dispatch with `secrets: inherit` removed is red with the
line above (run 33758511822), reproducing the historical fingerprint exactly -- a
`unity-test-runner` step that starts and completes in the same second while every
step before it succeeds, then upload-artifact failing on an empty path.

No behaviour change. Two documentation edits so the next fork PR does not cost
another day:

- docs/claude/ci-unity-license.md: a section on why fork PRs are red, the duration
  tell that separates it from a bad .ulf (and survives log expiry, because job
  metadata outlives logs), how to reach the licensed suite by dispatch, and the
  three run ids that pin it.
- .github/workflows/test_unity_plugin.yml: a comment at the `secrets:` declaration
  pointing at that section.

Whether a fork's Unity checks should be made passable is a separate policy call
under the required-status-check ruleset; it stays with #543 and #971.

Closes #973
@IvanMurzak

Copy link
Copy Markdown
Owner Author

Fresh-evidence dispatch has finished. Recording the numbers, since the whole diagnosis rests on
them.

33757559794
test_pull_request_manual.yml on main, commit 91e2472a, secrets present:
14/14 jobs green, 15.9 min wall. That is inside the 14–16 min band of the last green run
32788655839 (2026-08-24), and nowhere near the 1.3–3.2 min fail-fast band the fork runs sat in.

leg
6000.3.1f1 editmode on windows-mono / base 15.9 / 15.7 min
2023.2.22f1 editmode on windows-mono / base 12.5 / 11.7 min
2022.3.62f3 editmode on windows-mono / base 12.4 / 12.4 min
the six standalone legs 5.0 – 6.0 min
test-cli (20, 22) 0.5 min each

Before the dispatch, test_pull_request_manual.yml had never rungh run list --workflow test_pull_request_manual.yml returned [], so this run id cannot be a stale match for an older
one.

The Unity legs ran and passed. Nothing was skipped, and nothing about the licensed path needed
changing to get here.

Accuracy corrections to the fork-PR section, from two report-only review helpers
plus this pass's own verification against the GitHub API. Documentation only; no
behavioural change.

docs/claude/ci-unity-license.md
- #543 is CLOSED (state_reason=completed, 2026-03-15), so "tracked in issue #543
  and PR #971" sent a reader to a six-month-dead issue. PR #971 is the live one.
- The run-33758511822 table row said "this same workflow", whose nearest
  antecedent is row 1's test_pull_request.yml. The run actually used
  test_pull_request_manual.yml (API: path=.github/workflows/
  test_pull_request_manual.yml, head_branch=ci/p0-fork-license-repro). Named it.
- "starts and completes in the same second" is falsified by the section's own
  cited runs: 32992648441's 6000.3.1f1-editmode/windows-mono leg is
  04:58:48Z -> 04:58:49Z, and on the repro run 33758511822 one of the two legs is
  13:01:37Z -> 13:01:38Z. Restated as a bound ("within a second").
- "the whole job is 1-3 minutes" understated the measured spread; job wall clocks
  on 32992648441 run 1m23s - 3m14s. Now 1-4 minutes.
- The bad-.ulf ordering claim ("pulls the editor image first, then fails inside
  the container") was a hypothesis in declarative register - no bad-.ulf run is on
  record. Kept the diagnostic, marked it unverified, and grounded the "minutes"
  half on run 33757559794, whose licensed step takes 9-13 min per leg.
- Duration does not discriminate a fork PR from a same-repo run whose secret was
  deleted: UNITY_LICENSE is required:false, so both deliver an empty string and
  fail identically. Points at the head-repository check as the discriminator.
- "every run ... was red" was wrong for 4 of 7. All seven runs in the window are
  forks and none went green, but they are 3 failure / 2 cancelled / 2
  action_required. Corrected, and documented action_required (GitHub's
  first-time-contributor approval gate) as a second fork-PR shape with a
  different signature - conclusion is null and nothing ran.
- refs/pull/<n>/merge only exists while GitHub can compute a clean test-merge,
  neither ref is fetched by default, and workflow_dispatch takes only a branch or
  tag - so --ref refs/pull/<n>/head is rejected. That is what makes the deferral
  to #971 legible rather than unexplained.
- Gave "all 12 legs" its derivation (6 caller jobs x the 2-way platform matrix) so
  it self-corrects if the three commented-out playmode jobs are ever enabled.
- The pre-existing refresh procedure told a reader with an invalid-license run to
  re-issue the .ulf, which is the exact trap this section exists to prevent; added
  a one-clause back-pointer.

.github/workflows/test_unity_plugin.yml
- The comment blamed "secrets: inherit", a construct this file does not contain -
  it is the caller's. Attributed it to test_pull_request.yml.

NOT changed, after verification: "the same commit 91e2472 the fork PRs branched
from". One helper reported this as unverifiable and recommended weakening it. It
is exactly right - PRs #922/#967/#968 all have base.sha=91e2472a, and
`git merge-base --is-ancestor 91e2472 refs/pull/<n>/head` holds for all three
with the merge-base being 91e2472 itself. The claim stands as written.

Gates: Suite 4 (CI-surface) green - check_nuget_gate.py exit 0 ("NuGet gate OK:
15 pins, generation UNITY_MCP_DEPS_3, propagation consistent"), workflow YAML
re-parses with all three secrets keys and three inputs intact. Suites 1-3 not
applicable (nothing under Unity-MCP-Plugin/**, nothing under cli/); the
substantive verification of a CI-docs change is the PR's own CI run.
scan-pipeline-leaks.py: 0 hits over 108 added lines, controls 104/104.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bac1LKpVobv1i1FNGCRNvM
@IvanMurzak
IvanMurzak merged commit b1e617a into main Sep 3, 2026
17 checks passed
@IvanMurzak
IvanMurzak deleted the worktree-p0-unity-ci-regreen branch September 3, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: the 12 red test-unity legs since 2026-08-25 are fork PRs with no secrets, not a CI regression

1 participant