docs(ci): the red test-unity legs are fork PRs with no secrets, not a regression - #974
Merged
Conversation
… regression Every `test-pull-request` run between 2026-08-25 and 2026-08-29 showed all 12 `test-unity-*` legs red at `game-ci/unity-test-runner`, which reads as a week-long CI outage. Every pull request in that window came from a fork, and GitHub withholds repository secrets from a `pull_request` run whose head repository is a fork, so `secrets: inherit` passes empty strings and game-ci aborts before it does any work: ##[error]Missing Unity License File and no Serial was found. If this is a personal license, make sure to follow the activation steps and set the UNITY_LICENSE GitHub secret or enter a Unity serial number inside the UNITY_SERIAL GitHub secret. The licensed path is healthy and no secret needs re-issuing. Discriminator, one variable, same commit 91e2472 the fork PRs branched from: a dispatch of test_pull_request_manual.yml on main with the secrets present is green (run 33757559794), and the same dispatch with `secrets: inherit` removed is red with the line above (run 33758511822), reproducing the historical fingerprint exactly -- a `unity-test-runner` step that starts and completes in the same second while every step before it succeeds, then upload-artifact failing on an empty path. No behaviour change. Two documentation edits so the next fork PR does not cost another day: - docs/claude/ci-unity-license.md: a section on why fork PRs are red, the duration tell that separates it from a bad .ulf (and survives log expiry, because job metadata outlives logs), how to reach the licensed suite by dispatch, and the three run ids that pin it. - .github/workflows/test_unity_plugin.yml: a comment at the `secrets:` declaration pointing at that section. Whether a fork's Unity checks should be made passable is a separate policy call under the required-status-check ruleset; it stays with #543 and #971. Closes #973
Owner
Author
|
Fresh-evidence dispatch has finished. Recording the numbers, since the whole diagnosis rests on
Before the dispatch, The Unity legs ran and passed. Nothing was skipped, and nothing about the licensed path needed |
Accuracy corrections to the fork-PR section, from two report-only review helpers plus this pass's own verification against the GitHub API. Documentation only; no behavioural change. docs/claude/ci-unity-license.md - #543 is CLOSED (state_reason=completed, 2026-03-15), so "tracked in issue #543 and PR #971" sent a reader to a six-month-dead issue. PR #971 is the live one. - The run-33758511822 table row said "this same workflow", whose nearest antecedent is row 1's test_pull_request.yml. The run actually used test_pull_request_manual.yml (API: path=.github/workflows/ test_pull_request_manual.yml, head_branch=ci/p0-fork-license-repro). Named it. - "starts and completes in the same second" is falsified by the section's own cited runs: 32992648441's 6000.3.1f1-editmode/windows-mono leg is 04:58:48Z -> 04:58:49Z, and on the repro run 33758511822 one of the two legs is 13:01:37Z -> 13:01:38Z. Restated as a bound ("within a second"). - "the whole job is 1-3 minutes" understated the measured spread; job wall clocks on 32992648441 run 1m23s - 3m14s. Now 1-4 minutes. - The bad-.ulf ordering claim ("pulls the editor image first, then fails inside the container") was a hypothesis in declarative register - no bad-.ulf run is on record. Kept the diagnostic, marked it unverified, and grounded the "minutes" half on run 33757559794, whose licensed step takes 9-13 min per leg. - Duration does not discriminate a fork PR from a same-repo run whose secret was deleted: UNITY_LICENSE is required:false, so both deliver an empty string and fail identically. Points at the head-repository check as the discriminator. - "every run ... was red" was wrong for 4 of 7. All seven runs in the window are forks and none went green, but they are 3 failure / 2 cancelled / 2 action_required. Corrected, and documented action_required (GitHub's first-time-contributor approval gate) as a second fork-PR shape with a different signature - conclusion is null and nothing ran. - refs/pull/<n>/merge only exists while GitHub can compute a clean test-merge, neither ref is fetched by default, and workflow_dispatch takes only a branch or tag - so --ref refs/pull/<n>/head is rejected. That is what makes the deferral to #971 legible rather than unexplained. - Gave "all 12 legs" its derivation (6 caller jobs x the 2-way platform matrix) so it self-corrects if the three commented-out playmode jobs are ever enabled. - The pre-existing refresh procedure told a reader with an invalid-license run to re-issue the .ulf, which is the exact trap this section exists to prevent; added a one-clause back-pointer. .github/workflows/test_unity_plugin.yml - The comment blamed "secrets: inherit", a construct this file does not contain - it is the caller's. Attributed it to test_pull_request.yml. NOT changed, after verification: "the same commit 91e2472 the fork PRs branched from". One helper reported this as unverifiable and recommended weakening it. It is exactly right - PRs #922/#967/#968 all have base.sha=91e2472a, and `git merge-base --is-ancestor 91e2472 refs/pull/<n>/head` holds for all three with the merge-base being 91e2472 itself. The claim stands as written. Gates: Suite 4 (CI-surface) green - check_nuget_gate.py exit 0 ("NuGet gate OK: 15 pins, generation UNITY_MCP_DEPS_3, propagation consistent"), workflow YAML re-parses with all three secrets keys and three inputs intact. Suites 1-3 not applicable (nothing under Unity-MCP-Plugin/**, nothing under cli/); the substantive verification of a CI-docs change is the PR's own CI run. scan-pipeline-leaks.py: 0 hits over 108 added lines, controls 104/104. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bac1LKpVobv1i1FNGCRNvM
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-unity-*legs are not a CI regression. Everytest-pull-requestrunbetween 2026-08-25 and 2026-08-29 came from a fork, and GitHub withholds repository
secrets from a
pull_requestrun whose head repository is a fork — sosecrets: inherithands
test_unity_plugin.ymlan emptyUNITY_LICENSE/UNITY_EMAIL/UNITY_PASSWORDandgame-ci/unity-test-runneraborts before it pulls an image. The licensed path is healthy andno secret needs re-issuing.
docs/claude/ci-unity-license.md— including the duration tellthat separates it from a bad
.ulfand that survives log expiry, because job metadataoutlives logs — and points at it from a comment on
test_unity_plugin.yml'ssecrets:block.Cause class
Not (a) license/activation, not (b) action/runtime, not (c) test-project content, not (d)
Editor/product code. It is a fifth class the diagnosis had to add: fork PRs receive nosecrets. It presents license-shaped — game-ci's error names
UNITY_LICENSE— but is notcredential-shaped: the stored secrets are valid and untouched, so the owner gate G-P0 is not
exercised and
Editor/was never involved.Evidence line (verbatim, run
33758511822, job100658764301)Its
env:block in that same log readsUNITY_LICENSE:,UNITY_EMAIL:andUNITY_PASSWORD:with nothing after the colon.
Why the premise was wrong
Head repository of every run in the window (
gh api repos/IvanMurzak/Unity-MCP/actions/runs/<id>):32788655839IvanMurzak/Unity-MCP(same repo)32822206166Nghaiz/Unity-MCP(fork)32989259890akimaleo/Unity-MCP(fork)32992648441zorionarrillaga/Unity-MCP(fork)33250121859zorionarrillaga/Unity-MCP(fork)The last green run is the last same-repository run, and there was no same-repository PR in
between — so the red sample contains fork PRs and nothing else. Read as a time series it looks
exactly like a regression; it is a sampling artefact.
Two independent checks rule the alternatives out. Nothing in the repo changed: the only commit
touching
.github/between the last green and the first red is91e2472a(release 0.90.0),whose entire
.github/diff is three lines of.github/nuget-gate.lock, andtest_unity_plugin.ymlwas last modified 2026-07-20 (06ca3e1a). Nothing upstream changedeither: the action is SHA-pinned, and the pinned editor images
ubuntu-2022.3.62f3-base-3,ubuntu-2022.3.62f3-windows-mono-3andubuntu-6000.3.1f1-base-3were last pushed 2026-04-13,2026-04-13 and 2026-03-19 respectively — four months before the break.
The original logs are expired (HTTP 410), but job metadata is not. In
32992648441every Unityleg's
Run game-ci/unity-test-runner@08fd329f…step started and completed in the samesecond (
04:57:53Z -> 04:57:53Z) withFree disk spaceandactions/cachesucceeding beforeit. An action that rejects its inputs immediately never reached an activation attempt.
Discriminator — both halves, one variable, same commit
Both dispatches ran
test_pull_request_manual.ymlagainst91e2472a, the commit the fork PRsbranched from. The only difference between them is whether the Unity job received the secrets.
main, workflow unmodified33757559794test_pull_request_manual.ymlkept one Unity job and dropped itssecrets: inheritline33758511822The red half reproduces the historical fingerprint rather than merely failing: step 7
13:01:37Z -> 13:01:38Z,upload-artifactfailing straight after it onInput required and not supplied: path(nothing was produced to upload), and step numberingjumping 8 → 14 — the same shape
32992648441recorded. The throwaway branch has been deleted;its harness was one deletion of
secrets: inherit, so it reproduces from this description.Caveat stated plainly: the 2026-08-25..27 logs are gone, so the error text cannot be
compared byte-for-byte against them. What is compared is the structural fingerprint, which the
API still serves for both.
What this PR deliberately does not do
Making a fork's Unity checks pass would satisfy the required-status-check ruleset with jobs
that compiled nothing — a leg that stops failing because it stopped running is worse than a red
one, and it trades away the only mechanical guarantee that a fork's code builds. That is a
policy decision for the owner, open in PR #971 — issue #543, which first asked for it, was closed as completed in
2026-03 — and is out of scope here.
Test plan
python .github/scripts/check_nuget_gate.pylocally →NuGet gate OK: 15 pins, generation UNITY_MCP_DEPS_3, propagation consistent.(exit 0)test_unity_plugin.ymlre-parses as YAML with its threesecrets:keys intact (the edit is comment-only)Unity-MCP-Plugin/**orcli/**<?xml,LicenseVersion,UNITY_PASSWORD=→ 0 hits; no secret value appears in this PR, its commits, or its logstest_pull_request.ymlrun green on all jobs — nuget gate,test-cli(20 + 22) and all 12test-unity-*legs actually running and passing: run33758936539, 17/17 checksgreen, the twelve Unity legs 3m43s—15m44s — the band of the last green run, not the
1.3—3.2 min fail-fast band.
Closes #973