diff --git a/.agents/skills/qv-mobile-test-dispatch/SKILL.md b/.agents/skills/qv-mobile-test-dispatch/SKILL.md new file mode 100644 index 0000000000..42def85f29 --- /dev/null +++ b/.agents/skills/qv-mobile-test-dispatch/SKILL.md @@ -0,0 +1,209 @@ +--- +name: qv-mobile-test-dispatch +description: Start an AWS Device Farm mobile integration test for an addon and pick the right prebuild source, so the run tests the binary the developer means rather than the published release. Covers run ids, GPR dev builds, published pins, test filters, device names, and reading the result. Use when someone asks to run mobile tests, test an addon on a device/phone, test a native change on mobile, or invokes /qv-mobile-test-dispatch. +disable-model-invocation: true +--- + +# mobile-test-dispatch + +Mobile integration tests run on **AWS Device Farm**, which is billed per device +minute. They do not run automatically on PRs — someone dispatches them by hand, +choosing one platform, the device(s), and usually a test filter. + +The part that goes wrong is **which binary ends up on the phone**. A dispatch +does not compile the addon; it installs a prebuilt one. Get that wrong and the +run is green against code nobody changed. + +Canonical reference: [`docs/ci/MOBILE-ON-DEMAND.md`](../../../docs/ci/MOBILE-ON-DEMAND.md). +Read it once per session before answering detailed questions; this skill is the +operating procedure, that doc is the source of truth. + +## When to use this skill + +- "Run mobile tests for ``" +- "Test my native change on a device / on a phone" +- "Why did my mobile run test the wrong build?" +- "How do I get a run id?" + +## Safety rules + +- **Device Farm costs money.** Never dispatch the full suite to explore. Always + pass a `tests` filter and the smallest device set that answers the question. +- **`llm-llamacpp` is sharded** (7 Android groups, 13 iOS groups). An empty + `tests` filter fans out the whole set as separate Device Farm runs. Always + filter for LLM. +- **One platform per dispatch.** Android and iOS are separate runs. +- **A second dispatch of the same workflow on the same branch cancels the + first.** To cover both platforms, either wait, or use different addons in + parallel. +- Never dispatch on someone's behalf without telling them it bills Device Farm. + +## Step 1 — decide which binary should be tested + +| goal | input | +|---|---| +| my own PR's native change | `prebuild_run_id=` | +| a build from another branch, or a published release | `package=@tetherto/-mono@` or `package=@qvac/@` | +| just the published release | leave both empty | + +`prebuild_run_id` and `package` are **mutually exclusive** — setting both fails +with a message telling you to clear one. + +The input is named `package` on most addons but **`package_spec`** on +`asr-ggml`, `audiogen-ggml`, `tts-ggml`. `prebuild_run_id` is the same everywhere. + +## Step 2 — get a run id (only for the `prebuild_run_id` route) + +**First: the PR must have built prebuilds at all.** The prebuild stage is +label-gated by `ci-router` — it runs only when the PR carries `prebuilds`, +`run-desktop-addon-tests`, or `run-mobile-addon-tests`. With none of those there +is no bundle and no run id. Add the `prebuilds` label and let CI re-run. + +**Then:** open the PR's Checks tab, click the run that built the prebuilds, and +take the number at the end of its URL. + +Do **not** filter by the addon's own workflow name. Which workflow built the +bundle varies — `on-pr-nx.yml` for most addons, `on-pr-.yml` for some, +`on-merge-.yml` for a branch build. Scope by the PR's head commit: + +```bash +PKG=llm-llamacpp # the package directory name, i.e. packages/ +PR=1234 + +SHA=$(gh pr view "$PR" --repo tetherto/qvac --json headRefOid --jq .headRefOid) +for rid in $(gh api "repos/tetherto/qvac/actions/runs?head_sha=$SHA&per_page=100" \ + --jq '.workflow_runs[].id'); do + gh api "repos/tetherto/qvac/actions/runs/$rid/artifacts?per_page=100" \ + --jq ".artifacts[]|select(.name==\"prebuilds-$PKG\" and .expired==false)|.name" \ + 2>/dev/null | grep -q . && { echo "$rid"; break; } +done + +# An empty result must not be dispatched: prebuild_run_id="" is the unchanged +# path and quietly resolves @latest, which is the failure this route closes. +``` + +Nothing printed means either the label is missing, or — on the nx path — that run +only built the addons it considered affected and yours was not one. The dispatch +failure message lists which addons a run did build. + +## Step 3 — pick a valid test filter + +`tests` is a mocha `--grep` over runner **names**, not file names. A name that +matches nothing is rejected up front by `validate-devices`, for free, with the +list of valid names — so a wrong guess costs nothing but a round trip. + +Read the names from the same source `validate-devices` uses: + +```bash +# sharded addons (llm-llamacpp, diffusion-cpp, tts-ggml, audiogen-ggml, vla, ...) +jq -r '(.android//{})|[..|strings]|unique|.[]' packages//test/mobile/test-groups.json + +# single-spec addons +grep -oE '\brun[A-Z][A-Za-z0-9_]*' packages//test/mobile/integration.auto.cjs | sort -u +``` + +If a name is rejected on device with +`[prestage] FATAL: tests grep // matched no known runner`, it is in neither +the addon's `test-groups.json` nor its `integration.auto.cjs` — i.e. a typo. Take +a name from the commands above. (That FATAL used to fire for *valid* runners too, +because the prestage generator kept its own list; `readKnownRunners()` now reads +`test-groups.json` directly.) + +## Step 4 — dispatch + +```bash +gh workflow run integration-mobile-test-.yml --repo tetherto/qvac --ref \ + -f platform=Android \ + -f devices_custom="Google Pixel 9" \ + -f device_model_operator=EQUALS \ + -f tests= \ + -f prebuild_run_id= +``` + +- `devices_custom` takes a comma-separated list and overrides the `device` + dropdown. Names are full fleet names (`Google Pixel 9`, `Apple iPhone 16 Pro`). +- `device_model_operator=EQUALS` bills exactly that model; `CONTAINS` may pick a + different variant. +- `ref` selects the JS harness, tests and app — **not** the native binary. It and + the prebuild source are deliberately independent. + +## Step 5 — read the result + +The build job's setup phase prints the provenance: + +``` +Verified: prebuilds come from run — artifact 'prebuilds-', +workflow '', head , branch (), +``` + +Check the **head SHA** is the commit you meant — a run id resolves whether or not +it built the code under review. + +Warnings worth acting on: + +- `run concluded 'failure'` — the source run was red. Its prebuild job may + still be the green part, but confirm. +- `run built code from the FORK ''` — normal for a fork PR (the repo + is fork-first), but confirm you meant that contributor's code. + +The run-id path **fails closed** — a wrong, private, unfinished or expired run id +fails the run with the reason rather than falling back to `@latest`. + +## Per-addon notes + +| addon | note | +|---|---| +| `llm-llamacpp` | sharded — always pass `tests` | +| `asr-ggml`, `audiogen-ggml` | `@qvac/*` publishes **no mobile prebuilds**, so an empty input cannot work. Use `prebuild_run_id` or the GPR `-mono` build. | +| `audiogen-ggml` | pins its composite actions to the default branch, so `prebuild_run_id` only works once that support is on `main`; it fails loudly with instructions until then | +| `vla` | package dir is `packages/vla-ggml`, workflow slug is `vla` | +| `decoder-audio` | no native prebuild of its own (rides `bare-ffmpeg` from npm). `package` has no effect; use `ref`. | +| `inference-addon-cpp` | compiles its own prebuilds in-run from the dispatched `ref`, so no prebuild input is needed or offered | + +## Reading a failure — where the logs are + +The `console-logs-*` artifact on the run is where everything lands. The +`test-results.json` in it only records the harness assertion +(`expect(received).toBe(expected)` at `app.test.js`), which is identical for +every failure and never says why. The real reason is in the app's own output, +and the file differs per platform. + +| what | Android | iOS | +|---|---|---| +| JS / bare runtime, TAP lines, the failure | `logcat_full.txt`, `bare` tag | `bare_console.log` | +| **native C++ / engine output** | `logcat_full.txt`, `bare` tag, `[C++ TEST]` prefix | `bare_console.log`, `[C++ TEST]` prefix | +| app shell | `logcat_full.txt`, `ReactNativeJS` tag | `bare_console.log` | +| device/OS noise | `logcat_full.txt` (most of it) | `iOS_appium.log` | + +```bash +gh run download --repo tetherto/qvac --dir ./logs + +# Android — the bare runtime carries BOTH the JS and the C++ output +grep -aE "E bare|I bare" logs/**/*logcat_full.txt | head -40 # test + errors +grep -a "\[C++ TEST\]" logs/**/*logcat_full.txt | head -40 # native/engine + +# iOS — same two, one file +grep -aE "error|not ok" logs/**/*bare_console.log | head -40 +grep -a "\[C++ TEST\]" logs/**/*bare_console.log | head -40 +``` + +Traps that cost real time: + +- **Use `logcat_full.txt`, not `Logcat.logcat`.** They are different files; + the latter is a smaller capture and does not carry the bare output. +- **Grep the `bare` tag, not TAP markers or the package name.** The runtime + prints through logcat, so `TAP version`/`ok 1` never appear as raw lines. +- Native C++ lines are prefixed `[C++ TEST] [INFO]: [Llama.cpp] ...` on both + platforms — the engine logs through the same channel, not a separate tag. +- There is **no `bare_console.log` on Android**, by construction: the app writes + it into its private data dir, which adb cannot read and `run-as` refuses on a + release-signed APK. That is expected — logcat is the Android channel. + +A real example, the whole reason a run went red, invisible in `test-results.json`: + +``` +E bare: Test 'runFitStubTest' failed: AddonError: ADDON_NOT_FOUND: + Cannot find addon '.' from @qvac/model-fit/binding.js + Candidates: - linked:libqvac__model-fit.0.12.0.so + [cause]: Error: dlopen fail +``` diff --git a/.agents/skills/qv-mobile-test-dispatch/agents/openai.yaml b/.agents/skills/qv-mobile-test-dispatch/agents/openai.yaml new file mode 100644 index 0000000000..5b1f887a91 --- /dev/null +++ b/.agents/skills/qv-mobile-test-dispatch/agents/openai.yaml @@ -0,0 +1,2 @@ +policy: + allow_implicit_invocation: false diff --git a/.agents/skills/qv-skill-list/SKILL.md b/.agents/skills/qv-skill-list/SKILL.md index fd052ab11e..4b9a9dd250 100644 --- a/.agents/skills/qv-skill-list/SKILL.md +++ b/.agents/skills/qv-skill-list/SKILL.md @@ -42,6 +42,7 @@ When unsure which skill fits, scan the tables below or ask: *"which qv skill sho | [`qv-devops-pr-status`](../qv-devops-pr-status/SKILL.md) | Team DevOps PR dashboard: re-review, stale, needs-review, conflicts. | DevOps pod PR queue health. **Manual:** `/qv-devops-pr-status` | | [`qv-devops-why-my-pr-not`](../qv-devops-why-my-pr-not/SKILL.md) | Diagnose missing CI checks or merge blockers (labels, CODEOWNERS, approvals). | "Why aren't checks running?" / "Why can't I merge?" **Manual:** `/qv-devops-why-my-pr-not` | | [`qv-devops-daily-update`](../qv-devops-daily-update/SKILL.md) | Slack standup (Done / Planned / Blockers) from PRs, reviews, CI. | DevOps EOD or standup. **Manual:** `/qv-devops-daily-update` | +| [`qv-mobile-test-dispatch`](../qv-mobile-test-dispatch/SKILL.md) | Start an addon mobile (Device Farm) test and pick the right prebuild source; where the Android/iOS and C++ logs are. | "Run mobile tests for X" / "test my native change on a device" / reading a mobile failure. **Manual:** `/qv-mobile-test-dispatch` | --- @@ -128,6 +129,7 @@ Rule nudge: `.cursor/rules/qip-triage.mdc` | SDK team PR board | `qv-sdk-pr-status` | | DevOps team PR board | `qv-devops-pr-status` | | Why CI/merge is blocked | `qv-devops-why-my-pr-not` | +| Run mobile tests on a device | `qv-mobile-test-dispatch` | | Write SDK PR body | `qv-sdk-pr-create` | | Sync SDK models.ts from registry | `qv-sdk-update-models` | | Write addon PR body | `qv-addon-pr-create` | diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 901b522d00..6f03d3c01a 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -11,6 +11,8 @@ Read the relevant references before changing CI: - [`../docs/ci/SELF-HOSTED-RUNNERS.md`](../docs/ci/SELF-HOSTED-RUNNERS.md) for persistent runners and workspace cleanup. - [`../docs/ci/TEAMS.md`](../docs/ci/TEAMS.md) for approval ownership. +- [`../docs/ci/MOBILE-ON-DEMAND.md`](../docs/ci/MOBILE-ON-DEMAND.md) for dispatching + addon mobile (Device Farm) tests and choosing the prebuild source. - [`../docs/agent-automation.md`](../docs/agent-automation.md) for automation safety. When editing workflows or composite actions: diff --git a/.github/actions/run-mobile-integration-tests/setup/action.yml b/.github/actions/run-mobile-integration-tests/setup/action.yml index 2be796f775..d983ba9331 100644 --- a/.github/actions/run-mobile-integration-tests/setup/action.yml +++ b/.github/actions/run-mobile-integration-tests/setup/action.yml @@ -4,7 +4,8 @@ description: | Frees up disk space (Linux runners), checks out the qvac-test-addon-mobile framework into ./test-framework, sets up Node.js, installs the Expo CLI, - resolves the addon's prebuilds (PR artifacts -> npm fallback -> committed), + resolves the addon's prebuilds (named run id -> PR artifacts -> npm fallback + -> committed), and prunes desktop prebuilds to keep the runner from running out of disk. Prebuild precedence: CI artifacts win by default (artifact-first). A caller @@ -16,9 +17,12 @@ description: | integration-mobile-test-llm-llamacpp.yml does. The pinned-package path resolves BOTH registries: @qvac/* from npmjs.org and - @tetherto/-mono from GitHub Packages. The latter is the only way to put - an unmerged native change on a device, since a standalone dispatch has no - prebuild artifacts of its own to fall back to. + @tetherto/-mono from GitHub Packages. That is how a build from ANOTHER + branch, or a published release, gets onto a device. + + To test YOUR OWN PR, prefer `prebuild-run-id`: it installs the prebuilds + artifact of a named run. It takes precedence over every other source and never + falls back — a run id that cannot be honoured fails the run. Pre-condition: the consumer has already checked out the addon repository at ./addon/. We do NOT check the addon out here so the consumer can run @@ -100,6 +104,34 @@ inputs: "true" to force-enable the fallback in a PR context. required: false default: "false" + prebuild-run-id: + description: | + A run id whose `prebuilds-` artifact holds the native binaries to + install (falls back to the legacy `prebuilds` name). The route for testing + your own PR on a device. + + Highest precedence of every prebuild source: when set, the same-run + artifact downloads and the npm/GPR fallback are skipped, and resolution + fails closed rather than sliding back to @latest. Mutually exclusive with + `package-version` / `force-npm-prebuild`. + + The provenance line names the head SHA and the repository the binaries came + from; a fork-built run warns but is not refused. `ref` and the prebuild run + are independent, so check the printed head SHA is the one you meant. + + Needs `actions: read` on the calling job; `pat-token` carries it. Empty + (default) leaves every existing path unchanged. + required: false + default: "" + +outputs: + prebuild-source-run-id: + description: | + The run id the prebuilds were installed from, or empty when another source + was used. A caller that passed `prebuild-run-id` can assert this matches: + an older copy of this action silently ignores the input (GitHub only warns + on an unknown input), which would install @latest instead. + value: ${{ steps.prebuild_run.outputs.source_run_id }} runs: using: composite @@ -150,8 +182,78 @@ runs: echo "Installing global dependencies..." npm install -g --force @expo/cli@latest + # Resolution happens before anything is downloaded, so a bad run id costs + # nothing and no branch below can rescue it into an @latest run. Node rather + # than `gh`, which is not guaranteed on the self-hosted runners. + - name: Resolve prebuilds from a run id + id: prebuild_run + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id != '' + shell: bash + env: + PREBUILD_RUN_ID: ${{ inputs.prebuild-run-id }} + ADDON_WORKDIR: ${{ inputs.addon-workdir }} + PLATFORM: ${{ inputs.platform }} + PACKAGE_VERSION: ${{ inputs.package-version }} + FORCE_NPM_PREBUILD: ${{ inputs.force-npm-prebuild }} + GITHUB_TOKEN: ${{ inputs.pat-token }} + REPO: ${{ github.repository }} + ACTION_PATH: ${{ github.action_path }} + run: | + node "$ACTION_PATH/resolve-prebuild-run.mjs" + # download-artifact overwrites the files it carries but leaves the rest, + # so a committed prebuilds/ dir or a leftover from an earlier job on the + # same self-hosted runner could shadow the resolved run. Runs only after + # resolution succeeds, so a rejected run id deletes nothing. + rm -rf "addon/$ADDON_WORKDIR/prebuilds" + + # No continue-on-error: the run id is an explicit instruction, so a failed + # download must fail the run. + - name: Download prebuilds (from the resolved run) + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id != '' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # 8.0.1 + with: + # By id, not name: a re-run can leave two live `prebuilds-` rows + # under one run id. A single artifact-ids value extracts straight into + # `path`, exactly as `name` does. + artifact-ids: ${{ steps.prebuild_run.outputs.artifact_id }} + run-id: ${{ steps.prebuild_run.outputs.source_run_id }} + github-token: ${{ inputs.pat-token }} + path: addon/${{ inputs.addon-workdir }}/prebuilds + + # A partial prebuild matrix (a cancelled leg) still publishes a bundle, just + # without this platform's dir — which the generic "Verify and prepare + # prebuilds" step below would accept, since it only checks non-emptiness. + - name: Verify the resolved run's prebuilds cover this platform + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id != '' + shell: bash + working-directory: addon/${{ inputs.addon-workdir }} + env: + EXPECTED_DIRS: ${{ steps.prebuild_run.outputs.expected_dirs }} + SOURCE_RUN_ID: ${{ steps.prebuild_run.outputs.source_run_id }} + SOURCE_HEAD_SHA: ${{ steps.prebuild_run.outputs.head_sha }} + SOURCE_ARTIFACT: ${{ steps.prebuild_run.outputs.artifact_name }} + PLATFORM: ${{ inputs.platform }} + run: | + MISSING="" + for dir in $EXPECTED_DIRS; do + if [ ! -d "prebuilds/$dir" ] || [ -z "$(ls -A "prebuilds/$dir" 2>/dev/null)" ]; then + MISSING="$MISSING $dir" + fi + done + if [ -n "$MISSING" ]; then + echo "::error::Run $SOURCE_RUN_ID's '$SOURCE_ARTIFACT' artifact has no$MISSING, so it cannot build for $PLATFORM." + echo "That run's prebuild matrix did not produce this platform — a cancelled or" + echo "filtered leg publishes a bundle without it. Pick a run whose $PLATFORM" + echo "prebuild leg succeeded, or re-run that leg and use the new run id." + echo "Directories the artifact does contain:" + ls -A prebuilds 2>/dev/null || echo " (none)" + exit 1 + fi + echo "Using prebuilds from run $SOURCE_RUN_ID (head $SOURCE_HEAD_SHA), artifact '$SOURCE_ARTIFACT':" + ls -la prebuilds/ + - name: Download Android prebuilds (from artifacts) - if: inputs.skip-prebuilds != 'true' && inputs.force-npm-prebuild != 'true' && inputs.platform == 'Android' + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id == '' && inputs.force-npm-prebuild != 'true' && inputs.platform == 'Android' uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # 8.0.1 with: path: addon/${{ inputs.addon-workdir }}/prebuilds @@ -160,7 +262,7 @@ runs: continue-on-error: true - name: Download iOS prebuilds (from artifacts) - if: inputs.skip-prebuilds != 'true' && inputs.force-npm-prebuild != 'true' && inputs.platform == 'iOS' + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id == '' && inputs.force-npm-prebuild != 'true' && inputs.platform == 'iOS' uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # 8.0.1 with: path: addon/${{ inputs.addon-workdir }}/prebuilds @@ -197,7 +299,7 @@ runs: # npm. No-op for addons / runs that never produced this artifact (e.g. # genuine workflow_dispatch), where the npm fallback below still applies. - name: Download merged prebuilds artifact (fallback when per-matrix artifacts absent) - if: inputs.skip-prebuilds != 'true' && inputs.force-npm-prebuild != 'true' && steps.check_prebuilds.outputs.present != 'true' + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id == '' && inputs.force-npm-prebuild != 'true' && steps.check_prebuilds.outputs.present != 'true' uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # 8.0.1 with: name: ${{ steps.check_prebuilds.outputs.merged }} @@ -215,7 +317,7 @@ runs: # published package is the intended behaviour, so the fallback stays on. # `allow-npm-fallback: 'true'` force-enables it even in a PR context. - name: Download prebuilds (from npm — fallback when no artifacts found) - if: inputs.skip-prebuilds != 'true' && (inputs.force-npm-prebuild == 'true' || inputs.allow-npm-fallback == 'true' || (github.event_name != 'pull_request' && github.event_name != 'pull_request_target')) + if: inputs.skip-prebuilds != 'true' && inputs.prebuild-run-id == '' && (inputs.force-npm-prebuild == 'true' || inputs.allow-npm-fallback == 'true' || (github.event_name != 'pull_request' && github.event_name != 'pull_request_target')) shell: bash working-directory: addon/${{ inputs.addon-workdir }} env: diff --git a/.github/actions/run-mobile-integration-tests/setup/resolve-prebuild-run.mjs b/.github/actions/run-mobile-integration-tests/setup/resolve-prebuild-run.mjs new file mode 100644 index 0000000000..426021456a --- /dev/null +++ b/.github/actions/run-mobile-integration-tests/setup/resolve-prebuild-run.mjs @@ -0,0 +1,378 @@ +// Resolves the prebuilds artifact of a named workflow run so a mobile dispatch +// can install the binaries that run already built. +// +// Helpers are pure and the HTTP client is injected, so the decision surface is +// covered by test/resolve-prebuild-run.test.mjs without a network or a token. +// `gh` is not used: mobile jobs run on self-hosted runners where the CLI is not +// guaranteed, while Node comes from the setup action's own setup-node step. + +import { appendFileSync, realpathSync } from 'node:fs' +import { fileURLToPath } from 'node:url' + +const MAX_ARTIFACT_PAGES = 10 + +// Platform -> the prebuild dir the mobile build consumes. The setup action fans +// these out to the other ABIs afterwards, so only the built dir matters here. +export const PLATFORM_PREBUILD_DIRS = { + Android: ['android-arm64'], + iOS: ['ios-arm64'], +} + +// The run id arrives from a workflow_dispatch input and ends up in an API path. +export function parseRunId(raw) { + const value = String(raw ?? '').trim() + return /^[1-9][0-9]*$/.test(value) ? value : null +} + +// Matches how reusable-prebuilds.yml names the merged bundle. +export function mergedArtifactName(addonWorkdir) { + const segments = String(addonWorkdir ?? '') + .split('/') + .filter((segment) => segment !== '' && segment !== '.') + const name = segments[segments.length - 1] + if (!name || name === '..') return null + return `prebuilds-${name}` +} + +// Bare `prebuilds` is the legacy name, kept so an older run id still resolves. +export function candidateArtifactNames(addonWorkdir) { + const merged = mergedArtifactName(addonWorkdir) + return merged ? [merged, 'prebuilds'] : ['prebuilds'] +} + +export function platformPrebuildDirs(platform) { + return PLATFORM_PREBUILD_DIRS[String(platform ?? '').trim()] ?? [] +} + +// A live artifact wins in candidate order. "Expired" is only reported when every +// candidate is expired, because expired and never-existed need different advice. +export function selectArtifact(artifacts, candidates) { + const rows = Array.isArray(artifacts) ? artifacts : [] + + // A re-run leaves the earlier attempt's artifacts under the same run id, so a + // run can hold two live rows with the same name. Take the NEWEST: first-match + // would freeze whichever the API happened to list first (id-ascending in + // practice, i.e. the pre-re-run binary) while the provenance line printed the + // same head_sha either way — a stale .bare on the device, and a log that looks + // right. created_at breaks the tie when ids are not comparable. + const newest = (a, b) => { + const at = Date.parse(a?.created_at ?? '') + const bt = Date.parse(b?.created_at ?? '') + if (Number.isFinite(at) && Number.isFinite(bt) && at !== bt) return at > bt ? a : b + return (Number(a?.id) || 0) >= (Number(b?.id) || 0) ? a : b + } + + for (const name of candidates) { + const live = rows + .filter((row) => row?.name === name && row?.expired !== true) + .reduce((best, row) => (best ? newest(best, row) : row), null) + if (live) return { name, id: live.id ?? null, expired: false } + } + for (const name of candidates) { + if (rows.some((row) => row?.name === name)) { + return { name, id: null, expired: true } + } + } + return null +} + +// A run id and a pinned package are two answers to "which binary goes on the +// phone", so a caller that sets both is told to choose rather than guessed at. +export function conflictingSource({ packageVersion, forceNpmPrebuild }) { + const pinned = String(packageVersion ?? '').trim() + if (pinned !== '') return `package/package_spec='${pinned}'` + if (String(forceNpmPrebuild ?? '').trim() === 'true') { + return 'force-npm-prebuild=true' + } + return null +} + +// A fork PR's on-pr run is pull_request_target, so it appears in the base repo's +// run list while its head_repository is the fork. This warns rather than +// refusing: the repo is fork-first (docs/gitflow.md), and a fork's prebuilds only +// exist once the merge/release team approved `fork-ci` on that run. +export function sourceRepositoryWarning(run, repo) { + const headRepo = run?.head_repository?.full_name + if (!headRepo) { + return ( + `run ${run?.id} reports no head repository, so which repository built these ` + + 'binaries cannot be confirmed from the API.' + ) + } + if (headRepo !== repo) { + return ( + `run ${run?.id} built code from the FORK '${headRepo}', not '${repo}'. ` + + 'That is normal for a fork PR and the run passed fork-ci approval to build ' + + "at all — but confirm you meant that contributor's code." + ) + } + return null +} + +export function formatProvenance(run, artifactName) { + const name = run?.name ?? 'unknown workflow' + const sha = run?.head_sha ?? 'unknown' + const branch = run?.head_branch ?? 'unknown' + const conclusion = run?.conclusion ?? run?.status ?? 'unknown' + return ( + `Verified: prebuilds come from run ${run?.id} — artifact '${artifactName}', ` + + `workflow '${name}', head ${sha}, branch ${branch} ` + + `(${run?.head_repository?.full_name ?? 'unknown repo'}), ${conclusion}` + ) +} + +// A red run can still hold good prebuilds: the prebuild job uploads before the +// desktop tests and lint that colour the run. Warn instead of refusing. +export function conclusionWarning(run) { + const conclusion = String(run?.conclusion ?? '').trim() + if (conclusion === '' || conclusion === 'success') return null + return ( + `run ${run?.id} concluded '${conclusion}'. Its prebuilds artifact exists and will be used; ` + + 'check that the prebuild job itself is the green one before trusting the result.' + ) +} + +// Which addons a run built a bundle for. A real nx run carries 65+ artifacts, so +// listing them all buries the answer, which is usually "nx decided your addon +// wasn't affected". +export function describeAvailableBundles(artifacts, limit = 12) { + const rows = Array.isArray(artifacts) ? artifacts : [] + if (rows.length === 0) return ['That run published no artifacts at all.'] + + const bundles = [ + ...new Set( + rows + .map((row) => row?.name) + .filter((name) => typeof name === 'string' && name.startsWith('prebuilds-')) + // The per-PR reuse marker is also a `prebuilds-` artifact but is not a + // bundle, and would read as an addon name. + .filter((name) => !name.startsWith('prebuilds-cache-pr-')) + .map((name) => name.slice('prebuilds-'.length)), + ), + ].sort() + + if (bundles.length === 0) { + return [ + `That run published ${rows.length} artifact(s) but no prebuilds bundle — it does not build prebuilds.`, + "Use the addon's on-pr-.yml (or prebuilds-.yml) run instead.", + ] + } + + const shown = bundles.slice(0, limit).join(', ') + const rest = bundles.length > limit ? `, +${bundles.length - limit} more` : '' + return [ + `That run built prebuilds for: ${shown}${rest}.`, + 'On the nx path a run only builds the addons it decided were affected, so yours may not be there.', + ] +} + +class ResolveError extends Error { + constructor(message, hints = []) { + super(message) + this.hints = hints + } +} + +async function apiJson(request, url, token) { + const response = await request(url, { + headers: { + accept: 'application/vnd.github+json', + authorization: `Bearer ${token}`, + 'x-github-api-version': '2022-11-28', + }, + }) + if (response.status === 404) return { notFound: true, body: null } + if (!response.ok) { + throw new ResolveError( + `GitHub API returned ${response.status} for ${url}`, + response.status === 403 || response.status === 401 + ? [ + "The job needs 'actions: read' and a token that carries it.", + "Add `actions: read` to the build job's permissions block.", + ] + : [], + ) + } + return { notFound: false, body: await response.json() } +} + +async function listArtifacts(request, apiUrl, repo, runId, token) { + const artifacts = [] + for (let page = 1; page <= MAX_ARTIFACT_PAGES; page += 1) { + const url = `${apiUrl}/repos/${repo}/actions/runs/${runId}/artifacts?per_page=100&page=${page}` + const { notFound, body } = await apiJson(request, url, token) + // An empty body and an empty run are different facts; conflating them + // reported "published no artifacts at all" for a failed listing. + if (notFound || !body) { + throw new ResolveError(`Could not list run ${runId}'s artifacts (the API returned no listing).`, [ + `Check https://github.com/${repo}/actions/runs/${runId}`, + "If the run exists, the token may lack 'actions: read'.", + ]) + } + const rows = body.artifacts ?? [] + artifacts.push(...rows) + if (rows.length < 100) return artifacts + } + throw new ResolveError( + `Run ${runId} has more than ${MAX_ARTIFACT_PAGES * 100} artifacts, so the listing was truncated.`, + ['The bundle may exist beyond the page bound — raise MAX_ARTIFACT_PAGES in resolve-prebuild-run.mjs.'], + ) +} + +// Resolves everything the action needs before a byte is downloaded. Never falls +// back to another prebuild source: a run id that cannot be honoured fails. +export async function resolvePrebuildRun({ env, request }) { + const rawRunId = env.PREBUILD_RUN_ID ?? '' + const runId = parseRunId(rawRunId) + if (!runId) { + throw new ResolveError( + `prebuild_run_id must be a numeric GitHub Actions run id, got '${rawRunId}'.`, + [ + 'Copy it from the run URL: https://github.com///actions/runs/.', + 'Or: gh run list --workflow on-pr-.yml --branch --status success --json databaseId', + ], + ) + } + + const conflict = conflictingSource({ + packageVersion: env.PACKAGE_VERSION, + forceNpmPrebuild: env.FORCE_NPM_PREBUILD, + }) + if (conflict) { + throw new ResolveError( + `prebuild_run_id and a pinned package are mutually exclusive (${conflict}).`, + [ + `prebuild_run_id installs run ${runId}'s prebuilds artifact; package/package_spec installs a published or GPR build.`, + 'Clear whichever one you did not mean and dispatch again.', + ], + ) + } + + const token = String(env.GITHUB_TOKEN ?? '').trim() + if (!token) { + throw new ResolveError('prebuild_run_id needs a token with actions:read, but none was supplied.', [ + 'The calling workflow must pass pat-token (secrets.GITHUB_TOKEN is enough).', + "The build job also needs 'actions: read' in its permissions block.", + ]) + } + + const repo = String(env.REPO ?? '').trim() + if (!repo) throw new ResolveError('REPO must name the repository holding the run.') + + const platform = String(env.PLATFORM ?? '').trim() + const expectedDirs = platformPrebuildDirs(platform) + if (expectedDirs.length === 0) { + throw new ResolveError(`Unknown platform '${platform}' — expected Android or iOS.`) + } + + const apiUrl = String(env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, '') + + const { notFound, body: run } = await apiJson( + request, + `${apiUrl}/repos/${repo}/actions/runs/${runId}`, + token, + ) + if (notFound || !run) { + throw new ResolveError(`Run ${runId} does not exist in ${repo}, or this token cannot see it.`, [ + `Check https://github.com/${repo}/actions/runs/${runId}`, + `The id must be a run in ${repo}'s own Actions list — a run from a fork's own`, + 'Actions tab is a different repository and does not resolve here.', + ]) + } + + const candidates = candidateArtifactNames(env.ADDON_WORKDIR) + const artifacts = await listArtifacts(request, apiUrl, repo, runId, token) + const selected = selectArtifact(artifacts, candidates) + + if (!selected) { + // An unfinished run is the likeliest reason a bundle is missing, and saying + // "does not build prebuilds" there sends the reader to the wrong run. + const status = String(run.status ?? '').trim() + if (status !== '' && status !== 'completed') { + throw new ResolveError( + `Run ${runId} is still '${status}', so its prebuilds artifact has not been uploaded yet.`, + [ + `Watch it: https://github.com/${repo}/actions/runs/${runId}`, + 'The prebuild job uploads part-way through the run — wait for it, then dispatch again with the same run id.', + ], + ) + } + throw new ResolveError( + `Run ${runId} has no ${candidates.map((name) => `'${name}'`).join(' or ')} artifact.`, + [ + `That run is '${run.name ?? 'unknown'}' (${run.path ?? 'unknown path'}).`, + ...describeAvailableBundles(artifacts), + ], + ) + } + + if (selected.expired) { + throw new ResolveError(`Run ${runId}'s '${selected.name}' artifact has expired.`, [ + 'Artifacts expire (retention is set per repository), so a run id stops being usable once its prebuilds are gone.', + 'Re-run the prebuild job on your PR and dispatch against the new run id.', + ]) + } + + return { + runId, + artifactName: selected.name, + artifactId: selected.id, + headSha: run.head_sha ?? '', + headBranch: run.head_branch ?? '', + expectedDirs, + provenance: formatProvenance({ ...run, id: runId }, selected.name), + warnings: [ + sourceRepositoryWarning({ ...run, id: runId }, repo), + conclusionWarning({ ...run, id: runId }), + ].filter(Boolean), + } +} + +function writeOutputs(outputPath, outputs) { + if (!outputPath) return + const lines = Object.entries(outputs) + .map(([key, value]) => `${key}=${value}`) + .join('\n') + appendFileSync(outputPath, `${lines}\n`) +} + +export async function main({ env = process.env, request = fetch, log = console } = {}) { + try { + const resolved = await resolvePrebuildRun({ env, request }) + for (const warning of resolved.warnings) log.log(`::warning::${warning}`) + log.log(resolved.provenance) + writeOutputs(env.GITHUB_OUTPUT, { + artifact_name: resolved.artifactName, + // The download selects by id: a re-run leaves the earlier attempt's + // artifacts under the same run id, so a run can hold two live rows with + // the same name. + artifact_id: resolved.artifactId ?? '', + source_run_id: resolved.runId, + head_sha: resolved.headSha, + head_branch: resolved.headBranch, + expected_dirs: resolved.expectedDirs.join(' '), + }) + return 0 + } catch (error) { + log.log(`::error::${error.message}`) + for (const hint of error.hints ?? []) log.log(hint) + return 1 + } +} + +// Realpaths, not URL strings: import.meta.url is the percent-encoded realpath +// while argv[1] is the literal path, so a naive comparison breaks whenever a +// segment is a symlink (macOS /tmp -> /private/tmp). That failure was silent — +// main() never ran and the step still exited 0. +function invokedAsScript() { + if (!process.argv[1]) return false + try { + return realpathSync(fileURLToPath(import.meta.url)) === realpathSync(process.argv[1]) + } catch { + return false + } +} + +if (invokedAsScript()) { + process.exitCode = await main() +} diff --git a/.github/actions/run-mobile-integration-tests/setup/test/resolve-prebuild-run.test.mjs b/.github/actions/run-mobile-integration-tests/setup/test/resolve-prebuild-run.test.mjs new file mode 100644 index 0000000000..29907da77f --- /dev/null +++ b/.github/actions/run-mobile-integration-tests/setup/test/resolve-prebuild-run.test.mjs @@ -0,0 +1,682 @@ +// Guards the prebuild-run-id resolution used by the mobile setup action. This is +// the path that puts an unmerged native change on a device, so its failure modes +// matter more than its happy path: it must never quietly degrade into testing +// the published release. The HTTP client is injected, so nothing here needs a +// network, a token or a real run. +import test from 'node:test' +import assert from 'node:assert/strict' +import { mkdtempSync, readFileSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { + candidateArtifactNames, + describeAvailableBundles, + conclusionWarning, + conflictingSource, + sourceRepositoryWarning, + formatProvenance, + main, + mergedArtifactName, + parseRunId, + platformPrebuildDirs, + resolvePrebuildRun, + selectArtifact, +} from '../resolve-prebuild-run.mjs' + +const REPO = 'tetherto/qvac' +const RUN_ID = '33179656677' +const HEAD_SHA = '1d2c3b4a5f6e7d8c9b0a1f2e3d4c5b6a7f8e9d01' + +function run(overrides = {}) { + return { + id: Number(RUN_ID), + name: 'On PR Trigger (LLM)', + path: '.github/workflows/on-pr-llm-llamacpp.yml', + head_sha: HEAD_SHA, + head_branch: 'feat/backend-selection', + head_repository: { full_name: REPO }, + status: 'completed', + conclusion: 'success', + ...overrides, + } +} + +// Stands in for the REST API. Records every URL so a test can prove which calls +// were made — in particular that NOTHING is requested when validation fails. +function fakeApi({ runResponse, artifactPages = [[]], status = 200 }) { + const calls = [] + let page = 0 + + const respond = (body, code = 200) => ({ + ok: code >= 200 && code < 300, + status: code, + json: async () => body, + }) + + return { + calls, + request: async (url) => { + calls.push(url) + if (status !== 200) return respond({ message: 'boom' }, status) + if (url.includes('/artifacts')) { + const artifacts = artifactPages[page] ?? [] + page += 1 + return respond({ artifacts }) + } + if (runResponse === null) return respond({ message: 'Not Found' }, 404) + return respond(runResponse ?? run()) + }, + } +} + +function baseEnv(overrides = {}) { + return { + PREBUILD_RUN_ID: RUN_ID, + ADDON_WORKDIR: 'packages/llm-llamacpp', + PLATFORM: 'Android', + PACKAGE_VERSION: '', + FORCE_NPM_PREBUILD: 'false', + GITHUB_TOKEN: 'ghs-test-token', + REPO, + ...overrides, + } +} + +async function expectFailure(env, api) { + await assert.rejects(() => resolvePrebuildRun({ env, request: api.request }), (error) => { + assert.ok(error.message, 'the failure carries a message') + return true + }) +} + +test('parseRunId accepts only a positive integer run id', () => { + assert.equal(parseRunId('33179656677'), '33179656677') + assert.equal(parseRunId(' 42 '), '42') + + // Every one of these arrives from a dispatch input and would otherwise be + // pasted into an API path. + for (const hostile of [ + '', + '0', + '-1', + '12.5', + '4242abc', + '4242/../../secrets', + '4242?per_page=1', + 'latest', + '../../etc/passwd', + null, + undefined, + ]) { + assert.equal(parseRunId(hostile), null, `must reject ${JSON.stringify(hostile)}`) + } +}) + +test('the artifact name is derived the way reusable-prebuilds.yml uploads it', () => { + assert.equal(mergedArtifactName('packages/llm-llamacpp'), 'prebuilds-llm-llamacpp') + assert.equal(mergedArtifactName('packages/vla/'), 'prebuilds-vla') + assert.equal(mergedArtifactName('./packages/ocr-ggml'), 'prebuilds-ocr-ggml') + assert.equal(mergedArtifactName(''), null) + assert.equal(mergedArtifactName('..'), null) + + // The legacy bare name stays as a fallback so an older run id still resolves. + assert.deepEqual(candidateArtifactNames('packages/tts-ggml'), [ + 'prebuilds-tts-ggml', + 'prebuilds', + ]) +}) + +test('platform maps to the prebuild dir the mobile build actually consumes', () => { + assert.deepEqual(platformPrebuildDirs('Android'), ['android-arm64']) + assert.deepEqual(platformPrebuildDirs('iOS'), ['ios-arm64']) + // Case matters — the workflows pass the matrix value verbatim. + assert.deepEqual(platformPrebuildDirs('android'), []) + assert.deepEqual(platformPrebuildDirs(''), []) +}) + +test('a live artifact wins over an expired one, in candidate order', () => { + const selected = selectArtifact( + [ + { name: 'prebuilds-llm-llamacpp', expired: true, id: 1 }, + { name: 'prebuilds', expired: false, id: 2 }, + ], + candidateArtifactNames('packages/llm-llamacpp'), + ) + // The preferred name is expired, so the legacy name is used rather than + // failing on retention while a usable bundle sits right there. + assert.deepEqual(selected, { name: 'prebuilds', id: 2, expired: false }) + + assert.deepEqual( + selectArtifact( + [{ name: 'prebuilds-llm-llamacpp', expired: false, id: 7 }, { name: 'prebuilds', id: 8 }], + candidateArtifactNames('packages/llm-llamacpp'), + ), + { name: 'prebuilds-llm-llamacpp', id: 7, expired: false }, + ) +}) + +test('all-expired is distinguishable from never-existed', () => { + const expired = selectArtifact( + [{ name: 'prebuilds-llm-llamacpp', expired: true, id: 1 }], + candidateArtifactNames('packages/llm-llamacpp'), + ) + assert.equal(expired.expired, true) + + const absent = selectArtifact( + [{ name: 'perf-report-android', expired: false, id: 9 }], + candidateArtifactNames('packages/llm-llamacpp'), + ) + assert.equal(absent, null) +}) + +test('conflictingSource names whichever competing pin was set', () => { + assert.equal(conflictingSource({ packageVersion: '', forceNpmPrebuild: 'false' }), null) + assert.match( + conflictingSource({ packageVersion: '@qvac/llm-llamacpp@0.47.0' }), + /@qvac\/llm-llamacpp@0\.47\.0/, + ) + assert.match( + conflictingSource({ packageVersion: '', forceNpmPrebuild: 'true' }), + /force-npm-prebuild=true/, + ) +}) + +test('provenance names the run, the artifact and the head SHA', () => { + const line = formatProvenance(run(), 'prebuilds-llm-llamacpp') + assert.match(line, new RegExp(RUN_ID)) + assert.match(line, new RegExp(HEAD_SHA)) + assert.match(line, /prebuilds-llm-llamacpp/) + assert.match(line, /On PR Trigger \(LLM\)/) +}) + +test('a non-success source run warns but is still usable', () => { + // The prebuild job uploads before desktop tests and lint, so a red run + // regularly holds good binaries. Refusing it would block the main use case. + assert.equal(conclusionWarning(run()), null) + assert.match(conclusionWarning(run({ conclusion: 'failure' })), /concluded 'failure'/) + assert.equal(conclusionWarning(run({ conclusion: null })), null) +}) + +// A fork PR's on-pr run sits in the base repo's run list while its +// head_repository is the fork (verified live: on-pr-nx.yml carries runs with +// head_repository 'ogad-tether/qvac'). This warns rather than refusing — the +// repo is fork-first, and a fork bundle only exists once the merge/release team +// approved `fork-ci` on that run. +test('a fork-built source run is surfaced, not silently accepted', () => { + assert.equal(sourceRepositoryWarning(run(), REPO), null) + + const forkWarning = sourceRepositoryWarning( + run({ head_repository: { full_name: 'ogad-tether/qvac' } }), + REPO, + ) + assert.match(forkWarning, /FORK 'ogad-tether\/qvac'/) + // The warning must not read as an error: this is the documented workflow. + assert.match(forkWarning, /normal for a fork PR/) + assert.match(forkWarning, /fork-ci/) + + // A payload with no head_repository must not be read as "same repo". + assert.match(sourceRepositoryWarning(run({ head_repository: null }), REPO), /cannot be confirmed/) + assert.match(sourceRepositoryWarning({ id: 1 }, REPO), /cannot be confirmed/) +}) + +test('a fork-built run still resolves, with the warning attached', async () => { + const api = fakeApi({ + runResponse: run({ head_repository: { full_name: 'ogad-tether/qvac' } }), + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }) + + const resolved = await resolvePrebuildRun({ env: baseEnv(), request: api.request }) + assert.equal(resolved.artifactName, 'prebuilds-llm-llamacpp') + assert.ok( + resolved.warnings.some((warning) => warning.includes("FORK 'ogad-tether/qvac'")), + `the fork must be called out, got ${JSON.stringify(resolved.warnings)}`, + ) +}) + +test('a fork run that ALSO concluded red carries both warnings', () => { + // The two warnings are independent; neither may swallow the other. + return resolvePrebuildRun({ + env: baseEnv(), + request: fakeApi({ + runResponse: run({ + head_repository: { full_name: 'ogad-tether/qvac' }, + conclusion: 'failure', + }), + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }).request, + }).then((resolved) => { + assert.equal(resolved.warnings.length, 2, JSON.stringify(resolved.warnings)) + }) +}) + +test('the provenance line names the repository the code came from', () => { + assert.match(formatProvenance(run(), 'prebuilds-llm-llamacpp'), /\(tetherto\/qvac\)/) +}) + +test('resolves a real on-pr run to its merged bundle', async () => { + const api = fakeApi({ + artifactPages: [ + [ + { name: 'perf-report-android', expired: false, id: 1 }, + { name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }, + ], + ], + }) + + const resolved = await resolvePrebuildRun({ env: baseEnv(), request: api.request }) + + assert.equal(resolved.artifactName, 'prebuilds-llm-llamacpp') + assert.equal(resolved.runId, RUN_ID) + assert.equal(resolved.headSha, HEAD_SHA) + assert.deepEqual(resolved.expectedDirs, ['android-arm64']) + assert.match(resolved.provenance, new RegExp(`run ${RUN_ID}`)) + assert.deepEqual(resolved.warnings, []) + assert.ok( + api.calls.some((url) => url === `https://api.github.com/repos/${REPO}/actions/runs/${RUN_ID}`), + `the run itself is fetched:\n${api.calls.join('\n')}`, + ) +}) + +test('a malformed run id fails before any request is made', async () => { + const api = fakeApi({}) + const env = baseEnv({ PREBUILD_RUN_ID: 'my-branch' }) + + await assert.rejects( + () => resolvePrebuildRun({ env, request: api.request }), + /must be a numeric GitHub Actions run id/, + ) + assert.deepEqual(api.calls, [], 'nothing is requested for a rejected run id') +}) + +test('prebuild_run_id together with a pinned package is an error, not a precedence puzzle', async () => { + const api = fakeApi({}) + + await assert.rejects( + () => + resolvePrebuildRun({ + env: baseEnv({ PACKAGE_VERSION: '@tetherto/llm-llamacpp-mono@0.47.0-tmp.runid-1' }), + request: api.request, + }), + /mutually exclusive/, + ) + await assert.rejects( + () => + resolvePrebuildRun({ + env: baseEnv({ FORCE_NPM_PREBUILD: 'true' }), + request: api.request, + }), + /mutually exclusive/, + ) + assert.deepEqual(api.calls, [], 'a conflicting request never reaches the API') +}) + +test('a missing token fails closed and names the permission', async () => { + const api = fakeApi({}) + + await assert.rejects( + () => resolvePrebuildRun({ env: baseEnv({ GITHUB_TOKEN: '' }), request: api.request }), + /actions:read/, + ) + assert.deepEqual(api.calls, []) +}) + +test('an unknown platform fails before any request', async () => { + const api = fakeApi({}) + await assert.rejects( + () => resolvePrebuildRun({ env: baseEnv({ PLATFORM: 'Windows' }), request: api.request }), + /expected Android or iOS/, + ) + assert.deepEqual(api.calls, []) +}) + +test('a run id that does not exist points the reader at the run URL', async () => { + const api = fakeApi({ runResponse: null }) + + await assert.rejects( + () => resolvePrebuildRun({ env: baseEnv(), request: api.request }), + (error) => { + assert.match(error.message, new RegExp(`Run ${RUN_ID} does not exist`)) + assert.ok( + error.hints.some((hint) => hint.includes(`/actions/runs/${RUN_ID}`)), + 'the hint links the run', + ) + return true + }, + ) +}) + +test('403 on the run lookup blames the missing actions:read permission', async () => { + const api = fakeApi({ status: 403 }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.ok( + error.hints.some((hint) => hint.includes('actions: read')), + `hints should name the permission, got ${JSON.stringify(error.hints)}`, + ) + return true + }) +}) + +test('an expired artifact fails closed and says so', async () => { + // The whole failure mode this ticket exists to stop: retention lapses, and + // the run must NOT slide back to @latest. + const api = fakeApi({ + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: true, id: 2 }]], + }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.match(error.message, /has expired/) + assert.ok( + error.hints.some((hint) => hint.includes('retention')), + 'the hint explains retention', + ) + assert.ok( + error.hints.some((hint) => hint.includes('new run id')), + 'the hint says what to do next', + ) + return true + }) +}) + +// A real nx run carries 65+ artifacts, so listing them all buries the answer. +test('the not-found hint names the addons a run DID build, not every artifact', () => { + const artifacts = [ + { name: 'prebuilds-llm-llamacpp' }, + { name: 'prebuilds-ocr-ggml' }, + { name: 'prebuilds-llm-llamacpp' }, // duplicated across legs in real runs + { name: 'prebuild-llm-llamacpp-win32-x64' }, // per-matrix leg, not a bundle + { name: 'prebuilds-cache-pr-4519-8b692edcec92ec00' }, // reuse marker + { name: 'coverage-report' }, + { name: 'ocr-ggml-perf-report-linux-x64' }, + ] + + const hints = describeAvailableBundles(artifacts) + assert.match(hints[0], /prebuilds for: llm-llamacpp, ocr-ggml\./) + assert.ok(!hints.join(' ').includes('cache-pr'), hints.join(' ')) + assert.ok(!hints.join(' ').includes('coverage-report'), hints.join(' ')) + assert.ok(!hints.join(' ').includes('win32-x64'), 'a per-matrix leg is not a bundle') + assert.ok( + hints.join('\n').length < 400, + `the hint must stay readable, got ${hints.join('\n').length} chars`, + ) +}) + +test('a long bundle list is capped rather than dumped', () => { + const artifacts = Array.from({ length: 30 }, (_, index) => ({ name: `prebuilds-addon-${index}` })) + const hints = describeAvailableBundles(artifacts, 5) + assert.match(hints[0], /\+25 more/) +}) + +test('a run with artifacts but no bundle says so plainly', () => { + const hints = describeAvailableBundles([{ name: 'coverage-report' }, { name: 'logs' }]) + assert.match(hints[0], /2 artifact\(s\) but no prebuilds bundle/) + assert.match(hints[1], /on-pr-\.yml/) +}) + +test('a run with no artifacts at all says that instead of an empty list', () => { + assert.deepEqual(describeAvailableBundles([]), ['That run published no artifacts at all.']) +}) + +test('a run with no prebuilds artifact lists what it does have', async () => { + const api = fakeApi({ + runResponse: run({ name: 'Docs website health check', path: '.github/workflows/docs.yml' }), + artifactPages: [[{ name: 'link-report', expired: false, id: 5 }]], + }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.match(error.message, /has no 'prebuilds-llm-llamacpp' or 'prebuilds' artifact/) + assert.ok( + error.hints.some((hint) => hint.includes('no prebuilds bundle')), + `the hint must say the run builds no bundles, got ${JSON.stringify(error.hints)}`, + ) + assert.ok( + error.hints.some((hint) => hint.includes('Docs website health check')), + 'the hint names the run the reader actually picked', + ) + return true + }) +}) + +test('a run that published nothing says that instead of listing an empty set', async () => { + const api = fakeApi({ artifactPages: [[]] }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.ok(error.hints.some((hint) => hint.includes('no artifacts at all'))) + return true + }) +}) + +// `gh run list --limit 1` returns the newest run, which on an active branch is +// usually still building. "Does not build prebuilds" is wrong there. +test('a still-running source run says so instead of "builds no prebuilds"', async () => { + for (const status of ['in_progress', 'queued', 'waiting']) { + const api = fakeApi({ + runResponse: run({ status, conclusion: null }), + artifactPages: [[{ name: 'coverage-report', expired: false, id: 1 }]], + }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.match(error.message, new RegExp(`still '${status}'`)) + assert.ok( + error.hints.some((hint) => hint.includes('dispatch again with the same run id')), + `the hint must say to wait and retry, got ${JSON.stringify(error.hints)}`, + ) + assert.ok( + !error.hints.some((hint) => hint.includes('does not build prebuilds')), + 'must not blame the workflow for an unfinished run', + ) + return true + }) + } +}) + +test('a completed run with no bundle still blames the workflow, not the clock', async () => { + const api = fakeApi({ + runResponse: run({ status: 'completed', conclusion: 'success' }), + artifactPages: [[{ name: 'coverage-report', expired: false, id: 1 }]], + }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request: api.request }), (error) => { + assert.match(error.message, /has no 'prebuilds-llm-llamacpp' or 'prebuilds' artifact/) + return true + }) +}) + +// An empty body and an empty run are different facts. +test('a failed artifact listing is not reported as an empty run', async () => { + const calls = [] + const request = async (url) => { + calls.push(url) + if (url.includes('/artifacts')) { + return { ok: true, status: 200, json: async () => null } + } + return { ok: true, status: 200, json: async () => run() } + } + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request }), (error) => { + assert.match(error.message, /Could not list run .* artifacts/) + assert.ok(error.hints.some((hint) => hint.includes('actions: read'))) + return true + }) +}) + +test('a truncated artifact listing fails loudly rather than guessing', async () => { + // Every page full to the page bound: the bundle may sit just past it, so + // "no prebuilds here" would be a guess presented as a fact. + const fullPage = Array.from({ length: 100 }, (_, index) => ({ + name: `filler-${index}`, + expired: false, + id: index, + })) + const request = async (url) => ({ + ok: true, + status: 200, + json: async () => (url.includes('/artifacts') ? { artifacts: fullPage } : run()), + }) + + await assert.rejects(() => resolvePrebuildRun({ env: baseEnv(), request }), (error) => { + assert.match(error.message, /listing was truncated/) + return true + }) +}) + +test('created_at breaks the tie when it disagrees with id order', async () => { + const api = fakeApi({ + artifactPages: [ + [ + { name: 'prebuilds-llm-llamacpp', expired: false, id: 999, created_at: '2026-01-01T00:00:00Z' }, + { name: 'prebuilds-llm-llamacpp', expired: false, id: 111, created_at: '2026-06-01T00:00:00Z' }, + ], + ], + }) + const resolved = await resolvePrebuildRun({ env: baseEnv(), request: api.request }) + assert.equal(resolved.artifactId, 111, 'the later created_at wins over the larger id') +}) + +test('the resolved artifact ID is surfaced so the download cannot pick another row', async () => { + // A re-run leaves the earlier attempt's artifacts under the same run id, so a + // run can hold two live rows with the same name. + const api = fakeApi({ + artifactPages: [ + [ + { name: 'prebuilds-llm-llamacpp', expired: false, id: 111 }, + { name: 'prebuilds-llm-llamacpp', expired: false, id: 222 }, + ], + ], + }) + + const resolved = await resolvePrebuildRun({ env: baseEnv(), request: api.request }) + // The NEWEST row, not the first listed: the API lists id-ascending, so + // first-match would pin the pre-re-run binary while the provenance line + // printed the same head_sha either way. + assert.equal(resolved.artifactId, 222, 'the newest matching row wins') +}) + +test('the artifact listing is paginated, so the bundle is found past page 1', async () => { + // A real LLM run carries well over 100 artifacts (per-matrix prebuilds, perf + // reports, device logs), so a single-page lookup would miss the bundle. + const firstPage = Array.from({ length: 100 }, (_, index) => ({ + name: `perf-report-${index}`, + expired: false, + id: index, + })) + const api = fakeApi({ + artifactPages: [firstPage, [{ name: 'prebuilds-llm-llamacpp', expired: false, id: 999 }]], + }) + + const resolved = await resolvePrebuildRun({ env: baseEnv(), request: api.request }) + assert.equal(resolved.artifactName, 'prebuilds-llm-llamacpp') + assert.equal( + api.calls.filter((url) => url.includes('/artifacts')).length, + 2, + 'the second page is requested', + ) +}) + +test('GITHUB_API_URL is honoured so GHES is not hardcoded to api.github.com', async () => { + const api = fakeApi({ + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }) + + await resolvePrebuildRun({ + env: baseEnv({ GITHUB_API_URL: 'https://ghe.example/api/v3/' }), + request: api.request, + }) + + assert.ok( + api.calls.every((url) => url.startsWith('https://ghe.example/api/v3/repos/')), + `all calls should hit the configured API host:\n${api.calls.join('\n')}`, + ) +}) + +test('iOS resolves the ios-arm64 dir', async () => { + const api = fakeApi({ + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }) + const resolved = await resolvePrebuildRun({ + env: baseEnv({ PLATFORM: 'iOS' }), + request: api.request, + }) + assert.deepEqual(resolved.expectedDirs, ['ios-arm64']) +}) + +test('main writes the step outputs the action consumes', async () => { + const directory = mkdtempSync(join(tmpdir(), 'qvac-prebuild-run-')) + const outputFile = join(directory, 'github-output') + const logs = [] + const api = fakeApi({ + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }) + + try { + const code = await main({ + env: baseEnv({ GITHUB_OUTPUT: outputFile }), + request: api.request, + log: { log: (line) => logs.push(line) }, + }) + + assert.equal(code, 0, logs.join('\n')) + const written = readFileSync(outputFile, 'utf8') + // Exactly the names action.yml reads back. + assert.match(written, /^artifact_name=prebuilds-llm-llamacpp$/m) + assert.match(written, /^artifact_id=2$/m) + assert.match(written, new RegExp(`^source_run_id=${RUN_ID}$`, 'm')) + assert.match(written, new RegExp(`^head_sha=${HEAD_SHA}$`, 'm')) + assert.match(written, /^expected_dirs=android-arm64$/m) + assert.ok( + logs.some((line) => line.includes(`run ${RUN_ID}`) && line.includes(HEAD_SHA)), + `the provenance line is printed:\n${logs.join('\n')}`, + ) + } finally { + rmSync(directory, { recursive: true, force: true }) + } +}) + +test('main exits non-zero and emits ::error:: on failure, writing no outputs', async () => { + const directory = mkdtempSync(join(tmpdir(), 'qvac-prebuild-run-')) + const outputFile = join(directory, 'github-output') + const logs = [] + + try { + const code = await main({ + env: baseEnv({ PREBUILD_RUN_ID: 'nope', GITHUB_OUTPUT: outputFile }), + request: fakeApi({}).request, + log: { log: (line) => logs.push(line) }, + }) + + assert.equal(code, 1) + assert.ok( + logs.some((line) => line.startsWith('::error::')), + `the failure is annotated for the run summary:\n${logs.join('\n')}`, + ) + // No outputs means the download step gets an empty artifact name and + // cannot silently pull something else. + assert.throws(() => readFileSync(outputFile, 'utf8')) + } finally { + rmSync(directory, { recursive: true, force: true }) + } +}) + +test('main warns about a red source run without refusing it', async () => { + const logs = [] + const api = fakeApi({ + runResponse: run({ conclusion: 'failure' }), + artifactPages: [[{ name: 'prebuilds-llm-llamacpp', expired: false, id: 2 }]], + }) + + const code = await main({ + env: baseEnv(), + request: api.request, + log: { log: (line) => logs.push(line) }, + }) + + assert.equal(code, 0) + assert.ok(logs.some((line) => line.startsWith('::warning::')), logs.join('\n')) +}) + +// Keeps expectFailure referenced: a resolver that suddenly succeeds for an +// empty env would mean every guard above was bypassed. +test('an empty environment resolves nothing', async () => { + await expectFailure({}, fakeApi({})) +}) diff --git a/.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js b/.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js index b3d46df601..41d6affa76 100644 --- a/.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js +++ b/.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js @@ -90,11 +90,47 @@ exports.config = { // runs on crash paths where the WDIO command queue may have a pending // command stuck behind a long timeout (e.g. waitForDisplayed 60s on an // element that will never appear). Raw HTTP bypasses the queue. + // iOS reads bare_console.log from the app container. Android cannot: the app + // writes it to its private data dir, which adb cannot read and run-as + // refuses on a release-signed APK. Android's app output is in logcat under + // the `bare` tag instead, so this is not a gap. The one candidate below is + // the world-readable path an app could be changed to write to. + global.bareLogCandidates = function (isAndroid, bundleId) { + if (!isAndroid) return ['@' + bundleId + ':documents/bare_console.log']; + return ['/sdcard/Android/data/' + bundleId + '/files/bare_console.log']; + }; + global.flushBareLog = async function (reason) { if ('__ENABLE_FLUSH_BARE_LOG__' !== 'true') return; - try { + var isAndroid = (capabilities.platformName || '').toLowerCase() === 'android'; + var candidates = global.bareLogCandidates(isAndroid, BUNDLE_ID); + var lastError = null; + for (var ci = 0; ci < candidates.length; ci++) { + try { + await global.pullBareLog(reason, candidates[ci]); + return; + } catch (e) { + lastError = e; + } + } + if (isAndroid) { + console.log( + '[bare-log] ' + reason + ': no bare_console.log on Android; app-side output is in ' + + 'logcat_full.txt under the `bare` tag. Last error: ' + + (lastError ? lastError.message : 'none') + ); + return; + } + console.log( + '[bare-log] ' + reason + ' flush failed: ' + + (lastError ? lastError.message : 'no candidate path') + ); + }; + + global.pullBareLog = async function (reason, devicePath) { + { var http = require('http'); - var body = JSON.stringify({ path: '@' + BUNDLE_ID + ':documents/bare_console.log' }); + var body = JSON.stringify({ path: devicePath }); var b64 = await new Promise(function (resolve, reject) { var req = http.request({ hostname: '127.0.0.1', port: 4723, @@ -112,12 +148,19 @@ exports.config = { req.write(body); req.end(); }); + // Appium returns a base64 string on success, an error object on failure. + // Passing the object to Buffer.from threw a type error that replaced + // Appium's real reason. + if (typeof b64 !== 'string') { + var why = (b64 && (b64.message || b64.error)) || JSON.stringify(b64); + throw new Error('pull_file returned no base64 payload — ' + why); + } var text = Buffer.from(b64, 'base64').toString(); var logDir = process.env.DEVICEFARM_LOG_DIR || '.'; require('fs').writeFileSync(logDir + '/bare_console.log', text); - console.log('[bare-log] ' + reason + ' flush ok (' + text.length + ' bytes)'); - } catch (e) { - console.log('[bare-log] ' + reason + ' flush failed: ' + e.message); + console.log( + '[bare-log] ' + reason + ' flush ok (' + text.length + ' bytes) from ' + devicePath + ); } }; diff --git a/.github/scripts/test/mobile-prebuild-registry.test.mjs b/.github/scripts/test/mobile-prebuild-registry.test.mjs index 891c1f0bad..0f50044b62 100644 --- a/.github/scripts/test/mobile-prebuild-registry.test.mjs +++ b/.github/scripts/test/mobile-prebuild-registry.test.mjs @@ -649,3 +649,434 @@ test('every mobile dispatch input advertises the -mono GPR name', () => { `GPR dev builds are published as @tetherto/-mono:\n${offenders.join('\n')}`, ) }) + +// ── prebuild_run_id ───────────────────────────────────────────────────────── +// Resolution logic is unit-tested in +// .github/actions/run-mobile-integration-tests/setup/test/resolve-prebuild-run.test.mjs. +// Asserted HERE is the wiring: the route is only as good as the weakest workflow +// that forgot a piece of it, and a source not gated on the run id would shadow +// it silently. + +// Without the gate a run id could resolve and then be overwritten, or fall +// through to `npm pack @qvac/@latest` and go green against the release. +test('every other prebuild source is gated off when a run id is set', () => { + const source = read(ACTION) + const gated = [ + 'Download Android prebuilds (from artifacts)', + 'Download iOS prebuilds (from artifacts)', + 'Download merged prebuilds artifact (fallback when per-matrix artifacts absent)', + STEP, + ] + + for (const stepName of gated) { + const index = source.indexOf(`name: ${stepName}`) + assert.notEqual(index, -1, `step "${stepName}" exists`) + const condition = source.slice(index).match(/^\s*if:\s*(.+)$/m)?.[1] ?? '' + assert.ok( + condition.includes("inputs.prebuild-run-id == ''"), + `"${stepName}" must be skipped while prebuild-run-id is set, got: ${condition}`, + ) + } +}) + +test('the run-id steps are the first prebuild source and fail closed', () => { + const source = read(ACTION) + + const resolveIndex = source.indexOf('name: Resolve prebuilds from a run id') + const downloadIndex = source.indexOf("name: Download prebuilds (from the resolved run)") + const verifyIndex = source.indexOf("name: Verify the resolved run's prebuilds cover this platform") + const androidIndex = source.indexOf('name: Download Android prebuilds (from artifacts)') + + assert.ok(resolveIndex !== -1 && downloadIndex !== -1 && verifyIndex !== -1) + // Resolution happens before anything is downloaded, so a bad run id costs + // nothing, and before the artifact-first steps so it cannot be shadowed. + assert.ok( + resolveIndex < downloadIndex && downloadIndex < verifyIndex && verifyIndex < androidIndex, + 'order must be resolve -> download -> verify -> (gated) artifact-first steps', + ) + + // continue-on-error on the cross-run download would turn a missing artifact + // back into an @latest run. The other artifact downloads tolerate absence by + // design; this one must not. + const downloadStep = source.slice(downloadIndex, androidIndex) + assert.doesNotMatch( + downloadStep, + /continue-on-error/, + 'a named run id is an explicit instruction — a failed download must fail the run', + ) + assert.match(downloadStep, /run-id: \$\{\{ steps\.prebuild_run\.outputs\.source_run_id \}\}/) + // By ID, not name: a re-run leaves the earlier attempt's artifacts under the + // same run id, so a run can hold two live `prebuilds-` rows. Selecting + // by name would let this step extract a different one than the resolver + // validated and printed provenance for. + assert.match(downloadStep, /artifact-ids: \$\{\{ steps\.prebuild_run\.outputs\.artifact_id \}\}/) + assert.doesNotMatch( + downloadStep, + /^\s+name: /m, + 'selecting by name would reintroduce the ambiguity artifact_id removes', + ) +}) + +// Every mobile workflow sparse-checks out only +// .github/actions/run-mobile-integration-tests, so a move would leave the step +// calling a file that is not on disk — visible only at dispatch time. +test('the resolver is reachable from the callers own sparse checkout', () => { + const resolver = '.github/actions/run-mobile-integration-tests/setup/resolve-prebuild-run.mjs' + assert.ok(existsSync(join(root, resolver)), `${resolver} must exist`) + assert.match( + read(ACTION), + /node "\$ACTION_PATH\/resolve-prebuild-run\.mjs"/, + 'the step must invoke the resolver through github.action_path', + ) + + const workflows = spawnSync( + 'git', + ['ls-files', '.github/workflows/integration-mobile-test-*.yml'], + { cwd: root, encoding: 'utf8' }, + ).stdout.trim().split('\n').filter(Boolean) + + for (const relativePath of workflows) { + const source = read(relativePath) + if (!source.includes('prebuild-run-id:')) continue + assert.match( + source, + /sparse-checkout: \|\n\s+\.github\/actions\/run-mobile-integration-tests/, + `${relativePath} must sparse-check out the directory holding the resolver`, + ) + } +}) + +// Two addons deliberately have no run-id route. Pinning them means a third +// exclusion has to be a decision, not a silent omission. +const RUN_ID_EXEMPT = { + // Native code comes transitively from bare-ffmpeg, so setup skips every + // prebuild step (skip-prebuilds: 'true') and has nothing to install. + 'decoder-audio': /skip-prebuilds:\s*'true'/, + // Compiles its own prebuilds in prebuild-android / prebuild-ios jobs in the + // SAME run from the dispatched ref, so a dispatch already tests the branch's + // native code and the gap this route closes does not exist. + 'inference-addon-cpp': /^\s{2}prebuild-android:$/m, +} + +test('every mobile dispatch offers the run-id route, or is a pinned exemption', () => { + const workflows = spawnSync( + 'git', + ['ls-files', '.github/workflows/integration-mobile-test-*.yml'], + { cwd: root, encoding: 'utf8' }, + ).stdout.trim().split('\n').filter(Boolean) + + assert.ok(workflows.length >= 13, `found ${workflows.length} mobile workflows`) + + const missing = [] + for (const relativePath of workflows) { + const slug = relativePath.replace(/.*integration-mobile-test-|\.yml$/g, '') + const source = read(relativePath) + + if (slug in RUN_ID_EXEMPT) { + assert.match( + source, + RUN_ID_EXEMPT[slug], + `${slug} is exempt from the run-id route for a reason that no longer holds`, + ) + assert.ok( + !source.includes('prebuild_run_id'), + `${slug} is listed as exempt but now exposes prebuild_run_id — drop the exemption`, + ) + continue + } + + const problems = [] + if (!/^ prebuild_run_id:$/m.test(source)) problems.push('no prebuild_run_id input') + if (!source.includes('prebuild-run-id: ${{ inputs.prebuild_run_id }}')) { + problems.push('input not wired into setup') + } + // Without actions: read the lookup 403s and the download fails — after the + // dispatcher has already waited for a build. + if (!/^ actions: read$/m.test(source)) problems.push('no actions: read') + if (problems.length > 0) missing.push(`${slug}: ${problems.join(', ')}`) + } + + assert.deepEqual(missing, [], `incomplete prebuild_run_id wiring:\n${missing.join('\n')}`) +}) + +// The dispatch inputs are what people copy from, and the action rejects the +// combination, so the descriptions must say so. +test('the run-id input documents its precedence and the mutual exclusion', () => { + const workflows = spawnSync( + 'git', + ['ls-files', '.github/workflows/integration-mobile-test-*.yml'], + { cwd: root, encoding: 'utf8' }, + ).stdout.trim().split('\n').filter(Boolean) + + for (const relativePath of workflows) { + const source = read(relativePath) + const match = source.match(/^ prebuild_run_id:\n description: "([^"]*)"/m) + if (!match) continue + + const description = match[1] + assert.match(description, /Mutually exclusive/, `${relativePath} must state the exclusion`) + assert.match(description, /precedence over/, `${relativePath} must state the precedence`) + assert.match( + description, + /fails the run/, + `${relativePath} must say a bad run id fails rather than falling back`, + ) + } +}) + +test('the documented route is the one docs/ci/MOBILE-ON-DEMAND.md tells people to use', () => { + const docs = read('docs/ci/MOBILE-ON-DEMAND.md') + assert.match(docs, /prebuild_run_id/, 'the docs must document the input') + // The GPR pin stays documented for the cross-branch / published cases. + assert.match(docs, /@tetherto\/-mono/) +}) + +// The generic "Verify and prepare prebuilds" step only asserts prebuilds/ is +// non-empty, which a bundle missing this platform passes. Run the real shell. +const VERIFY_STEP = "Verify the resolved run's prebuilds cover this platform" +const verifyScript = extractRunBlock(ACTION, VERIFY_STEP) + +assert.ok( + !verifyScript.includes('${{'), + 'the verify step body must stay free of GitHub expressions so it is testable as plain shell', +) + +function runVerify({ dirs = ['android-arm64'], expected = 'android-arm64', platform = 'Android' } = {}) { + const directory = mkdtempSync(join(tmpdir(), 'qvac-prebuild-run-verify-')) + + for (const dir of dirs) { + mkdirSync(join(directory, 'prebuilds', dir), { recursive: true }) + writeFileSync(join(directory, 'prebuilds', dir, 'addon.bare'), 'mock') + } + + const result = spawnSync( + 'bash', + ['--noprofile', '--norc', '-e', '-o', 'pipefail', '-c', verifyScript], + { + cwd: directory, + encoding: 'utf8', + env: { + ...process.env, + EXPECTED_DIRS: expected, + SOURCE_RUN_ID: '33179656677', + SOURCE_HEAD_SHA: 'deadbeefcafe', + SOURCE_ARTIFACT: 'prebuilds-llm-llamacpp', + PLATFORM: platform, + }, + }, + ) + + rmSync(directory, { recursive: true, force: true }) + return { status: result.status, output: `${result.stdout}${result.stderr}` } +} + +test('the resolved bundle passes when it carries this platform, and says where it came from', () => { + const run = runVerify({ dirs: ['android-arm64', 'ios-arm64'] }) + + assert.equal(run.status, 0, run.output) + // The run id and head SHA land next to the file list, so the log shows what + // was installed without cross-referencing an earlier step. + assert.match(run.output, /run 33179656677/) + assert.match(run.output, /deadbeefcafe/) +}) + +test('a bundle missing this platform fails instead of building around a gap', () => { + // The exact shape of a prebuild run whose iOS leg was cancelled. + const run = runVerify({ dirs: ['android-arm64'], expected: 'ios-arm64', platform: 'iOS' }) + + assert.notEqual(run.status, 0, run.output) + assert.match(run.output, /::error::/) + assert.match(run.output, /cannot build for iOS/) + // Naming what IS there is what turns this into a one-look diagnosis. + assert.match(run.output, /android-arm64/) +}) + +test('an empty platform dir counts as missing, not present', () => { + const directory = mkdtempSync(join(tmpdir(), 'qvac-prebuild-run-verify-')) + mkdirSync(join(directory, 'prebuilds/android-arm64'), { recursive: true }) + + const result = spawnSync( + 'bash', + ['--noprofile', '--norc', '-e', '-o', 'pipefail', '-c', verifyScript], + { + cwd: directory, + encoding: 'utf8', + env: { + ...process.env, + EXPECTED_DIRS: 'android-arm64', + SOURCE_RUN_ID: '1', + SOURCE_HEAD_SHA: 'abc', + SOURCE_ARTIFACT: 'prebuilds-llm-llamacpp', + PLATFORM: 'Android', + }, + }, + ) + + rmSync(directory, { recursive: true, force: true }) + assert.notEqual(result.status, 0, `${result.stdout}${result.stderr}`) +}) + +// download-artifact overwrites the files it carries and leaves the rest, so a +// committed prebuilds/ dir or a leftover from an earlier job on the same +// self-hosted runner could survive and be linked into the app. +test('the run-id path clears prebuilds/ so nothing can shadow the resolved run', () => { + const source = read(ACTION) + const resolveStep = source.slice( + source.indexOf('name: Resolve prebuilds from a run id'), + source.indexOf('name: Download prebuilds (from the resolved run)'), + ) + + assert.match( + resolveStep, + /rm -rf "addon\/\$ADDON_WORKDIR\/prebuilds"/, + 'the resolve step must clear prebuilds/ before the artifact is extracted', + ) + // Ordering matters: clearing before a failed resolve would delete the tree + // for a run that is about to be rejected anyway, and `-e` makes the node + // invocation the gate. + assert.ok( + resolveStep.indexOf('resolve-prebuild-run.mjs') < resolveStep.indexOf('rm -rf'), + 'the clear must happen only after resolution succeeds', + ) +}) + +// The repo is fork-first, so a fork-built source run is the normal case and must +// not be refused — what matters is that the dispatcher can see it. Behaviour +// lives in the resolver's unit tests; this pins the surfacing. +test('the resolver reports which repository built the prebuilds', () => { + const resolver = read( + '.github/actions/run-mobile-integration-tests/setup/resolve-prebuild-run.mjs', + ) + + assert.match( + resolver, + /export function sourceRepositoryWarning/, + 'the check must stay a named, separately testable function', + ) + // The provenance line itself must carry the head repository, so the fact is + // present even when no warning fires. + const provenance = resolver.slice( + resolver.indexOf('export function formatProvenance'), + resolver.indexOf('export function conclusionWarning'), + ) + assert.match( + provenance, + /head_repository/, + 'the provenance line must name the repository the binaries were built from', + ) + // Fork-built runs are the documented norm here, so this must not hard-fail. + assert.ok( + !/throw new ResolveError\(`Refusing prebuilds/.test(resolver), + 'a fork-built run must warn, not be refused — this repo is fork-first', + ) +}) + +// audiogen-ggml pins its composite actions to the DEFAULT BRANCH as a +// supply-chain guard, so it runs main's setup action rather than the PR's. An +// older copy has no `prebuild-run-id` input, and GitHub only WARNS on an unknown +// input — observed live: the run logged "Unexpected input(s) 'prebuild-run-id'" +// and then "downloading @qvac/audiogen-ggml@latest from npm", i.e. it silently +// tested the published release. That is the failure this route exists to remove, +// so the workflow must assert the input was honoured. +test('a workflow pinning the composite to the default branch asserts the run id took effect', () => { + const workflows = spawnSync( + 'git', + ['ls-files', '.github/workflows/integration-mobile-test-*.yml'], + { cwd: root, encoding: 'utf8' }, + ).stdout.trim().split('\n').filter(Boolean) + + const offenders = [] + for (const relativePath of workflows) { + const source = read(relativePath) + if (!source.includes('prebuild-run-id:')) continue + + // Does this workflow load the setup composite from the default branch? + const pinned = /ref: \$\{\{ github\.event\.repository\.default_branch \}\}/.test(source) + if (!pinned) continue + + const asserts = + source.includes('steps.setup.outputs.prebuild-source-run-id') && + // always(): when the input is ignored, setup fails first, so a plain + // conditional would skip the step that explains why. + /if: always\(\) && inputs\.prebuild_run_id != ''/.test(source) + if (!asserts) offenders.push(relativePath) + } + + assert.deepEqual( + offenders, + [], + 'a default-branch-pinned workflow silently ignores prebuild-run-id until the action is on main,\n' + + 'so it must assert steps.setup.outputs.prebuild-source-run-id matches the request:\n' + + offenders.join('\n'), + ) +}) + +// The assertion above is only possible because the composite exposes what it +// actually used. +test('the setup action exposes the run id it installed from', () => { + const action = read(ACTION) + assert.match(action, /^outputs:$/m, 'the action must declare outputs') + assert.match( + action, + /prebuild-source-run-id:[\s\S]*?value: \$\{\{ steps\.prebuild_run\.outputs\.source_run_id \}\}/, + 'prebuild-source-run-id must surface the resolver output', + ) +}) + +// Appium's pull_file returns `value` as a base64 string on success and as an +// error object on failure. Handing the object to Buffer.from threw "The first +// argument must be of type string...", which replaced Appium's real reason — +// observed on every Android Device Farm run, pass or fail, while iOS logged +// "flush ok". The app-side log is the only place a failing runner says why it +// failed, so masking that error makes every Android failure untriageable. +test('the bare-log flush reports Appium\'s real error, not a type error', () => { + const template = read( + '.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js', + ) + const flush = template.slice( + template.indexOf('global.flushBareLog'), + template.indexOf('global.isAndroid'), + ) + + assert.match( + flush, + /typeof b64 !== 'string'/, + 'the payload must be type-checked before Buffer.from', + ) + assert.match( + flush, + /pull_file returned no base64 payload/, + 'the thrown message must name the real failure', + ) + // The guard has to come first, or Buffer.from still throws the type error. + assert.ok( + flush.indexOf("typeof b64 !== 'string'") < flush.indexOf("Buffer.from(b64"), + 'the type check must precede the Buffer.from it protects', + ) +}) + +// iOS reads the app-side log fine. Android cannot with the Device Farm artifact: +// adb hits "Permission denied" on the app's private data dir and run-as is +// refused with "package not debuggable" on a release-signed APK — both observed +// on real runs. So Android tries only the world-readable external path and +// otherwise states plainly that the log is unavailable, rather than burning +// several doomed pulls per run and reporting a confusing error. +test('the bare-log pull is platform-appropriate and explains the Android gap', () => { + const template = read( + '.github/actions/run-mobile-integration-tests/upload-to-devicefarm/wdio.template.js', + ) + + assert.match(template, /global\.bareLogCandidates = function \(isAndroid, bundleId\)/) + // iOS keeps the container form that works. + assert.match(template, /return \['@' \+ bundleId \+ ':documents\/bare_console\.log'\]/) + // Android: exactly one candidate, the adb-readable external path. + assert.match(template, /return \['\/sdcard\/Android\/data\/' \+ bundleId \+ '\/files\/bare_console\.log'\]/) + // No run-as: it cannot work on a release-signed APK. + assert.doesNotMatch(template, /command: 'run-as'/) + // The Android branch must say why, and point at where the output actually is: + // the bare runtime logs to logcat, so logcat_full.txt carries the reason. + assert.match(template, /no bare_console\.log on Android/) + // Match on facts, not on how the comment happens to wrap. + assert.match(template, /release-signed APK/) + assert.match(template, /logcat_full\.txt under the `bare` tag/) +}) diff --git a/.github/workflows/integration-mobile-test-asr-ggml.yml b/.github/workflows/integration-mobile-test-asr-ggml.yml index 7d50ba6c8e..dfaab7ed7a 100644 --- a/.github/workflows/integration-mobile-test-asr-ggml.yml +++ b/.github/workflows/integration-mobile-test-asr-ggml.yml @@ -88,7 +88,12 @@ on: required: false default: "" package_spec: - description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/asr-ggml@... for npm or @tetherto/asr-ggml-mono@... for GPR to force-install a specific build." + description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/asr-ggml@... for npm or @tetherto/asr-ggml-mono@... for GPR to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-asr-ggml run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package_spec`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package_spec`." type: string required: false default: "" @@ -186,6 +191,8 @@ jobs: timeout-minutes: ${{ !inputs.run_rtf_benchmarks && 210 || 180 }} permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -322,6 +329,9 @@ jobs: # equivalent package_spec input. Pin either path explicitly. package-version: ${{ inputs.prebuild_package || inputs.package_spec }} force-npm-prebuild: ${{ (inputs.prebuild_package != '' || inputs.package_spec != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # The release environment authorizes GitHub OIDC to assume the scoped AWS # role (no long-lived keys); us-west-2 so the presigned parakeet URLs diff --git a/.github/workflows/integration-mobile-test-audiogen-ggml.yml b/.github/workflows/integration-mobile-test-audiogen-ggml.yml index 0be4dff111..3e9d44bba6 100644 --- a/.github/workflows/integration-mobile-test-audiogen-ggml.yml +++ b/.github/workflows/integration-mobile-test-audiogen-ggml.yml @@ -81,7 +81,12 @@ on: required: false default: "" package_spec: - description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/audiogen-ggml@... for npm or @tetherto/audiogen-ggml-mono@... for GPR to force-install a specific build." + description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/audiogen-ggml@... for npm or @tetherto/audiogen-ggml-mono@... for GPR to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-audiogen-ggml run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package_spec`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package_spec`." type: string required: false default: "" @@ -185,6 +190,8 @@ jobs: timeout-minutes: 120 permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -234,6 +241,7 @@ jobs: fetch-depth: 0 - name: Setup mobile test environment + id: setup uses: ./trusted-actions/.github/actions/run-mobile-integration-tests/setup with: platform: ${{ matrix.platform }} @@ -244,6 +252,40 @@ jobs: # workflow_call passes prebuild_package; workflow_dispatch passes package_spec. package-version: ${{ inputs.prebuild_package || inputs.package_spec }} force-npm-prebuild: ${{ (inputs.prebuild_package || inputs.package_spec) != '' && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} + + # This addon pins the composite actions to the DEFAULT BRANCH (the + # supply-chain guard above), so it runs whatever version of the setup + # action is on main — not this ref's. An older copy has no + # `prebuild-run-id` input, and GitHub only WARNS on an unknown input, so + # the run would quietly install @qvac/audiogen-ggml@latest instead of the + # binaries you asked for. Turn that into a failure. + # always(): when the input is ignored the setup step fails first with a + # message that says nothing about the cause. Run regardless so it is stated. + - name: Assert the requested prebuild run id was honoured + if: always() && inputs.prebuild_run_id != '' + env: + REQUESTED: ${{ inputs.prebuild_run_id }} + RESOLVED: ${{ steps.setup.outputs.prebuild-source-run-id }} + run: | + # The resolver emits a trimmed id, so compare trimmed: a stray pasted + # space would otherwise fail here after the build was paid for. + REQUESTED="${REQUESTED#"${REQUESTED%%[![:space:]]*}"}" + REQUESTED="${REQUESTED%"${REQUESTED##*[![:space:]]}"}" + if [ "$RESOLVED" = "$REQUESTED" ]; then + echo "prebuilds came from run $RESOLVED, as requested" + exit 0 + fi + echo "::error::prebuild_run_id=$REQUESTED was requested but the prebuilds did not come from it (resolved '$RESOLVED')." + echo "If the log shows \"Unexpected input(s) 'prebuild-run-id'\": this addon loads its" + echo "composite actions from the default branch, so it only supports prebuild_run_id" + echo "once that support is on the default branch. @qvac/audiogen-ggml publishes no" + echo "prebuilds, so @latest cannot work either — use the GPR route until then:" + echo " -f package_spec=@tetherto/audiogen-ggml-mono@" + echo "See docs/ci/MOBILE-ON-DEMAND.md." + exit 1 - name: Build mobile app id: build diff --git a/.github/workflows/integration-mobile-test-bci-whispercpp.yml b/.github/workflows/integration-mobile-test-bci-whispercpp.yml index 6299ca1dc4..d97430ec49 100644 --- a/.github/workflows/integration-mobile-test-bci-whispercpp.yml +++ b/.github/workflows/integration-mobile-test-bci-whispercpp.yml @@ -72,7 +72,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/bci-whispercpp@ (published) or @tetherto/bci-whispercpp-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/bci-whispercpp@ (published) or @tetherto/bci-whispercpp-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-bci-whispercpp run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -162,6 +167,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -218,6 +225,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call keeps its prebuild_package/artifact-first path. package-version: ${{ inputs.platform != '' && inputs.package || inputs.prebuild_package }} force-npm-prebuild: ${{ ((inputs.platform != '' && inputs.package != '') || inputs.prebuild_package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # `npm run test:mobile:generate` downloads models + fixtures # (BCI signal traces, whisper weights) into test/mobile/testAssets/ diff --git a/.github/workflows/integration-mobile-test-classification-ggml.yml b/.github/workflows/integration-mobile-test-classification-ggml.yml index 63edf387a2..b08ef52dde 100644 --- a/.github/workflows/integration-mobile-test-classification-ggml.yml +++ b/.github/workflows/integration-mobile-test-classification-ggml.yml @@ -70,7 +70,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/classification-ggml@ (published) or @tetherto/classification-ggml-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/classification-ggml@ (published) or @tetherto/classification-ggml-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-classification-ggml run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -160,6 +165,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -216,6 +223,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call is untouched (artifact-first). package-version: ${{ inputs.platform != '' && inputs.package || '' }} force-npm-prebuild: ${{ (inputs.platform != '' && inputs.package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # The shared composite handles prebuild fan-out, but classification-ggml # also needs `weights/*.gguf` copied into `test/mobile/testAssets/.gguf.bin` diff --git a/.github/workflows/integration-mobile-test-diffusion-cpp.yml b/.github/workflows/integration-mobile-test-diffusion-cpp.yml index 66cd142452..59dd8bd25c 100644 --- a/.github/workflows/integration-mobile-test-diffusion-cpp.yml +++ b/.github/workflows/integration-mobile-test-diffusion-cpp.yml @@ -77,7 +77,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/diffusion-cpp@ (published) or @tetherto/diffusion-cpp-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/diffusion-cpp@ (published) or @tetherto/diffusion-cpp-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-diffusion-cpp run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -219,6 +224,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -275,6 +282,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call keeps its prebuild_package/artifact-first path. package-version: ${{ inputs.platform != '' && inputs.package || inputs.prebuild_package }} force-npm-prebuild: ${{ ((inputs.platform != '' && inputs.package != '') || inputs.prebuild_package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # Presign the already-seeded (by the seed-models job) US-bucket objects into # a map, then bake them into the manifest before the build. Presign-only: no diff --git a/.github/workflows/integration-mobile-test-embed-llamacpp.yml b/.github/workflows/integration-mobile-test-embed-llamacpp.yml index 873a35cac8..c878a04837 100644 --- a/.github/workflows/integration-mobile-test-embed-llamacpp.yml +++ b/.github/workflows/integration-mobile-test-embed-llamacpp.yml @@ -82,7 +82,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/embed-llamacpp@ (published) or @tetherto/embed-llamacpp-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/embed-llamacpp@ (published) or @tetherto/embed-llamacpp-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-embed-llamacpp run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -214,6 +219,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -270,6 +277,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call is untouched (artifact-first). package-version: ${{ inputs.platform != '' && inputs.package || '' }} force-npm-prebuild: ${{ (inputs.platform != '' && inputs.package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # The mobile perf benchmark shards (benchmark-perf-*.test.js) are not # committed — they are generated from test/integration/_benchmark-matrix.js. diff --git a/.github/workflows/integration-mobile-test-llm-llamacpp.yml b/.github/workflows/integration-mobile-test-llm-llamacpp.yml index 772026ed48..aff0dfa939 100644 --- a/.github/workflows/integration-mobile-test-llm-llamacpp.yml +++ b/.github/workflows/integration-mobile-test-llm-llamacpp.yml @@ -168,7 +168,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/llm-llamacpp@ (published) or @tetherto/llm-llamacpp-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/llm-llamacpp@ (published) or @tetherto/llm-llamacpp-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-llm-llamacpp run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -330,6 +335,8 @@ jobs: test-failed: ${{ steps.monitor.outputs.test-failed }} permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -409,6 +416,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call keeps addon_npm_version/artifact-first. force-npm-prebuild: ${{ ((inputs.platform != '' && inputs.package != '') || inputs.addon_npm_version != '') && 'true' || 'false' }} package-version: ${{ inputs.platform != '' && inputs.package || inputs.addon_npm_version }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} pat-token: ${{ secrets.GITHUB_TOKEN }} # The mobile perf benchmark shards (benchmark-perf-*.test.js) are not diff --git a/.github/workflows/integration-mobile-test-model-fit.yml b/.github/workflows/integration-mobile-test-model-fit.yml index 1383cadd5a..f1cd74077e 100644 --- a/.github/workflows/integration-mobile-test-model-fit.yml +++ b/.github/workflows/integration-mobile-test-model-fit.yml @@ -79,7 +79,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/model-fit@ (published) or @tetherto/model-fit-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/model-fit@ (published) or @tetherto/model-fit-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-model-fit run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -164,6 +169,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -220,6 +227,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call is untouched (artifact-first). package-version: ${{ inputs.platform != '' && inputs.package || '' }} force-npm-prebuild: ${{ (inputs.platform != '' && inputs.package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} - name: Build mobile app id: build diff --git a/.github/workflows/integration-mobile-test-ocr-ggml.yml b/.github/workflows/integration-mobile-test-ocr-ggml.yml index 226381e049..213d974cf5 100644 --- a/.github/workflows/integration-mobile-test-ocr-ggml.yml +++ b/.github/workflows/integration-mobile-test-ocr-ggml.yml @@ -78,7 +78,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/ocr-ggml@ (published) or @tetherto/ocr-ggml-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/ocr-ggml@ (published) or @tetherto/ocr-ggml-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-ocr-ggml run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -165,6 +170,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -246,6 +253,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call keeps its prebuild_package/artifact-first path. package-version: ${{ inputs.platform != '' && inputs.package || inputs.prebuild_package }} force-npm-prebuild: ${{ ((inputs.platform != '' && inputs.package != '') || inputs.prebuild_package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} - name: Build mobile app id: build diff --git a/.github/workflows/integration-mobile-test-translation-nmtcpp.yml b/.github/workflows/integration-mobile-test-translation-nmtcpp.yml index a995e49ac6..e0bf77430a 100644 --- a/.github/workflows/integration-mobile-test-translation-nmtcpp.yml +++ b/.github/workflows/integration-mobile-test-translation-nmtcpp.yml @@ -74,7 +74,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/translation-nmtcpp@ (published) or @tetherto/translation-nmtcpp-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/translation-nmtcpp@ (published) or @tetherto/translation-nmtcpp-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-translation-nmtcpp run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -167,6 +172,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -232,6 +239,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call is untouched (artifact-first). package-version: ${{ inputs.platform != '' && inputs.package || '' }} force-npm-prebuild: ${{ (inputs.platform != '' && inputs.package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} - name: Build mobile app id: build diff --git a/.github/workflows/integration-mobile-test-tts-ggml.yml b/.github/workflows/integration-mobile-test-tts-ggml.yml index 7acc1ce753..eda64a8621 100644 --- a/.github/workflows/integration-mobile-test-tts-ggml.yml +++ b/.github/workflows/integration-mobile-test-tts-ggml.yml @@ -80,7 +80,12 @@ on: required: false default: "" package_spec: - description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/tts-ggml@... for npm or @tetherto/tts-ggml-mono@... for GPR to force-install a specific build." + description: "Package to test (name@version). Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; use @qvac/tts-ggml@... for npm or @tetherto/tts-ggml-mono@... for GPR to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-tts-ggml run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package_spec`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package_spec`." type: string required: false default: "" @@ -175,6 +180,8 @@ jobs: timeout-minutes: ${{ !inputs.run_rtf_benchmarks && 180 || 150 }} permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -233,6 +240,9 @@ jobs: # manual validation cannot silently ignore the requested native build. package-version: ${{ inputs.prebuild_package || inputs.package_spec }} force-npm-prebuild: ${{ (inputs.prebuild_package != '' || inputs.package_spec != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # The release environment authorizes GitHub OIDC to assume the scoped AWS # role used for model-manifest signing; no long-lived AWS keys are stored. diff --git a/.github/workflows/integration-mobile-test-vla.yml b/.github/workflows/integration-mobile-test-vla.yml index 816a64ab92..b901329d9d 100644 --- a/.github/workflows/integration-mobile-test-vla.yml +++ b/.github/workflows/integration-mobile-test-vla.yml @@ -70,7 +70,12 @@ on: required: false default: "" package: - description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/vla-ggml@ (published) or @tetherto/vla-ggml-mono@ (GPR) to force-install a specific build." + description: "Full NPM package spec to test. Leave EMPTY only when this workflow is called from a run that built prebuilds — a standalone dispatch has none, so empty resolves the PUBLISHED @latest, not your branch's native code; set @qvac/vla-ggml@ (published) or @tetherto/vla-ggml-mono@ (GPR) to force-install a specific build. Prefer `prebuild_run_id` for your own PR: it installs the prebuilds your on-pr run already built, with no publish step." + type: string + required: false + default: "" + prebuild_run_id: + description: "Run id whose `prebuilds` artifact to install — the route for testing YOUR OWN PR on a device. Point it at the on-pr-vla run that already built your prebuilds (the number at the end of that run's URL); no tmp-* branch, no publish, no hand-assembled package name. Takes precedence over `package`, and fails the run if that artifact is missing or expired rather than falling back to @latest. Mutually exclusive with `package`." type: string required: false default: "" @@ -195,6 +200,8 @@ jobs: # silently turns failed runs green. permissions: contents: read + # prebuild_run_id reads another run and downloads its artifact. + actions: read packages: read pull-requests: write id-token: write @@ -251,6 +258,9 @@ jobs: # device. See docs/ci/MOBILE-ON-DEMAND.md. workflow_call is untouched (artifact-first). package-version: ${{ inputs.platform != '' && inputs.package || '' }} force-npm-prebuild: ${{ (inputs.platform != '' && inputs.package != '') && 'true' || 'false' }} + # Dispatch-only: under workflow_call this input does not exist and + # resolves empty, leaving the artifact-first path untouched. + prebuild-run-id: ${{ inputs.prebuild_run_id }} # OIDC session for the SmolVLA/GR00T S3 presign (the us-west-2 Device Farm # session is acquired separately in upload-to-devicefarm). Must run BEFORE diff --git a/.github/workflows/on-pr-shared-ci-infra.yml b/.github/workflows/on-pr-shared-ci-infra.yml index 2f96d0b178..d41195fda2 100644 --- a/.github/workflows/on-pr-shared-ci-infra.yml +++ b/.github/workflows/on-pr-shared-ci-infra.yml @@ -11,6 +11,7 @@ on: - ".github/workflows/on-pr-shared-ci-infra.yml" - ".github/actions/cache-models/**" - ".github/actions/run-mobile-integration-tests/seed-and-presign-models/**" + - ".github/actions/run-mobile-integration-tests/setup/**" - "scripts/perf-report/**" - "scripts/ci/**" - "packages/ocr-ggml/.agent/setup.sh" @@ -48,6 +49,12 @@ jobs: run: node --test .github/actions/cache-models/test/*.test.mjs - name: Run seed-and-presign action unit tests run: node --test .github/actions/run-mobile-integration-tests/seed-and-presign-models/test/*.test.mjs + # The prebuild_run_id resolver decides which native binary reaches a + # device. Nothing else runs it pre-merge: the mobile workflows are + # dispatch-only, so a broken resolver would first surface as a failed + # manual run, or worse as a run that installed the wrong build. + - name: Run mobile setup prebuild-run resolver unit tests + run: node --test .github/actions/run-mobile-integration-tests/setup/test/*.test.mjs # The per-addon benchmark aggregators are plain Node scripts shared by the # benchmark-performance-*.yml workflows, which only run on demand — so without diff --git a/docs/ci/MOBILE-ON-DEMAND.md b/docs/ci/MOBILE-ON-DEMAND.md index ffeae446c4..3c67fe13c8 100644 --- a/docs/ci/MOBILE-ON-DEMAND.md +++ b/docs/ci/MOBILE-ON-DEMAND.md @@ -16,6 +16,64 @@ This applies to all 14 mobile addons: `asr-ggml`, `audiogen-ggml`, 2. Click **Run workflow** and fill in the inputs (below). 3. Click **Run workflow**. +> Agents (Claude Code, Codex, Cursor) can walk you through this: the +> `qv-mobile-test-dispatch` skill in `.agents/skills/` is the operating procedure +> built on this page. + +### Quick start — test your own PR on a device + +The common case, end to end. Device Farm is billed per device minute, so filter +to one test and one device unless you need more. + +```bash +ADDON=llm-llamacpp # workflow slug: integration-mobile-test-$ADDON.yml +PKG=llm-llamacpp # package dir: packages/$PKG (vla is the odd one: vla vs vla-ggml) +PR=1234 +BRANCH=$(git branch --show-current) + +# 0. Your PR must carry the `prebuilds` label (or run-desktop/run-mobile-addon-tests), +# or CI builds no prebuilds and there is no run id to point at. + +# 1. The run that built YOUR addon's bundle for THIS commit, whatever workflow built it. +SHA=$(gh pr view "$PR" --repo tetherto/qvac --json headRefOid --jq .headRefOid) +for rid in $(gh api "repos/tetherto/qvac/actions/runs?head_sha=$SHA&per_page=100" \ + --jq '.workflow_runs[].id'); do + gh api "repos/tetherto/qvac/actions/runs/$rid/artifacts?per_page=100" \ + --jq ".artifacts[]|select(.name==\"prebuilds-$PKG\" and .expired==false)|.name" \ + 2>/dev/null | grep -q . && { RUN_ID=$rid; break; } +done +echo "run id: $RUN_ID" + +# 2. A valid test filter (a mocha --grep over runner NAMES). +jq -r '(.android//{})|[..|strings]|unique|.[]' packages/$PKG/test/mobile/test-groups.json 2>/dev/null \ + || grep -oE '\brun[A-Z][A-Za-z0-9_]*' packages/$PKG/test/mobile/integration.auto.cjs | sort -u + +# 3. Dispatch. Android and iOS are separate runs, and a second dispatch of the +# same workflow on the same branch cancels the first. +gh workflow run integration-mobile-test-$ADDON.yml --repo tetherto/qvac --ref "$BRANCH" \ + -f platform=Android \ + -f devices_custom="Google Pixel 9" \ + -f device_model_operator=EQUALS \ + -f tests= \ + -f prebuild_run_id="${RUN_ID:?refusing to dispatch with an empty run id}" +``` + +Then check the build job's setup step printed the run and commit you meant: + +``` +Verified: prebuilds come from run — artifact 'prebuilds-', …, head , branch (), success +``` + +**Common stops**, all of which fail fast and for free: + +| message | meaning | +|---|---| +| `tests filter '' matches none of the N known runners` | wrong runner name — the error lists the valid ones | +| `Run has no 'prebuilds-' … That run built prebuilds for: …` | that run did not build your addon (nx only builds affected ones) | +| `Run is still ''` | the prebuild job has not uploaded yet — wait, same run id | +| `prebuild_run_id and a pinned package are mutually exclusive` | clear whichever of the two you did not mean | +| `[prestage] FATAL: tests grep // matched no known runner` | the name is in neither the addon's `test-groups.json` nor its `integration.auto.cjs` — a typo; take one from the lists above | + ### Inputs | Input | What it does | @@ -25,7 +83,8 @@ This applies to all 14 mobile addons: `asr-ggml`, `audiogen-ggml`, | **devices_custom** | A free-text field for one **or more** device models, comma-separated (e.g. `Pixel 9, Pixel 8`). When set, it **overrides** the dropdown. Use it for new/uncommon devices or to run several at once. | | **device_model_operator** | How the model name is matched: `EQUALS` (**default** — that exact fleet model only; dropdown values are exact fleet names) or `CONTAINS` (any model containing the value — Device Farm picks by availability, so `Pixel 9` can also match `Pixel 9 Pro`). Default is `EQUALS` so a single-device run bills exactly the model you picked. | | **tests** | Optional test filter — see [below](#the-tests-filter). Empty = the full mobile suite. | -| **package** (or **package_spec**) | Which build to actually put on the phone — see [below](#which-build-gets-tested). Default **empty** resolves the **published `@qvac/@latest`** on a manual run, *not* your branch — a manual dispatch builds no prebuild artifacts of its own. To test unmerged native code you must pin a GPR dev build; see [Testing unmerged / unpublished native code](#testing-unmerged--unpublished-native-code). | +| **prebuild_run_id** | The run id whose `prebuilds` artifact holds the native binaries to install. **This is the route for testing your own PR on a device** — see [below](#testing-unmerged--unpublished-native-code). Outranks `package`, and a wrong or expired run id **fails the run** instead of quietly resolving `@latest`. | +| **package** (or **package_spec**) | Which *published* build to put on the phone — see [below](#which-build-gets-tested). Default **empty** resolves the **published `@qvac/@latest`** on a manual run, *not* your branch — a manual dispatch builds no prebuild artifacts of its own. Use it for a release or a build from **another** branch; for your own PR prefer `prebuild_run_id`. Mutually exclusive with it. | | **ref** | Git ref to check out for the **test harness / app** (not the native binary — see below). | ### Device selection: dropdown + free-text @@ -155,11 +214,38 @@ If in doubt, run once **without** a filter and open the Device Farm run's `bare_console.log` / the "Run → tests" legend on the job summary — it enumerates the `run*` names that executed, which you can then narrow with `tests`. +### Where the logs are when a run fails + +`test-results.json` only records the harness assertion, which is the same for +every failure. The reason is in the app's own output: + +| what | Android | iOS | +|---|---|---| +| JS / bare runtime, TAP, the failure | `logcat_full.txt`, `bare` tag | `bare_console.log` | +| **native C++ / engine** | `logcat_full.txt`, `bare` tag, `[C++ TEST]` prefix | `bare_console.log`, `[C++ TEST]` prefix | + +```bash +gh run download --repo tetherto/qvac --dir ./logs +grep -aE "E bare|I bare" logs/**/*logcat_full.txt # Android: test + native +grep -a "\[C++ TEST\]" logs/**/*bare_console.log # iOS: native +``` + +Use `logcat_full.txt`, **not** the smaller `Logcat.logcat`, and grep the `bare` +tag rather than TAP markers — the runtime prints through logcat, so `ok 1` never +appears as a raw line. There is no `bare_console.log` on Android by construction +(private app data, unreadable by adb on a release-signed APK). + ### Which build gets tested A manual run does **not** compile the native addon — it installs a **prebuilt** -one. Which prebuild depends on the `package` / `package_spec` input: - +one. Sources are tried in this order: + +- **`prebuild_run_id=`** → install the `prebuilds` artifact **that run + already built**. Highest precedence: when set, every source below is skipped, + and resolution **fails closed** — a run id that is wrong, private, builds no + prebuilds, or whose artifact has expired fails the run rather than sliding + back to `@latest`. This is the route for testing **your own PR**; see + [below](#testing-unmerged--unpublished-native-code). - **Empty (default)** → artifact-first resolution: prebuild artifacts **from the same run**, then the published **`@qvac/@latest`** if there are none. A standalone dispatch builds no prebuilds of its own, so in practice **empty @@ -174,12 +260,126 @@ one. Which prebuild depends on the `package` / `package_spec` input: leftovers or do not exist. Setting any non-empty spec flips `force-npm-prebuild` on. +`prebuild_run_id` and `package` are two different answers to "which binary goes +on the phone", so setting **both is an error** — the run fails and tells you to +clear one, rather than picking for you. + ### Testing unmerged / unpublished native code `--ref ` gives you the branch's JS harness, tests and app — but **never** -its compiled `.bare`. If your change touches `addon/src/**`, you must pin a GPR -dev build, or the run exercises your new tests against the **published** engine -and passes for the wrong reason. +its compiled `.bare`. If your change touches `addon/src/**`, the run otherwise +exercises your new tests against the **published** engine and passes for the +wrong reason. That has happened: a PR ran mobile on five addons, went green on +all of them, and every run had `package` empty — so each one installed the +published release instead of the ~300 lines of new C++ under review. + +Two routes. Pick by **whose build you need**. + +#### Route A — your own PR: `prebuild_run_id` (use this) + +Your PR's `on-pr-` run already compiled the prebuilds. Point the mobile +dispatch at that run and it installs those exact binaries — **no `tmp-*` branch, +no On Merge dispatch, no wait for a publish, no hand-assembled package name.** + +**First, your PR must have built prebuilds at all.** The prebuild stage is +**label-gated** by `ci-router`: it only runs when the PR carries `prebuilds`, +`run-desktop-addon-tests`, or `run-mobile-addon-tests`. With none of those there +is no bundle and no run id to point at — add the `prebuilds` label and let CI +re-run first. + +**Where the run id comes from.** Easiest: open the PR's **Checks** tab, click the +run that built the prebuilds, and take the number at the end of its URL +(`.../actions/runs/`). + +Do **not** assume it is your addon's own workflow. Which workflow builds the +bundle varies — `on-pr-nx.yml` for most addons, `on-pr-.yml` for some, +`on-merge-.yml` for a branch build — so filtering by workflow name is +unreliable. Scope by your PR's head commit instead: + +```bash +PKG=llm-llamacpp # the package directory name, i.e. packages/ +PR=4519 # your PR number + +SHA=$(gh pr view "$PR" --repo tetherto/qvac --json headRefOid --jq .headRefOid) +RUN_ID=$(for rid in $(gh api "repos/tetherto/qvac/actions/runs?head_sha=$SHA&per_page=100" \ + --jq '.workflow_runs[].id'); do + gh api "repos/tetherto/qvac/actions/runs/$rid/artifacts?per_page=100" \ + --jq ".artifacts[]|select(.name==\"prebuilds-$PKG\" and .expired==false)|.name" \ + 2>/dev/null | grep -q . && { echo "$rid"; break; } +done) + +# An empty RUN_ID would dispatch the "unchanged" path and quietly resolve +# @qvac/@latest — the published-release-goes-green failure this whole +# route exists to close. Stop instead. +[ -n "$RUN_ID" ] || { echo "no run for $SHA carries prebuilds-$PKG (is the 'prebuilds' label on the PR?)" >&2; exit 1; } +echo "$RUN_ID" +``` + +That finds the run carrying **your addon's** bundle for **this commit**, whatever +built it. If it prints nothing, either the label is missing or — on the nx path — +that run only built the addons it considered affected, and yours was not one. The +dispatch failure message lists which addons a run did build, so a wrong guess +tells you where to look. + +gh workflow run integration-mobile-test-$WF.yml --repo tetherto/qvac --ref $BRANCH \ + -f platform=Android \ + -f devices_custom="Google Pixel 9" \ + -f device_model_operator=EQUALS \ + -f prebuild_run_id=$RUN_ID +``` + +The build job's *Resolve prebuilds from a run id* step prints the provenance: + +``` +Verified: prebuilds come from run 33179656677 — artifact 'prebuilds-llm-llamacpp', +workflow 'On PR Trigger (LLM)', head 1d2c3b4…, +branch feat/backend-selection (tetherto/qvac), success +``` + +That line names the **head SHA** the binaries were built from. Check it against +your branch tip: a run id resolves whether or not it built the commit you meant, +so this is what tells you the binaries are the ones under review. The **`ref`** +input and the prebuild run are deliberately independent — that is what lets you +test a JS-only fix against prebuilds from an earlier commit — so nothing can +infer the mismatch for you. + +This route **fails closed** by design. A run id that does not exist, that you +cannot read, that built no prebuilds, or whose artifact has aged out of +retention fails the run with the reason and what to do about it. It never falls +back to `@latest` — that silent fallback is the bug this route exists to remove. + +Three things to know: + +- **Artifacts expire.** Retention is set per repository, so an old run id stops + working. Re-run the prebuild job on your PR and use the new run id. +- **The artifact must cover your platform.** A prebuild run whose iOS leg was + cancelled still publishes a bundle, just without `ios-arm64`; the run fails + with the directories the artifact does contain, rather than building an app + around a missing binary. +- **`audiogen-ggml` is the exception.** It loads its composite actions from the + default branch (a supply-chain guard for its `release`-environment job), so it + only honours `prebuild_run_id` once that support is on the default branch. + Until then the run fails with an explicit message rather than quietly + installing `@latest` — use `-f package_spec=@tetherto/audiogen-ggml-mono@` + in the meantime. Note that `@qvac/audiogen-ggml` publishes **no prebuilds**, so + an empty input cannot work for this addon at all. +- **Check which repository built it.** This repo is fork-first, so a PR's + `on-pr` run is usually `pull_request_target` on a *fork* — it appears in this + repo's run list while its head repository is the contributor's fork. That is + the normal case and is not blocked: a fork's prebuilds only exist because the + merge/release team already approved `fork-ci` on that run. But the binaries do + get bundled into an app and executed on org devices, so the provenance line + names the head repository and a run from a **different** repository than the + one you expected raises a `::warning::`. Read it before trusting a green run. + +It also works for **`ocr-ggml` and `translation-nmtcpp`**, which Route B cannot +serve at all (see the note at the end of that section). + +#### Route B — a build from another branch, or a published release + +Route A needs a run you can point at. When you need someone *else's* branch, an +older commit whose artifact has expired, or a specific published version, pin a +package instead. **Step 1 — publish a dev build of your branch.** Push it as `tmp-`; the addon's *On Merge Trigger* workflow builds the prebuilds and publishes @@ -262,32 +462,35 @@ Verified: prebuilds come from @tetherto/-mono@ (pinned, GitHub P If you instead see `downloading @qvac/@latest from npm (registry.npmjs.org)`, the pin did not arrive and you are testing the published release. -**The input is named `package` on most addons but `package_spec` on three:** +**The Route B input is named `package` on most addons but `package_spec` on three.** +`prebuild_run_id` (Route A) has the same name everywhere: | input | addons | |---|---| | `-f package=` | `bci-whispercpp`, `classification-ggml`, `decoder-audio`, `diffusion-cpp`, `embed-llamacpp`, `llm-llamacpp`, `model-fit`, `ocr-ggml`, `translation-nmtcpp`, `vla` | | `-f package_spec=` | `asr-ggml`, `audiogen-ggml`, `tts-ggml` | | *(no such input)* | `inference-addon-cpp` | +| `-f prebuild_run_id=` | every addon above **except** `decoder-audio` and `inference-addon-cpp` (see *Addons that need none of this*) | **Addons that need none of this:** `inference-addon-cpp` compiles its `.bare` in the same run from `ref`, and `decoder-audio` has no native prebuild of its own (it rides on `bare-ffmpeg`'s, so its `package` input does not change what is -tested) — for both, plain `--ref ` is enough. +tested) — for both, plain `--ref ` is enough, which is why neither +exposes `prebuild_run_id`. -> **Two addons cannot do this today.** `ocr-ggml` and `translation-nmtcpp` never -> publish a GPR dev build — their `publish-gpr` job is skipped on every push -> because it depends transitively on the `release-merge-guard` job, which is -> skipped on any non-`release-*` branch, and unlike `build` it does not opt out -> with `!cancelled()`. `@tetherto/ocr-ggml-mono` has therefore never existed, and -> `@tetherto/translation-nmtcpp-mono` is frozen at 2026-06-30. Until that is -> fixed there is no way to put unmerged native code for those two on a device. +> **Two addons publish no mobile prebuilds to npm.** `@qvac/asr-ggml` and +> `@qvac/audiogen-ggml` ship none, so an **empty** input cannot work for them — +> the run fails with "No prebuilds directory found in package". Their +> `@tetherto/-mono` dev builds *do* carry prebuilds, so Route B works; so +> does Route A. (`@qvac/decoder-audio` also ships none, but it needs no prebuilds +> of its own — see below.) > The **`ref`** input defaults to **blank**, so the run checks out the branch you > dispatch from (`gh workflow run … --ref ` — no `-f ref=` needed). Pass > `-f ref=` only to override it. `ref` drives the JS test harness and > the app; it does **not** drive the native prebuild, which always comes from an -> artifact or a package (see above). +> artifact (this run's, or the run named by `prebuild_run_id`) or a package +> (see above). > > The `tests`-filter / shard-count validation reads the runner list from the > **same commit** the build executes, so if your branch renames or adds runners diff --git a/packages/model-fit/test/integration/fit-stub.test.js b/packages/model-fit/test/integration/fit-stub.test.js index df279d3ce8..4cab3e8b2b 100644 --- a/packages/model-fit/test/integration/fit-stub.test.js +++ b/packages/model-fit/test/integration/fit-stub.test.js @@ -15,6 +15,7 @@ const test = require('brittle') const fs = require('bare-fs') +const path = require('bare-path') const process = require('bare-process') const { fitParams, FIT_STATUS } = require('../../index.js') const { ensureModelPath } = require('./utils') @@ -42,13 +43,20 @@ async function ensureFixtures() { // what creates that directory. FIT_MODEL_PATH skips the download, so on a // fresh checkout the directory is not there — this file is the first to write // into it rather than only read from it. - fs.mkdirSync(fixtureDir(), { recursive: true }) + // + // Take it from the model path rather than re-deriving it: on a device this + // file runs from the read-only app bundle, so the derived location resolves + // inside the bundle and mkdirSync fails. + const baseDir = path.dirname(fullPath) + fs.mkdirSync(fixtureDir(baseDir), { recursive: true }) fixtures = { fullPath, - stubPath: writeFitStub(fullPath, fixturePath('fit-stub.gguf')), - fullSplit: writeSplit(fullPath, fixturePath('split-full'), { splitCount: SPLIT_COUNT }), - stubSplit: writeSplit(fullPath, fixturePath('split-stub'), { + stubPath: writeFitStub(fullPath, fixturePath('fit-stub.gguf', baseDir)), + fullSplit: writeSplit(fullPath, fixturePath('split-full', baseDir), { + splitCount: SPLIT_COUNT + }), + stubSplit: writeSplit(fullPath, fixturePath('split-stub', baseDir), { splitCount: SPLIT_COUNT, stub: true }) diff --git a/packages/model-fit/test/integration/gguf.js b/packages/model-fit/test/integration/gguf.js index cd1cea6d6a..d150f1ba51 100644 --- a/packages/model-fit/test/integration/gguf.js +++ b/packages/model-fit/test/integration/gguf.js @@ -408,14 +408,20 @@ function allKvs(meta) { return meta.kvs.map((raw) => ({ raw })) } -/** Directory the fixtures are written to, beside the downloaded test model. */ -function fixtureDir() { - return path.resolve(__dirname, '../model') +/** + * Directory the fixtures are written to, beside the downloaded test model. + * + * Pass `baseDir` — the directory the model actually landed in. Re-deriving it + * from `__dirname` points inside the read-only app bundle on a device, where + * creating it fails. Defaults to the checkout layout for desktop callers. + */ +function fixtureDir(baseDir) { + return baseDir || path.resolve(__dirname, '../model') } /** Absolute path of a fixture beside the downloaded test model. */ -function fixturePath(name) { - return path.join(fixtureDir(), name) +function fixturePath(name, baseDir) { + return path.join(fixtureDir(baseDir), name) } module.exports = { diff --git a/packages/vla-ggml/scripts/__tests__/generate-prestage-block.test.js b/packages/vla-ggml/scripts/__tests__/generate-prestage-block.test.js index 81e3f24c49..38401b4d20 100644 --- a/packages/vla-ggml/scripts/__tests__/generate-prestage-block.test.js +++ b/packages/vla-ggml/scripts/__tests__/generate-prestage-block.test.js @@ -19,7 +19,8 @@ const { MODEL_SHARDS, buildManifest, buildScript, - formatYamlBlock + formatYamlBlock, + readKnownRunners } = require('../generate-prestage-block') function withAssetsDir(fn) { @@ -170,6 +171,47 @@ test('shard grep is a regex: partial matches stage, an unbaked-but-known shard f assert.match(out(typo), /matched no known runner \(test-groups <-> model-map drift\)/) }) +// A runner with no model is still a scheduled runner. The oracle used to be +// built from MODEL_SHARDS — the subset that HAS a model — so every model-less +// runner looked like drift and failed closed on device: +// -f tests=runEsmNamedExportsTest +// [prestage] FATAL: tests grep /runEsmNamedExportsTest/ matched no known runner +// observed on a real Device Farm run. The oracle reads test-groups.json instead. +test('readKnownRunners covers every scheduled runner, not just the ones with models', () => { + const known = readKnownRunners() + for (const name of ['runAddonTest', 'runGrootTest', 'runEsmNamedExportsTest', 'runPi05Test']) { + assert.ok(known.includes(name), `${name} must be a known runner, got ${known.join(', ')}`) + } +}) + +test('readKnownRunners falls back to the model list when test-groups.json is unreadable', () => { + // An empty oracle would fail every grep closed, which is worse than a narrow one. + const known = readKnownRunners('/nonexistent/test-groups.json') + assert.deepEqual([...known].sort(), MODEL_SHARDS.map((s) => s.test).sort()) +}) + +test('a model-less runner stages nothing instead of failing closed', () => { + const man = { + runAddonTest: [{ name: 'smolvla.gguf', url: 'https://x/smolvla.gguf' }], + runGrootTest: [{ name: 'groot.gguf', url: 'https://x/groot.gguf' }] + } + const script = buildScript(Buffer.from(JSON.stringify(man)).toString('base64')) + const out = (r) => `${r.stdout}${r.stderr}` + + for (const runner of ['runEsmNamedExportsTest', 'runPi05Test']) { + const result = runWithStubs(script, { grep: runner }) + assert.equal(result.status, 0, `${runner} must not fail closed: ${out(result)}`) + assert.match(out(result), /known runner with no baked URL yet/) + assert.match(out(result), /0 model\(s\) for 0 test\(s\)/) + assert.doesNotMatch(out(result), /model-map drift/) + } + + // The oracle still fails closed on a name that is in neither list. + const typo = runWithStubs(script, { grep: 'runEsmNamedExport' + 'sTypo' }) + assert.notEqual(typo.status, 0) + assert.match(out(typo), /matched no known runner \(test-groups <-> model-map drift\)/) +}) + test('buildScript ios backend uses pymobiledevice3 apps push into Documents', () => { const man = { runAddonTest: [{ name: 'smolvla.gguf', url: 'https://x/smolvla.gguf' }] } const b64 = Buffer.from(JSON.stringify(man)).toString('base64') diff --git a/packages/vla-ggml/scripts/generate-prestage-block.js b/packages/vla-ggml/scripts/generate-prestage-block.js index 0f1a363c29..19f00de876 100644 --- a/packages/vla-ggml/scripts/generate-prestage-block.js +++ b/packages/vla-ggml/scripts/generate-prestage-block.js @@ -17,6 +17,7 @@ const fs = require('fs') const path = require('path') const DEFAULT_ASSETS_DIR = path.resolve(__dirname, '../test/mobile/testAssets') +const DEFAULT_TEST_GROUPS = path.resolve(__dirname, '../test/mobile/test-groups.json') const IOS_BUNDLE_ID = 'io.tether.test.qvac' // `test` is the test-groups.json function name the composite bakes into each @@ -26,14 +27,43 @@ const MODEL_SHARDS = [ { test: 'runGrootTest', name: 'groot-q5_vf16.gguf', urlsFile: 'groot-urls.json' } ] -// The static set of mobile runner names — the drift oracle. buildManifest below -// only bakes shards whose presigned URL already exists, and pi05 is deferred on -// mobile, so a grep that matches a KNOWN runner but no manifest key is a legit -// "URL not staged yet -> network fallback". A grep that matches NO known runner -// at all is a test-groups <-> model-map drift and must fail closed. Emitted as a -// single-quoted JS array literal so it stays safe inside the `node -e "…"` arg. -function knownRunnersLiteral() { - return '[' + MODEL_SHARDS.map((s) => `'${s.test}'`).join(',') + ']' +// The set of mobile runner names — the drift oracle. A grep matching a KNOWN +// runner with no manifest key is a legit "no URL staged -> network fallback"; +// one matching NO known runner is drift and must fail closed. +// +// Read from test-groups.json, the actual runner list, NOT from MODEL_SHARDS, +// which is only the subset that has a model. Deriving it from MODEL_SHARDS made +// every model-less runner look like drift and fail on device. +function readKnownRunners(testGroupsPath = DEFAULT_TEST_GROUPS) { + const names = new Set(MODEL_SHARDS.map((s) => s.test)) + let groups + try { + groups = JSON.parse(fs.readFileSync(testGroupsPath, 'utf8')) + } catch (_) { + // Fall back to the model list: an empty oracle would fail every grep closed. + return [...names] + } + const walk = (node) => { + if (typeof node === 'string') { + if (/^[A-Za-z_$][\w$]*$/.test(node)) names.add(node) + return + } + if (Array.isArray(node)) return node.forEach(walk) + if (node && typeof node === 'object') return Object.values(node).forEach(walk) + } + walk(groups) + return [...names].sort() +} + +// Single-quoted JS array literal so it stays safe inside the `node -e "…"` arg. +function knownRunnersLiteral(testGroupsPath = DEFAULT_TEST_GROUPS) { + return ( + '[' + + readKnownRunners(testGroupsPath) + .map((n) => `'${n}'`) + .join(',') + + ']' + ) } // Build the { : [{ name, url }] } manifest from the bundled *-urls.json. @@ -220,4 +250,11 @@ function main() { if (require.main === module) main() -module.exports = { MODEL_SHARDS, buildManifest, buildScript, formatYamlBlock, IOS_BUNDLE_ID } +module.exports = { + MODEL_SHARDS, + buildManifest, + buildScript, + formatYamlBlock, + readKnownRunners, + IOS_BUNDLE_ID +}