Skip to content

QVAC-19792 feat: consolidate the ts-checks producers into on-pr-ts-nx - #4497

Merged
sidj-thr merged 6 commits into
mainfrom
fix/ts-checks-nx-global-inputs
Sep 17, 2026
Merged

sidj-thr merged 6 commits into
mainfrom
fix/ts-checks-nx-global-inputs

Conversation

@sidj-thr

@sidj-thr sidj-thr commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Replaces the eight on-pr-<pkg>-ts.yml producers with one nx-driven workflow, so the checks on-pr-nx awaits are always produced.

🎯 What problem does this PR solve?

on-pr-nx awaits a ts-check for every nx-affected package with hasTsChecks, but each producer triggered on its own package's paths. nx selects a package for reasons those paths never see, so the check is never produced and the await burns the hard 25-minute TIMEOUT_MS in await-ts-checks/await.mjs. merge-guard / validate-pr fails with it, because sanity-checks-status folds in needs.ts-checks.result.

Two mechanisms have hit production, hours apart:

Global inputs. #4454, run 35089161423pnpm-lock.yaml marks all 14 packages affected while triggering no producer. Five of six awaits timed out. Also hit #4439 twice and #4389; both are SDK-side PRs that touch no native addon at all.

Dependency edges. #4445 touches only packages/registry-server/**, including client/index.d.ts, which publishes @qvac/registry-client. Six packages depend on it, so nx is right to select them — their types really could break. The bug is that the checks proving it never run.

An earlier commit on this branch added pnpm-lock.yaml, package.json and nx.json to each producer. That closes the first mechanism and not the second, and nx has others still (project-glob-changes, workspace-json-changes). Chasing each with static paths is whack-a-mole where every miss costs 25 minutes.

📝 How does it solve it?

on-pr-ts-nx.yml triggers broadly and lets nx narrow:

on:
  pull_request:
    paths:
      - "packages/**"          # superset of on-pr-nx's own trigger
      - ".github/workflows/on-pr-ts-nx.yml"
      - ".github/workflows/reusable-ts-checks.yml"

A matrix job runs nx-project-matrix with target: on-pr, filters to hasTsChecks, and eight static jobs gate on the result:

  llm-pr-head-ts-checks:
    needs: matrix
    if: contains(fromJSON(needs.matrix.outputs.tspackages), 'llm-llamacpp')
    uses: ./.github/workflows/reusable-ts-checks.yml

Producer and consumer now resolve against the same nx graph and the same on-pr target, so a package selected for any reason is selected on both sides. Only affected packages run a ts job, which is why this is cheaper than broadening the eight producers to packages/**.

Same graph and target is not the same as identical inputs. This workflow computes its changed files from nx-set-shas' git range; on-pr-nx feeds the API file list to nx --stdin. Base-sha resolution and rename handling can diverge at the edges, and that asymmetry is inherent to the privileged/unprivileged split rather than something this PR removes.

It stays on pull_request because these run PR-head code and must remain unprivileged. That is also the event where nx-set-shas works natively.

Job ids are load-bearing. A reusable call publishes <job id> / ts-checks, and on-pr-nx polls those exact strings. All eight are preserved, including diffusion's, which predates the reusable, runs inline with name: diffusion-pr-head-ts-checks, and has no / ts-checks suffix. Its steps are copied verbatim; migrating it onto the reusable would rename a status check and is a separate decision.

The eight producers are deleted in the same commit. Both publish identical check names, so keeping them would run every ts check twice on the same SHA.

Check names

Not a detail: these strings are the entire contract between producer and consumer, matched literally. A rename does not error, it makes the await poll for something that will never appear and fail 25 minutes later, on an unrelated PR.

Every name is unchanged by this PR. The published name is <job id> / ts-checks for the seven that call the reusable, and the job's name: for diffusion, which runs inline.

Package Check name Published by
classification-ggml classification-ggml-pr-head-ts-checks / ts-checks job id + reusable
diffusion-cpp diffusion-pr-head-ts-checks name: override, inline job, no suffix
embed-llamacpp embed-pr-head-ts-checks / ts-checks job id + reusable
llm-llamacpp llm-pr-head-ts-checks / ts-checks job id + reusable
model-fit model-fit-pr-head-ts-checks / ts-checks job id + reusable
ocr-ggml ocr-ggml-pr-head-ts-checks / ts-checks job id + reusable
translation-nmtcpp translation-nmtcpp-pr-head-ts-checks / ts-checks job id + reusable
vla-ggml vla-pr-head-ts-checks / ts-checks job id + reusable

Diffusion's exception is why the job id in on-pr-ts-nx.yml is ts-checks with a name: override rather than diffusion-pr-head-ts-checks: it reproduces the old workflow's structure exactly, and changing it would rename the check.

Each name has two or three pollers, all of which must keep resolving:

  • on-pr-nx.yml — the jq map that builds matrix.tsCheckName
  • the legacy on-pr-<pkg>.yml consumers — a literal check_name: each, eight of them, deleted by #3976
  • .github/scripts/test/await-ts-checks.test.mjs — hard-codes llm-pr-head-ts-checks / ts-checks

That last one is easy to miss, which is the point of the next section.

How to verify

.github/scripts/test/ts-check-names.test.mjs is added here and checks four things:

  1. every name on-pr-nx awaits is published by on-pr-ts-nx
  2. every literal ts-check name anywhere in the repo is published by something, so no poller is orphaned
  3. every producer job gates on the nx-affected list and needs the matrix job
  4. on-pr-ts-nx's trigger paths are a superset of on-pr-nx's
node .github/scripts/test/ts-check-names.test.mjs

The fourth is the invariant the whole design rests on, and it closes the trigger loop the way 1–2 close the name loop. Narrow the producer below the consumer and on-pr-nx runs while on-pr-ts-nx does not, so nx never narrows anything and every await times out. That is precisely the regression the superseded approach shipped, and nothing asserted it until now. A trailing /** is treated as covering everything beneath it, so a broad producer glob still satisfies a narrower consumer path.

It is mutation-tested rather than merely passing:

Mutation Result
rename a job id 2 of 4 fail
drop diffusion's name: override 2 of 4 fail
remove a job's if: gate 1 of 4 fails
narrow the producer to one package 1 of 4 fails
consumer gains an unmirrored path 1 of 4 fails
unmodified 4 pass

🧪 How was it tested?

Probed with the eight producers deleted, so nothing else could have produced the checks:

Result
on-pr-ts-nx matrix success, selected ["ocr-ggml"]
ocr-ggml-pr-head-ts-checks / ts-checks success
other seven producer jobs skipped, correctly
await ts-checks (ocr-ggml) / await-ts-checks success

The two sides agreed exactly: on-pr-nx under pull_request_target logged Affected (on-pr, transitive): ["ocr-ggml"], and on-pr-ts-nx under pull_request computed the same. That agreement is what the design turns on, though as noted above it rests on a shared graph and target rather than on identical changed-file inputs.

The run pages show the published names directly. The producer run lists ocr-ggml-pr-head-ts-checks / ts-checks as succeeded and the other seven job names as skipped, so all eight can be read off one page and compared against the table above. The consumer's await ts-checks (ocr-ggml) / await-ts-checks resolved in 7 seconds, against the 25-minute timeout it hits today.

Check names asserted mechanically, see the section above. Eight for eight against on-pr-nx's map, plus a whole-repo scan for orphaned pollers, plus mutation testing to prove the assertion can actually fail.

Static: actionlint -shellcheck= clean. ts-check-names 4/0, await-ts-checks 13/0, ci-trust-policy 74/0, prebuild-status 19/0, publish-gate-policy 5/0, publish-gpr-skip-policy 33/0, runner-names 13/0. No reference to any deleted file remains anywhere in the repo.

Expect one noisy CI run on this PR

Deleting the eight files matches .github/workflows/*<pkg>*.yml in the legacy on-pr-<pkg>.yml consumers, so six of them wake up. Those packages are not nx-affected, the new producer correctly skips them, and their awaits have nothing to wait for. Observed on the probe: six On PR Trigger (<pkg>) runs whose Await PR-head TypeScript checks jobs would time out.

It is one-time, confined to this PR, and disappears once #3976 removes those legacy workflows. Landing this after #3976 avoids it entirely.

on-pr-nx awaits a ts-check for every nx-affected package with hasTsChecks, but
each on-pr-<pkg>-ts.yml only triggers on its own package's paths. nx treats
pnpm-lock.yaml, the root package.json and nx.json as inputs to every project, so
a change to any of them marks all 14 packages affected while triggering none of
the producers. The awaits then burn their 25 minute timeout and fail, taking
merge-guard with them via sanity-checks-status.

First hit on #4454, run 35089161423: 23 changed paths, 14 affected, 5 of 6
awaits timed out. That PR cannot avoid it, since it adds a dependency to
packages/llm-llamacpp/package.json and CONTRIBUTING requires the lockfile be
regenerated in the same commit.

Narrowing the nx side is not an option: nx.json fans out through a hard-coded
implicits map, and the root package.json through the js plugin's npm-packages
locator. Neither reads namedInputs, so only the producers can be aligned.

Also adds the missing reusable-ts-checks.yml path to diffusion-cpp, which the
other seven already had, so a change to the shared reusable stops silently
skipping diffusion's checks.

Costs one short ts job per package on dependency-bump PRs.
@sidj-thr
sidj-thr requested review from a team as code owners September 16, 2026 12:26
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Workflow security (shadow mode)

zizmor found 1124 finding(s) in .github/ (highest severity: high). This check is warn-only and does not block the merge.

Findings are annotated inline on the changed files and listed in the job summary.

Reproduce locally:

pipx run zizmor==1.27.0 --offline .github/

@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

License compliance — findings detected (warn-only)

Critical: 0 · High: 2 · Medium: 0

Dependency License Scope Severity Outcome
actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd (none detected) runtime High blocks
actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f (none detected) runtime High blocks

How to resolve a blocking finding:

  • Remove or replace the disallowed dependency, or
  • If the license is genuinely acceptable, run the compliance SKILL and record the decision in .github/license-allowlist.yml (CODEOWNERS-reviewed), or
  • For a one-off, a maintainer can apply the license-override label (High findings only; Critical cannot be overridden).

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

  • ./docs/website
  • ./packages/fabric/test/integration
  • ./packages/llm-llamacpp/benchmarks/server
  • ./packages/llm-llamacpp/benchmarks/performance
  • ./packages/inference-addon-cpp/mobile
  • ./packages/asr-ggml/benchmarks/server
  • ./packages/embed-llamacpp/benchmarks/server
  • ./packages/embed-llamacpp/benchmarks/performance
  • ./packages/sdk/e2e
  • ./packages/vla-ggml/sim/server
  • ./.github/actions/release-merge-guard

Supersedes the three-path change earlier on this branch. Adding pnpm-lock.yaml,
package.json and nx.json to each producer closed the global-input fan-out, but
not the dependency-edge one: #4445 touches only registry-server and nx correctly
selects the six packages depending on @qvac/registry-client, whose producers
still never fire. Chasing each nx selection mechanism with static paths is
whack-a-mole, and every miss costs a 25 minute timeout.

on-pr-ts-nx triggers broadly on packages/** and lets nx narrow, so producer and
consumer run the same nx computation against the same target and cannot select
different sets. Only affected packages run a ts job, so this is cheaper than
broadening the eight producers to packages/**.

Stays on pull_request: these run PR-head code and must remain unprivileged.
That is also where nx-set-shas works natively.

Job ids are load-bearing, since a reusable call publishes "<job id> / ts-checks"
and on-pr-nx polls those exact strings. All eight are preserved, including
diffusion's inline job, which predates the reusable and has no "/ ts-checks"
suffix.

Deletes the eight per-package producers in the same commit: both publish
identical check names, so keeping them would run every ts check twice.
@sidj-thr sidj-thr changed the title QVAC-19792 fix: fire the ts-checks producers on nx's global inputs QVAC-19792 feat: consolidate the ts-checks producers into on-pr-ts-nx Sep 16, 2026
…s-nx

A renamed job id stops satisfying its pollers silently; the only symptom is an
await burning its 25 minute timeout on someone else's PR days later. Reads the
awaited names out of on-pr-nx's jq map, derives the published names from the
producer's job ids and name: overrides, and scans the repo for any other
literal poller. Verified to fail on a renamed job id, a dropped name: override
and a missing if: gate.
…nx's

Review feedback. Tests 1-2 close the check-name loop and test 3 the per-job
gating, but nothing asserted trigger parity, which is the invariant the design
rests on: narrow the producer below the consumer and on-pr-nx runs while
on-pr-ts-nx does not, so nx never narrows anything and every await times out.
That is the regression the superseded approach shipped.

Compares both paths: blocks, treating a trailing /** as covering everything
beneath it so a broad producer glob still satisfies a narrower consumer path.
Verified to fail both when the producer is narrowed to one package and when the
consumer gains an unmirrored path.

Also softens the header comment: the two share the graph and the on-pr target,
but not the changed-file input, since this workflow uses nx-set-shas' git range
while on-pr-nx feeds the API file list to nx --stdin.
@sidj-thr
sidj-thr merged commit 9e17163 into main Sep 17, 2026
85 of 92 checks passed
@sidj-thr
sidj-thr deleted the fix/ts-checks-nx-global-inputs branch September 17, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants