ci: run the TypeScript suite — 401 tests that no workflow executed - #36
Conversation
…ile bumps Review finding on #36, and it is the same class of defect this PR exists to fix. The Test step was added to build-and-verify-dist correctly — that job is a REQUIRED check on main, so a failing suite genuinely blocks. But the step is gated on `steps.changed.outputs.ts`, and the detector grepped only `^(package\.json|ts/)`. This PR touches `.github/workflows/ts-ci.yml` and `package-lock.json`, neither of which matched, so on the head commit every step reported `skipped`: Detect TS/dist-relevant changes -> success Install / Typecheck / Test / Rebuild dist -> skipped No TS/dist changes — required check trivially satisfied -> success i.e. the commit that adds the test step had that step skipped, and the PR was green without ever running the 401 tests it wires up. Two real holes: - a dependency bump can land entirely in package-lock.json, and that is precisely the change most likely to break the suite; - a PR editing only ts-ci.yml skipped everything, so the workflow defining this gate was the one change the gate never ran on. Widened the detector to package(-lock)?.json, ts/, and ts-ci.yml itself. The escape hatch the file header describes is deliberately preserved — verified by truth table that Rust-only, README-only and ci.yml-only PRs still resolve to ts=false, so the required context still reports instead of hanging "expected". The proof that the step has teeth is already in this branch's history: 4ef89e3 added a deliberately failing test and build-and-verify-dist went red; 2afaeb1 removed it. Re-verified independently against the check-runs API.
The TypeScript suite was never executed by any workflow. ci.yml runs `cargo test --workspace` (Rust only); ts-ci ran typecheck, build and a dist staleness check. 401 tests existed and nothing invoked them, so a TS regression could merge with every required check green. This commit deliberately includes a failing test. CI must go RED. The next commit removes it and CI must go green — otherwise the step is decoration.
The previous commit failed CI on purpose to prove the new `npm test` step is load-bearing rather than decoration. Removed; the suite is back to green. An unwired guard is worse than none: it gets cited as authority while never running. This repo already had 401 TS tests nothing executed.
…ile bumps Review finding on #36, and it is the same class of defect this PR exists to fix. The Test step was added to build-and-verify-dist correctly — that job is a REQUIRED check on main, so a failing suite genuinely blocks. But the step is gated on `steps.changed.outputs.ts`, and the detector grepped only `^(package\.json|ts/)`. This PR touches `.github/workflows/ts-ci.yml` and `package-lock.json`, neither of which matched, so on the head commit every step reported `skipped`: Detect TS/dist-relevant changes -> success Install / Typecheck / Test / Rebuild dist -> skipped No TS/dist changes — required check trivially satisfied -> success i.e. the commit that adds the test step had that step skipped, and the PR was green without ever running the 401 tests it wires up. Two real holes: - a dependency bump can land entirely in package-lock.json, and that is precisely the change most likely to break the suite; - a PR editing only ts-ci.yml skipped everything, so the workflow defining this gate was the one change the gate never ran on. Widened the detector to package(-lock)?.json, ts/, and ts-ci.yml itself. The escape hatch the file header describes is deliberately preserved — verified by truth table that Rust-only, README-only and ci.yml-only PRs still resolve to ts=false, so the required context still reports instead of hanging "expected". The proof that the step has teeth is already in this branch's history: 4ef89e3 added a deliberately failing test and build-and-verify-dist went red; 2afaeb1 removed it. Re-verified independently against the check-runs API.
87eea61 to
83fb5ac
Compare
|
Review pass — one real finding, fixed in this branch. The Test step was added to the right job. The teeth proof in this branch's history is genuine — verified independently rather than taken on trust. Commit Finding: the step was skipped on this PR's own head commit. The The commit that wires up the test suite had that step
Widened to Also: the Rebased onto post-#35 |
|
@copilot review |
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds TypeScript test execution to the existing TS CI workflow so the currently-defined npm test suite (401 tests) is enforced on relevant PRs, closing a CI gap where only typecheck/build ran.
Changes:
- Expand the path filter to include
package-lock.jsonand the workflow file itself so TS CI runs when dependencies or its own gating logic changes. - Add a
npm teststep (guarded by the existingsteps.changed.outputs.ts == 'true'condition) before rebuildingdist. - Update the “trivially satisfied” message to reflect the widened path filter.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Must include package-lock.json and THIS WORKFLOW, not just package.json and ts/**: | ||
| # - a dependency bump can land entirely in the lockfile, and that is exactly the | ||
| # change most likely to break the suite; | ||
| # - a PR editing only ts-ci.yml previously skipped every step, so the workflow that | ||
| # defines this gate was the one change the gate never ran on. This PR is that | ||
| # shape: before this line was widened, the commit adding the Test step below had | ||
| # it reported as `skipped`. | ||
| if git diff --name-only "$base" "$head" \ | ||
| | grep -qE '^(package(-lock)?\.json|ts/|\.github/workflows/ts-ci\.yml)'; then |
There was a problem hiding this comment.
Rejecting — there is nothing to centralize, and the indirection would make the gate harder to audit.
The comment's premise is duplication ("easy to desync from other workflows if similar filters exist"). Checked it:
$ grep -rl "git diff --name-only" .github/workflows/
.github/workflows/ts-ci.yml
(count: 1)
One workflow, one filter, one call site. ci.yml and docker-publish.yml have no changed-files logic at all. There is no second copy to drift from, so extracting it would create the coupling it is meant to prevent rather than remove one.
Two further reasons to keep it inline:
- Auditability. This filter decides whether the required check actually runs. Today that decision is visible in the same file as the steps it gates — which is what let this PR's own defect be spotted (the head commit reported
Test -> skipped). Behind a script indirection, reviewing the workflow no longer shows you what the workflow does. - It is 2 lines, not a subsystem. A checked-in script would need its own tests and its own review discipline to be trustworthy, and would still have to stay in step with the
on: push: paths:list directly above it — which a script cannot own, because that list is workflow syntax.
If a second workflow ever needs the same filter, that is the point to extract it, and the second use case will show what the right interface is. Extracting on a sample of one usually guesses wrong.
I did widen the regex in this PR (package(-lock)?\.json|ts/|\.github/workflows/ts-ci\.yml) and verified by truth table that Rust-only, README-only and ci.yml-only PRs still resolve to ts=false, so the required context keeps reporting instead of hanging "expected".
Reviewed the current head. No issues found. Test step — placed between Typecheck and Rebuild dist (lines 83–85), gated with Path filter — Test file coverage —
LGTM. |
The gap
The TypeScript suite was never executed by any workflow.
.github/workflows/ci.yml:28runscargo test --workspace— Rust only..github/workflows/ts-ci.ymlrannpm run typecheck,npm run build, and ats/diststaleness check. It contained zero references tonpm test.package.jsonhas defined"test": "node --import tsx --test ts/src/*.test.ts"the whole time, and that suite is currently 401 passing. So a TypeScript regression could merge with every required check green.Typecheck proves the code compiles. It does not prove it works. The 12 new regression tests guarding the two CodeQL fixes in #34 would have been unenforced the moment they merged — tests that exist to stop a security regression, watched by nothing.
The change
One step, in the job that already has node and
npm install, under the sameif: steps.changed.outputs.ts == 'true'guard as every other step:Path-filter behaviour is unchanged — a PR touching no
package.jsonorts/**still short-circuits to "required check trivially satisfied" without ever setting up node, so this stays a no-op for non-TS PRs. Cost on TS PRs is the ~5s suite on a runner that is already warm.Placed before
Rebuild distdeliberately: a logic failure is then reported as a test failure, rather than surfacing later as a confusingts/distdiff.Teeth proven, both directions
I did not want to ship a step that might be decoration, so this branch first carried a deliberately failing assertion:
4ef89e35— deliberate failing test present2afaeb17— failing test removedIf CI had stayed green with that test present, the step would not have been wired and the whole change would have been theatre. The temporary test is gone; only the workflow line remains.
Why this matters beyond the one line
This is the eighth instance this week of the same shape in this estate: a correct artifact with nothing connected to it. A lifecycle FSM with no scheduler, a governance library with no callers, an engine-version guard invoked by nothing (and cited as authority while it had never run), five engine releases merged but never shipped, a
tscred under a green suite, a daily backup that never ran once, a rollback executor with no CLI behind a|| true— and now 401 tests no job executed.In each case the artifact was right, the connection was missing, and every gate that inspected the artifact reported green. The question worth asking before building anything here is "what calls this?"