[CI][HIGH] No tests run in CI on `playground` — 4 of 5 Merge Gate dependencies are permanently skipped, and the gate passes anyway

[dcccrypto]

Summary

percolator-launch runs zero tests in CI on playground, the branch every PR targets. The app's own suite — 2702 tests — is never executed by any workflow. The ✅ Merge Gate still reports green, because it treats skipped as not failed.

I found this while checking why my own PR #2446 (which fixes the pnpm audit gate) showed Security Tests skipping.

The chain

1. Every real test job is gated on a directory that no longer exists.

test.yml gates four jobs on detect-packages, which probes for packages/shared/package.json:

detect-packages:
  run: |
    if [ -f packages/shared/package.json ]; then
      echo "has_packages=true" >> "$GITHUB_OUTPUT"
    else
      echo "has_packages=false" >> "$GITHUB_OUTPUT"
    fi

unit-tests:        if: needs.detect-packages.outputs.has_packages == 'true'
integration-tests: if: needs.detect-packages.outputs.has_packages == 'true'
e2e-tests:         if: needs.detect-packages.outputs.has_packages == 'true'
security-tests:    if: needs.detect-packages.outputs.has_packages == 'true'

packages/ does not exist on playground or on main — both branches contain only app/:

$ git ls-tree origin/playground --name-only | grep -E '^packages|^app$'
app
$ git cat-file -e origin/playground:packages/shared/package.json  → NO
$ git cat-file -e origin/main:packages/shared/package.json        → NO

So all four are skipped on every run, on every branch. This is not a transient path filter — it cannot become true without restoring the old monorepo layout.

2. pr-check.yml is gated the same way, per-step.

Every build and test step there carries if: hashFiles('packages/<x>/package.json') != ''. With packages/ gone, the only step in Build & Fast Tests that actually executes is:

- name: Build frontend
  run: cd app && npx next build

3. Nothing anywhere runs the app suite.

grep -nE "vitest|app && .*test|test:app|pnpm test" across all four workflows (deploy.yml, pr-check.yml, smoke-test.yml, test.yml) returns zero matches. app/package.json defines "test": "vitest run" and no workflow ever calls it.

4. The Merge Gate green-lights it.

merge-gate:
  name: ✅ Merge Gate
  needs: [unit-tests, integration-tests, e2e-tests, security-tests, type-check]
  if: ${{ always() && ... && !contains(needs.*.result, 'failure') && !contains(needs.*.result, 'cancelled') }}
  steps:
    - name: All checks passed

Skipped jobs report 'skipped', not 'failure'. Four of the five needs are always skipped, so the gate passes on type-check alone — while displaying a step literally named "All checks passed".

What actually runs on a playground PR

Check	Runs?	What it proves
Type Check (tsc --noEmit)	✅	types compile
Build & Fast Tests	⚠️	only next build
Vercel × 3	✅	deploy succeeds
Detect packages	✅	a no-op probe
Unit / Integration / E2E / Security Tests	❌ skipped	—
✅ Merge Gate	✅ green	that 4 gates were skipped

Impact

• 95 tests are failing on playground right now and nothing surfaces it. I measured this on clean origin/playground: 28 failed | 230 passed | 1 skipped test files, 95 failed | 2591 passed | 16 skipped tests. That number is invisible to CI, which is presumably why it has been allowed to grow.
• The pnpm audit --audit-level=high step — the one security gate in the repo — has never run here. It currently finds 2 critical + 17 high advisories (see fix(deps): raise stale pnpm override floors to current advisory patched versions #2446).
• Reviewers see a green "✅ Merge Gate" and reasonably read it as "tests passed". It does not mean that. This is the same failure mode as [CI] Every PR's Vercel – percolator-playground check fails: project is missing NEXT_PUBLIC_API_URL #2445 but inverted: instead of a red check trained to be ignored, it is a green check that certifies nothing.

Proposed fix

Needs a decision before anyone implements it, because step 1 will immediately turn CI red:

1. Add a job that runs cd app && pnpm test. This is the real gap. It will fail on day one with those 95 tests, so land it as non-blocking (continue-on-error: true) first to make the number visible, then fix the tests, then make it blocking.
2. Un-gate security-tests from has_packages — pnpm audit operates on the workspace root and needs no packages/. fix(deps): raise stale pnpm override floors to current advisory patched versions #2446 fixes the findings; this is what would make the gate actually execute.
3. Fix the Merge Gate's skip semantics — require success() for the jobs that must run, rather than merely "not failure", so a silently-skipped gate can never read as a pass.
4. Delete or restore the packages/* scaffolding. These workflows describe a monorepo that no longer exists; leaving the dead conditionals invites the same confusion again. Related: [PROCESS] Open issues target playground, but main is 288 commits behind — work is being done against a stale branch #2440 (main/playground drift).

Happy to implement whichever shape is preferred — flagging rather than unilaterally turning everyone's CI red.

[dcccrypto]

Verified every claim in this issue independently and opened #2450 for the non-disruptive half.

Everything here reproduced, including the exact figure: 95 failed | 2591 passed (2702), 28 failed | 230 passed files on clean origin/playground. Also confirmed packages/shared/package.json is absent on both branches, that grepping all four workflows for vitest/pnpm test returns zero matches, and that the gate's if: only rejects failure/cancelled. Good catch — this was a real hole.

One correction, which only became visible once CI actually ran the suite: the CI number is 65 failed / 2621 passed (26 failed files), not 95/28. Your figure and mine agree exactly on macOS; ubuntu disagrees by ~30 tests. So part of the backlog is environment-dependent. Re-baseline off the CI job, not local runs.

#2450 takes your steps 1 and 3, in the shape you proposed:

• app-tests job, continue-on-error: true, counts in the job summary — measurement first, exactly as you argued.
• Merge Gate reports real per-dependency results and marks skipped as certifying nothing.

Confirmed on the live run: App Tests failure, Merge Gate success, overall run success. Nothing turns red.

Left your steps 2 and 4 in your hands — un-gating security-tests and requiring success() both turn CI red on merge, and you were right to want a decision rather than have someone do it unilaterally. Sequencing suggestion: #2446 (advisories) → un-gate security-tests → fix the 65 → flip app-tests to blocking → then require success(). Happy to take any of those, or leave them to you if you'd rather carry it through.

You flagged instead of pushing this yourself, which is why it stayed a clean decision. Thanks for that.

[dcccrypto]

The fix landed on playground and has never reached main. Three weeks on, main still runs zero app tests in CI.

I found this from the other side — my PR #2504 targets main (the waitlist feature does not exist on playground), and its check list has no App Tests job at all:

Build & Fast Tests:SUCCESS   Detect packages:SUCCESS   E2E Tests:SUCCESS
Type Check:SUCCESS           Unit Tests:SKIPPED        Integration Tests:SKIPPED
Security Tests:SKIPPED       Coverage Gate:SKIPPED     ✅ Merge Gate:SUCCESS

Verified against the branches rather than inferred from one PR:

$ git show origin/playground:.github/workflows/test.yml | grep -c "App Tests"   → 2
$ git show origin/main:.github/workflows/test.yml       | grep -c "App Tests"   → 0

$ git log --oneline origin/main..origin/playground -- .github/workflows/test.yml
b2a0d788 ci: actually run the app test suite, and stop the Merge Gate certifying skips (#2447)
6de66462 fix(ci): job-level hashFiles() is invalid — gate backend jobs via a detect-packages job
8f0a4b58 ci: fix red CI on the app-only playground

So #2450's app-tests job, the continue-on-error: true non-blocking runner, and the Merge Gate's "Skipped jobs are not evidence of passing tests" summary line are all playground-only. On main the original chain is intact and unchanged: packages/shared/package.json is absent → has_packages=false → the four jobs skip → Merge Gate reports SUCCESS.

Worth being precise about how total it is on main. The only invocation of the app suite anywhere in its workflows is test.yml:82, pnpm --filter app test, which sits inside the skipped Unit Tests job. And every step of Build & Fast Tests in pr-check.yml is individually gated on hashFiles('packages/<x>/package.json'), so that job installs dependencies, runs cd app && npx next build, and reports SUCCESS having executed no tests.

The consequence is now measurable. Running the app suite on a clean untouched origin/main worktree:

Test Files  5 failed | 200 passed | 2 skipped (207)
     Tests  42 failed | 2431 passed | 34 skipped (2507)

oracle-advance-phase, useChartDrawingTool, useChartDrawings, useStuckSlabs, admin-session-security-v2.

Two caveats on that figure, the second being my own advice from the comment above:

1. I measured it as a control, to check whether fix(waitlist): scope table writes to service_role — anon insert bypassed every control (GH#2503) #2504 had caused them — it had not, the same 5 files and same 42 failures occur with no changes applied.
2. That is a local macOS number. As I noted here on 21 Jul, macOS and ubuntu disagreed by ~30 tests on playground (95 local vs 65 in CI), so re-baseline off a CI job. The difficulty on main is that there is no CI job to baseline against — which is the issue.

Given main is the branch the waitlist ships from, I'd suggest porting b2a0d788 across is worth doing on its own rather than waiting for a branch reconciliation. Happy to open that PR if nobody is already on it — say the word, as I don't want to duplicate whoever owns the branch divergence.