ci-local.sh misses seven scripts CI runs, and skips a required check without saying so

[vladimirrott]

Re-measured 2026-09-09 at 5673d20. Both totals moved again, and the
drift is the same seven scripts. Twenty local, twenty-seven in CI, which is
every *.test.sh on disk. CONTRIBUTING.md moved from :182 to :212.

$ grep -oE 'tests/(release|e2e)/[a-z0-9-]+\.test\.sh' scripts/ci-local.sh | sort -u | wc -l
20
$ grep -rhoE 'tests/(release|e2e)/[a-z0-9-]+\.test\.sh' .github/workflows/ | sort -u | wc -l
27
$ ls tests/release/*.test.sh tests/e2e/*.test.sh | wc -l
27

CONTRIBUTING.md:212 asks for one thing before a PR:

Run scripts/ci-local.sh. A compile error in a crate your machine cannot build is still a compile error, and finding it costs a review round.

A green ci-local.sh does not mean a green board, in two independent ways.

1. The script list drifted from what CI runs

run_hygiene_group names twenty test scripts by hand (ci-local.sh:265-285).
CI runs twenty-seven. Seven run in CI and nowhere locally:

cassette-replay-parity.test.sh
codex-plugin-manifest.test.sh
grub-kargs-edit.test.sh
log-edit.test.sh
mount-edit.test.sh
no-secrets.test.sh
rmswap.test.sh

Four of those seven guard privileged helper scripts under packaging/, which is
the surface where a mistake costs the most.

This is the defect class this tracker keeps finding: a hand-maintained list that
drifted from what is on disk. STORIES=(1 2 3 4 5 6 7 8 9 10 11) against twelve
exec-*.sh files (#330) and six claim files against sixteen were both this.

2. A required check skips silently

run_postgres_contract_group records SKIP when neither docker nor podman is
installed:

if [[ -z "$runtime" ]]; then
    record SKIP "${label} (no SYSKNIFE_TEST_POSTGRES_URL and no docker/podman found)"
    return
fi

postgres-contract is one of the five required checks. A contributor with no
container runtime gets an all-green local board and a red required check on push,
with nothing in the local output saying which gate did not run.

Why it matters

Both failures point the same way: the contributor did what CONTRIBUTING.md
asked, believes the board is green, and finds out from CI. That is a review round
per occurrence, and it lands on the people least able to absorb it.

Scope

• Replace the hand-written list with a glob over tests/release/*.test.sh and
  tests/e2e/*.test.sh, with an explicit skip list for anything deliberately
  excluded and a comment saying why. A new test then runs locally the day it
  lands, rather than the day someone remembers.
• Make the Postgres skip loud: print, in the final summary, that a required
  check did not run and name the two ways to satisfy it
  (SYSKNIFE_TEST_POSTGRES_URL, or install podman). A SKIP line in the middle
  of a long run is not a warning.
• Consider trying podman before docker. Docker's published ports reset the
  connection on at least one maintainer machine, so the current preference order
  picks the runtime that does not work and reports failure rather than falling
  through.

This overlaps #331, which asks for a guard that every tests/release/*.test.sh
is invoked by a gate. That one is about CI; this one is about the command
contributors are told to run. Doing #331 first makes the glob here the obvious
implementation, so they are worth reading together.

Tests first

A guard that lists both sets and fails when they differ:

comm -13 <(grep -o 'tests/\(release\|e2e\)/[a-z-]*\.test\.sh' scripts/ci-local.sh | sort -u) \
         <(grep -ro 'tests/\(release\|e2e\)/[a-z-]*\.test\.sh' .github/workflows/ | cut -d: -f2- | sort -u)

Prove it bites by deleting one run_step line and watching it name that file.
Assert the extraction found something first: a regex that matches nothing makes
comm report no difference and the guard passes over an empty comparison, which
is the same vacuity the guard exists to prevent.

Difficulty

medium. The glob is short. Deciding what belongs in the skip list, and proving
the guard cannot pass over an empty set, is the work.

Getting started

CONTRIBUTING.md
has the build and test commands. No CLA and no copyright waiver. The project is MIT.

[vladimirrott]

@xianjianlf2 #346 is close to what you did in #316.

That PR took provider keys out of ssh argv, where any local user could read them from /proc/*/cmdline. This one is the same kind of gap on the contributor side: CONTRIBUTING.md tells people to run scripts/ci-local.sh before opening a PR, and that script names its seventeen test scripts by hand while CI runs twenty-four. Four of the seven it misses guard the privileged helpers under packaging/, so the surface where a mistake costs most is the surface nobody checks locally.

The second half is the one I would look at first. postgres-contract is a required check, and ci-local.sh records a mid-run SKIP for it when no container runtime is installed. A green local board and a red required check on push, with nothing loud enough to explain why.

Labelled medium. Say so here and it is yours.

[vladimirrott]

Re-measured on 5a9450a. The divergence set is exactly the seven scripts you listed. Both totals have moved by one:

	issue body	5a9450a
named in ci-local.sh	17	16
referenced by a workflow	24	23
on disk	not stated	23

$ grep -o 'tests/\(release\|e2e\)/[a-z0-9-]*\.test\.sh' scripts/ci-local.sh | sort -u | wc -l
16
$ grep -ro 'tests/\(release\|e2e\)/[a-z0-9-]*\.test\.sh' .github/workflows/ | cut -d: -f2- | sort -u | wc -l
23
$ ls tests/release/*.test.sh tests/e2e/*.test.sh | wc -l
23

Your exact [a-z-]* regex gives 16 and 23 as well, so the body's counts came from a tree with one more of each rather than from a different pattern. The seven-script set is unchanged: cassette-replay-parity, codex-plugin-manifest, grub-kargs-edit, log-edit, mount-edit, no-secrets, rmswap.

Three things for the scope.

The drift is one-sided. ci.yml references all 23 on-disk scripts and carries no paths: filter, so comm -23 of disk against ci.yml is empty and every script runs on every PR. ci-local.sh is the only short list, which makes the glob a strict improvement with nothing to reconcile on the CI side.

registry-manifest.test.sh reads as unwired if you grep the workflows and de-duplicate by filename, because it appears in publish-mcp.yml:89 and release.yml:52 as well as ci.yml:57. It is wired. Worth knowing before the skip list gets written against that reading.

PR #341 adds tests/e2e/story-runner-verdicts.test.sh and wires it into ci.yml and ci-local.sh in the same commit, which is what the glob would stop depending on an author to remember.

On the Postgres half: postgres-contract is not vacuous, and both tokens are load-bearing. Measured here without a database reachable, at 5a9450a:

SYSKNIFE_REQUIRE_POSTGRES=1, --include-ignored   -> running 11 tests, 5 passed; 6 failed   rc=101
--include-ignored only                           -> running 11 tests, 11 passed; 0 failed  rc=0
URL set, SYSKNIFE_REQUIRE_POSTGRES=1, no flag    -> running 11 tests, 6 passed; 5 ignored  rc=0

The second and third lines are the two false-green shapes (#313 and #315). Both print running 11 tests, so that string does not discriminate between a real run and a skipped one. 0 ignored in the test result line is the literal that does. Worth putting in the loud-skip message this issue asks for, so a contributor reading local output can tell which of the two they got.

[vladimirrott]

Re-measured at 60af0adb. Both counts moved by one since the last check, and the
seven-script divergence set is unchanged.

	issue body	5a9450a	60af0adb
named in ci-local.sh	17	16	17
referenced by a workflow	24	23	24
on disk	not stated	23	24

$ comm -23 <(ls tests/release/*.test.sh tests/e2e/*.test.sh | sort) \
           <(grep -oE '(tests/(release|e2e)/[a-z0-9-]+\.test\.sh)' scripts/ci-local.sh | sort -u)
tests/release/cassette-replay-parity.test.sh
tests/release/codex-plugin-manifest.test.sh
tests/release/grub-kargs-edit.test.sh
tests/release/log-edit.test.sh
tests/release/mount-edit.test.sh
tests/release/no-secrets.test.sh
tests/release/rmswap.test.sh

The new script is tests/e2e/story-runner-verdicts.test.sh from #341, and it
landed wired into both ci.yml and ci-local.sh, so the gap did not widen.
Nothing on disk is missing from a workflow:

$ comm -23 <(ls tests/release/*.test.sh tests/e2e/*.test.sh | sort) \
           <(grep -rhoE '(tests/(release|e2e)/[a-z0-9-]+\.test\.sh)' .github/workflows/*.yml | sort -u)
(no output)

Worth folding into the scope here: the hygiene script list is one of two
divergences. The postgres-contract job runs two cargo test commands and
ci-local.sh runs one of them, so a contributor with podman installed
reproduces five of the six live tests. That half is #362, with a patch.

A glob fixes the seventeen-versus-twenty-four count. It will not catch the
cargo test commands, because those are not *.test.sh files. Whoever takes
this is worth pointing at #362 so the two are not solved twice.

[vladimirrott]

Follow-up on the third scope bullet, the docker-before-podman preference. I got a
container runtime working on this machine and reproduced it, so it is measured
rather than suspected.

The machine has both CLIs. Only podman works:

$ command -v docker; command -v podman
/usr/local/bin/docker
/usr/bin/podman
$ docker info >/dev/null 2>&1; echo rc=$?
rc=1

have docker is true because the CLI is on PATH, so
run_postgres_contract_group picks docker and stops there:

### postgres-contract (optional)

==> postgres-contract: starting postgres:17-alpine via docker
failed to connect to the docker API at unix:///home/entropia/.docker/desktop/docker.sock; check if the path is correct and if the daemon is running: dial unix /home/entropia/.docker/desktop/docker.sock: connect: no such file or directory

Then record FAIL. It never tries podman, which is installed and works.

Swapping the two lines in the if have docker / elif have podman block, and
changing nothing else, runs the whole contract:

### postgres-contract (optional)

==> postgres-contract: starting postgres:17-alpine via podman
==> postgres-contract: cargo test -p sysknife-daemon --test postgres_store --locked -- --include-ignored
running 11 tests
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.02s

podman 4.9.3, rootless, postgres:17-alpine on 127.0.0.1:5433, container
started and removed by the script.

The developer this bites has a docker CLI left over from a Docker Desktop
install and a working podman alongside it, which is a common state on Ubuntu.
Probing whether the runtime answers (docker info, podman info) beats probing
whether its binary exists, and it makes the preference order stop mattering.

11 passed; 0 ignored is the line that says the contract ran. 0 ignored
is the part to check: a run can print running 11 tests and still have
exercised nothing (#315).

[vladimirrott]

@be-student held this for you as a follow-up to your #376. It is the same ci-local.sh you were in, one layer up: it misses seven scripts CI runs and skips a required check silently. No pressure, and no deadline. Reply here that you want it and it is formally yours.

[vladimirrott]

Re-measured at 049bc1f. Both totals moved by one again since 60af0adb, and
the seven-script divergence set has not changed since the issue was written.

	issue body	5a9450a	60af0adb	049bc1f
named in ci-local.sh	17	16	17	18
referenced by a workflow	24	23	24	25
on disk	not stated	23	24	25

$ ls tests/release/*.test.sh tests/e2e/*.test.sh | sort > /tmp/ns-audit/disk.txt; wc -l < /tmp/ns-audit/disk.txt
25
$ grep -oE 'tests/(release|e2e)/[A-Za-z0-9_.-]+\.test\.sh' scripts/ci-local.sh | sort -u > /tmp/ns-audit/local.txt; wc -l < /tmp/ns-audit/local.txt
18
$ grep -rhoE 'tests/(release|e2e)/[A-Za-z0-9_.-]+\.test\.sh' .github/workflows/ | sort -u > /tmp/ns-audit/wf.txt; wc -l < /tmp/ns-audit/wf.txt
25
$ comm -13 /tmp/ns-audit/local.txt /tmp/ns-audit/wf.txt
tests/release/cassette-replay-parity.test.sh
tests/release/codex-plugin-manifest.test.sh
tests/release/grub-kargs-edit.test.sh
tests/release/log-edit.test.sh
tests/release/mount-edit.test.sh
tests/release/no-secrets.test.sh
tests/release/rmswap.test.sh

Nothing arrived unwired. The one new script since the last count,
tests/release/markdown-link-files.test.sh from #376, landed in ci.yml and in
ci-local.sh on the same commit:

$ comm -23 /tmp/ns-audit/disk.txt /tmp/ns-audit/wf.txt
$ comm -23 /tmp/ns-audit/disk.txt <(cat /tmp/ns-audit/local.txt /tmp/ns-audit/wf.txt | sort -u)

Both empty.

One thing worth knowing before you start on the glob, because it makes the case
sharper. I audited no-secrets.test.sh this week, one of the seven, and it has
a hole: deleting the Slack pattern from scripts/check_no_secrets.sh leaves the
gate green. Details and a patch are in #387. Nothing
for you to do there, and the two do not touch the same lines, but it is the
concrete answer to "why does it matter that seven scripts run only in CI": those
are the seven nobody exercises while they are working on them.

[vladimirrott]

@be-student releasing this reservation.

#346 was already held by another contributor when I reserved it for you on 7 September, and I told neither of you. That is mine to own: the tool that was supposed to prevent it reported how many issues were free without saying which, so the offers got picked by hand.

You have #377 open with a review on it, and #369 stays reserved for you, which is the one that actually matches what you have been landing here. Nothing lost.

[vladimirrott]

@xianjianlf2 checking in, and nothing hangs on it. I offered this on 2 September and have not seen a branch. The issue is not going anywhere.

Two things that might be what stalled it.

Both counts moved again today, and the drift did not. run_hygiene_group names twenty scripts by hand, CI runs twenty-seven, which is every *.test.sh on disk, and the seven that run in CI and nowhere locally are the same seven names as when I filed it. The body carries today's figures and the commands behind them. A count that no longer matches the body is my drift rather than your mistake.

The one open decision is answered. The third scope bullet asked whether ci-local.sh should keep preferring docker over podman; I answered that on this thread on 3 September, so nothing waits on me.

It stays yours. I dropped the claimed label because you never claimed it, and the label was telling other contributors something you had not said. The issue is still available to you, and handing it back costs nothing.

[vladimirrott]

@xianjianlf2 freeing this one, and nothing is owed.

I offered it on 2 September and have not heard back. Eight days is where I stop holding an issue silently, because other people read this tracker and a reservation they cannot see costs them.

Say the word and it is yours again. I re-measured the finding today at f5dffd8 and the number in the title is still exact:

CI runs 29 script(s); ci-local mentions 25
in CI but not mentioned anywhere in ci-local.sh:
  tests/release/cassette-replay-parity.test.sh
  tests/release/codex-plugin-manifest.test.sh
  tests/release/grub-kargs-edit.test.sh
  tests/release/log-edit.test.sh
  tests/release/mount-edit.test.sh
  tests/release/no-secrets.test.sh
  tests/release/rmswap.test.sh

Seven, as filed.

[vladimirrott]

@QinXi-ai holding this for you, offered as part of a batch on #417.

Released from @xianjianlf2 this morning with nothing owed. I re-measured it today at f5dffd8 and the seven in the title is still exact:

CI runs 29 script(s); ci-local mentions 25
missing: cassette-replay-parity, codex-plugin-manifest, grub-kargs-edit,
         log-edit, mount-edit, no-secrets, rmswap

You have spent four PRs inside this repo's CI in two days, so you know better than I do which of the seven belong in the local run and which are genuinely CI-only. That judgement is most of the work.

Take it or leave it, and no is a complete answer.