ci-local reports PASS after a root-only assertion silently skipped, and nothing compares how CI invokes a test with how ci-local does

[vladimirrott]

scripts/ci-local.sh prints ci-local: PASS after running a test that skipped the assertion CI runs it for, and says nothing about the difference.

What happens

CI runs one release test with elevated privileges, and only one:

.github/workflows/ci.yml
  - name: Check bounded actions and removal of shell grants
    run: sudo -n bash tests/release/action-steps.test.sh

run_shell_tests discovers the same file by glob and runs every test the same way:

run_step "hygiene: ${test#"$repo_root/"}" bash "$test"

So locally it runs as whoever you are. tests/release/action-steps.test.sh carries

@unittest.skipUnless(hasattr(os, 'geteuid') and os.geteuid() == 0,
                     'real credential-drop test requires a root test subprocess')
def test_real_drop_cannot_regain_root_or_write_through_a_symlink(self):

and that is the case the sudo -n exists for. Unprivileged, it skips:

test_real_drop_cannot_regain_root_or_write_through_a_symlink ... skipped
Ran 11 tests in 0.018s
OK (skipped=1)

run_step sees exit 0, records a pass, and the summary ends:

=========================================
ci-local: PASS

The one assertion that proves the helper cannot regain root or write through a symlink never executed, and nothing in the output says so.

Why this shape and not another

required_skips already exists for exactly this problem, and is used for exactly one thing:

$ grep -n 'required_skips' scripts/ci-local.sh
79:required_skips=()
351:        required_skips+=(postgres-contract)
375:        required_skips+=(postgres-contract)
444:if ((${#required_skips[@]} > 0)); then
445:    printf 'WARNING: REQUIRED CI check(s) did not run: %s\n' "${required_skips[*]}"

(Line numbers recounted at b2c823e6 on 2026-09-16; they were 350, 374, 443 and
444 when I filed this. Everything else below still lands.)

A missing Postgres downgrades the run to INCOMPLETE. A missing root does not, and the reason is that the sudo step arrived after that machinery did.

The part that will not stay one test

Today the class has exactly one member. sudo -n bash tests/ matches one line in the workflows, and one test file carries a root condition. Nothing detects the next one. A second privileged step added to ci.yml gets the same silent gap on the day it lands, and the local run keeps saying PASS.

tests/release/ci-local.test.sh compares the discovered set against the workflow set by path and asserts they match. It does not compare how each side invokes them, which is where the divergence lives.

Suggested shape

Read the invocation out of the workflows rather than hardcoding a list, the way discovery already reads the file set:

• extract tests/(release|e2e)/*.test.sh from .github/workflows/*.yml together with whether the step runs it under sudo
• when ci-local runs one of those without being root, add it to required_skips and let the existing summary do the rest
• extend tests/release/ci-local.test.sh so a new privileged step in a workflow fails the guard until ci-local knows about it

The point is that the guard is derived from the workflow, so it cannot go stale the way a hardcoded list does.

Reproducing

bash tests/release/action-steps.test.sh; echo "rc=$?"
# OK (skipped=1), rc=0

sudo -n bash tests/release/action-steps.test.sh 2>&1 | tail -3
# Ran 11 tests, no skip

Tests first

Break what the guard protects. Add a second privileged step to a fixture
workflow, sudo -n bash tests/release/<something>.test.sh, and require
ci-local run as a non-root user to name it in required_skips and finish
INCOMPLETE rather than PASS. Write it before the fix and watch it print
ci-local: PASS, which is the defect. The existing Postgres path is the worked
example of the behaviour you want:

$ grep -n 'required_skips' scripts/ci-local.sh
79:required_skips=()
351:        required_skips+=(postgres-contract)
375:        required_skips+=(postgres-contract)
444:if ((${#required_skips[@]} > 0)); then
445:    printf 'WARNING: REQUIRED CI check(s) did not run: %s\n' "${required_skips[*]}"

Break the guard's own input. This check has one specific way of being
useless: the workflow scan finds no privileged steps and the run reports no
skips, which is indistinguishable from a clean pass. Assert the count. A fixture
workflow set containing two sudo -n bash tests/... steps must yield two
detections, so a regex that stops matching (sudo --non-interactive, a step
that sets sudo through a shell variable, a run: block spanning lines) turns
red rather than quietly reporting nothing to skip.

Extend tests/release/ci-local.test.sh in the same change, so a privileged step
added to a workflow tomorrow fails the guard until ci-local knows about it.
Today it compares the two sides by path and not by invocation, which is why this
gap exists.

Difficulty

medium. Bash and two workflow readers, no Rust and no VM. The work is deriving
the privileged set from .github/workflows/*.yml rather than hardcoding the one
file that carries it today, and proving the derivation fails loudly when it
selects nothing.

Getting started

CONTRIBUTING.md
has the build and test commands. The two that matter here are
bash scripts/ci-local.sh and bash tests/release/ci-local.test.sh. You do not
need a root password to work on this; you need to run ci-local without one
and watch what it says. No CLA and no copyright waiver. The project is MIT.

[vladimirrott]

Rechecked at b2c823e6. The defect reproduces and four line numbers had moved one or two lines, so I corrected them in the body and added the sections it was missing.

$ grep -n 'required_skips' scripts/ci-local.sh
79:required_skips=()
351:        required_skips+=(postgres-contract)
375:        required_skips+=(postgres-contract)
444:if ((${#required_skips[@]} > 0)); then
445:    printf 'WARNING: REQUIRED CI check(s) did not run: %s\n' "${required_skips[*]}"
$ grep -n 'sudo -n bash tests' .github/workflows/ci.yml
122:        run: sudo -n bash tests/release/action-steps.test.sh
$ grep -n 'run_step "hygiene' scripts/ci-local.sh
280:        run_step "hygiene: ${test#"$repo_root/"}" bash "$test"

Still one privileged step in CI, still discovered and run unprivileged by ci-local, and required_skips still reserved for Postgres. The body now carries Tests first, Difficulty and Getting started, which it never had.

[vladimirrott]

Recording an offer I made somewhere this issue could not see it. I held #433 for @vsolano9 on 15 September, in my review on #432:

One more of the same shape when #407 closes, and I will hold it for you. #433 is a ci-local run reporting PASS after a root-only assertion skipped itself, with nothing comparing what ran against what was meant to run. That is your floor argument moved one layer up. Say the word and it is yours.

That comment lives on a pull request thread, so nothing here recorded it, and my own tooling listed #433 as free this morning. One line on the issue itself fixes that for everyone reading the tracker, me included.

@vsolano9 no deadline attached. Say the word and I will assign it; say nothing and it stays held for you.

Still reproduces at 30b9e315, so the body needs no correction:

$ grep -n 'sudo -n bash tests' .github/workflows/*.yml
.github/workflows/ci.yml:122:        run: sudo -n bash tests/release/action-steps.test.sh
$ out="$(bash tests/release/action-steps.test.sh 2>&1)"; rc=$?; printf '%s\n' "$out" | tail -4; echo "rc=$rc"
----------------------------------------------------------------------
Ran 11 tests in 0.016s

OK (skipped=1)
rc=0

CI runs that file under sudo -n and reaches eleven tests. A contributor runs it as themselves, skips the credential-drop assertion, and ci-local prints PASS over the gap.