RE001 regressed by #773: a "documentation-only" PR deleted the quoted-uses and block-scalar handling — E2E Elixir job red on main #826

Summary

E2E — Elixir Scanner Pipeline (tests.yml) is red on main @ 54f14dad.
mix test reports 2 failures of 1581. Both are the same rule,
Hypatia.Rules.ResearchExtensions.re001_missing_harden_runner/1, and they fail
in opposite directions — one false positive, one false negative.

The cause is PR #773 (3f30047, 2026-09-12), which deleted 104 lines of
structural job analysis and replaced it with two flat regexes — while its own
description states "Validation was not run; this is a documentation-only
change." It is not documentation-only. It reverted the logic of two earlier
bug fixes without touching the tests that guard them.

This failure is newly visible, not newly introduced. See the timeline below.

The two failures

Reproduced locally at 9de66da (content-identical to main for this tree)
with mix test: 1593 tests, 2 failures.

1. False positive — quoted uses: is not recognised as harden-runner

test/research_extensions_test.exs:100, "passes when harden-runner uses values are quoted"

code:  assert ResearchExtensions.re001_missing_harden_runner(repo) == []
left:  [ %{rule: "RE001", line: 6, ...}, %{rule: "RE001", line: 11, ...} ]
right: []

The fixture installs harden-runner as - uses: "step-security/harden-runner@main"
and - uses: 'step-security/harden-runner@main'. The detector at
lib/rules/research_extensions.ex:199-200 is:

Regex.match?(~r/^\s+(?:-\s+)?uses:\s*step-security\/harden-runner@/m, active_content)

\s* is followed by " or ', not s, so the match fails and the rule fires
on a job that is hardened. Both jobs report ⇒ 2 findings where 0 are correct.

2. False negative — text inside a block scalar is read as configuration

test/research_extensions_test.exs:120, "does not treat nested multiline values as runner configuration or hardening"

** (MatchError) no match of right hand side value: []
code: [finding] = ResearchExtensions.re001_missing_harden_runner(repo)

The fixture puts the literal text runs-on: ubuntu-latest and
- uses: step-security/harden-runner@main inside an env: block scalar in
one job and inside a run: | block scalar in another. Because the detector is a
flat regex over the job's raw text, it reads both as real workflow keys:

• job nested-values — has no real runs-on:; the regex finds the one inside
  the env: scalar, so local_runner? is wrongly true.
• job exposed — has a real runs-on: and a real secrets.EXPOSED, but no
  real harden-runner step; the regex finds the one inside the run: | scalar,
  so installs_harden? is wrongly true and the finding is suppressed.

Result []; exactly one finding (the exposed job) is correct. A genuinely
unhardened job that references a secret is silently passed because a comment,
example or heredoc elsewhere in the job mentions harden-runner.

Root cause: what #773 deleted

git show 3f30047 -- lib/rules/research_extensions.ex — 10 insertions,
104 deletions. Eight private functions were removed:

direct_job_field?/2      harden_runner_step?/1
job_field_indent/1       lines_in_steps/2
step_indent/2            harden_runner_uses?/1
block_scalar?/1

Two of those are precisely the earlier fixes, by name and by purpose:

Both fixes landed 2026-09-10 with the tests that now fail. #773 removed the
implementations and left the tests, so the tests correctly caught the
regression — nothing ran them.

The replacement also discards indentation awareness (direct_job_field? checked
that runs-on: sat at the job's own field indent; ^\s+runs-on: matches at any
depth), so the reusable-workflow exemption is weaker than documented too.

Timeline — why nobody saw it

Measured from actions/runs/<id>/jobs (job count, not just conclusion):

So it was visibly red on 09-16 and 09-17 only, and then disappeared behind
two distinct startup deaths back to back:

1. 09-17 → 09-18, recorded honestly as conclusion: startup_failure;
2. 09-19 onwards, after tests.yml acquired the duplicate with: key (fix(ci): pin third-party actions to full commit SHAs #802) —
   and ⚠ these record conclusion: failure, not startup_failure. The only
   tell is jobs = 0. A run list alone cannot distinguish this second death
   from a genuine test failure, which is exactly why the red looked continuous. fix(ci): remove the duplicate 'with:' block from six workflows (#822) #824 removed the duplicate key
   and the job runs again, which is why this surfaces today.

Secondary defect: the failure is undiagnosable from CI

tests/e2e.sh:145-152 runs mix test --trace, captures it, and on failure
prints:

echo "--- mix test output (tail 120) ---"
tail -120 "$_mix_log"

That block carries a comment explaining it is itself a fix for an earlier
version that discarded the output entirely. It is still insufficient: ExUnit
prints failure diagnostics inline, as they occur, and --trace emits two
lines per test across 1581 tests, so the last 120 lines contain only the tail of
the trace plus the summary line. Neither failure block appears in the CI log
for any of the runs above — grep -E '^\s*[0-9]+\) test ' over the full job
log returns nothing. The identity of the two failures above was obtained by
reproducing locally, not from CI.

Acceptance criteria

• re001_missing_harden_runner/1 recognises uses: values that are
  unquoted, single-quoted and double-quoted.
• It ignores runs-on: and uses: text appearing inside block scalars
  (|, >, and their chomping/indent variants) in env:, run: or any
  other value.
• runs-on: is recognised only at the job's own field indentation, so the
  reusable-workflow exemption matches the documented behaviour.
• Both tests at test/research_extensions_test.exs:100 and :120 pass, and
  the remaining RE001 tests stay green.
• Mutant: re-introduce the #773 regex form and confirm both tests go
  red again — a passing suite proves nothing until the mutant dies.
• tests/e2e.sh surfaces the actual failure on a red run: print the ExUnit
  failure blocks (or drop --trace so the tail contains them, or upload the
  full log as an artifact). Verify by planting a failing test and reading the
  failing job's log for its assertion text.
• E2E — Elixir Scanner Pipeline reports jobs > 0 and success on
  main.

Note on the fix

The straightforward repair is to restore the eight helpers #773 deleted while
keeping its documentation wording, since the prose change was the stated intent
and is independently fine. The logic hunk is the whole regression.

Related: #814 (the other 10 newly-visible reds — a testcontainers resolver
conflict, a different cause). Not a merge blocker per the standing ruling that a
new finding becomes an issue with acceptance criteria.

🤖 Generated with Claude Code

https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV