Repository navigation
Every ExUnit test is dead in CI: the only mix test sits behind a needs: edge to a failing job #836
Description
The finding
Every ExUnit test in this repository is dead in CI. Not failing — never
executed, while the workflow, the job and the check-run all still appear.
There is exactly one mix test invocation in the whole tree:
.github/workflows/tests.yml:347 run: mix test --include integration
It lives in the integration-tests job, which declares needs: [build-images].
Build Test Images fails. A needs: edge to a failing job does not fail the
dependent job — it makes it skipped.
Measured on the run for 3bf9674a:
Build Test Images failure
Integration Tests skipped <- the only home of mix test
Why this matters more than a red suite
A red suite is visible. A needs:-skipped job looks like it participated: it is
listed, it has a name, it reports a conclusion. And a skipped conclusion
satisfies a required status check, so this can sit behind a green ruleset
indefinitely.
Concretely, it already has: test/unified-api-adapter-contract_test.exs is the
ABI wire-contract guard that #120 repointed from the dead src/abi/Types.idr
onto the live src/Hypatia/ABI/Types.idr, closing instance 21 of the
guard-reads-the-wrong-thing trap. That fix is correct in the tree and vacuous
in CI, and nothing anywhere said so.
Note also that mix test excludes 242 :verisim_data tests unconditionally
(test/test_helper.exs), so even once it runs, the denominator is ~83%.
Acceptance criteria
Integration Testsreaches conclusionsuccesson a run againstmain— not
skipped, notcancelled. Either fixbuild-images, or movemix testout
from behind it into a job with no dependency on image building. The Elixir
unit suite does not need ArangoDB; only the:integration-tagged subset does.- The fix is proven by a mutant: break one assertion in
test/unified-api-adapter-contract_test.exs(e.g. change one expected
connector name), push, and confirm the check goes red. Revert and confirm
green. A suite that cannot be made to fail is not evidence it runs. - The run prints its denominator — the
N tests, M failuresline must appear
in the log withN > 0. A zero-test run exits 0 and reads as success. - A workflow-level assertion that no job carrying a test runner can resolve
skippedsilently: either drop theneeds:edge, or add a final job with
if: always()that fails whenneeds.integration-tests.result == 'skipped'. - State plainly in the job or the README whether the 242
:verisim_datatests
are in scope. If they stay excluded, the exclusion is named, not implicit.
Probe for re-checking this class of fault anywhere
git grep -n 'mix test\|cargo test\|zig build test\|bun test' -- .github/workflows
gh run view <run-id> --repo O/R --json jobs --jq '.jobs[]|"\(.conclusion)\t\(.name)"' | sortAsk "did it execute", never "does it exist".
Found while preparing #835.
Activity
hyperpolymath commentedon Sep 22, 2026
Cured by #842 (
450734eonmain) -- as a side effect, not by direct action.This issue's mechanism was a
needs:edge fromintegration-teststo a failingbuild-images, making the onlymix testin the repo resolve toskipped.#842 cured the
actions.lockfault that hadtests.ymlstartup-dead (#837);with the workflow alive,
Build Test Imagespasses, so the edge is nolonger broken and the dependent job runs.
Measured, run
35770348007, jobIntegration Tests(success)Step
Run Elixir integration tests->success. The step ranmix test --include integration:1593 tests executed. Per the standing rule that a green suite proves nothing
without its denominator, that number -- not the job's conclusion -- is the
evidence.
test/unified-api-adapter-contract_test.exs, the ABI wire-contractguard #120 repointed onto the live
src/Hypatia/ABI/Types.idr, is among them,so the instance-21 fix is now genuinely exercised rather than correct-in-tree
and vacuous in CI.
The general lesson, which outlives this issue
A
skippedconclusion satisfies a required status check, and astartup-dead workflow reports no jobs at all. Both look like "not red" and
neither is evidence. When a cure lands upstream in a
needs:DAG, the verdictof every node below it re-opens and must be re-measured -- carrying forward
the old conclusions is how a tier stays dead after its cause is fixed.
The same re-measurement surfaced
End-to-End Tests, which by the identicalmechanism had never executed either. It now runs and fails (
docker composecannot find its configuration file, so the E2E suite is still skipped). That is
tracked separately as #843 -- this issue covers the ExUnit tier, which is now
demonstrably alive.
🤖 Generated with Claude Code
https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV