test_lint_targets whitaker-fallback cases fail on a clean main checkout

[leynos]

Running the template test suite (make test) on an unmodified main checkout (commit 7d7eb00) fails three parametrised cases:

FAILED tests/test_template/test_lint_targets.py::test_makefile_resolves_whitaker_fallback[path]
FAILED tests/test_template/test_lint_targets.py::test_makefile_resolves_whitaker_fallback[home]
FAILED tests/test_template/test_lint_targets.py::test_makefile_resolves_whitaker_fallback[missing]

For the path and home cases, the stub whitaker script is never executed, so the marker file the stub should create does not exist:

AssertionError: expected generated lint target to execute path Whitaker
assert False
 +  where False = exists()
 +    where exists = PosixPath('.../whitaker-ran').exists

The make lint invocation in the generated project returns 0 without running the stub, suggesting the Makefile's Whitaker resolution (or the test's PATH/HOME sandboxing) does not behave as the test expects on this environment.

Environment: Rocky Linux 10, uvx --with pytest-copier --with pyyaml --with syrupy --with make-parser --with hypothesis pytest tests/, Python via uv, GNU make from /usr/bin.

Observed while validating #52 — the failures reproduce identically on a clean main checkout (verified via git stash), so they are unrelated to that change. The remaining 47 tests pass (1 skipped).

[coderabbitai]

🔗 Related PRs

leynos/stilyagi#10 - Wire Makefile and CI smoke path (1.2.3) [closed]
leynos/agent-template-python#13 - Add generated workflow validation coverage [merged]
leynos/stilyagi#22 - Add maturin and PyO3 compatibility tests [closed]
leynos/dev-env-rocky#111 - Install profiling tools [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

Diagnose first: read the recipe's echoed Whitaker binary: value to prove that an inherited environment variable, not the $(or ...) chain, is short-circuiting resolution — the only mechanism consistent with all three cases exiting 0 without running the stub.

Make the fallback test hermetic by stripping WHITAKER/MAKEFLAGS/MFLAGS/MAKELEVEL from the subprocess environment before overriding HOME/PATH/CARGO, keeping the template and its snapshot untouched.

Extract the sanitised-environment logic into a reusable test helper and enrich assertion output so any recurrence is self-diagnosing.

Retain a clearly gated contingency to harden template/Makefile.jinja (with snapshot and contract updates) only if diagnosis proves the template chain, rather than the environment, is at fault.

Design Choices

Design Choice 1: Whether the defect lives in the test's environment sandboxing or in the Makefile's resolution chain

Options Considered:

1. Make the test hermetic — strip resolution-controlling variables from the subprocess environment before applying HOME/PATH/CARGO overrides, leaving the Makefile untouched.
2. Harden the Makefile's $(or ...) chain in template/Makefile.jinja to ignore inherited overrides and re-derive WHITAKER unconditionally, then regenerate the snapshot and contracts.

Chosen Option: 1

Rationale: The symptom pattern is decisive: all three cases fail, and make lint exits 0 even in the missing case (which must fail). Static analysis of the $(or ...) chain shows every branch — including the path case's absolute-path stub — should execute the stub or fail non-zero. The only mechanism that makes all three exit 0 without running the stub is the ?= default being pre-satisfied by an inherited environment variable (a leaked WHITAKER, or a make-injected override such as MAKEFLAGS/MFLAGS/MAKELEVEL) that resolves to an exit-0 no-op. The test spreads **os.environ wholesale, which permits exactly this leakage. The leaked-environment mechanism is the only explanation consistent with all three symptoms, and sanitising the test's environment fixes the failures with zero template churn and no snapshot regeneration; retain option 2 as the documented contingency if Phase 1 proves the Makefile itself is at fault.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Reproduce and confirm the root cause

Establish, on the affected environment, exactly what WHITAKER resolves to and which inherited variable defeats the fallback, before changing behaviour. This phase produces evidence, not fixes.

Task 1: Capture the resolved Whitaker value on the failing host

Surface what the recipe actually runs so the failure is no longer opaque.

• Run the three parametrised cases and capture full stdout/stderr; the lint recipe already prints Whitaker binary: <value> via the @echo line, so the resolved value is directly observable.
• Record whether the printed value is the stub path (expected), the bare literal whitaker, or an unexpected third value (e.g. true, /usr/bin/true, or another no-op) that would explain an exit-0 with no marker.

Task 2: Identify the leaking environment variable

Pin down which inherited variable pre-satisfies the ?= default or overrides it.

• Inspect the environment used by test_makefile_resolves_whitaker_fallback for WHITAKER, MAKEFLAGS, MFLAGS, and MAKELEVEL, since these can override a ?= default or inject a command-line-level variable into the child make.
• Confirm the hypothesis: an inherited value resolves to an exit-0 command, so make never evaluates the command -v/$(wildcard) branches and the stub never executes — reproducing the identical symptom across path, home, and missing.
• If, and only if, the captured value proves the Makefile chain itself misbehaves (stub reachable yet unused with a clean environment), escalate to the Phase 2 contingency (Task 2.3).

🤖 Prompt for AI agents

Implement Phase 1: reproduce and confirm the root cause of the three failing
`test_makefile_resolves_whitaker_fallback` cases, without changing any behaviour
yet — this phase is diagnostic only.

Run the three parametrised cases (`path`, `home`, `missing`) and capture the
full `stdout`/`stderr`. The `lint` recipe already prints `Whitaker binary:
<value>` via an `@echo` line, so surface this value directly.

Record whether the printed value is the stub path (expected), the bare literal
`whitaker`, or an unexpected third value (e.g. `true`, `/usr/bin/true`, or
another no-op) that would explain an exit-0 outcome with no marker file created.

Inspect the environment used by the test for `WHITAKER`, `MAKEFLAGS`, `MFLAGS`,
and `MAKELEVEL`, since any of these can override a `?=` default or inject a
command-line-level variable into the child make process.

Confirm the hypothesis that an inherited value resolves to an exit-0 command,
causing make to skip evaluating the `command -v`/`$(wildcard)` branches
entirely, so the stub never executes — this must reproduce the identical symptom
across all three cases (`path`, `home`, `missing`).

Only if the captured evidence proves the Makefile chain itself misbehaves (i.e.
the stub is reachable yet unused even with a clean environment), escalate to the
Phase 2 contingency task (Task 2.3) that hardens the template.

Phase 2: Make the behavioural test hermetic

Remove the environment leakage so the fallback test exercises exactly one resolution branch per case, independent of the host or how the suite was launched (uvx pytest directly or via make test).

Task 1: Sanitise the subprocess environment in the fallback test

Stop inheriting variables that control Whitaker resolution.

• Update tests/test_template/test_lint_targets.py so the env passed to subprocess.run([make, "lint"], ...) is built from os.environ with WHITAKER, MAKEFLAGS, MFLAGS, and MAKELEVEL explicitly removed before the HOME/PATH/CARGO overrides are applied.
• Preserve the existing minimal PATH construction (/usr/bin:/bin, optionally prefixed with path_bin) so bash and coreutils remain reachable while the stub whitaker is the only resolvable one in each branch.

Task 2: Extract a reusable sanitised-environment helper

Capture the pattern once so future generated-project make invocations inherit the same hygiene.

• Add a small helper in tests/helpers/ (alongside rendering.py) that returns a copy of os.environ with the resolution-controlling variables stripped, accepting caller-supplied overrides for HOME/PATH/CARGO.
• Consume this helper from test_makefile_resolves_whitaker_fallback; document its intent (hermetic make invocation for generated projects) in the helper's docstring.

Task 3: Contingency — harden the Makefile resolution (only if Phase 1 requires it)

Apply solely if Task 1.2 proves the template chain, not the environment, is at fault.

• Adjust template/Makefile.jinja so WHITAKER is re-derived from the command -v/$(wildcard)/literal chain rather than honouring an inherited value, keeping the missing case failing clearly at execution time.
• Regenerate tests/test_template/__snapshots__/test_snapshots.ambr and update the substring assertions in tests/helpers/tooling_contracts/makefile.py to match the revised recipe text.

🤖 Prompt for AI agents

Implement Phase 2: make the behavioural test hermetic so the fallback test
exercises exactly one resolution branch per case, regardless of host or how the
suite was launched (`uvx pytest` directly or via `make test`).

Update `tests/test_template/test_lint_targets.py` so the `env` passed to
`subprocess.run([make, "lint"], ...)` is built from `os.environ` with
`WHITAKER`, `MAKEFLAGS`, `MFLAGS`, and `MAKELEVEL` explicitly removed before
applying the `HOME`/`PATH`/`CARGO` overrides. Preserve the existing minimal
`PATH` construction (`/usr/bin:/bin`, optionally prefixed with `path_bin`) so
bash and coreutils remain reachable while the stub whitaker is the only
resolvable one in each branch.

Extract this sanitisation logic into a reusable helper in `tests/helpers/`
(alongside `rendering.py`) that returns a copy of `os.environ` with the
resolution-controlling variables stripped, accepting caller-supplied overrides
for `HOME`/`PATH`/`CARGO`. Consume this helper from
`test_makefile_resolves_whitaker_fallback`, and document its intent (hermetic
make invocation for generated projects) in the helper's docstring.

Apply the following contingency task only if Phase 1 diagnosis proves the
template chain, rather than the environment, is at fault: adjust
`template/Makefile.jinja` so `WHITAKER` is re-derived from the `command
-v`/`$(wildcard)`/literal chain rather than honouring an inherited value,
keeping the `missing` case failing clearly at execution time. If this
contingency is triggered, regenerate
`tests/test_template/__snapshots__/test_snapshots.ambr` and update the substring
assertions in `tests/helpers/tooling_contracts/makefile.py` to match the revised
recipe text.

Phase 3: Verify and guard against regression

Prove the fix on the affected host and lock the resolved value into failure output for future diagnosis.

Task 1: Improve failure diagnostics

Make any future recurrence self-explaining.

• Extend the assertion messages in test_makefile_resolves_whitaker_fallback to include the captured stdout (which carries the Whitaker binary: <value> line), so a failing run immediately reports what was resolved rather than only that the marker is absent.

Task 2: Re-run the full suite and confirm stability

Validate on the reporting environment and confirm no collateral churn.

• Run the whole tests/ suite under uvx --with pytest-copier --with pyyaml --with syrupy --with make-parser --with hypothesis pytest tests/ on Rocky Linux 10; confirm the three fallback cases pass and the previously passing 47 remain green.
• Confirm the snapshot (test_snapshots.ambr) and tooling_contracts/makefile.py need no changes under the chosen environment-sanitisation approach; treat any required snapshot change as a signal that the contingency (Task 2.3) was silently triggered and must be reviewed.

🤖 Prompt for AI agents

Implement Phase 3: verify the fix on the affected host and lock the resolved
value into failure output for future diagnosis.

Extend the assertion messages in `test_makefile_resolves_whitaker_fallback` to
include the captured `stdout` (which carries the `Whitaker binary: <value>`
line), so that a failing run immediately reports what was resolved rather than
only reporting that the marker is absent.

Run the whole `tests/` suite under `uvx --with pytest-copier --with pyyaml
--with syrupy --with make-parser --with hypothesis pytest tests/` on Rocky Linux
10; confirm the three fallback cases pass and the previously passing 47 tests
remain green.

Confirm that the snapshot (`test_snapshots.ambr`) and
`tooling_contracts/makefile.py` need no changes under the chosen
environment-sanitisation approach. Treat any required snapshot change as a
signal that the contingency (Task 2.3) was silently triggered, and flag it for
review.

Research

This repository is a Copier template that renders downstream Rust projects. The rendered Makefile (template/Makefile.jinja) resolves the whitaker linter through a GNU-Make $(or ...) chain: WHITAKER ?= $(or $(shell command -v whitaker 2>/dev/null),$(wildcard $(USER_WHITAKER)),whitaker), invoked from the lint target.

The behavioural test in tests/test_template/test_lint_targets.py renders a project, then runs make lint in a sandboxed subprocess by spreading **os.environ and overriding HOME, PATH, and CARGO. Static contracts live in tests/helpers/tooling_contracts/makefile.py, and the rendered target is pinned byte-for-byte in tests/test_template/__snapshots__/test_snapshots.ambr.

The sibling test_clippy_runs relies on an unrestricted ambient toolchain and does not isolate the environment, so only the fallback test exercises resolution behaviourally.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: reproduce and confirm the root cause of the three failing
`test_makefile_resolves_whitaker_fallback` cases, without changing any behaviour
yet — this phase is diagnostic only.

Run the three parametrised cases (`path`, `home`, `missing`) and capture the
full `stdout`/`stderr`. The `lint` recipe already prints `Whitaker binary:
<value>` via an `@echo` line, so surface this value directly.

Record whether the printed value is the stub path (expected), the bare literal
`whitaker`, or an unexpected third value (e.g. `true`, `/usr/bin/true`, or
another no-op) that would explain an exit-0 outcome with no marker file created.

Inspect the environment used by the test for `WHITAKER`, `MAKEFLAGS`, `MFLAGS`,
and `MAKELEVEL`, since any of these can override a `?=` default or inject a
command-line-level variable into the child make process.

Confirm the hypothesis that an inherited value resolves to an exit-0 command,
causing make to skip evaluating the `command -v`/`$(wildcard)` branches
entirely, so the stub never executes — this must reproduce the identical symptom
across all three cases (`path`, `home`, `missing`).

Only if the captured evidence proves the Makefile chain itself misbehaves (i.e.
the stub is reachable yet unused even with a clean environment), escalate to the
Phase 2 contingency task (Task 2.3) that hardens the template.
===============================================================================

Task: 2

Implement Phase 2: make the behavioural test hermetic so the fallback test
exercises exactly one resolution branch per case, regardless of host or how the
suite was launched (`uvx pytest` directly or via `make test`).

Update `tests/test_template/test_lint_targets.py` so the `env` passed to
`subprocess.run([make, "lint"], ...)` is built from `os.environ` with
`WHITAKER`, `MAKEFLAGS`, `MFLAGS`, and `MAKELEVEL` explicitly removed before
applying the `HOME`/`PATH`/`CARGO` overrides. Preserve the existing minimal
`PATH` construction (`/usr/bin:/bin`, optionally prefixed with `path_bin`) so
bash and coreutils remain reachable while the stub whitaker is the only
resolvable one in each branch.

Extract this sanitisation logic into a reusable helper in `tests/helpers/`
(alongside `rendering.py`) that returns a copy of `os.environ` with the
resolution-controlling variables stripped, accepting caller-supplied overrides
for `HOME`/`PATH`/`CARGO`. Consume this helper from
`test_makefile_resolves_whitaker_fallback`, and document its intent (hermetic
make invocation for generated projects) in the helper's docstring.

Apply the following contingency task only if Phase 1 diagnosis proves the
template chain, rather than the environment, is at fault: adjust
`template/Makefile.jinja` so `WHITAKER` is re-derived from the `command
-v`/`$(wildcard)`/literal chain rather than honouring an inherited value,
keeping the `missing` case failing clearly at execution time. If this
contingency is triggered, regenerate
`tests/test_template/__snapshots__/test_snapshots.ambr` and update the substring
assertions in `tests/helpers/tooling_contracts/makefile.py` to match the revised
recipe text.
===============================================================================

Task: 3

Implement Phase 3: verify the fix on the affected host and lock the resolved
value into failure output for future diagnosis.

Extend the assertion messages in `test_makefile_resolves_whitaker_fallback` to
include the captured `stdout` (which carries the `Whitaker binary: <value>`
line), so that a failing run immediately reports what was resolved rather than
only reporting that the marker is absent.

Run the whole `tests/` suite under `uvx --with pytest-copier --with pyyaml
--with syrupy --with make-parser --with hypothesis pytest tests/` on Rocky Linux
10; confirm the three fallback cases pass and the previously passing 47 tests
remain green.

Confirm that the snapshot (`test_snapshots.ambr`) and
`tooling_contracts/makefile.py` need no changes under the chosen
environment-sanitisation approach. Treat any required snapshot change as a
signal that the contingency (Task 2.3) was silently triggered, and flag it for
review.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!