FailureMode.Continue does what its name suggests for plan sequencing: after a
plan fails, the remaining plans in the target still run. That part is useful and
is exactly what I needed.
The problem is what it does to the outcome. Verified with a deliberately
three-plan probe, where the middle plan is guaranteed to fail:
Target PlanFailureProbe => _ => _
.FailureMode(FailureMode.Continue)
.Executes(() => new[]
{
DotNetTool.Plan("--version"),
DotNetTool.Plan("nonexistent-subcommand-xyz"), // fails
DotNetTool.Plan("--info"),
});
Observed:
- plan 1 runs
- plan 2 fails, and Tamp logs
==> PlanFailureProbe FAILED (exit 1)
- plan 3 still runs (good, this is the point of the mode)
- Build Summary reports
PlanFailureProbe ✓ Done
- the Tamp process exits 0
So Tamp knows the plan failed and says so mid-log, then discards that fact in
both the summary and the exit code.
For a test-running target this is a live footgun: a suite with failing tests
produces a green build and a zero exit status. In CI that is indistinguishable
from a pass. I had to add a separate downstream target with
AssuredAfterFailure() that re-reads the TRX result files and throws, purely to
put the failure back.
Suggested fix, any of:
- Keep running the remaining plans but still mark the target failed and exit
non-zero. That seems like the intuitive reading of "continue".
- Add a distinct mode (
ContinueThenFail, or a FailureMode.Continue
parameter) so the swallow-the-failure behaviour is opt-in rather than implied.
- At minimum, document the exit-code behaviour prominently, since the current
name reads as "keep going", not "keep going and forget it went wrong".
FailureMode.Continuedoes what its name suggests for plan sequencing: after aplan fails, the remaining plans in the target still run. That part is useful and
is exactly what I needed.
The problem is what it does to the outcome. Verified with a deliberately
three-plan probe, where the middle plan is guaranteed to fail:
Observed:
==> PlanFailureProbe FAILED (exit 1)PlanFailureProbe ✓ DoneSo Tamp knows the plan failed and says so mid-log, then discards that fact in
both the summary and the exit code.
For a test-running target this is a live footgun: a suite with failing tests
produces a green build and a zero exit status. In CI that is indistinguishable
from a pass. I had to add a separate downstream target with
AssuredAfterFailure()that re-reads the TRX result files and throws, purely toput the failure back.
Suggested fix, any of:
non-zero. That seems like the intuitive reading of "continue".
ContinueThenFail, or aFailureMode.Continueparameter) so the swallow-the-failure behaviour is opt-in rather than implied.
name reads as "keep going", not "keep going and forget it went wrong".