Containerized baseline test suite — prove future changes are improvements

[BaseInfinity]

Problem

No repeatable, isolated baseline exists to prove a change improves the harness. Tests run in the host shell, share state with parallel sessions, and measure code correctness — not harness behavior.

The user's ask: "prove any future changes are good for the harness" with containers for session isolation.

Design

Dockerfile.test

• Base: node:20-slim (matches consumer environments)
• Install: bash, python3-yaml, jq, git, shellcheck
• Copy: repo working tree
• No runtime deps (the package has none)

scripts/run-tests-containerized.sh

• Builds sdlc-wizard-test:local
• Runs the full test suite inside the container
• Outputs structured results (pass/fail counts, timing)
• Safe for parallel sessions — each gets its own container

Mutation baseline

• Known-bad mutations (wrong model, missing hook, broken gate) injected before test run
• Each mutation MUST fail at least one test
• If a mutation passes all tests, the test suite has a coverage gap → file it
• Baseline recorded as tests/fixtures/mutation-baseline.json

Brain wiring proof

• Verify the escalation ladder is wired: builder → first brain → second brain
• Verify REVIEW_MODEL env var passes through to the launcher
• Verify advisor model is set correctly
• All provable without live API calls (stub-based, like test-run-review-leg.sh)

Acceptance criteria

• Dockerfile.test builds and runs the full suite
• scripts/run-tests-containerized.sh is the single entry point
• Mutation baseline: ≥5 known-bad mutations, each caught
• Parallel sessions cannot interfere (no shared state)
• Brain wiring assertions pass
• Docker 24.0.6 confirmed available on host

Blocks

• Nothing. This is additive.

Related

• CI/CD audit: 60+ suites, brittle string guards, solo-maintainer overhead #714 — CI/CD audit (the containerized suite could replace or supplement CI)
• Set the Opus 4.6 line in stone: name the fallback, prove it with a canary, and cut release/opus-4.6 only if the canary says to #689 §7.2 — lifecycle canary (the container is the isolated environment for it)