Skip to content

CI: Fail on missing preconditions and exceeded boundaries - #1754

Draft
Steven Shriver (stshrive) wants to merge 1 commit into
agent-substrate:mainfrom
stshrive:dev/harden-preconditions
Draft

Steven Shriver (stshrive) wants to merge 1 commit into
agent-substrate:mainfrom
stshrive:dev/harden-preconditions

Conversation

@stshrive

@stshrive Steven Shriver (stshrive) commented Sep 18, 2026

Copy link
Copy Markdown

This change addresses six ways CI pipeline could report false success or hang on broken infrastructure. Importantly, this change does two things to reduce load on infrastructure:

  1. It prevents jobs from running for the default action limit of 360m by pinning the timeout to 45m.
  2. It limits pull_request jobs from continuing execution once superseded by new changes.
  • Silent container test skipping: Tests now explicitly fail if CI or REQUIRE_DOCKER is set (while still skipping on local machines lacking Docker), with the check implemented in dockerenv to avoid an import cycle between storetest and atepg.
  • Unbounded trust bundle wait: Enforced a shared 120-second timeout across both bundles (overridable via ATE_INSTALL_TRUST_BUNDLE_TIMEOUT) and added diagnostic dumping of bundles, controller pods, and logs before returning a non-zero exit code.
  • Missing sandbox preflight validation: Added early preflight checks for /dev/kvm and SandboxConfig/microvm that fail fast and print actionable remediation instructions.
  • Skipped migration checks on main: Configured the migration immutability check to run on pushes to main to catch modified migrations at the point of merge.
  • Post-merge vulnerability scanning: Configured govulncheck to run on pull requests to catch vulnerable dependencies prior to merging into main.
  • Missing job timeouts and concurrency limits: Defined explicit timeout-minutes (45m and 120m bounds) and added concurrency groups that automatically cancel superseded pull request runs without canceling runs on main.

Fixes #1747

  • Tests pass
    • Silent container skipping, unbounded trust bundles, and missing sandbox were all forced locally and confirmed to exist with changes here resolving each.
    • The latter half of the scenarios exist in CI only due to being GH Action trigger issues.

Six ways the pipeline reported false success or hung on broken
infrastructure.

- **Silent container test skipping**: Tests now explicitly fail if
  `CI` or `REQUIRE_DOCKER` is set (while still skipping on local
  machines lacking Docker), with the check implemented in
  `dockerenv` to avoid an import cycle between `storetest` and
  `atepg`.
- **Unbounded trust bundle wait**: Enforced a shared 120-second
  timeout across both bundles (overridable via
  `ATE_INSTALL_TRUST_BUNDLE_TIMEOUT`) and added diagnostic dumping
  of bundles, controller pods, and logs before returning a non-zero
  exit code.
- **Missing sandbox preflight validation**: Added early preflight
  checks for `/dev/kvm` and `SandboxConfig/microvm` that fail fast
  and print actionable remediation instructions.
- **Skipped migration checks on main**: Configured the migration
  immutability check to run on pushes to main to catch modified
  migrations at the point of merge.
- **Post-merge vulnerability scanning**: Configured `govulncheck`
  to run on pull requests to catch vulnerable dependencies prior
  to merging into main.
- **Missing job timeouts and concurrency limits**: Defined explicit
  timeout-minutes (45m and 120m bounds) and added concurrency
  groups that automatically cancel superseded pull request runs
  without canceling runs on main.
@BenTheElder

Copy link
Copy Markdown
Collaborator

Post-merge vulnerability scanning: Configured govulncheck to run on pull requests to catch vulnerable dependencies prior to merging into main.

We intentionally did not do this. Vulns are point-in-time X sources, not just sources.
We don't want to stop merging other PRs when there's a vuln in main's deps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing preconditions are not a hard-stop for some test causing low-fidelity signals in the CI/testing infra.

2 participants