CI: Fail on missing preconditions and exceeded boundaries - #1754
Draft
Steven Shriver (stshrive) wants to merge 1 commit into
Draft
Steven Shriver (stshrive) wants to merge 1 commit into
Steven Shriver (stshrive) wants to merge 1 commit into
Conversation
Six ways the pipeline reported false success or hung on broken infrastructure. - **Silent container test skipping**: Tests now explicitly fail if `CI` or `REQUIRE_DOCKER` is set (while still skipping on local machines lacking Docker), with the check implemented in `dockerenv` to avoid an import cycle between `storetest` and `atepg`. - **Unbounded trust bundle wait**: Enforced a shared 120-second timeout across both bundles (overridable via `ATE_INSTALL_TRUST_BUNDLE_TIMEOUT`) and added diagnostic dumping of bundles, controller pods, and logs before returning a non-zero exit code. - **Missing sandbox preflight validation**: Added early preflight checks for `/dev/kvm` and `SandboxConfig/microvm` that fail fast and print actionable remediation instructions. - **Skipped migration checks on main**: Configured the migration immutability check to run on pushes to main to catch modified migrations at the point of merge. - **Post-merge vulnerability scanning**: Configured `govulncheck` to run on pull requests to catch vulnerable dependencies prior to merging into main. - **Missing job timeouts and concurrency limits**: Defined explicit timeout-minutes (45m and 120m bounds) and added concurrency groups that automatically cancel superseded pull request runs without canceling runs on main.
Collaborator
We intentionally did not do this. Vulns are point-in-time X sources, not just sources. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change addresses six ways CI pipeline could report false success or hang on broken infrastructure. Importantly, this change does two things to reduce load on infrastructure:
45m.CIorREQUIRE_DOCKERis set (while still skipping on local machines lacking Docker), with the check implemented indockerenvto avoid an import cycle betweenstoretestandatepg.ATE_INSTALL_TRUST_BUNDLE_TIMEOUT) and added diagnostic dumping of bundles, controller pods, and logs before returning a non-zero exit code./dev/kvmandSandboxConfig/microvmthat fail fast and print actionable remediation instructions.govulncheckto run on pull requests to catch vulnerable dependencies prior to merging into main.Fixes #1747