Skip to content

Track v0.4.0 production graduation #112

Open
Open
Track v0.4.0 production graduation#112
Assignees
ErenAri

Description

@ErenAri
ErenAri
opened on Jul 29, 2026

Objective

Graduate only the supported CLI + GitHub Action + disposable QEMU/KVM validation boundary to stable v0.4.0. The API/UI demo, runtime loading, agent, registry, SaaS, Firecracker, and virtme-ng remain experimental and explicitly out of scope.

This issue is the audit trail and checklist. Manual runs are shakeouts only and do not count as scheduled graduation campaigns. The earlier Yusuf-dependent review contract was superseded by the solo-maintainer design merged in #114; historical comments remain below as audit history.

Solo-maintainer release controls

  • main requires strict status checks, admin enforcement, conversation resolution, and blocks force-pushes/deletion, with zero required human approvals
  • production-release has no human reviewers, a 15-minute wait timer, no admin bypass, and an exact v* deployment policy
  • Active immutable v* tag ruleset blocks update/deletion and has no bypass actors
  • Tag workflow can only build, attest, sign, and stage a private draft
  • Separate workflow_dispatch promotion must run on the exact tag and re-verify the candidate run ID, full commit, digest, typed confirmation, draft assets, checksums, attestations, Sigstore signatures, image version, and release channel
  • Live scripts/check-production-environment.sh passes in solo-maintainer mode
  • Deprecated actions/attest-sbom wrapper migrated to pinned actions/attest in ci: migrate SBOM attestations to actions/attest #116; all 10 PR checks and post-merge workflows passed

Release candidate

  • Immutable annotated v0.4.0-rc.3 tag resolves to cba1e09537f2e05e4ba7278efdb5c2d044d1889b
  • Candidate run 30552866250 passed release quality, amd64/ARM64 build, native ARM64 execution, positive VM validation, classified-negative VM validation, multiarch image/version verification, signing, provenance, SBOM, and evidence attestation
  • Candidate evidence binds run 30552866250, attempt 1, the exact tag/commit, prerelease channel, and image digest sha256:842ebbc5746dcfcf98823407a762bdf379e24beb8ea20941ff77f9a74a1a2ac6
  • Exact-input promotion run 30553637272 waited 15 minutes, re-verified all bindings, and succeeded
  • GitHub release is a public prerelease with ten verified assets
  • GHCR RC tag is amd64+arm64 at the evidence digest; stable 0.4 was not created and latest stayed at sha256:4858cedde87681b1af4876bb8c869c7c1df0f5721764448a47042fe616e65102

Pre-campaign operational work

  • Deploy RC3 to the public demo at bpfcompat.kernelguard.net while keeping it visibly Technical Preview and outside the production claim
  • Verify public /livez, /readyz, HTTPS/security headers, KVM, watchdog, runtime-execute disabled, exact version/commit/checksum, and retained rollback binary
  • Resolve or explicitly disposition the seven stale enterprise-kernel baselines in Stale kernel baselines (weekly freshness report) #89 before campaign 1
  • Execute and record a deliberate release-promotion rollback drill using immutable before/after image digests, exact operator, commands, elapsed recovery, signature/provenance/version verification, and [bpfcompat-rollback-drill:v1]
  • Exercise and record the promotion incident path, proving an invalid deployment remains fail-closed

Four scheduled campaigns

Only schedule events from .github/workflows/latest-kernel-compatibility.yml count. Each must be unique, chronological, 6–8 days apart, complete within 120 minutes, have target p95 <= 12 minutes, and contain zero infra_error targets.

  • Campaign 1 — scheduled run 30790938510 on 2026-08-03
  • Campaign 2 — Monday 2026-08-10 03:21 UTC
  • Campaign 3 — Monday 2026-08-17 03:21 UTC
  • Campaign 4 — Monday 2026-08-24 03:21 UTC

For every run, record the workflow URL, artifact name, commit SHA, start time, report SHA-256, target status summary, infrastructure-error count, p95 target duration, and total workflow duration. Any supported-boundary behavior change after the window starts requires a new RC and restarts the window; documentation/release-tooling-only changes do not.

Independent ecosystem and consumer evidence

  • Capture the first successful scheduled Falco run after falcosecurity/libs@f6fed8e / falcosecurity/libs#3061
  • Verify the Falco artifact hash and expanded RHEL-family profile coverage
  • Run clean-install, source-build, Action, container, and external-consumer canaries against RC3
  • Verify 24-hour and 72-hour post-publication canaries; investigate every failure before graduation

Post-v0.4 ecosystem follow-up (not a v0.4 gate)

Confirmed adoption is a 1.0 ecosystem-readiness goal, not a technical prerequisite for the supported v0.4 boundary and not a substitute for its evidence.

  • Invite at least two controlled pilot evaluations through the adopter issue form without claiming adoption or endorsement
  • Record any confirmed public pilot only with explicit listing permission in ADOPTERS.md

Stable-release gate

  • Download the four campaign artifacts, expanded Falco artifact, candidate evidence, rollback note, incident note, and canary evidence into one private evidence directory
  • Build readiness/input.json with exact relative paths and SHA-256 values plus the solo operator confirmation marker
  • Run scripts/production-readiness-report.sh readiness/input.json readiness/bpfcompat-0.4.0-readiness.md without the test-only attestation bypass
  • Commit docs/releases/bpfcompat-0.4.0-readiness.{md,json} in the final release PR
  • Set stable_version: 0.4.0, release_version: 0.4.0, and release_channel: stable
  • Tag only the reviewed merge commit, complete the exact-input wait-gated promotion, and publish
  • Verify release assets, signatures, SBOM, provenance, image aliases, installer defaults, Action consumption, and stable canaries at 24h and 72h

Earliest defensible stable release: 2026-08-24/25, only if every gate passes and the campaign window is not reset.

Activity

  1. ErenAri
    self-assigned this
    on Jul 29, 2026

  2. ErenAri commented on Jul 29, 2026

    @ErenAri
    ErenAri
    on Jul 29, 2026
    ContributorAuthor

    RC1 audit update

    • PR Prepare v0.4.0 release candidate quality gates #111 merged with independent approval and all required checks green.
    • The immutable v0.4.0-rc.1 tag was created on 720addf2a9af4180620aa8df89982a4efc8875ae.
    • Release run 30492851666 stopped in the pre-build quality gate. No candidate assets, draft/public release, GHCR release tag, moving alias, or promotion deployment was created.
    • Root cause: tag variables leaked into a stable-metadata test fixture; the checker also had an undeclared/fail-open rg dependency.
    • Fix and immutable v0.4.0-rc.2 metadata are under independent review in Fix tag release quality gate #113. The rc.1 tag will not be moved or reused.

  3. ErenAri commented on Jul 29, 2026

    @ErenAri
    ErenAri
    on Jul 29, 2026
    ContributorAuthor

    @yusuf-demirel4 v0.4.0-rc.2 has reached the separate protected promotion gate in release run 30495097626. Before approving production-release, please inspect the draft assets and release-candidate-evidence.json. Completed successfully:

    • release-quality, coverage, vulnerability, and live environment checks
    • attested amd64 CLI, arm64 CLI, static validator, and CycloneDX SBOM
    • exact native ARM64 execution
    • pinned positive VM validation
    • expected classified-negative VM validation with no infrastructure error
    • multiarchitecture image build, version check, signature, and provenance
    • attested candidate promotion evidence and private draft release

    Please approve the environment only if the evidence is acceptable; otherwise reject it and record the concern. The release and RC image tag remain unpublished until this decision.

  4. ErenAri commented on Jul 29, 2026

    @ErenAri
    ErenAri
    on Jul 29, 2026
    ContributorAuthor

    Independent RC2 draft verification

    I downloaded the private draft assets and verified them independently while the protected environment remains pending:

    • SHA256SUMS passes for amd64 CLI, arm64 CLI, and static validator
    • amd64 binary reports v0.4.0-rc.2, commit ceddee960e99, Go 1.25.12
    • GitHub attestations verify for both CLIs, validator, and release-candidate-evidence.json, bound to release-artifacts.yml
    • candidate evidence binds run 30495097626, full commit ceddee960e997d0499edbe74c688f035dfa805f8, tag v0.4.0-rc.2, channel prerelease, and image digest sha256:8f03b6c5bb6f3fae79b78f26979d523f4c449080eefd509657c05e805bc55e8c
    • keyless Sigstore signatures verify for SHA256SUMS and the CycloneDX SBOM with the exact tag-workflow identity
    • the candidate container signature verifies at the same immutable digest

    The draft is still private and the promotion job remains waiting; only @yusuf-demirel4 can approve it.

  5. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    RC3 publication and demo deployment evidence

    • Candidate run: 30552866250, exact commit cba1e09537f2e05e4ba7278efdb5c2d044d1889b.
    • Candidate evidence image: ghcr.io/kernel-guard/bpfcompat@sha256:842ebbc5746dcfcf98823407a762bdf379e24beb8ea20941ff77f9a74a1a2ac6.
    • Promotion run: 30553637272; it waited 15 minutes, then re-verified the exact run, commit, digest, assets, checksums, attestations, Sigstore identities, image version, draft state, and channel before publication.
    • Public prerelease: v0.4.0-rc.3. The RC image is amd64+arm64 at the evidence digest. 0.4 remains absent and latest remains sha256:4858cedde87681b1af4876bb8c869c7c1df0f5721764448a47042fe616e65102.
    • Public assets pass SHA256SUMS; amd64 reports v0.4.0-rc.3 (cba1e09537f2) built with Go 1.25.12.
    • Demo deployment intentionally preserved the previous binary/source. The first health probe encountered a restart-time connection refusal and the deployment automatically restored v0.3.2-1-gc940cdc; public and local recovery were verified. A corrected bounded retry then deployed RC3 successfully.
    • Demo now returns livez=ok, readyz.version=v0.4.0-rc.3, remains visibly Technical Preview, serves HSTS/CSP/frame/referrer/content-type protections, has KVM and the healthcheck timer active, and reports runtime_execute_enabled=false.
    • Exact deployed binary SHA-256: 07a8394d512f8dec2a3a773473cd4c210fa82825586cce03bee8daac3d64a0a0. Previous deployment retained under /var/lib/bpfcompat-deployments/20260730T150745Z for rollback.

  6. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    Rollback and fail-closed incident evidence

    • Protected rollback workflow: run 30557817015. It waited 5 minutes in the isolated production-drill environment, then completed successfully.
    • [bpfcompat-rollback-drill:v1] evidence names operator ErenAri, stable digest sha256:4858cedde87681b1af4876bb8c869c7c1df0f5721764448a47042fe616e65102, RC3 digest sha256:842ebbc5746dcfcf98823407a762bdf379e24beb8ea20941ff77f9a74a1a2ac6, and exact completion time 2026-07-30T15:45:40Z.
    • Before either move, both immutable endpoints passed exact Sigstore identity, SLSA v1 provenance, and embedded-version verification.
    • The dedicated alias moved stable → RC3 in 924 ms, then RC3 → stable in 846 ms. Independent post-run inspection confirms both rollback-drill-v0.4.0-rc.3 and protected latest resolve to the original stable digest. Evidence JSON SHA-256: 6dea4ec98f0debff511b5cd7a86d2eebbe0189d95264cca718371d7f6695ea30.
    • Fail-closed incident exercise: run 30556520652 deliberately dispatched promotion from main. The production-release deployment policy permits only exact v* tags, so GitHub rejected the job before allocating a runner; it had zero steps and completed failure in about five seconds.
    • After the incident exercise, the public RC3 prerelease and its ten assets remained intact, RC image alias 0.4.0-rc.3 remained at the attested digest, stable latest remained unchanged, and no 0.4 alias was created.

    The rollback drill is intentionally isolated from real release aliases; it proved the mutation/recovery mechanics without weakening or exercising the stable channel.

  7. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    Exact RC3 canary evidence

    • Final authenticated canary: run 30560118663, completed successfully at 2026-07-30T16:14:20Z.
    • [bpfcompat-rc-canary:v1] binds immutable tag v0.4.0-rc.3, commit cba1e09537f2e05e4ba7278efdb5c2d044d1889b, and image digest sha256:842ebbc5746dcfcf98823407a762bdf379e24beb8ea20941ff77f9a74a1a2ac6.
    • Clean installer, source build, published Action, signed amd64+arm64 container, and external consumers all report pass.
    • Exact-tag ecosystem run 30560146169 passed Falco scap-open modern_bpf loading, KubeArmor build-on-target system_monitor loading, and Inspektor Gadget published OCI gadget validation.
    • Evidence manifest SHA-256: 00679feba40d98eeb6c91cffa5a6726827d7c939c1b43a16df87f4aa50bb9ddc; all seven component hashes independently reverified from the downloaded artifact.
    • Two earlier shakeouts failed closed before product validation: run 30559109069 hit a transient cosign download reset, and run 30559621578 proved unauthenticated attestation lookup is rejected. PRs ci: use attestation path in installer canary #123 and ci: authenticate installer attestation check #124 removed the redundant bootstrap and supplied the read-only workflow token; the final run proves the corrected path.

    The T+24h and T+72h observations remain deliberately unchecked until their scheduled runs complete.

  8. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    Pre-campaign kernel freshness resolved

    • PR #125 merged the functionally validated baselines at 8e492e8aa272ae7fd4ae392128e193a53c910c6b.
    • Seven-target enterprise refresh 30558174135 passed Ubuntu, AlmaLinux, CentOS Stream, Rocky, Amazon Linux 2, Amazon Linux 2023, and the Oracle base target; report SHA-256 4c1083c5203a2970c1a64ac5c6434af71cbecabce6c442ba9bf04d9cb119193e.
    • Exact Oracle UEK run 30559962217 installed Oracle-signed packages, rebooted into 6.12.0-204.92.4.4.el9uek.x86_64, attached the required BPF program, and captured the required execve trace event; report SHA-256 c967c8fd950bbec181b75e80c9ac5677d183ac42291d008823bc50c3c239f158.
    • Authoritative post-merge freshness run 30560775651 checked 46 profiles and found zero stale. Downloaded freshness.json SHA-256: 0985d31f8e121eb30c06af641fa455a16ba6379004fc8891d468ea44965e2c73.
    • Stale kernel baselines (weekly freshness report) #89 closed automatically from the workflow. Subscription-walled RHEL remains explicitly uncovered; no Oracle result is presented as RHEL evidence.

  9. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    Pre-campaign technical rollup

    • Local make production-tech-check on main@8e492e8aa272ae7fd4ae392128e193a53c910c6b reported ready at 2026-07-30T16:19:40Z: Go tests/vet, hostile artifact/configuration suite, release consistency, generated documentation, and required support/SLO/incident/schema/upgrade contracts all passed.
    • GitHub stability run 30560941863 independently reported READY at the same commit for build, vet, gofmt, unit tests, govulncheck, release consistency, and generated docs.
    • Downloaded stability report SHA-256: 1795157fee53e932f4d2cdfc75c67b9cb9e687721c63e11ee101c2c4d39fb7a3.

    This is a manual pre-campaign shakeout; it does not substitute for the four dated scheduled VM campaigns.

  10. ErenAri commented on Jul 30, 2026

    @ErenAri
    ErenAri
    on Jul 30, 2026
    ContributorAuthor

    Graduation evidence contract hardened

    PR #126 merged at 7075bde70566334f8c96337f9739d36a6b311b9f with every protected check green.

    The production-readiness generator and stable tag gate now require and validate:

    • exactly three canary manifests in manual, t-plus-24h, t-plus-72h order;
    • schedule provenance for both timed observations;
    • the same RC tag, commit, and image digest as the attested candidate;
    • five passing paths per canary: installer, source, Action, container, and external consumers;
    • unique canary workflow run IDs and exact evidence SHA-256 values;
    • separately hashed [bpfcompat-promotion-incident:v1] fail-closed evidence.

    Regression tests prove missing T+72h, a manual run presented as T+24h, and unmarked incident evidence are rejected. The stable workflow also refuses readiness JSON without all three canaries and completed incident evidence.

    The ignored private bundle at evidence/graduation-v0.4.0/ is seeded with the attested candidate, rollback artifact, manual canary and all component files, and incident note. Candidate attestation verified with checksum-verified GitHub CLI 2.96.0. Future scheduled artifacts remain intentionally absent.

  11. ErenAri commented on Aug 6, 2026

    @ErenAri
    ErenAri
    on Aug 6, 2026
    ContributorAuthor

    Elapsed scheduled-gate audit — 2026-08-06

    RC observation windows: PASS

    • T+24h run 30649359059 was a real schedule event. It completed all five paths successfully and emitted [bpfcompat-rc-canary:v1] with milestone t-plus-24h, exact RC3 commit cba1e09537f2e05e4ba7278efdb5c2d044d1889b, and image digest sha256:842ebbc5746dcfcf98823407a762bdf379e24beb8ea20941ff77f9a74a1a2ac6. Evidence SHA-256: 0134dc4ddef908ca3284612953245ea26aea00aa4b3fab575a92b9445930c7c5. Nested ecosystem run: 30649370618.
    • T+72h run 30756195754 was a real schedule event with the same exact RC bindings and all five paths passing. Evidence SHA-256: bb57039cc075995cd0d5f520f7b3adea62bd20ce901768c4b40162dd3c4fce37. Nested ecosystem run: 30756205291.
    • Every downloaded component hash for both canaries reverified successfully. Both artifacts are now in the ignored private graduation bundle.

    Campaign 1: PASS

    • Scheduled campaign run 30790938510, artifact bpfcompat-latest-kernel-30790938510, commit 7075bde70566334f8c96337f9739d36a6b311b9f.
    • Report start: 2026-08-03T06:42:56Z; workflow duration: 3m02s.
    • Four targets passed: Ubuntu 24.04/6.8, 24.10/6.11, 25.04/6.14, and 25.10/6.17. Required failures: 0. Infrastructure errors: 0. Target p95: 42,689 ms.
    • Report SHA-256, reverified against production-campaign.json: 1ca1d6b2f61c413e90c140bc9ef12906a6ad41c3c48b9ac1afae73100e44e4fe.

    First qualifying upstream Falco schedule: FAIL before VM execution

    • falcosecurity/libs run 30792162435 ran at 1a1d1e7ec9374cff24e9d03f55422590acb8fc92, which is 18 commits after and directly descended from f6fed8edf6a43c3ecbba4bcd44e7df175785d08e.
    • The Falco scap-open build succeeded. No compatibility target ran and no report artifact was produced.
    • Root cause is the upstream pin to bpfcompat v0.3.6 (f1ba21f): that Action downloads the amd64 CLI and validator, then runs sha256sum -c SHA256SUMS; the release checksum file also lists the arm64 CLI, which was not downloaded, so verification fails on the missing file.
    • This is not evidence of a kernel, BPF attach, or Falco loader regression. RC3 already replaces the affected path with selected-asset checksum and GitHub-attestation verification.
    • Both Falco completion checkboxes remain open. A focused upstream pin update and a later successful schedule run are still required.

  12. ErenAri commented on Aug 6, 2026

    @ErenAri
    ErenAri
    on Aug 6, 2026
    ContributorAuthor

    2026-08-06 consumer-pin remediation

    The v0.3.6 prebuilt consumer failure has been remediated at both controlled call sites without changing supported runtime behavior:

    The first #129 check set was damaged solely by the active GitHub Actions major outage: unrelated jobs failed during runner setup while resolving pinned Actions with HTTP Service Unavailable. A branch-only empty retry commit created a clean nine-check set and will be omitted by squash merge. Those checks are currently queued behind the outage; branch protections remain intact.

    Manual Falco validation proves the fix but does not satisfy the graduation gate. Keep the Falco items unchecked until falcosecurity/libs#3076 merges and the first subsequent upstream schedule run succeeds with a verified artifact.

  13. ErenAri commented on Aug 9, 2026

    @ErenAri
    ErenAri
    on Aug 9, 2026
    ContributorAuthor

    2026-08-09 no-wait completion audit

    • Merged evidence-only profile/catalog refresh #131 at 5aa81a235de62d941eedf1fd7f2d6ec636c1130b. It does not modify application code or matrices/latest-kernel-sweep.yaml, so Campaign 1 remains valid.
    • Current-main hosted KVM dress rehearsal 31319665219 passed all four Ubuntu targets with zero required failures and zero infrastructure errors. Every target passed load, required attach, and the execve behavior assertion. Report SHA-256: d4b568579134792c112b9a33fe92d7419a288c5854a316e293c3823e654e6f8f; campaign manifest SHA-256: d257fec790010c4262f3edc893d900f1a181e0e03bd3f91da7655da655b15f1b. Its recorded event is correctly workflow_dispatch, so this is a shakeout and does not count as Campaign 2.
    • The two remaining Amazon public-image lags now have a complete implementation in draft #133. Real KVM runs installed, rebooted into, and behavior-validated exact AL2 5.10.260-259.1061 and AL2023 6.1.177-224.371; the branch freshness oracle reports 0 stale of 57 checked. test: validate current Amazon kernels #133 remains post-v0.4 because it changes supported executor behavior and merging it now would reset the campaign window.
    • Certification-like Falco conformance remains complete and green in draft #130, also staged post-v0.4 for the same boundary reason.

    The only v0.4 gates still open are future scheduled Campaigns 2–4, upstream maintainer merge/workflow approval for falcosecurity/libs#3076 followed by a successful upstream scheduled run, and final evidence assembly/publication after those inputs exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions