fix(ci): harden release and evaluation acceptance - #117
Merged
Conversation
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
Successive clean acceptance runs exposed defects that unit-only gates could not reveal:
Completecondition hid an already-terminal failed restore;The apparent Ollama runtime regression reproduced across versions and was traced to evaluator/App parity and prompt behavior. The corrected exact-head run now qualifies 0.32.5 without weakening a trajectory or laundering cost evidence.
How
The Platform workflow now validates the shared deployment as a private, caller-specific system and always proves cleanup. Evaluation constructs the same policy-bearing App used in production, retains upstream evaluator instrumentation, runs cases serially against the immutable Qwen model, and emits a minimized release verdict. Cost comparison accepts only complete critical transcripts.
The Release workflow starts from protected
main, requires successful exact-SHA CI, Docs, Scan, Eval, and Platform runs, preflights both images before registry writes, preserves source and published digest chains, and reconciles only resources whose ownership is proven. For issue-based freshness evidence, GitHub renders both the candidate template and issue; only genuine rendered task-list checkboxes with an exact visible-label inventory are accepted.Test plan
mise run install:maintainermise run formatmise run checkmise run test— 630 passed, 95.53% branch coveragemise run scan— gitleaks and Trivy cleanmise run build— docs plus agent and MLflow imagesCloud scope
The optional GKE lab was applied, deployed, exercised, and destroyed in
agentops-open-courseat source7757ef97ba2e54dc30e1026df0f57ddd3ffae3cd. A fresh read-only audit found no course cluster, disk, repository, address, bucket, instance, forwarding rule, router, or course IAM binding afterward. That is real GCP boundary evidence for7757ef9, not an exact-head cloud claim for9ac1ae7; the final branch is instead qualified by the exact-head account-free Platform run above.