Skip to content

fix(ci): harden release and evaluation acceptance - #117

Merged
fmind merged 10 commits into
mainfrom
fix/platform-probe-dns
Jul 31, 2026
Merged

fix(ci): harden release and evaluation acceptance#117
fmind merged 10 commits into
mainfrom
fix/platform-probe-dns

Conversation

@fmind

@fmind fmind commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What

  • harden exact-head Platform acceptance: DNS-preserving network isolation, measured MLflow memory, private exposure, PII-stable canaries, fail-fast restore evidence, sanitized diagnostics, and context-correct teardown
  • restore production App policy parity in ADK and MLflow evaluation paths, make recall and guarded-action trajectories unambiguous, and prevent failed critical transcripts from becoming cost baselines
  • qualify Ollama 0.32.5 with immutable model lineage, strict trajectories, deterministic scorers, groundedness, and a reviewed complete cost baseline
  • harden v0.5.0 publication with exact-main workflow evidence, ownership-proven reconciliation, source-bound image artifacts, strict JSON types, and GitHub-rendered freshness checklists
  • add convention, render, mutation, evaluator-parity, runtime, reconciliation, and adversarial regressions for those contracts

Why

Successive clean acceptance runs exposed defects that unit-only gates could not reveal:

  • replacing all probe egress also removed DNS;
  • the digest-pinned MLflow image was OOM-killed below its measured 2 GiB startup ceiling;
  • root-scoped Skaffold teardown resolved relative build contexts outside this repository;
  • numeric telemetry and backup markers were correctly redacted as financial identifiers;
  • waiting only for a Job's Complete condition hid an already-terminal failed restore;
  • broad Kubernetes diagnostics could preserve unsanitized termination messages;
  • ADK 2.6's stock evaluator and the MLflow harness rebuilt a bare-agent runner after policy moved to the App boundary;
  • an instruction contradiction and failed historical trajectory could contaminate the cost baseline;
  • raw Markdown freshness parsing accepted exact checkbox labels hidden in HTML blocks or unclosed comments;
  • permissive JSON coercion accepted malformed release-evidence scalar types.

The apparent Ollama runtime regression reproduced across versions and was traced to evaluator/App parity and prompt behavior. The corrected exact-head run now qualifies 0.32.5 without weakening a trajectory or laundering cost evidence.

How

The Platform workflow now validates the shared deployment as a private, caller-specific system and always proves cleanup. Evaluation constructs the same policy-bearing App used in production, retains upstream evaluator instrumentation, runs cases serially against the immutable Qwen model, and emits a minimized release verdict. Cost comparison accepts only complete critical transcripts.

The Release workflow starts from protected main, requires successful exact-SHA CI, Docs, Scan, Eval, and Platform runs, preflights both images before registry writes, preserves source and published digest chains, and reconciles only resources whose ownership is proven. For issue-based freshness evidence, GitHub renders both the candidate template and issue; only genuine rendered task-list checkboxes with an exact visible-label inventory are accepted.

Test plan

  • mise run install:maintainer
  • mise run format
  • mise run check
  • mise run test — 630 passed, 95.53% branch coverage
  • mise run scan — gitleaks and Trivy clean
  • mise run build — docs plus agent and MLflow images
  • exact-head CI 30668146441, Docs 30668146440, Scan 30668146468, and CodeQL 30668144649 green
  • exact-head Platform 30668170928 green, including isolation, load budgets, telemetry, backup/restore, and complete teardown
  • exact-head Eval 30668170951 green with all six signals and independently validated sanitized evidence

Cloud scope

The optional GKE lab was applied, deployed, exercised, and destroyed in agentops-open-course at source 7757ef97ba2e54dc30e1026df0f57ddd3ffae3cd. A fresh read-only audit found no course cluster, disk, repository, address, bucket, instance, forwarding rule, router, or course IAM binding afterward. That is real GCP boundary evidence for 7757ef9, not an exact-head cloud claim for 9ac1ae7; the final branch is instead qualified by the exact-head account-free Platform run above.

@fmind fmind changed the title fix(platform): harden acceptance and teardown fix(ci): harden release acceptance Jul 31, 2026
@fmind fmind changed the title fix(ci): harden release acceptance fix(ci): harden release and evaluation acceptance Jul 31, 2026
@fmind
fmind merged commit ba34f8f into main Jul 31, 2026
14 checks passed
@fmind
fmind deleted the fix/platform-probe-dns branch July 31, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant