Skip to content

Release v0.6.0: expanded real-model evidence, adapter lint, reporting, governance - #35

Merged
sergeyizmailov merged 2 commits into
mainfrom
suite-results
Jul 24, 2026
Merged

Release v0.6.0: expanded real-model evidence, adapter lint, reporting, governance#35
sergeyizmailov merged 2 commits into
mainfrom
suite-results

Conversation

@sergeyizmailov

Copy link
Copy Markdown
Owner

Summary

Release v0.6.0 — consolidated release PR (parallel reviewer + this pass, all verified locally).

Real-model evidence (Priority 0/3 of the pre-submission review):

  • Reviewed Llama 3.3 70B expanded-suite report: 80 trials across the four representative pairs — 30/40 attacks produced the expected violations (V1/V2 10/10, V6-as-V2 10/10, V7 10/10; V3 not exercised on the first prompt), 0/40 benign false positives, 40/40 benign task success, 0 errors.
  • Separately preserved 20-run V3 follow-up with the strengthened prompt: 10/10 expected V3 violations, 10/10 benign clean.
  • Qwen expanded-suite first attempt is not published (4 attack-trial timeouts + harness_commit misattribution in the pre-fix harness); it is preserved privately and the rerun is one dispatch of the now per-scenario real-model-benchmarks workflow. Every published report carries a review block with source hash and an explicit corrections log; per-run records are never modified.

Feature and hardening changes (already reviewed in #31/#32/#34): validate-adapter linter, --junit-detail, contract-validation wording, GOVERNANCE.md, ruff baseline pinned + CI gate, dependabot cooldown, harness hardening (HARNESS_COMMIT at process start, endpoint URL validation, progress flushing), benchmark review protocol, per-scenario benchmark workflow with concurrency control.

Version: 0.6.0 across pyproject.toml, __init__.py, CITATION.cff, CHANGELOG (0.6.0 section).

Verification

  • 364 tests pass, corpus 75/75 in all defense modes, ruff check . clean, actionlint clean, repo-wide zizmor clean, git diff --check clean, wheel builds as delegationbench-0.6.0.
  • Suite aggregates independently recomputed from raw run records before commit.
  • No credentials in any artifact; endpoint redacted.

…, governance

Consolidated release commit (parallel reviewer + this pass, all verified):
- validate-adapter trace linter with --strict; malformed-input hardening,
  nonce-reuse and principal-mismatch findings.
- Expanded real-model suite harness (V1/V2, V3, V6, V7 pairs) with
  reviewed Llama 3.3 70B evidence: 80 trials across four pairs (30/40
  expected violations, 0/40 benign FPs, 40/40 benign success) plus a
  20-run strengthened-prompt V3 follow-up (10/10 V3 detected, 10/10
  benign clean). First-pass V3 prompt-sensitivity documented; Qwen
  expanded-suite first attempt preserved privately (4 timeouts +
  harness_commit misattribution), rerun pending via the now per-scenario
  benchmark workflow.
- --junit-detail levels, corpus contract-validation wording, explicit
  bundled-corpus fallback message.
- GOVERNANCE.md; ruff baseline pinned and gated in CI; dependabot
  cooldown; benchmark review protocol; harness hardening
  (HARNESS_COMMIT at start, endpoint URL validation, progress flushing).
- Version 0.6.0 across pyproject, __init__, CITATION.cff, CHANGELOG.
@sergeyizmailov
sergeyizmailov enabled auto-merge (rebase) July 24, 2026 18:02
@sergeyizmailov
sergeyizmailov merged commit bf5d661 into main Jul 24, 2026
10 checks passed
@sergeyizmailov
sergeyizmailov deleted the suite-results branch July 24, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant