policy: mcptest policy simulate OSS governance gate (WOR-1421)#70
Open
rickcrawford wants to merge 2 commits into
Open
policy: mcptest policy simulate OSS governance gate (WOR-1421)#70rickcrawford wants to merge 2 commits into
mcptest policy simulate OSS governance gate (WOR-1421)#70rickcrawford wants to merge 2 commits into
Conversation
A deterministic, offline policy simulator: it reads a small declarative policy file plus saved mcptest artifacts (run report, judge certification, conformance report, security report, model-compat diff, evidence artifact), extracts named facts from each, evaluates the policy rules against them, applies expiring waivers, and prints a pass/fail/warn verdict with a deterministic exit code. No network, no live run, so a team can gate a release locally before adopting heavier enterprise policy infrastructure. - mcptest-core policy.rs (pure): PolicyFile/PolicyRule (one comparator each: max/min/equals/one_of; severity fail|warn) + PolicyWaiver (rule/owner/reason/ expiry/issue), seven facts_from_* extractors over serde_json::Value, and evaluate() producing a PolicyOutcome. A failing rule with an active waiver is Waived; an expired waiver fails closed; a missing fact is Unevaluated and fails closed (a missing input never silently passes). - mcptest policy simulate (cli/args/policy.rs + handlers/policy.rs): reads the policy YAML and whichever --artifact files are supplied, evaluates, renders pretty/json. Dry-run always exits 0; --gate exits 1 on fail. - examples/policy/policy.yml worked example (skipped by the examples gate, not a run suite), docs/policy-simulator.md with the full fact catalog, cli-reference + command-groups + help template, llms regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-policy-simulator # Conflicts: # docs-site/llms-full.sha256
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A deterministic, offline policy simulator: reads a small declarative policy file plus saved mcptest artifacts, extracts named facts, evaluates the rules, applies expiring waivers, and prints a pass/fail/warn verdict with a deterministic exit code. It answers "would this release pass our quality/security/judge-certification requirements" without any external service, so a team can gate locally before adopting heavier enterprise policy infra.
Design
mcptest-core/src/policy.rs(pure, fully unit-tested):PolicyFile/PolicyRule(each names afact+ one comparator:max/min/equals/one_of;severity: fail|warn) /PolicyWaiver(rule/owner/reason/expiry/issue). Sevenfacts_from_*extractors overserde_json::Value(run report, judge cert incl. computedjudge.expired, conformance, security severity counts, model-compat, evidence).evaluate()→PolicyOutcome.Waived; an expired waiver fails closed; a rule whose fact is missing isUnevaluatedand fails closed (a missing input never silently passes). Dependency-free RFC3339 parsing for waiver expiry.mcptest policy simulate(cli/args/policy.rs+handlers/policy.rs): reads the policy YAML and whichever--artifactfiles are supplied, renders pretty/json. Dry-run always exits0;--gateexits1on fail.Tests
13 core unit tests (every comparator, computed expiry, active vs expired waiver, missing-fact, warn-severity, security tallies, RFC3339 epochs) + 6 CLI integration tests (pass, gate-fail, dry-run-still-zero, active waiver, expired waiver fail-closed, json). Help-drift guard satisfied.
Docs / examples
docs/policy-simulator.md(full fact catalog + comparator/waiver/exit-code model),examples/policy/policy.yml(worked example; added to the examples-gate skip-list since it is a policy file, not a run suite), cli-reference + command-groups + help template, llms regen.Acceptance
Local simulation over saved artifacts (no external services) ✓; gates on conformance/security severity/drift/judge-cert/model-compat/evidence ✓; cited per-rule rows + deterministic exit codes ✓; explicit, expiring, auditable waivers ✓.
Gate
clippy
-D warnings,cargo doc -D warnings, module-size, em-dash, examples gate, all tests green locally; merged up to current main (#69).