Review-response hardening: external validity, attribution rigor, scanner fix - #30
Closed
zacharyr0th wants to merge 18 commits into
Closed
Review-response hardening: external validity, attribution rigor, scanner fix#30zacharyr0th wants to merge 18 commits into
zacharyr0th wants to merge 18 commits into
Conversation
Update the pinned runtime artifacts so Mise can verify the selected Python releases.
The scanner inherited the built-in accounts.tenant_id column for any custom (domain_path) domain with no tenancy config, flagging every trace as tenant-scope-missing. builtin_domain_tenant_column() now returns a column only for a built-in domain with no domain_path, and sql_preserves_tenant_scope() skips the check (rather than reporting a violation) when no tenancy basis is configured. Adds table_tenant_columns for per-table tenancy. Built-in behavior and true-positive detection are unchanged; a real repo scan (metricflow) drops from 163 to 95 findings with zero spurious tenant-scope findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the strawman baselines with two a competitor would deploy: conventional_test_suite (a spec-derived engineer test suite, 1579/1720) and property_differential (Cedar-style pairwise differential, 899/1720). Adds evaluate_false_positives(), which the existing framework lacked, to score baselines on clean traces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… metrics - compound: stack 2-3 distinct-surface skews per case; first-transition attribution is stable under composition. - counterfactual: validate attribution interventionally (repair the attributed layer -> witness must vanish; repair another -> it must persist), replacing circular localization accuracy. Teeth-tested. - minimization: per-witness reduction ratios and 1-minimality; the bounded reducer reaches 1-minimality on the standard suites but does not guarantee it. Wires compound/counterfactual/minimization-report CLI subcommands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- adversarial clean controls: 1000+ clean controls per domain; 0 detector false positives vs 285 for naive denial-flagging. The shipped 80-case suite stays byte-identical. - scalability: deterministic pairwise covering-array generator (~90% fewer cases, coverage-verified) and flat per-case throughput curves. - difficulty tiers from the baseline kill matrix, for a leaderboard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Property-tests (Hypothesis, 400 examples) plus an exhaustive sweep that a witness always implies a contract violation; characterizes completeness per witness class rather than claiming it globally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ClickHouseAdapter over the HTTP interface (stdlib only, no new dependency), DDL row-policy fixture, env-gated integration tests (verified against ClickHouse 25.6), evidence script, and a CI job. - CI now runs on push and pull_request (was workflow_dispatch only); the PostgreSQL integration job runs by default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mutates the scanner's adapters in process and classifies the effect: 16 of 18 adapter mutations silently corrupt scan output (hide or invent findings), 1 is loud. Documents the risk and mitigations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Asks a model to emit semantic queries from paraphrase sets under a manifest-derived prompt with a repair budget, and probes whether a version-skewed manifest changes the emitted plan. No paid runs are made (guarded behind an explicit opt-in flag); stub results verify the harness only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A self-contained INSERT/UPDATE/DELETE containment model with its own witness classes, surfaces, operators, simulator, and first-transition detector; write containment via database WITH CHECK. 48/48 killed, 0 false positives, 100% localization. Read pipeline untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- incident_reconstruction: 19 real, cited public faults (RLS CVEs, incident writeups) reconstructed as deterministic fixtures (19/19 killed, 100% localization); 6 dropped honestly with reasons. - spec_blind: 42 mutants authored from the contract spec without detector access; the detector agrees on 39/42, and the 3 misses expose a genuine contract ambiguity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scans metricflow, cube, WrenAI, and midday. Honest outcome: 0 new real bugs, a real-input false-positive measurement (~1.4%), a true-positive demo on cube's own broken ACL fixtures, and 5 documented scanner gaps (2 since fixed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README and evidence snapshot now lead with the 159-miss gap and frame 1720/1720 as a construction-consistency check. Adds the new comparator rows to the baselines table, an Extended Studies index, and review-response.md mapping every review item to what changed (with a paper-grade classification). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bump the Python package to 1.1.0 and record the review-response changes in the changelog. PyPI package only; the npm runtime and gateway packages are unchanged this cycle. Updates the composite-action tag references to v1.1.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zacharyr0th
marked this pull request as ready for review
July 22, 2026 04:48
Member
Author
|
Superseded by the validated linear release in #32, which is now merged to main with the same release tree. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Responds to the external review of the PolicyStrata artifact. 13 commits, almost entirely additive. Each commit is one feature cluster (code + tests + script + doc); the last commit is the 1.1.0 release bump.
What changed
conventional_test_suite(1579/1720) andproperty_differential(899/1720) replace the strawmen.accounts.tenant_idcolumn for custom (domain_path) domains with no tenancy config; adds per-table tenancy config. metricflow drops 163 → 95 findings, 0 spurious; built-in behavior and true-positive detection unchanged.docs/review-response.mdmaps every review item to what changed and classifies each study as paper-grade / supporting / future-work.Release
Bumped to 1.1.0 (PyPI only — the npm runtime and gateway packages are unchanged this cycle). Tag
v1.1.0is pushed; the CHANGELOG and composite-action tag references are updated. Publishing to PyPI is a manualpublish.ymlworkflow_dispatchfrom the tag (trusted publishing), to be run after merge.Verification
uv run pytest: 328 passed, 8 skipped (DB integration tests need env vars).scripts/reproduce-final.sh: 1720/1720 killed, 0/80 clean-control false positives — unchanged by the additions; frozen-suite verify passes with 0 hash mismatches.Honest caveats
🤖 Generated with Claude Code