Add mixture-aware post-failure design diagnostic

docs/design_mixture_diagnostic.md

@@ -0,0 +1,121 @@
+# Post-failure mixture-aware design diagnostic
+
+This diagnostic is intentionally separate from the immutable N=60 stress failure
+frozen at commit `d1251dddeaa706d79feaaec99185dcc4236aa4a3`. It does not
+replace, tune, or reinterpret that result. Its method and random streams are
+committed before the new evaluation is run.
+
+## Locked method
+
+The diagnostic is restricted to the chronologically earlier heuristic maximin
+allocation at N=60. It uses three-fold cross-fitting, so every observation is
+scored only by models fitted without that observation. The pure family contains
+the six prespecified single-candidate regressions.
+
+The targeted composite family contains all 15 unordered candidate pairs. For
+each training fold, an intercept and two nonnegative slopes are fitted. A free
+total amplitude makes this a nonnegative cone over each pairwise convex simplex;
+a pure candidate is a boundary case. The maximum held-out score across all 15
+pairs is the familywise composite statistic.
+
+For each possible pure winner, calibration under that matched pure generator
+sets separate upper thresholds for:
+
+- the maximum pairwise-composite score improvement over the pure model; and
+- a cross-fitted residual lack-of-fit ratio, defined as validation squared error
+  divided by the variance fitted on the corresponding training fold.
+
+Every upper threshold uses the one-based order statistic
+`ceil((n+1)(1-alpha))`. With `n=200` and familywise `alpha=0.05`, this is
+rank 191. The pure-over-null statistic is the maximum over all six pure
+candidates under the null. The composite statistic is the maximum over all 15
+pairs, calibrated separately under each matched pure candidate. The
+winner/runner statistic is the best-minus-second-best pure score under the null;
+the residual threshold is candidate-specific.
+
+The independent audit measures four distinct forms of power, each with a 95%
+Wilson interval:
+
+- correct pure-call retention, separately for each pure candidate;
+- correct abstention under the null;
+- correct abstention, separately for each of the 15 mixture pairs; and
+- correct abstention for the fixed out-of-span probe.
+
+A candidate or contrast is enabled only when its own audit Wilson lower bound is
+at least 0.70. An underpowered pure winner is forced to abstain. Pair/null/probe
+evaluation rates remain descriptive but receive the status
+`mandatory_abstain_underpowered` when their corresponding audit gate fails.
+Thus an easy contrast cannot license a claim for a difficult pair. These power
+rules were fixed before evaluation and are not relaxed after seeing its output.
+
+The fixed streams and replicate counts are:
+
+| Purpose | Seed | Replicates |
+|---|---:|---:|
+| threshold calibration | 196613 | 200 |
+| independent calibration audit | 262147 | 200 |
+| one-time evaluation | 324949 | 200 |
+
+All three streams are disjoint. Evaluation data do not set thresholds, choose
+models, change the power rule, or alter the allocation.
+
+## Scope and identifiability
+
+The extension is targeted to positive two-candidate mixtures. It is not a
+general open-set classifier. The nonnegative sign restriction follows the
+declared positive-effect simulation and would need separate justification for
+an empirical model with unoriented effects.
+
+Cross-fitting uses all 60 observations once as held-out predictions, but it
+cannot create information absent from the design. The immutable diagnostic
+already shows that several mixtures are only weakly separated from their best
+pure affine approximation on this support. The artifact therefore reports a
+population oracle log-score-gap index for every pair. Low finite-sample audit
+power forces abstention; it is not treated as evidence that the pure hypothesis
+is correct.
+
+The fixed orthogonalized nonlinear residual remains one bounded out-of-span
+probe. The residual lack-of-fit gate is calibrated for the matched Gaussian
+simulation, not arbitrary biological misspecification, serial dependence,
+animal hierarchy, indicator dynamics, or an executable sequential protocol.
+Passing this diagnostic would support only the declared simulation family and
+would not constitute a main-paper robustness claim without an independent
+freeze.
+
+## Frozen one-time result
+
+The one-time evaluation was produced at commit
+`564ab3cfe42f67867d70a5d5869c7c4401d94c8e` with configuration digest
+`338ac9eea51f37d09b46303bdec66d5acc8d5e6fb16740c9afb036de3063c5a7`.
+All six matched-pure contrasts and the null contrast passed their independent
+audit-power gates. Nine of the 15 pairwise mixtures passed their own audit gate;
+the other six are reported as `mandatory_abstain_underpowered`. The fixed
+out-of-span probe also failed its audit-power gate and is therefore disabled.
+
+Across the enabled pairwise-mixture contrasts, the largest evaluation
+false-pure-call rate was 0.195 (95% Wilson upper bound 0.2554). The null
+false-pure-call rate was 0.005 (upper bound 0.0278). The disabled out-of-span
+probe had a descriptive false-pure-call rate of 0.73 and cannot support an
+open-set claim. The frozen result therefore establishes partial, targeted
+sensitivity with mandatory abstention, not general open-set robustness.
+
+## Reproduction
+
+From a clean checkout of the exact producer commit:
+
+```bash
+bayesian-ach-mixture-diagnostic \
+  --repo-root . \
+  --code-sha <producer-commit> \
+  --baseline-artifact results/design-open-set-stress-n60 \
+  --locked-allocation /absolute/path/optimal_design_allocation_seed7.csv \
+  --locked-allocation-sha256 a823be49faf6c6cbebf60b11d4b5ca895cf7734d6e9c577ee98f97a5907b69b2 \
+  --locked-design-code-sha 1b2028929ac6ebc1cce0882f0c22af9918044342 \
+  --locked-allocation-seed 7 \
+  --output /absolute/path/mixture-aware-diagnostic
+```
+
+The command verifies the immutable baseline, exact allocation hash, explicit
+seed metadata, source commit, deterministic allocation reconstruction, clean
+worktree, and producer commit. It writes checksum-bound tables and immediately
+runs the independent artifact verifier.

pyproject.toml

@@ -71,6 +71,8 @@ plot = ["matplotlib>=3.8"]
 bayesian-ach = "bayesian_ach.cli_ext:main"
 bayesian-ach-design = "bayesian_ach.design_cli:main"
 bayesian-ach-design-stress = "bayesian_ach.design_stress_cli:main"
+bayesian-ach-mixture-diagnostic = "bayesian_ach.design_mixture_diagnostic_cli:main"
+bayesian-ach-verify-mixture-diagnostic = "bayesian_ach.design_mixture_diagnostic_cli:verify_main"
 bayesian-ach-replay = "bayesian_ach.replay_cli:main"
 
 [tool.setuptools.packages.find]

results/design-mixture-diagnostic-n60/SHA256SUMS.csv

@@ -0,0 +1,10 @@
+path,bytes,sha256
+summary.json,4231,1311330a9f7b3278eb16dbb48d65b57b8337a695c6ad2026cac1efbff795b3c2
+thresholds.csv,870,2da2db3544398e84e124ae591ad3acac08d0ff9d716871067795fba630b9410f
+calibration_audit.csv,5328,1333d0eae21afa6f87675db935d2ce69e026f0149d7d9c6707341a25bfdac444
+pure_evaluation.csv,1192,58b2d40de305d5f4c789b3b92c0110bdd74e98c8649928f8cea1c6810cded2d1
+null_evaluation.csv,279,782463aeb6919c0753ebe05212d15744a46e8df2e4d71a4a4d279b46741df074
+mixture_evaluation.csv,3154,37a864c5a8410b7ea91fcefee4a4ae2df11fc1a1d397fb4f7651520abbcf5a3e
+out_of_span_evaluation.csv,500,181099f69267c8dc119d45fbd7b66a0a003cc05912f5dae459ef2742c5f07a9f
+geometry.csv,1722,30bcf7c226d17f8a543cf08f58052ab33529b68d9e1a585c05da7ce3f4001d49
+artifact_manifest.json,3488,bf08f458a3f34339a62c8ae74fc0a1cafdf7f74f21e939d174df86b021bc21b7

results/design-mixture-diagnostic-n60/artifact_manifest.json

@@ -0,0 +1,101 @@
+{
+  "artifact": "post_failure_mixture_aware_design_diagnostic",
+  "configuration": {
+    "alpha": 0.05,
+    "budget": 60,
+    "calibration_audit_replicates": 200,
+    "calibration_audit_seed": 262147,
+    "calibration_replicates": 200,
+    "confidence_level": 0.95,
+    "design": "maximin_optimized",
+    "effect_size": 1.0,
+    "evaluation_replicates": 200,
+    "evaluation_seed": 324949,
+    "folds": 3,
+    "minimum_pure_retention_wilson_lower": 0.7,
+    "minimum_rejection_power_wilson_lower": 0.7,
+    "noise_std": 1.0,
+    "threshold_seed": 196613
+  },
+  "configuration_sha256": "338ac9eea51f37d09b46303bdec66d5acc8d5e6fb16740c9afb036de3063c5a7",
+  "files": [
+    {
+      "bytes": 4231,
+      "path": "summary.json",
+      "sha256": "1311330a9f7b3278eb16dbb48d65b57b8337a695c6ad2026cac1efbff795b3c2"
+    },
+    {
+      "bytes": 870,
+      "path": "thresholds.csv",
+      "sha256": "2da2db3544398e84e124ae591ad3acac08d0ff9d716871067795fba630b9410f"
+    },
+    {
+      "bytes": 5328,
+      "path": "calibration_audit.csv",
+      "sha256": "1333d0eae21afa6f87675db935d2ce69e026f0149d7d9c6707341a25bfdac444"
+    },
+    {
+      "bytes": 1192,
+      "path": "pure_evaluation.csv",
+      "sha256": "58b2d40de305d5f4c789b3b92c0110bdd74e98c8649928f8cea1c6810cded2d1"
+    },
+    {
+      "bytes": 279,
+      "path": "null_evaluation.csv",
+      "sha256": "782463aeb6919c0753ebe05212d15744a46e8df2e4d71a4a4d279b46741df074"
+    },
+    {
+      "bytes": 3154,
+      "path": "mixture_evaluation.csv",
+      "sha256": "37a864c5a8410b7ea91fcefee4a4ae2df11fc1a1d397fb4f7651520abbcf5a3e"
+    },
+    {
+      "bytes": 500,
+      "path": "out_of_span_evaluation.csv",
+      "sha256": "181099f69267c8dc119d45fbd7b66a0a003cc05912f5dae459ef2742c5f07a9f"
+    },
+    {
+      "bytes": 1722,
+      "path": "geometry.csv",
+      "sha256": "30bcf7c226d17f8a543cf08f58052ab33529b68d9e1a585c05da7ce3f4001d49"
+    }
+  ],
+  "inputs": [
+    {
+      "checksum_table_sha256": "44a5188c43bda52e6fc9dc7007cf2de44a9671e9c5477ac88c3173c06cfdbd80",
+      "kind": "immutable_original_design_stress_failure",
+      "manifest_sha256": "d840a2ec34f5a386109c7f985033b53144fdcc1b1a0e0592b3274c0e38902b64",
+      "producer_commit": "c71695fda83ae93407599a909097962ee3fa9e0e",
+      "verified_payload_count": 9
+    },
+    {
+      "allocation_bytes": 14450,
+      "allocation_file": "optimal_design_allocation_seed7.csv",
+      "allocation_file_seed_field_present": false,
+      "allocation_seed": 7,
+      "allocation_seed_source": "explicit_cli_metadata",
+      "allocation_sha256": "a823be49faf6c6cbebf60b11d4b5ca895cf7734d6e9c577ee98f97a5907b69b2",
+      "construction_contract": {
+        "all_three_allocations_reconstructed": true,
+        "allocation_seed": 7,
+        "comparator_cap_semantics": "the maximin cap does not apply to the deterministic coupled-novelty or uniform-factorial constructors",
+        "maximin_max_point_fraction": 0.15,
+        "maximin_maximum_count_by_budget": {
+          "60": 9
+        }
+      },
+      "design_budgets": {
+        "coupled_novelty": 60,
+        "maximin_optimized": 60,
+        "uniform_factorial": 60
+      },
+      "kind": "chronologically_locked_primary_design_allocation",
+      "source_code_sha": "1b2028929ac6ebc1cce0882f0c22af9918044342",
+      "source_repository": "IPS-Stuttgart/Bayesian-ACh"
+    }
+  ],
+  "producer_commit": "564ab3cfe42f67867d70a5d5869c7c4401d94c8e",
+  "producer_git_dirty": false,
+  "repository": "IPS-Stuttgart/Bayesian-ACh",
+  "schema_version": 1
+}

results/design-mixture-diagnostic-n60/calibration_audit.csv

@@ -0,0 +1,24 @@
+scenario,candidate,audit_measure,replicates,successes,wrong_pure_calls,abstentions,rate,wilson_lower,wilson_upper,minimum_wilson_lower,contrast_enabled,reasons
+matched_pure,innovation_l2,correct_pure_retention_rate,200,158,5,37,0.79,0.7283538314009755,0.84071587930021,0.7,True,"{""pairwise_composite_better"":1,""pure_ambiguity"":29,""pure_call"":163,""residual_lack_of_fit"":7}"
+matched_pure,surprise,correct_pure_retention_rate,200,164,4,32,0.82,0.7608852497127511,0.8670537414057983,0.7,True,"{""pairwise_composite_better"":2,""pure_ambiguity"":25,""pure_call"":168,""residual_lack_of_fit"":5}"
+matched_pure,gain,correct_pure_retention_rate,200,188,0,12,0.94,0.8980683070423372,0.9653478057456681,0.7,True,"{""pairwise_composite_better"":3,""pure_call"":188,""residual_lack_of_fit"":9}"
+matched_pure,update_l2,correct_pure_retention_rate,200,181,0,19,0.905,0.8563983883518006,0.9383368972826135,0.7,True,"{""pairwise_composite_better"":4,""pure_ambiguity"":7,""pure_call"":181,""residual_lack_of_fit"":8}"
+matched_pure,information_gain,correct_pure_retention_rate,200,188,0,12,0.94,0.8980683070423372,0.9653478057456681,0.7,True,"{""pairwise_composite_better"":3,""pure_ambiguity"":1,""pure_call"":188,""residual_lack_of_fit"":8}"
+matched_pure,change_probability,correct_pure_retention_rate,200,183,0,17,0.915,0.8681041292843443,0.9462542498225244,0.7,True,"{""pairwise_composite_better"":7,""pure_call"":183,""residual_lack_of_fit"":10}"
+null,null,correct_abstention_rate,200,197,3,197,0.985,0.9568342712073097,0.9948857622067417,0.7,True,"{""null_not_rejected"":196,""pure_ambiguity"":1,""pure_call"":3}"
+fifty_fifty_mixture,innovation_l2+surprise,correct_abstention_rate,200,86,114,86,0.43,0.36334322333434416,0.4992951223584731,0.7,False,"{""null_not_rejected"":1,""pairwise_composite_better"":3,""pure_ambiguity"":80,""pure_call"":114,""residual_lack_of_fit"":2}"
+fifty_fifty_mixture,innovation_l2+gain,correct_abstention_rate,200,168,32,168,0.84,0.7828592983793289,0.8843258796841297,0.7,True,"{""pairwise_composite_better"":103,""pure_ambiguity"":64,""pure_call"":32,""residual_lack_of_fit"":1}"
+fifty_fifty_mixture,innovation_l2+update_l2,correct_abstention_rate,200,174,26,174,0.87,0.8163364812775159,0.9097179772033067,0.7,True,"{""pairwise_composite_better"":101,""pure_ambiguity"":73,""pure_call"":26}"
+fifty_fifty_mixture,innovation_l2+information_gain,correct_abstention_rate,200,139,61,139,0.695,0.6280144790202163,0.7546358436926496,0.7,False,"{""pairwise_composite_better"":49,""pure_ambiguity"":89,""pure_call"":61,""residual_lack_of_fit"":1}"
+fifty_fifty_mixture,innovation_l2+change_probability,correct_abstention_rate,200,166,34,166,0.83,0.7718411637168737,0.8757209208741302,0.7,True,"{""pairwise_composite_better"":98,""pure_ambiguity"":66,""pure_call"":34,""residual_lack_of_fit"":2}"
+fifty_fifty_mixture,surprise+gain,correct_abstention_rate,200,187,13,187,0.935,0.8919809207009312,0.961623645350847,0.7,True,"{""pairwise_composite_better"":66,""pure_ambiguity"":121,""pure_call"":13}"
+fifty_fifty_mixture,surprise+update_l2,correct_abstention_rate,200,163,37,163,0.815,0.7554293723884824,0.8626980719938395,0.7,True,"{""pairwise_composite_better"":74,""pure_ambiguity"":88,""pure_call"":37,""residual_lack_of_fit"":1}"
+fifty_fifty_mixture,surprise+information_gain,correct_abstention_rate,200,123,77,123,0.615,0.5459986722024608,0.6796669027307678,0.7,False,"{""pairwise_composite_better"":43,""pure_ambiguity"":78,""pure_call"":77,""residual_lack_of_fit"":2}"
+fifty_fifty_mixture,surprise+change_probability,correct_abstention_rate,200,161,39,161,0.805,0.7445595562538726,0.8539447946559949,0.7,True,"{""null_not_rejected"":1,""pairwise_composite_better"":109,""pure_ambiguity"":49,""pure_call"":39,""residual_lack_of_fit"":2}"
+fifty_fifty_mixture,gain+update_l2,correct_abstention_rate,200,108,92,108,0.54,0.47082289188953336,0.6076694820002854,0.7,False,"{""pairwise_composite_better"":41,""pure_ambiguity"":64,""pure_call"":92,""residual_lack_of_fit"":3}"
+fifty_fifty_mixture,gain+information_gain,correct_abstention_rate,200,143,57,143,0.715,0.6488465397039482,0.7730499699538271,0.7,False,"{""pairwise_composite_better"":60,""pure_ambiguity"":83,""pure_call"":57}"
+fifty_fifty_mixture,gain+change_probability,correct_abstention_rate,200,200,0,200,1.0,0.9811546736227335,1.0,0.7,True,"{""pairwise_composite_better"":95,""pure_ambiguity"":105}"
+fifty_fifty_mixture,update_l2+information_gain,correct_abstention_rate,200,94,106,94,0.47,0.40204754873265425,0.5390831708499817,0.7,False,"{""pairwise_composite_better"":25,""pure_ambiguity"":66,""pure_call"":106,""residual_lack_of_fit"":3}"
+fifty_fifty_mixture,update_l2+change_probability,correct_abstention_rate,200,198,2,198,0.99,0.9642782382838231,0.9972533418664555,0.7,True,"{""pairwise_composite_better"":110,""pure_ambiguity"":88,""pure_call"":2}"
+fifty_fifty_mixture,information_gain+change_probability,correct_abstention_rate,200,191,9,191,0.955,0.9167032959981803,0.976147456998507,0.7,True,"{""pairwise_composite_better"":133,""pure_ambiguity"":58,""pure_call"":9}"
+out_of_span_probe,full_grid_orthogonalized_tanh_surprise,correct_abstention_rate,200,52,148,52,0.26,0.20413830063574687,0.3249074560253411,0.7,False,"{""null_not_rejected"":31,""pure_ambiguity"":18,""pure_call"":148,""residual_lack_of_fit"":3}"

results/design-mixture-diagnostic-n60/geometry.csv

@@ -0,0 +1,16 @@
+first_candidate,second_candidate,best_pure_candidate,best_pure_affine_residual,true_pair_affine_residual,crossfit_oracle_log_score_gap_index,below_five_nat_power_index
+innovation_l2,surprise,surprise,0.06476225526808417,2.5106422755032005e-31,1.8825461925239708,True
+innovation_l2,gain,update_l2,0.2377395072664464,1.8957570419251692e-31,6.398602138782985,False
+innovation_l2,update_l2,update_l2,0.2576905807957704,6.575124830138023e-32,6.878315004395995,False
+innovation_l2,information_gain,information_gain,0.22866732483457436,2.3189189901515054e-31,6.177903187677524,False
+innovation_l2,change_probability,change_probability,0.2715733105685233,1.2703433913178208e-31,7.2076488292956435,False
+surprise,gain,update_l2,0.3334070092601475,3.315809387586691e-31,8.632119836108483,False
+surprise,update_l2,update_l2,0.2882713806954706,6.993725703550589e-31,7.599039143577237,False
+surprise,information_gain,information_gain,0.17513118479245662,1.4426499236756656e-31,4.841393639879367,True
+surprise,change_probability,change_probability,0.24059459272699613,1.431864715987097e-31,6.467723248570308,False
+gain,update_l2,gain,0.15154391801250797,2.865420764592673e-31,4.233107381393687,True
+gain,information_gain,update_l2,0.24549715570701747,3.96330703488968e-31,6.586043161858336,False
+gain,change_probability,gain,0.719444663119841,2.762008232078141e-31,16.260041052371072,False
+update_l2,information_gain,update_l2,0.13734921104922596,1.9408238026238305e-31,3.861009040227646,True
+update_l2,change_probability,update_l2,0.6095014120398538,1.0145670551702333e-30,14.277733472165602,False
+information_gain,change_probability,change_probability,0.4585249642762335,3.867138932999944e-31,11.322768797515211,False