v3.46 de-overfit: fix probes 4.22/4.23/4.24 (held-out prompts / paraphrase queries / 4 domains); audit on v3.44-Trained ckpt = 19/26

[FluffyAIcode]

Scope

Runner + spec changes only. No SUT modification. Fixes circular / selection-biased test designs in three cipher probes identified in review of v3.45-Runner-Update.

Overfit critique being addressed

probe	circular element in pre-v3.46 design	v3.46 fix
4.22	3 prompts hand-selected for "Qwen's unconditional top-12 is function-token-dominated". Selection bias: PASS on these 3 does not imply generalization.	Add held-out set B ("Tell me about", "Please describe", "Explain how"); require both sets pass independently.
4.23	Query = mem.source_text. The query contains the rare tokens that the tail slot is tested against. Round-trip.	Query = corpus_paraphrase_music() (token-disjoint paraphrases). Identify dominant memory from ctx.diag; evaluate tail slot against that memory's rare_keyword_ids. Inline check that query tokens are disjoint from rare keywords.
4.24	2 domains (music, space) with labels assigned by CIPHER_MUSIC_KEYWORDS / CIPHER_SPACE_KEYWORDS matching. The keyword lists were hand-written against the same corpora. Circular labeling.	4 domains (music, space, cooking, finance). Label by source-text identity against runner-owned corpus tuples, not keyword matching. Cooking and finance are held-out: never in any CIPHER_*_KEYWORDS list, never referenced by cases 4.1–4.19. Pass requires both (a) 4-domain loo_nn >= 0.65 and (b) held-out-2-domain loo_nn >= 0.70.

Result on the same v3.44-Trained checkpoint

19 / 26 pass, identical to v3.45-Runner-Update. No case changed pass/fail status. The meaning of each result is stronger under the de-overfit metrics.

Key numeric outcomes:

4.22 PASS (selection bias refuted)

• Set A (hand-picked): delta = 11.0, margin_wins = 3/3
• Set B (held-out generic): delta = 10.0, margin_wins = 3/3
• Held-out set passes at ~90% the magnitude of selected set

4.23 FAIL (circularity removed, residual gap reduced)

• Median rank of best rare: 759 / 151936 (was 4291 under v3.45 round-trip → 5.7× improvement)
• Paraphrase query successfully retrieves dominant memory (ctx.diag.dominant_per_batch[0] is set)
• Top-20 intersection still = 0 → direction is broadly correct but concentration insufficient at 60-step training

4.24 FAIL at 4-domain, PASS at held-out-2-domain — overfit hypothesis falsified

• loo_nn_accuracy_all_4 = 0.625 (threshold ≥ 0.65)
• loo_nn_accuracy_heldout_2 = 0.875 (threshold ≥ 0.70, PASS)
• Per-domain: cooking 4/4, finance 3/4, music 1/4, space 2/4

If the encoder were memorizing the music/space pair (test overfit), music and space should score highest and held-out domains should score at random. The opposite pattern is observed: held-out performs best, hand-crafted performs worst. This measurement falsifies the overfit hypothesis for 4.24. The FAIL on the 4-domain metric is a genuine capability gap in the hybrid encoder (β=0.8 collapses music/space hidden_means together); it is not a test-design artifact.

Spec updates (V331_BLACKBOX_TEST_SPEC.md)

Added "De-overfit notice" blocks to Sections 4.22, 4.23, 4.24 documenting the changes and the falsifiability tests now present in each probe.

Artifacts

• V331_BLACKBOX_TEST_SPEC.md (updated)
• v331_blackbox_eval.py (updated)
• reports/v346_deoverfit_blackbox/{report.json, report.md, runner.log, audit_feedback.md}

Dependencies

Builds on SPEC PR #18 (Section 1.1 / 4-meta.1) and PR #19 (v3.45 runner metrics). If those merge first, this one is a clean fast-forward.