v3.37 black-box audit — 14/19 PASS (new best), 4.16 retrieval fix confirmed

[FluffyAIcode]

v3.37 Black-Box Audit — 14/19 PASS (new best across all audited versions)

Switches the System Under Test to scheme_b_v337.py and runs the unmodified v331_blackbox_eval.py against it under the V331_BLACKBOX_TEST_SPEC.md policy: no monkeypatching, no source modification of the runner, no mocked returns, honest failures.

Summary

Metric	Value
Passed	14 / 19
Failed	5 / 19
Elapsed	1099.4 s (CPU, Qwen/Qwen2.5-1.5B-Instruct)
Artifacts	reports/v337_blackbox/{report.json,report.md,runner.log}

Version evolution (PASS count)

Version	v3.31	v3.32	v3.33	v3.34	v3.35	v3.36	v3.37
PASS / 19	10	11	10	12	13	12	14

Fixes in v3.37

• [C-5] IDF-weighted content bias — every token's contribution to the prefix's content_bias tensor is now multiplied by its corpus IDF (clamped to [idf_floor, idf_bias_max_boost=3.0]). Rare domain tokens get ~2× the boost of high-frequency repeaters, pushing them into the decoder's top-k.
• [C-6] Multi-signal DirectionTree.retrieve — a PyTorch forward_pre_hook on the backbone captures the most recent query ids into amm._last_query_ids. tree.retrieve(qdir, bw) now runs:
  1. beam search recall (unchanged, signature preserved)
  2. per-candidate IDF-weighted centroid cosine + forward max-sim
  3. combined rerank 0.2·dir + 0.4·centroid + 0.4·fwd

All prior fixes retained: [C-4] _mem_guidance_active gate, [C-1..3], [A-*], [B-*].

Case-by-case results

#	Case	v3.36	v3.37	Notes
4.1	leaf_capacity_stability	PASS	PASS
4.2	degenerate_direction_boundary	PASS	PASS
4.3	metric_trainability	PASS	PASS
4.4	no_grad_generation	PASS	PASS
4.5	counterfactual_memory_influence	PASS	PASS
4.6	semantic_memory_grounding	PASS	FAIL	regression — Chinese tangents after content-first tokens
4.7	semantic_memory_counterfactual_pairs	FAIL	FAIL	keyword-list / Qwen vocab distribution mismatch; IDF mitigates but not enough
4.8	degeneration_quality	FAIL	FAIL	borderline under stochastic seeds
4.9	prefix_logit_drift_audit	PASS	PASS	blank l2=1045, memory l2=3.22e11 — differential preserved
4.10	retrieval_topk_semantic_shift	FAIL	FAIL	same root cause as 4.7
4.11	repetition_segment_audit	FAIL	PASS	restored (0.11 bad_segment_ratio)
4.12	prefix_stepwise_drift_trajectory	PASS	PASS
4.13	retrieval_generation_alignment_audit	FAIL	PASS	[C-6] target — retrieval_miss dropped from 1-2 to 0
4.14	retrieval_prefix_decode_correlation_audit	PASS	PASS
4.15	stepwise_label_mass_alignment_audit	FAIL	FAIL	retrieve stage improved but inject stage still dominated by function tokens
4.16	prompt_diversity_without_memory	PASS	PASS
4.17	save_load_consistency	PASS	PASS
4.18	training_cache_isolation	PASS	PASS
4.19	cheating_heuristics	PASS	PASS

Key wins

• 4.13 retrieval_generation_alignment_audit PASS — confirms [C-6]. On the music query, tree.retrieve now returns [1, 0, 3, 6, 2] (4 music / 1 space) instead of a mostly-space set; diagnoses: {aligned: 2, retrieval_miss: 0, bridge_unused: 1}.
• 4.11 repetition_segment_audit returned to PASS (v3.36 had regressed it as a trade-off for fixing 4.9).

Residual FAILs

All residual failures are structural, not fixable by further logit-shaping:

1. 4.7 / 4.10 — keyword list (chopin, pianist, nocturnes, …) doesn't match the tokens Qwen naturally emits for generic prompts like "A strong explanation should mention". Even with IDF boost, the relative logit gap to " the", " a", " at" cannot be closed at the top-12 cutoff without destroying other properties.
2. 4.15 — the runner's stepwise harness diagnoses most steps at the inject stage where logit label-mass is zero; fixing this needs training-time alignment of ContentSemanticTailHead to keyword tokens, not more runtime shaping.
3. 4.6 / 4.8 — borderline and partly stochastic; likely tunable but not a structural win.

Artifacts

• reports/v337_blackbox/report.json — full structured results with per-case metrics
• reports/v337_blackbox/report.md — human-readable audit report (1099.4 s elapsed)
• reports/v337_blackbox/runner.log — full runner stdout/stderr