v3.38 black-box audit — 15/19 original PASS (new best), 1/2/4 probe PASS/NI/FAIL

[FluffyAIcode]

v3.38 Black-Box Audit — 26 cases, 1299.7s

Full run of v331_blackbox_eval.py (external runner) against scheme_b_v338.py as SUT, under strict no mocks / no fallbacks / no overfit / no simplification policy. Original cases 4.1–4.19 untouched; new cipher-system probes 4.20–4.26 appended.

Headline

Metric	Value
Original 19: PASS	15 / 19
Original 19: FAIL	4 / 19
Cipher probes (new): PASS	1 / 7
Cipher probes: not_implemented (honest, non-blocking)	2 / 7
Cipher probes: FAIL	4 / 7
Elapsed	1299.7 s, CPU, Qwen2.5-1.5B-Instruct

Version evolution (original-19 PASS count)

Version	v3.31	v3.32	v3.33	v3.34	v3.35	v3.36	v3.37	v3.38
PASS / 19	10	11	10	12	13	12	14	15

Movement on original-19 from v3.37 to v3.38

• 4.8 degeneration_quality FAIL → PASS (new: [D-4] anti-collapse). All metrics comfortably within thresholds.
• All other 18 original cases unchanged vs. v3.37.

Cipher probes

#	Probe	Result	Key metric
4.20	rerank_stability_probe	FAIL	Jaccard=1.0 on both paraphrase pairs (retrieval is stable at top-1), but the probe requires Spearman ≥ 0.5 on shared top-5, and upstream gating shrank top-5 to length 1 — Spearman is undefined on that. Diagnostic of over-filtering, not instability.
4.21	decode_repetition_feedback_probe	PASS	avg_max_repeat=1.67, no bigram repeats, no trigram locks. [D-4] end-to-end verified.
4.22	functional_token_suppression_probe	FAIL	Margin condition 0/3; avg_starter_delta=0.67 (need ≥1.5). L_functional_suppression is a training-time loss; this audit runs eval-only, so the bridge has not yet been optimized against it.
4.23	keyword_specific_tail_slot_probe	FAIL	mean_intersection=0 over 4 memories. Same root cause as 4.22: tail[1]'s rare-keyword KL needs training gradient, which this audit did not supply.
4.24	context_descriptor_cluster_probe	NOT_IMPLEMENTED (honest)	missing_api: MemEntry.context_descriptor field. v3.38 implements [D-3] as a per-query aggregate (MemLLM._compute_context_descriptor), not a per-memory stored field. Spec wording is per-memory, so reported as not_implemented per Section 5.
4.25	prefix_length_scaling_probe	FAIL	L_mem 8 → 16 gave 3 → 2 content starters in top-12 (expected +1). Slot-norm band ratio=1.001 (passes). Reads as capacity is not the bottleneck in the current training regime — the bridge is under-fit, not under-sized.
4.26	mixture_distribution_gate_probe	NOT_IMPLEMENTED (honest)	missing_api: DecodeContext.mixture_gate. v3.38 shapes logits additively (content_bias + suppression_bias + CFG). No convex mixture gate exists.

Interpretation

The structural pieces [D-3] and [D-4] delivered immediate eval-time wins (4.8 flipped, 4.21 passes, 4.10 drift audit stable). [D-1] and [D-2] are training-time losses — the wiring is in place (verified by python -c smoke checks on loss gradient flow, and by the fact that the probes compute without error), but the audit runner executes eval-only against the fresh initialization. Under the strict no-training policy, their effect is correctly invisible to the probes.

This cleanly separates two audit signals:

1. Decode-time structural fixes ([D-3], [D-4]): verified right now.
2. Training-time structural fixes ([D-1], [D-2]): wiring verified; convergence verification requires a training-phase audit, which this runner does not perform.

Artifacts

• reports/v338_blackbox/report.json — full structured results (8.6k lines)
• reports/v338_blackbox/report.md — per-case markdown (includes all 26)
• reports/v338_blackbox/runner.log — raw stdout from the 1299.7 s run