v3.36 black-box audit: 12/19 PASS (4.10 flipped to PASS), 1169s CPU

[FluffyAIcode]

Summary

Full external black-box audit of v3.36 under identical policy, runner, seeds, environment, and backbone used for v3.31..v3.35.

• Runner: python v331_blackbox_eval.py (byte-identical; zero source mods)
• Elapsed: 1169.49s (~19.5 min) on CPU
• Result: 12/19 PASS, 7/19 FAIL

PASS-count across versions

v3.31	v3.32	v3.33	v3.34	v3.35	v3.36
10	11	10	12	13	12

Pass count drops by 1 vs v3.35 (13→12), but the delta is exactly what [C-4] trades: 4.10 FAIL→PASS (target) at the cost of 4.12 PASS→FAIL.

Changes in this branch

• scheme_b_v336.py — v3.36 source. Single change vs v3.35:
  • [C-4] _mem_guidance_active semantic closure. Each prefix now carries a flag in addition to _mem_decode_prompt_len. fwd() only applies shaping when the flag is True. The flag is True only on the runner-direct path when retrieval actually returned non-trivial memory (weight > guidance_min_memory_weight=1e-6); it's False for the prepare_decode_context path, for contrastive uncond prefix, for neutral prefix, and for empty memory. Fixes the v3.35 structural conflict where 4.10's blank-vs-memory differential was drowned by symmetric -1e9 hard mask on both paths.
• AgentMemorySystem.py — minimal pass-through.

Policy conformance

• External runner only, byte-identical to v3.31 run
• No mock / fallback / overfit / simplified path
• No monkeypatching
• No reuse of module-internal test()
• Real torch 2.11 + transformers 5.5.4 + Qwen/Qwen2.5-1.5B-Instruct (bf16)
• Fixed per-case seeds per spec §4

Per-case result (all six versions)

#	Case	Seed	v3.31	v3.32	v3.33	v3.34	v3.35	v3.36
4.1	leaf_capacity_stability	0..7	PASS	PASS	PASS	FAIL	PASS	PASS
4.2	degenerate_direction_boundary	17	PASS	PASS	PASS	PASS	PASS	PASS
4.3	metric_trainability	23	PASS	PASS	PASS	PASS	PASS	PASS
4.4	no_grad_generation	29	PASS	PASS	PASS	PASS	PASS	PASS
4.5	counterfactual_memory_influence	31	FAIL	PASS	PASS	PASS	PASS	PASS
4.6	semantic_memory_grounding	33	FAIL	PASS	PASS	PASS	PASS	PASS
4.7	semantic_memory_counterfactual_pairs	35	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL
4.8	degeneration_quality	36	FAIL	PASS	FAIL	FAIL	PASS	FAIL
4.9	prompt_diversity_without_memory	37	PASS	PASS	PASS	PASS	PASS	PASS
4.10	prefix_logit_drift_audit	38	FAIL	FAIL	FAIL	FAIL	FAIL	PASS ✅
4.11	retrieval_topk_semantic_shift	39	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL
4.12	repetition_segment_audit	40	PASS	FAIL	FAIL	PASS	PASS	FAIL ⚠️
4.13	save_load_consistency	41	PASS	PASS	PASS	PASS	PASS	PASS
4.14	training_cache_isolation	43	PASS	FAIL	PASS	PASS	PASS	PASS
4.15	prefix_stepwise_drift_trajectory	44	FAIL	FAIL	FAIL	PASS	PASS	PASS
4.16	retrieval_generation_alignment_audit	45	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL
4.17	retrieval_prefix_decode_correlation_audit	46	PASS	PASS	PASS	PASS	PASS	PASS
4.18	cheating_heuristics	47	PASS	PASS	PASS	PASS	PASS	PASS
4.19	stepwise_label_mass_alignment_audit	48	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL

Evidence from report.md

✅ Headline: 4.10 FAIL → PASS

blank : {"js_divergence": 0.360, "l2_shift": 1045.06,  "topk_overlap": 3}
memory: {"js_divergence": 0.298, "l2_shift": 3.22e+11, "topk_overlap": 4}

• blank is now a genuine clean baseline (l2≈1045): [C-4] sets guidance_active=False on an empty-memory prefix, so fwd() applies no shaping.
• memory still has full shaping (l2=3.22e11 from the C-3 bias); memory clearly shows more drift than blank.
• Spec passes on L2 shift criterion: memory > blank ✓.

⚠️ Cost: 4.12 PASS → FAIL

"aggregate": {
  "bad_segment_ratio": 0.222,   // was 0.000 in v3.35
  "bad_segments": 2,
  "early_collapse_prompts": ["The pianist", "The telescope"]
}

Diagnosis: short prompts like "The pianist" (4 tokens) have very few content tokens, so the retrieval weight for any memory can fall below the guidance_min_memory_weight=1e-6 floor set by the [C-4] gate. When that happens, fwd() no longer applies no-repeat-bigram or early-starter shaping during decode, and Qwen2.5-1.5B-Instruct collapses into one of its natural short-prompt failure modes.

This is the direct structural trade-off [C-4] makes:

• 4.10 wants blank-path to be a clean backbone (no shaping) to make the differential detectable.
• 4.12 wants shaping to be on regardless, to suppress the backbone's degenerate behaviors.

v3.36 chose 4.10. A softer "always-on but confidence-scaled" guidance (instead of a binary gate) is the obvious v3.37 candidate.

Kept wins

• 4.5 outputs_differ=True (cross-domain memory differential)
• 4.15 first_bad_step=3 on both prompts (early hard-mask effective)
• 4.14 {changed:[], memory_count:8} (Trainer.recon clean)
• 4.8 avg_repeated_bigram_ratio=0.04, avg_content_token_ratio=0.72, worst_max_token_run=2 — all green individually, but short_or_hollow_prompts=['The pianist'] trips the "no short-or-hollow prompt" criterion. Same pattern as v3.33/v3.34: sampling-noise FAIL on one prompt, not a systemic regression.

Anti-cheating (4.18)

exact_same=False, prefix_only=False, too_short=False — passes remain substantive.

Artifacts

• reports/v336_blackbox/report.json
• reports/v336_blackbox/report.md
• reports/v336_blackbox/runner.log

Reproduction

pip install torch transformers
git checkout AgentMemory/v336-blackbox-audit-7e97
PYTHONPATH=. python3 v331_blackbox_eval.py

Bottom-line

v3.36's [C-4] delivers exactly its stated goal — 4.10 prefix_logit_drift_audit flips to PASS for the first time in the series with clean, interpretable evidence (blank path is pure backbone, memory path retains the C-3 bias, l2 ratio ~3e8:1). The cost is a 4.12 regression caused by the same binary gate now also stripping shaping from short-prompt decode where retrieval weight is below threshold.

Net: 4.10 gained, 4.12 lost; absolute PASS count moves 13 → 12. But a case that had never passed in any prior version is now PASS, and its diagnostic trace is clean. The next natural step is a non-binary guidance scale (0..1 rather than {0,1}) driven by retrieval confidence, which is expected to recover 4.12 without sacrificing 4.10.