Notes

Running notes for the position-vs-identity study. A fresh session should be able to pick up from this file alone. Literature is in lit/NOTES.md; the preliminary paper is paper/v1.pdf. Last updated 2026-09-17.

Amendments

Dated amendments live in DECISIONS.md. Amendment 1 (2026-09-06): retrain on a dataset with labels assigned uniformly at random per graph, so the two blocked label-based cells of experiment 3 can run. Outcome (2026-09-08): gate 1 failed on both seeds. Trained on relabeled data with the unchanged recipe (RunPod RTX 3090), the model memorizes the training graphs (99 and 97 percent on seen graphs) and is at chance on unseen ones (49 to 52 percent under four serialization seeds; readout ordering fails), with stage validation accuracy at chance from stage 0 on. The breadth-first label order of ProsQA is load-bearing for learning the task at this size and budget, not only for the trained model's competence. The two label cells were not run. Numbers and files in DECISIONS.md, Amendment 1, Outcome.

Amendment 2 (2026-09-08): names kept on tokens 0 and 1, concept labels random over 2..30 (data/relabel_names/, seed 20260908). Asks whether the root marker alone makes the task learnable; predictions and the gate in DECISIONS.md. Written, not trained.

Comparability of the evaluation across the original and retrained models (checked 2026-09-06): the accuracy is the argmax token against the target and does not depend on which tokens the readout counts. The inner-product readout of evaluate.py counted tokens 0..n-1 with n the graph's symbol count; in every original graph (14,785 train, 257 validation, 419 test) those are exactly the labels present in the graph. The readout is now written as "labels present in the graph", which is the identical definition on the original data and the right one when labels are random over 0..30. The readouts are therefore identical across the comparison; no rescoring of the original checkpoints is needed.

Status

Experiments 11 and 12 (2026-10-06): the v1 INLP arm and whole-path deletion, rerun on seeds 0 and 1. Full INLP removal at step 1 keeps the median (-0.06, -0.28) but flips 0.15 and 0.19 of graphs to the decoy, where rank-matched and size-matched random spans flip none (INLP minus size-matched +0.15 [+0.07, +0.24], +0.19 [+0.10, +0.29]); the flips stay at the fallback rate. Whole answer-path deletion at every intermediate step: medians -0.01 to -0.06, flips 0.04 to 0.12, below the fallback rate.
Experiment 10 (2026-10-05): the leftover under 8c's isolation mask. Of the share still right above chance, the candidates' incoming edges carry 0.76 [0.59, 0.96] pooled over four seeds, mostly the decoy's own; with them hidden too the answer goes with the label-id order (G 0.63) and a remainder of 0.18 [-0.01, 0.35] is neither shown nor excluded. No further experiments; the write-up is next.
Experiment 7 (2026-09-16): the four headline measurements on two released fine-tuned GPT-2 COCONUT checkpoints (no training). Both pass the regime check (98.0 with six latents; unchanged with the latents removed or every thought zeroed). At n=100 on both: no head at the latents follows edge content, the removal, the calibration mask and the final-thought carry-over change nothing on any graph, and the winner is decodable from the first thought; in the Jacobian basis the winner is at chance at every pass. The from-scratch circuit is absent from this regime. Complete (2026-09-17). See the experiment 7 block and its table. No further experiments; the write-up is next.
Experiments 0 to 6 are complete at n=100 on both seeds. Experiment 5 (2026-09-10): blocking every attention route onto the path edges adds nothing to the query-key removal. Experiment 6 (2026-09-11): the answer is carried in thought K and read out by the answer position's layer-1 MLP; the same-answer donor's thought K restores every removal-flipped graph (41 of 41 across seeds); token and edge masks at the final positions do nothing. Experiment 6b (2026-09-11): the decoy's incoming edges count against the decoy during the search, visible only under the removal; the winner is complete in thought K, largely present in K-1, absent in K-2; the label cue does not enter the trained model's choice. The recovery question is closed. Replication on seeds 2 and 3 (2026-09-15, DECISIONS.md, last entry): every headline cell replicates on both new seeds except the decoy-edge mask under the removal, which holds on seeds 0, 1 and 3 and not on seed 2. No further experiments; the write-up is next. Amendment 1 (shuffled-label retrain) failed its gate 1 on both seeds (2026-09-08); Amendment 2 (names kept) is written and not trained. Next step is the user's decision (see DECISIONS.md and the experiment 5 block). Experiment 5 is not designed yet; the open question left by 3 is where the answer is recovered when the thoughts' attention onto the answer path is removed (candidates: layer-1 edge reading, and the answer position's own content heads).
Finding worth its own line: ProsQA assigns concept ids in breadth-first order and the model uses label id as a depth cue, so consistent relabelings are off-distribution for it (experiment 3 block). This is why the paper's label-swap donors computed the opposite answer on only 79 percent of their own prompts.
Code for experiments 0-4 is written and tested (tests/: bit-exact equivalence with hooks idle, prompt renderer byte-identical to the vendor builder, hook plumbing, and every driver end to end on random weights; the smoke test writes to a scratch directory, not to results/). No experiment has run. scripts/make_numbers.py macros get registered once result files exist. No trained checkpoint was on this machine until 2026-09-06 02:17; ckpts/seed0/best.pt and ckpts/seed1/best.pt now come from the laptop. Every driver checks for its checkpoint and stops with a message if it is missing.
Retraining with periodic checkpoints (train.py --save-every) happens on the laptop when the user says so. Claude never launches training or a pilot.

Established (from paper/v1.pdf, treated as data)

A two-layer model trained from scratch on graph reachability with K continuous thoughts reaches 90-96 held-out accuracy depending on seed, and the inner-product readout orders nodes optimal > frontier > reachable > not reachable at every latent step.
The search frontier is linearly readable from the thought (probe AUC 0.998). The thought is necessary (zeroing at every step: -42.9 median change in T).
Per-branch linear edits (subtract, collapse, inject) do nothing in the input-embedding, probe, or causal-Jacobian basis, even with the full linearly readable subspace removed.
Whole-thought transplants at every intermediate step redirect the answer when the donor is a different graph with the same candidates (-91.7) and do nothing when the donor is the same graph with the candidate labels swapped (-0.0). A random donor destroys rather than redirects (e = 0.98).
Swapping the target and decoy directions in the causal-Jacobian basis at the final step flips the answer (-99.97); the same swap at intermediate steps does nothing.

One reframing, written 2026-09-06 before any new run: under both stories below, the candidates are absent from every intermediate frontier (they sit at depth K; the step K-1 frontier is their parents). So both stories predict the paper's label-swap null. That result did not separate them.

Open: what does a mid-search thought encode?

The position story: the thought remembers where to look in the prompt (which edge slots are consumed, which to read next); node names are read off the prompt at the last step. The identity story: the thought remembers which nodes are in the frontier, in a form the paper's linear edits could not reach. A hybrid is possible: positional mid-search, identity at the final step.

Two terms the paper does not use. An edge slot is the three-token span one edge occupies in the prompt (source target |). The memory cache is the set of keys and values the model stores at each position, which later positions read through attention.

Working rules

Predict only what a story forces; one line per story per cell; "no prediction" cells run as exploration. Never fill a table for completeness.
Pilot on the 10 pilot graphs; write here what the pilot changed; then run n=100 once on both seeds.
Effect sizes with 95% bootstrap intervals (2,000 resamples of graphs). Two counts with named cutoffs: flipped (change in T at or below -50; protects "the answer switched to the other candidate on this graph") and escaped (e at or above 0.5; protects "the model stopped answering the question").
Standing controls on every run: self-transplant (must be exactly zero; catches pass misalignment), reserialized baseline (a second fixed edge order, nothing fixed: the noise floor for "the prompt changed and the thought recomputed"), random donor (a random training graph's thoughts of the same step count: what breaking looks like on a vector the model actually produced), and the same-answer donor (a training graph with the same two candidates, the same correct answer and the same step count, transplanted at the intermediate passes and at all K). The same-answer donor can only move the answer by breaking the search, so in every cell a flip counts as redirection only if the same graph did not flip under it; each summary carries the cell's flip rate, the reference rate, and their paired difference with an interval (common.summarize, key "redirection"). No zero vectors as sensitivity anchors.
Every cell reported, nulls included. Numbers in prose point to a results file.
For any cell where graphs split (some flipped or escaped, others not), compare the two groups on the depth of the target (K), the number of branches (root out-degree), the slot of the target's parent edge in the list (absolute and as a fraction of the list), and the number of parent edges into the target. Report that split (stats.split_by_flag).

Sets and measures

Recipients: held-out test split (419 graphs, never trained on). Pilot = test graphs 400-409. n=100 = test graphs 0-99. Serialization pinned per graph with seed ((gi + 1000000) << 16) ^ base_seed; reserialized baseline adds 0xABCDEF (same formula as thoughts.pin_serialization).
Donors, averages, probe and Jacobian fits: training graphs.
Experiment 0 keeps the paper's sets for transplants and the final-step swap (training graphs 0-99). The per-branch subtraction moved off the paper's synthetic set (off-distribution for this model, see the experiment 0 block) to the natural recipients whose target has exactly one depth-1 ancestor along shortest paths; that ancestor is the answer branch, another child of the root is the sibling. About 70 percent of graphs qualify (210 of 300 training graphs in tests/test_prompts.py); skips are listed.
T = 100 x p(target) / (p(target) + p(decoy)) at the answer position; change in T is relative to the same graph's pinned baseline; e = 1 - p(target) - p(decoy). Every cell also reports where the probability went: target, decoy, other node token, non-node token.
For a renamed prompt, "target" means the target's new label. For a rewritten prompt (where the decoy becomes the right answer), "target" stays the original target, so following the edit shows as T falling to 0.
"Intermediates fixed" injects the original run's thoughts at passes 0..K-2. "All K fixed" also injects the final one. Both variants are run where the plan says so.
The GPT-2 models (experiment 7) use the same recipients in the model's natural-language format, K = 6 for every prompt, and a readout at the first name token after "### Root is a"; details in the experiment 7 block.

Experiment 0: setup and baseline

Design: retrain two seeds on the laptop (train.py --seed {0,1} --fixed-early-stages --full-task-patience 15 --save-every 10), then on each model: evaluate.py (accuracy, ordering), fit_probes.py (probe AUC), fit_jlens.py (causal-Jacobian basis), and baseline.py (per-branch subtraction with random-direction control; transplants: matched donor, label-swap donor, placebo donor, random donor, self, reserialized, at intermediate steps, first step only, final step only, all steps; final-step swap and intermediate swaps in the causal-Jacobian basis).

Predictions (not story-specific; this is the baseline the next paper needs): accuracy in the 90-96 range, ordering correct at every step, probe AUC near 0.998, subtraction near zero with control near zero, matched-donor transplant a large negative change in T with low e, label-swap near zero, random donor with high e, final-step swap near -100 and intermediate swaps near zero. Position story: all of the above. Identity story: all of the above.

Pilot outcome (seed 0, 2026-09-06; files in results/seed0/):

Accuracy 92.4 (387 of 419; paper 94.5), ordering correct at all four steps (evaluation.json).
Probes: 23 nodes, median held-out AUC 0.9972, minimum 0.63 (paper 0.9979 and 0.63; probe_basis_report.json).
Causal-Jacobian basis: cross-graph concentration by step 0.50, 0.49, 0.71, 0.93 and cosine to token embeddings 0.41, 0.55, 0.77, 0.78, the same as the paper's Table 14 to two decimals (jlens_basis_report.json).
Final-step swap: median -100.0 on 10 of 10; swaps at each intermediate step within 0.2 of zero (baseline_swap_pilot.json; paper -99.97 and ~0).
Transplants on training graphs 0-9 (baseline_transplant_pilot.json): matched donor at intermediate steps median -98.4, interval [-99.7, -25.8], 7 of 10 flipped, median e 0.001, and 2 graphs escaped (e 0.85, 0.93) instead of flipping (paper -91.7, e 0.002); first step only -7.2 (paper -2.9); final step only -99.8 (paper -99.95); all steps -99.9 (paper -99.97); label swap -0.0 with 1 of 10 flipped (paper -0.0, 2 percent); placebo 0.0; interior swap median -0.0 with 2 of 10 flipped (paper -0.08); random donor median -99.3 with median e 0.58, 6 of 10 flipped, 5 of 10 escaped (paper -30.4, e 0.98). Reserialized baseline and self-transplant exactly zero.
Subtraction on data/eval_graphs.json graphs 0-9: not interpretable. The model answers the decoy or a non-candidate on most of these graphs (119 of 500 correct over the whole file; the renderer matches the vendor path exactly, so this is the model, not the code). Where baseline T is high the subtraction is 0.0 in both bases (baseline_subtraction_pilot.json).

What the pilot changed:

The matched donor was mis-specified as a graph with the same correct answer; the paper's matched donor is a graph with the same two candidates whose correct answer is the recipient's decoy. Fixed in baseline.py; the same-answer donor is kept as a control (see the next point).
New observation on this model: a clean flip with low escape is not by itself redirection. The same-answer donor, which can only break the search, flips 4 of 10 graphs with e near 0 (median -25.6), and the random donor flips 4 graphs with e below 0.3. When the search does not find the target, this model often answers the other candidate. Consequence for experiment 3's reading: a cell "breaks" if it moves T or e; whether a flip is a redirection or a fallback is decided by comparing, per graph, with the same-answer donor and the random donor. The four-outcome reading below is amended accordingly.
The synthetic eval set is off-distribution for this model. Every ProsQA root is token 0 or 1, candidates are never 0 or 1, and the name tokens have no in-edges; the generator shuffles all labels. Relabeling the root to a name token lifts accuracy on the file from 119 to 179 of 500 only, so other conventions matter too (in ProsQA the unreachable component hangs off the second name token). Decided (user, 2026-09-06): no generator rebuild; the subtraction runs on natural test graphs where the target's depth-1 ancestor is unique. Its pilot is run before its n=100.
Accuracy gap (92.4 here vs 94.5 in the paper, same checkpoint and split): the vendor evaluation draws each graph's edge order from the unseeded global RNG, so every run sees different prompts. Evaluated under pinned serialization seeds 0-3 (evaluate.py --serialization-seed, files evaluation_ser<seed>.json): 396, 393, 390 and 388 of 419, that is 94.51, 93.79, 93.08 and 92.60, ordering correct at every step under every seed; the unseeded run gave 387 (92.36). The paper's 94.5 is inside this spread (it equals the seed-0 value), so the gap is serialization variance, not a different model. Go-ahead for n=100 on both seeds for everything that reproduced (2026-09-06).

Subtraction pilot under the new design (seed 0, test graphs 400-409, 9 with a unique ancestor; baseline_subtraction_pilot.json): median change 0.0 in the input-embedding and probe bases and under the random-direction control; one low-confidence graph (base T 81) moved by about 10 in both the real and the random cell. Same pattern as the paper; nothing changed after this pilot.

n=100 outcome, seed 0 (files in results/seed0/):

Transplants, training graphs 0-99 (baseline_transplant_n100.json): matched donor at intermediate steps median -95.1 [-98.2, -75.0], 61 percent flipped [52, 71], median e 0.002 (paper -91.7, e 0.002); first step only -1.2 (paper -2.9); final step only -99.9, 97 percent flipped (paper -99.95); all steps -100.0 (paper -99.97); label swap -0.0, 1 percent flipped (paper -0.0, 2 percent); placebo 0.0; interior swap -0.1, 16 percent flipped (paper -0.08); random donor -20.4, median e 0.92, 64 percent escaped (paper -30.4, e 0.98). Reserialized and self-transplant exactly zero on all 100.
Same-answer donor at intermediate steps (the new standing reference): median -0.7, 28 percent flipped [19, 37] with e near 0. Against that rate the matched donor's flips exceed the reference by +0.33 [+0.20, +0.44] (39 percent redirected), the final-step transplant by +0.68 [+0.59, +0.78], while the first-step transplant (+0.04 [-0.08, +0.17]) and the interior swap (-0.11) do not exceed it: their flips are fallback, not redirection.
Final-step swap (baseline_swap_n100.json): median -100.0, 98 percent flipped (paper -99.97); swaps at each intermediate step median -0.0.
Subtraction on natural test graphs 0-99, 68 with a unique ancestor (baseline_subtraction_n100.json): median change -0.00 [-0.00, -0.00] in the input-embedding basis, the probe basis, the random-direction control and the sibling subtraction; means -4.7 and -3.6 for the two answer-branch bases come from 3-4 low-confidence graphs (11 of 68 have base T below 90). The paper's null reproduces.

n=100 outcome, seed 1: evaluation done (unseeded 402 of 419, 95.9; seeds 0-3: 95.5, 96.7, 95.7, 95.5; the paper's 95.7 is inside; ordering correct at every step). Probes, Jacobian basis, transplant, swap and subtraction: done on CPU (files in results/seed1/). Probes: 23 nodes, median held-out AUC 0.9919, minimum 0.68. Causal-Jacobian concentration by step 0.50, 0.51, 0.71, 0.92 (seed 0: 0.50, 0.49, 0.71, 0.93). Transplants on training graphs 0-99: matched donor at intermediate steps -94.9 [-98.9, -63.0], 61 percent flipped, e near 0, +0.29 [+0.16, +0.43] beyond the same-answer reference (31 percent); first step only -1.4; final step only -99.9 (94 percent flipped); all steps -99.9; label swap -0.0 (1 percent flipped); placebo 0.0; interior swap -0.3 (22 percent flipped, below the reference); random donor -33.0 with e 0.77. Final-step swap -100.0 (98 percent flipped); intermediate swaps -0.0. Subtraction on 68 natural test graphs: medians -0.01, -0.00, +0.00, +0.00 for the input-embedding basis, probe basis, random control and sibling. Reserialized and self-transplant exactly zero on all 100.

Experiment 0 is complete on both seeds. Every paper measurement reproduces on both, with one addition: the same-answer donor's fallback rate (28 and 31 percent) is the reference against which redirection is counted.

Experiment 1: necessity, done properly

Design: seven conditions at every step of the same pinned graph, each run twice (all K passes; intermediate passes only): baseline; the average thought over 2,000 training graphs at the same pass (present but uninformative); Gaussian noise at the graph's own thought norm; the thought removed (latent tokens deleted; the prompt ends [R] root [A]); removed with length kept (latent tokens replaced by the pad token, attended); zero; random donor. Report the change in T, e, and the probability split.

Position story: no separating prediction. Identity story: no separating prediction. Shared expectation from the paper's transplant results: every uninformative substitute breaks the answer (the thought is informative, not a scaffold as in Kshirsagar's chess model). The two removal conditions are prompts the model never saw in training; they are reported as such.

Pre-pilot note (2026-09-06, before the pilot ran): the same-answer donor is now a cell here too and is the reference for fallback flips (28 percent on training graphs in experiment 0). Substitutes that carry no candidate (average thought, noise, zero) are expected to break mostly by escape or by fallback flips at about that rate; a cell whose flips exceed the reference would be surprising and would need the probability split to explain it.

Pilot outcome (seed 0, test graphs 400-409, results/seed0/necessity_pilot.json): every uninformative substitute breaks, as expected. At all K passes: average thought median -27.2 [-69.4, -5.4] with e 0.94 and 10 of 10 escaped (the mass, 0.91, lands on other node tokens: the model names nodes from the blurred thought); random donor -75.9 with e 1.00, 10 of 10 escaped; noise -5.3 but bimodal (4 flipped with e near 0, 3 escaped); zero -22.7 (3 flipped, 4 escaped; the paper's -42.9 was a median over 100 training graphs). Removing the latent tokens gives -87.4 with 7 of 10 flipped and e near 0: without thoughts the model answers the decoy, not chance. Pad tokens in place of the latents: -15.8, 4 flipped, 3 escaped. Same-answer donor: at intermediate passes -1.1 with 2 of 10 flipped (the fallback reference); at all K exactly no flips (the donor's final thought carries the shared answer). Self-transplant and reserialized exactly zero. What the pilot changed: nothing in the design. Two observations to carry: substitutes that contain node content (average, random donor) break by escape onto other nodes, substitutes without it (noise, zero) break by fallback flips; and "thought removed" is a systematic decoy answer. n=100 outcome (both seeds; results/seed0/necessity_n100.json, results/seed1/necessity_n100.json): every substitute breaks on both seeds. At all K passes, medians on seed 0 / seed 1: average thought -14.0 / -22.1, noise -13.6 / -11.7, zero -25.4 / -19.1, random donor -16.4 / -37.3; flip rates 34-43 percent, escape 29-90 percent. The same-answer donor at all K flips nothing on either seed (its final thought carries the shared answer), so all of those flips exceed the all-K reference (+0.35 to +0.44). At the intermediate passes the same-answer donor flips 19 / 26 percent and the substitutes exceed it by +0.11 to +0.27. Removing the latent tokens: -15.7 / -23.6 with 40 / 42 percent flipped; the pilot's "decoy on 7 of 10" was the small sample (decoy mass 0.22 / 0.27 at n=100). Pad tokens in place of the latents: -13.2 / -9.2. Where the mass goes replicates: node-bearing substitutes put 0.58-0.79 of the probability on other node tokens on both seeds; noise puts 0.30-0.43 there. Self-transplant and reserialized exactly zero on all 200 runs. Conclusion: the thought is necessary and informative on both seeds, as the paper found; no story-separating content here.

Experiment 2: which heads look by position and which by content

Design: for each of the 16 heads and each query position (edge tokens; intermediate latents; last latent; answer position), record attention over edge slots in two runs of the same graph: the original order, and a random reorder of the edge list with the original thoughts fixed at all K passes. Position score = correlation of attention over slots, slot for slot. Content score = the same after undoing the permutation, edge for edge. Also the share of attention mass that lands on edge slots at all, so heads that do not read edges are not over-interpreted.

Position story: the layer-2 heads that carry most attention from the intermediate latents onto edge slots keep their slots (position score near 1, content score near chance). Identity story: those heads follow the edges (content score near 1, position score near chance). Both stories: layer-1 heads at edge tokens attend within their own slot (Zhu et al.'s copy). A split between intermediate latents and the last latent or answer position is the hybrid signature.

Pre-pilot note (2026-09-06, before the pilot ran): scores are computed on the per-slot attention mass (sum over an edge's tokens). A head is read as an edge-reading head at a query class only if its mean slot mass there is at least 0.2 (the cutoff protects "this head attends to edges at all"); heads below it are reported but not classified.

Pilot outcome (seed 0, test graphs 400-409, results/seed0/heads_pilot.json):

Intermediate latents (thoughts 1..K-1 as queries): all 8 layer-2 heads put 0.89-0.97 of their attention on edge slots and follow the edges after the reorder: content scores 0.88-0.93, position scores -0.02 to +0.09. The identity story's prediction for this cell; the position story's is rejected on all 10 graphs for every layer-2 head. The same holds at the root query, the last latent and the answer position (content 0.62-1.00).
Layer 1 at the intermediate latents: the four edge-reading heads (mass 0.76-0.93) are mixed, content 0.49-0.67 and position 0.27-0.40; the heads with high position scores there carry almost no slot mass (0.01-0.12) and are not classified.
The method does find positional heads: at the answer position layer-1 head 2 has slot mass 0.99 with position score 0.99 and content 0.07, and heads 5 and 6 are mostly positional (0.78 and 0.64). Where they look is recorded from n=100 on (first-slot and last-slot mass).
Edge-token queries keep only 0.06-0.10 of their attention inside their own slot in both layers, so the layer-1 copy of Zhu et al.'s construction is not at the edge's target token in this model. What the pilot changed: the heads driver now also measures separator-token queries (the "|" token can see both endpoints of its edge) and reports each head's mass on the first and last slot, so positional heads can be located. Scores and cutoffs unchanged. n=100 outcome (both seeds; results/seed0/heads_n100.json, results/seed1/heads_n100.json):
Intermediate latents, layer 2: all 8 heads on both seeds carry 0.89-0.98 of their attention on edge slots and follow the edges. Content scores 0.85-0.90 (seed 0; intervals within [0.82, 0.92]) and 0.91-0.94 (seed 1; within [0.88, 0.96]); position scores between -0.02 and +0.03 on both. The identity story's prediction holds for every layer-2 head on both seeds; the position story's is rejected. The same holds at the last latent and, for the edge-reading layer-2 heads, at the answer position.
Layer 1 at the intermediate latents: edge-reading heads mixed (content 0.47-0.71, position 0.23-0.42); the heads with position scores above 0.6 carry 0.02-0.36 slot mass.
Zhu et al.'s layer-1 copy is at the separator token: from the "|" token, seven of eight layer-1 heads on seed 0 (0.90-0.94) and all eight on seed 1 (0.95-0.96) put their attention inside the edge's own slot; from the edge's target token only 0.05-0.11. Layer 2 does not copy (0.03-0.08).
A purely positional layer-1 head sits at the answer position on both seeds (seed 0 head 2: mass 0.99, position 0.99, content 0.00; seed 1 head 0: mass 0.97, position 0.87, content 0.05); it attends neither to the first nor the last slot in particular (0.06 and 0.00 of its mass), so it spreads over slots by position. What it does is an open item for experiment 4. Conclusion: the layer-2 query built from a mid-search thought matches edges by content, not by slot, on both seeds.

Experiment 3: the counterfactual set (thought fixed, one thing changed)

Design: every cell uses prompts.py, which renders the vendor format byte for byte with an explicit edge order. Each changed prompt is run three ways: thoughts free (that cell's reserialized-type baseline), intermediates fixed, all K fixed. Controls per graph: reserialized baseline, self-transplant, random donor at intermediates and at all K.

Cells and predictions:

Edges reordered (random permutation of slots), intermediates fixed. Position story: breaks; pointers land on other edges; e rises toward the random-donor level. Identity story: no effect, within the reserialized baseline. Separating, primary.
Labels renamed (one consistent relabeling of every node token, edge order kept), intermediates fixed. Position story: no effect (slots unchanged, candidates renamed too). Identity story: breaks (the thought names old labels that now denote other nodes). Separating, primary; mirror of the cell above.
Unreachable edges reordered among themselves (no reachable edge moves), intermediates fixed. Position story: no effect. Identity story: no effect. Fragility control: if this breaks, the thought is coupled to prompt content neither story needs.
Decoy swap (the edge into the target and an edge into the decoy trade slots; the graph is unchanged), all K fixed. Position story: clean flip to the decoy with low e (complete only when every parent edge of the target was swapped; n_swapped and complete are recorded per graph). Identity story: no effect. Separating for the final thought.
Decoy swap, intermediates fixed. Position story: no effect (the last hop is re-read from the prompt). Identity story: no effect. Consistency.
Non-candidate swap (as the decoy swap, but the swapped-in edge leads to an unreachable node that is not a candidate; the graph is unchanged), all K fixed. The cell reports p_watch, the probability the answer puts on that node. Position story: the final step dereferences the target's old slot and names that node, so p_watch is high (cutoff 0.5 protects "the model named the swapped-in node"); T may fall by fallback, but only p_watch is the signature, because a broken search cannot put mass on a node it has no way to select (the same-answer and random donors give the reference level of p_watch). Identity story: no effect, p_watch near zero. Separating for the final thought, and immune to the fallback ambiguity.
Non-candidate swap, intermediates fixed. Both stories: no effect. Consistency.
Last-hop rewrite (the cut edge into the target now points to the decoy, same slot; BFS confirms the target is unreachable and the decoy is at depth K), all K fixed. Position story: follows the edit (T falls to 0). Identity story: keeps the old answer. Separating for the final thought.
Single-edge rewrite at depth d, thoughts free. Both stories: follows the edit. Competence check (Jin et al.).
Single-edge rewrite at depth d, intermediates fixed, stratified by d. Both stories: old answer if d < K-1, new answer if d = K-1. Exploration: how much of the search is committed by step K-1.
Query-key subtraction (added 2026-09-06 at the user's request, before the pilot). The answer's edge is the edge (p, target) with p at depth K-1; it is read by the layer-2 attention of the query built from thought K-1 (the last intermediate thought, pass K-2). For each layer-2 head, the direction in thought space that the head's query matrix maps onto that edge's key is the query matrix applied to the key (the key averaged over the edge's tokens with the head's own attention as weights), scaled by the layer-norm gain and centered; this is the first-order direction, ignoring the norm rescaling of the layer norm and layer 1's indirect response. The cell subtracts that direction from thought K-1 (norm preserved, as the paper's SUBTRACT), per head and for all eight heads at once (their span removed). Controls: the same coefficients removed along matched random directions (per head and for the span), and the same construction for a non-answer edge (the frontier edge with the most attention that does not lead to the target, else the most attended other edge). Recorded per run: each head's attention from that query onto the answer's edge and the control edge, before and after; the removed coefficient as a fraction of the thought norm; the cosines between the eight directions and between each direction and the input-embedding directions of p and of the target. Identity story, read through attention geometry: attention to the answer's edge drops and the answer flips above the same-answer reference rate (redirection above zero), with the drop largest for the all-heads removal. Position story: attention drops too if the pointer is linear in the thought, since the direction is built from whatever the key carries; no prediction on the answer. Both controls: no drop, no flips beyond the reference. If nothing moves (attention unchanged, flips at the reference rate), the thought's causal content is not in this linear query-key geometry, which is where the paper's per-branch nulls would then be located, and the cell says so.

Pre-pilot note (2026-09-06): the predictions above and in experiment 4 stand as written; the same-answer donor is a standing control, and the non-candidate swap and query-key cells are included in the first pilot.

Added after the first two pilot runs of experiment 3 (2026-09-06, before the third run; see "What the pilot changed" below for why): the "labels renamed" cell cannot be run on this substrate, because the model relies on ProsQA's id-depth convention and fails a fully relabeled prompt with thoughts free. Two replacement cells that move only two labels, each run three ways and read against its own free twin (paired per graph: flips and moves of the fixed variant minus the free variant, beyond_free_twin):

Parent swap, one level up: the target's depth-(K-1) parent trades labels with a depth K-2 node. Identity story, intermediates fixed: breaks beyond the free twin (thought K-1 names the parent by a token that now denotes a node one level up, whose edges do not reach the target). Position story: nothing beyond the free twin. All K fixed: both stories nothing beyond the free twin (thought K names the target, whose label is untouched).
Parent swap, same depth: the parent trades labels with another depth K-1 node. Identity story: nothing beyond the free twin (the depth K-1 frontier is the same set of labels, so thought K-1 still names it). Position story: nothing. Control for the cell above. The free twins themselves are expected below baseline (a two-label swap cost 6 to 14 points of accuracy on 50 test graphs in the check that motivated this), which is why the reading is paired against them.

Pilot outcome, experiment 3 (seed 0, test graphs 400-409; three runs, all kept: results/seed0/counterfactuals_pilot_v1_root_relabeled.json, counterfactuals_pilot_v2_full_rename.json, counterfactuals_pilot.json):

Controls: reserialized and self-transplant exactly zero; random donor at intermediate passes -25.1 (4 flipped, 7 escaped), at all K -75.9 with e 1.00; same-answer donor 2 of 10 flipped at intermediates, none at all K.
Edges reordered: no effect in any variant (free, intermediates, all K: medians -0.0, no flips, no escapes, on all 10 graphs); the unreachable-only reorder likewise. The position story's primary prediction fails.
Decoy swap and non-candidate swap with all K fixed: no effect on any graph; p_watch 0.00 on all 10. The position story's prediction for the final thought fails. Intermediates fixed: no effect, as both stories said.
Last-hop rewrite, all K fixed: the old answer is kept on 7 of 7 (identity story's prediction; the position story's fails). Thoughts free, the model follows the edit on only 3 of 7 (median -14.8); intermediates fixed, 4 of 7 follow. Rewrites at depths 1, 2, 3 with thoughts free are followed on 2 of 4, 3 of 5, 1 of 3; with intermediates fixed the old answer stays on all, as both stories said. Following an edit is itself unreliable in this model.
Labels renamed: not runnable as a thought test. Run 1 gave the root a concept label and its free twin failed (e 1.00 on 6 of 10). Run 2 swapped the two name tokens and deranged the concepts; its free twin still failed, 6 of 10 answering the decoy with e near 0. Cause, checked in the data: ProsQA assigns concept ids in breadth-first order, so label 2 is always at depth 1 and the mean depth rises monotonically to 3.0 at label 25; the model learned that cue. Accuracy with thoughts free on 50 test graphs: original 44; concepts deranged into 2..30, 13; the graph's own labels permuted, 22; only the two names swapped, 44; two-label swaps 23-39 of 31-50 (74-82 percent); the paper's target-decoy swap 40 (the paper: 79 percent). The renamed cell stays in the driver as exploration only.
Replacement (run 3): parent swap one level up, free twin breaks on 4 of 10 (competence loss), intermediates fixed adds nothing beyond it (flips beyond the free twin -0.10 [-0.40, +0.20]; moves the same), all K nothing beyond. Same-depth parent swap: nothing in any variant, as both stories said. The identity story's prediction for the one-level-up swap is not confirmed at n=10; power is low because the free twin itself breaks on 4 graphs.
Query-key subtraction (qk_subtract/* cells): removing the span of the eight directions from thought K-1 cuts the answer edge's attention, summed over the eight layer-2 heads, from 2.10 to 0.38 (drop 1.71 [1.10, 2.36]) and leaves the control edge at 1.40 to 1.42; the answer flips on 4 of 10 with e near 0 (median -8.7; flips beyond the same-answer reference +0.20 [0.00, +0.50]). Matched random directions: attention 2.10 to 2.08, no flips. Non-answer edge's directions: that edge's attention 1.40 to 0.17, the answer edge's 2.10 to 1.90 (interval spans zero), no flips. Per head: heads 3 and 4 carry the most (drops 0.89 and 0.99; coefficients 0.25 and 0.28 of the thought norm); single-head removals rarely flip (head 4: 2 of 10). The eight directions are nearly orthogonal to each other (mean absolute cosine 0.20) and to the input embeddings of the edge's source (about 0.05) and of the target (about 0.1). This is the first linear edit of a thought with a causal effect; the paper's per-branch nulls were in bases these directions do not lie in. What the pilot changed: the renamed cell was fixed (name convention) and then demoted to exploration; the two parent-swap cells were added with predictions before run 3; every changed prompt is now also read against its free twin, paired per graph (beyond_free_twin). Nothing else. Reading of the mirror pair: "reordered does not break" is established at pilot level with no exception in 10 graphs; the renamed half cannot be run on this substrate. The identity story's positive support therefore rests on experiment 2 (n=100, both seeds), the query-key cell, and experiment 4.

Before the n=100 run (2026-09-06, user's decisions): the parent-swap cells are dropped (their free twin breaks too often to read anything from them). Two additions to the query-key cell, predictions first:

Per head, at n=100. Identity story: each single-head removal drops that head's own attention on the answer edge by an amount that tracks the head's coefficient in the thought; the answer flips above the same-answer reference only for the heads that carry most of the match (heads 3 and 4 in the pilot), and the all-heads removal flips more graphs than any single head. Position story: the same drop pattern; no prediction on the answer. Matched random per head: no drop, flips at the reference rate.
At every intermediate step (the paper's path elimination, asked in the right coordinates). At each pass k = 0..K-2 the answer-path edge read at that step is the most attended edge from a depth-(k+1) node on a shortest path to the target to a depth-(k+2) node on one (the last step's edge is the parent edge). From thought k+1 the span of the eight directions that its layer-2 heads map onto that edge's key is removed; all steps in one run, directions taken from the unedited run. Identity story: attention onto the answer-path edge drops at every step and the answer flips at least as often as in the last-step-only cell; the paper's identity-basis version of this removal did nothing, and this cell asks whether that null was the basis. Position story: attention drops if the pointer is linear in the thought; no prediction on the answer. Controls: matched random directions at every step (no drop, reference-rate flips); the most attended frontier edge off the answer path at every step (its own attention drops, the answer path's does not, no flips beyond the reference; skipped on graphs where some step has no such edge). A per-head every-step version is run as well, read like the per-head last-step cells.

n=100 outcome, experiment 3, seed 0 (results/seed0/counterfactuals_n100.json; seed 1 pending):

Controls: reserialized and self-transplant exactly zero on all 100. Random donor at intermediates -8.4 (30 percent flipped, 66 escaped), at all K -16.4 with e 1.00. Same-answer donor 19 percent flipped at intermediates, none at all K.
Edges reordered: no effect in any variant (flips 0 to 3 percent, no escapes; moves beyond the free twin -0.03 [-0.07, 0.00]). The unreachable-only reorder likewise. The position story's primary prediction fails at n=100.
Decoy swap and non-candidate swap: no effect in any variant; p_watch 0.000 on all 100 graphs. The position story's final-thought prediction fails at n=100.
Last-hop rewrite (68 constructible): all K fixed keeps the old answer on 68 of 68; thoughts free follow the edit on 29 percent, intermediates fixed on 34 percent (+0.04 [0.00, +0.10] beyond the free twin). Depth-1 and depth-2 rewrites are followed on 54 and 52 percent with thoughts free and on 0 percent with intermediates fixed, as both stories said.
Labels renamed (exploration; the free twin fails by the label cue, 49 percent flipped, 49 escaped). Read against its free twin, fixing the thoughts adds escape, not flips: moves beyond the free twin +0.17 [+0.08, +0.26] at intermediates and +0.32 [+0.23, +0.41] at all K, escape rising from 49 to 74 and 95 percent. A thought that names stale labels makes the model name other nodes; the position story predicted nothing beyond the free twin. Suggestive for the identity story, but not a clean cell.
Query-key subtraction, last step, all heads: the answer edge's attention (summed over the eight layer-2 heads) falls from 1.79 to 0.31 (drop 1.47 [1.30, 1.67]); the control edge is unchanged (1.11 to 1.13); matched random directions leave both (1.79 to 1.77); the non-answer edge's directions cut that edge to 0.13 and the answer edge to 1.68. The answer does not move beyond the fallback rate: median -0.2, 17 percent flipped, -0.02 [-0.11, +0.07] beyond the same-answer reference. The pilot's 4 of 10 was a small sample. Per head: the head-drops track the coefficients (head 4: 0.49, head 3: 0.39, head 0: 0.18, head 7: 0.22; heads 1, 2, 5 below 0.05) and no single head flips beyond the reference.
Query-key subtraction at every intermediate step, all heads: the answer-path edge's attention is removed at every step (mean per-step drop 1.60 of about 1.8; at the last step 1.79 to 0.13); the answer flips on 19 percent, 0.00 [-0.08, +0.08] beyond the reference. Random directions at every step: no drop, no flips. The off-path control cuts its own edge to 0.10 and, by overlap of directions, the path edge to 1.14, with 8 percent flips (below the reference). Per head: path drops 0.17 to 1.19, flips at most 12 percent, none beyond the reference. Reading: the linear geometry that drives the thought's layer-2 attention onto the answer path is real and removable, and removing it does not change the answer beyond fallback. The paper's "rewrite" null replicates in the right coordinates. The answer must be recovered elsewhere: layer 1 also reads edges (experiment 2), and the answer position has its own content heads. Where it is recovered is the next question, not answered here.
Splits: none of the story cells split (all at 0 or near 0); the query-key cells' flipped graphs are not distinguished by depth, branches, or the answer edge's slot (split tables in the file).

n=100 outcome, experiment 4b, seed 0 (results/seed0/cache_patch_n100.json; seed 1 pending): consistent patches from the reordered graph (keys and values together in layer 1, layer 2 or both; every latent-position slice) do nothing (flips 0 to 3 percent). Inconsistent ones break: layer-2 keys only -21.2, 40 percent flipped, +0.21 [+0.10, +0.32] beyond the reference; layer-2 values only -28.1, 40 percent, +0.21 [+0.09, +0.33]; values in both layers 6 percent (consistent again through the separator copy). The position story's keys-only prediction (no effect) fails; the identity story's (breaks) holds, as does its consistency reading. Random-graph cache: edge slots break in every variant (34 to 47 percent flipped, +0.15 to +0.29 beyond the reference); layer-1 values at all latents -35.5 with e 1.00 (43 percent flipped, 81 escaped) while layer-2 values at the latents and every intermediate-latent slice do nothing. Only the final latent's layer-1 values carry the answer to the answer position. Self patch exactly zero.

n=100 outcome, experiment 3, seed 1 (results/seed1/counterfactuals_n100.json): replicates seed 0 on every cell. Reordered, unreachable-only reorder, decoy swap and non-candidate swap: no effect in any variant (p_watch 0.000 on all 100). Last-hop rewrite with all K fixed keeps the old answer on 68 of 68; with thoughts free this model follows the edit on 53 percent (seed 0: 29). Renamed (exploration): fixing the thoughts again adds escape beyond the free twin, +0.19 [+0.11, +0.27] moves at intermediates and +0.28 [+0.20, +0.37] at all K, flips unchanged. Query-key subtraction, last step: the answer edge's attention falls 2.54 to 0.37 (drop 2.17 [1.95, 2.39]), controls flat; the answer -3.2 median, 15 percent flipped, -0.10 [-0.19, 0.00] beyond the same-answer reference (26 percent). Every step: attention 2.54 to 0.13, mean per-step path drop 1.99; answer -3.9, 22 percent flipped, -0.03 [-0.15, +0.09] beyond the reference. No single head flips beyond the reference on either seed. Same-answer donor at all K: no flips. Reserialized and self-transplant exactly zero.

n=100 outcome, experiment 4b, seed 1 (results/seed1/cache_patch_n100.json): replicates seed 0. Consistent patches from the reordered graph: no effect (edges, any layer, keys and values together: 1 to 3 percent flipped; latent slices 0 to 1). Layer-2 keys only -34.0, 43 percent flipped, +0.18 [+0.05, +0.32] beyond the reference; layer-2 values only -19.4, 35 percent, +0.10 [-0.03, +0.22]. Random-graph cache at the edge slots breaks (40 to 44 percent flipped); layer-1 values at all latents -61.8 with e 1.00 (52 percent flipped, 81 escaped); layer-2 values at the latents and every intermediate-latent slice do nothing. Self patch exactly zero.

Availability (from tests/test_prompts.py on training graphs 0-299, so the n=100 cells will have skips): reorder, rename, unreachable-only reorder, decoy swap and candidate swap are constructible on every graph; the last-hop rewrite on 201 of 300 (a cut edge into the target is needed); rewrites at depth 1 on 74, depth 2 on 97, depth 3 (when K=4) on 73. Skips are counted per cell in the results file.

Reading of the four outcomes of the mirror pair (written before the pilot):

Reordered breaks, renamed does not: the mid-search thought is positional. The model reads slots and dereferences names later. Then experiment 2 should show the thought-reading layer-2 heads as positional, and the keys-only cache patch (experiment 4b) should do nothing.
Renamed breaks, reordered does not: the mid-search thought names nodes. The paper's linear edits missed the code, not the content. Then experiment 2 should show content heads, and the keys-only cache patch should break the answer.
Both break. Two readings, separated by the escape levels. Plain fragility: any mismatch between a fixed thought and the prompt destroys the answer, so e in both cells sits at the random-donor level and the unreachable-edges reorder (which neither story cares about) also breaks. Mixed code: the thought carries both cues and the readout needs both, so removing one cue degrades rather than destroys: e sits between the reserialized baseline and the random-donor level, T moves toward 50 rather than to 0, and the unreachable-edges reorder stays at the reserialized baseline. If both cells destroy but the unreachable-edges reorder does not, the code is mixed and both cues are load-bearing. Amendment after the experiment-0 pilot: on this model a broken search often ends in a clean flip to the other candidate (same-answer donor: 4 of 10 flip with e near 0). So "breaks" means T or e moved; a flip counts as redirection only if the same graph does not also flip under the same-answer donor. The escape-level comparison in reading 3 uses the random donor's e and the same-answer donor's outcome as the two references, per graph.
Neither breaks. The fixed intermediate thoughts are not what the final steps consume, at least not slot-wise or name-wise. Check the random donor at intermediates first: if it breaks (as in the paper) while neither counterfactual does, the thought carries something invariant to both edge order and labels, an unlabeled search shape, and the all-K cells and experiment 4 become the main line. If the random donor also fails to break, the sustained-transplant result of the paper did not replicate on this model and experiment 0 needs a second look before anything else.

Pilot outcome: (not run) What the pilot changed: (not run) n=100 outcome: (not run)

Experiment 4: where the effect lives

4a, causal tracing by position. Corrupt the prompt with an on-manifold counterfactual whose answer differs (candidate swap: target and decoy tokens transposed in the edge list, question line unchanged; and the last-hop rewrite), run it, then restore the clean run's residual stream at one position and one level at a time (level 0 = input to layer 1, which for a latent position is the recycled thought plus its position embedding; level 1 = after layer 1; level 2 = after layer 2) and record T. Compare edge slots (source token, target token, separator), the question and root tokens, latent positions, and the answer position.

Position story: no separating prediction. Identity story: no separating prediction. Shared expectation: under the candidate swap the intermediate latent positions carry no effect (their thoughts are near-identical in the paper, cosine 0.9975); the effect sits at the edge slots holding the candidates and at the final thought, last latent, and answer position. This is the map that tells 4b which slices to patch. Conditional follow-up if experiment 3's reordered cell breaks: un-fix one intermediate thought at a time to find the step where the pointer bites.

4b, memory-cache patching. Donor: the same graph with edges reordered, run naturally. Recipient: original order, its own thoughts. Patch the donor's keys and/or values at chosen positions into the recipient's cache at every pass.

Keys only, edge slots, layer 2. Position story: no effect (the position part of each key is unchanged, so attention stays put and reads the recipient's own values). Identity story: breaks (keys now carry other edges' source nodes; attention follows them to slots whose values are different edges). Separating.
Values only, edge slots, layer 2. Both: breaks. Consistency.
Keys and values, latent positions 1..K-1. Position story: breaks. Identity story: no effect. Separating; the cache-level twin of experiment 3's primary cell.
The same three cells in layer 1: no prediction from either story; exploration.

For any carrier claimed, three checks (Ding et al.): the right patch moves the answer; the other slice from the same donor does not; replacing the slice with the same slice from a random graph's run breaks the recipient. Plus the self-patch (recipient's own cache back into itself), which must be exactly zero.

Pilot outcome, 4a (seed 0, test graphs 400-409, token granularity; results/seed0/tracing_pilot.json; candidate swap on 10, last-hop rewrite on 7). Mean recovery by position class and level:

Final latent, level 0 (the final recycled thought plus its position embedding): 0.98 and 0.99. Its levels 1 and 2: 0.00. So the final thought acts through the layer-1 keys and values written at the final latent position, which the answer position reads.
Intermediate latents: levels 0 and 1 about 0 (slightly negative under the rewrite, -0.13); level 2 (the output that becomes the next thought) 0.38 and 0.30, which is latent K-1's output, that is thought K, doing the work.
Changed edge slots: the target token at level 0 recovers 0.26 (candidate swap; several slots change) and 1.00 (rewrite; one slot changes); the separator at level 1 recovers 0.24 and 0.98; the source token 0.00. Candidates, root, and unchanged slots: 0.00. The edge's identity enters at its target token and is copied to the separator by layer 1, where layer 2 reads it; the same separator finding as experiment 2. No story-separating prediction was made for this map; it is consistent with the identity story's mechanism and it names the carriers for 4b.

Pilot outcome, 4b (results/seed0/cache_patch_pilot.json):

From the reordered donor: keys and values together, in layer 1, layer 2 or both, at the edge slots: no effect (no flips). Every patch of the latent positions (intermediate or all, keys, values or both): no effect. Under the position story the layer-2 cache of the reordered graph should have sent the pointers to other edges; it did not.
Inconsistent patches break: layer-2 keys only, median -4.0, 3 of 10 flipped (+0.10 beyond the reference, interval spanning zero); layer-2 values only, -62.6, 6 of 10 flipped. Values in both layers: no effect, because the separator's layer-1 copy then reads the donor's edge and the layer-2 keys become consistent with the values again. Reading: a consistent reordered cache is read correctly by content; only a cache whose keys and values disagree breaks the search. The keys-only prediction (identity: breaks; position: nothing) came out partial and cannot be called at n=10.
Random-graph cache (Ding's necessity): edge slots, layer-1 values or keys and values, -95 to -97 with 6 of 10 flipped; layer-2 keys, 4 of 10; all latents, layer-1 values -50 (5 flipped, 8 escaped) while layer-2 values at the latents and every intermediate-latent slice do nothing. Only the final latent's layer-1 values carry the answer to the answer position, matching 4a. Self patch exactly zero; same-answer donor 2 of 10. What the pilot changed: nothing in the design of 4a or 4b. After a memory kill lost a near-complete run, tracing now saves each graph as it finishes and resumes (tracing_<mode>_rows.jsonl). n=100 outcome, 4a, seed 0 (results/seed0/tracing_n100.json; candidate swap on 100 graphs, last-hop rewrite on 68). Mean recovery: final latent at level 0 (the final thought plus its position embedding) 0.98 [0.96, 0.99] and 0.95; its levels 1 and 2, 0.00. Intermediate latents: levels 0 and 1, 0.08 and -0.04; level 2 (the output that becomes the next thought), 0.47 and 0.34. Changed edge slots: target token at level 0, 0.34 (several slots change under the candidate swap) and 1.00 (one slot under the rewrite); separator at level 1, 0.31 and 0.96; source token 0.00. Candidates, root, unchanged slots: 0.00 (root at level 2 under the swap 0.03). The pilot's map holds at n=100. Seed 1 (results/seed1/tracing_n100.json) replicates: final latent at level 0, 1.02 [0.96, 1.09] and 1.00; its levels 1 and 2, 0.00; intermediate latents at level 2, 0.44 and 0.35, at levels 0 and 1, 0.03 and -0.04; changed slot's target token at level 0, 0.30 and 1.00; its separator at level 1, 0.27 and 0.95; source token, candidates, root and unchanged slots 0.00. Both seeds: the final thought does its work through the layer-1 keys and values written at the final latent position; an edge's identity enters at its target token and is copied to the separator by layer 1, where layer 2 reads it; the intermediate latents matter only through the thought they emit. n=100 outcome, 4b: see the experiment 3 block's neighbour above and the seed-0 block ("n=100 outcome, experiment 4b, seed 0").

Experiment 5: where the answer is recovered (masking), written 2026-09-10

Experiments 1-4 favored the identity story. Experiment 3's query-key cell left one loose end: removing the thought's linear match to the answer-path edges empties that attention in layer 2 (last step 1.79 to 0.31 and 2.54 to 0.37; every step: path attention 1.60 and 1.99 of about 1.8 and 2.5 gone), yet the answer stays at the fallback rate. The answer is recovered somewhere else. Experiment 2 showed two other routes that read edges by content: layer 1's edge-reading heads at the latent positions, and the answer position's own heads reading edges. This experiment blocks those routes (a mask on attention, attn_hooks.AttnHooks.add_mask) alone and in combination with the query-key removal, on the existing checkpoints.

Design. Recipients and controls as in experiment 3 (test graphs; pilot 400-409 on seed 0; n=100 on both seeds; reserialized, self-transplant, random donor, same-answer donor at intermediates and at all K; the query-key every-step removal, its matched random directions and the off-path control). Head sets are chosen per seed from experiment 2's n=100 summary (heads_n100.json), by a named cutoff: a head is an edge-reading head at a query class if its mean attention onto edge slots there is at least 0.5 (the cutoff protects "this head reads edges", stricter than experiment 2's 0.2 because here heads are removed, not scored). On seed 0 that gives layer-1 heads 3, 4, 6, 7 at the latents and layer-1 heads 2, 5, 6 plus layer-2 heads 0, 1, 2, 5, 6, 7 at the answer position; on seed 1 layer-1 heads 1, 3, 5, 6, 7 at the latents and layer-1 head 0 plus layer-2 heads 2, 4 at the answer position. The chosen sets are recorded in the results file. Masks act on the answer-path edges' slots (every shortest-path edge from depth k+1 to depth k+2, all steps) or, as control, on a matched set of off-path edges (the most attended frontier edges off the path, same count). Masking a slot blocks attention to all three of its tokens.

Cells (each is a mask set; "plus removal" adds the query-key every-step all-heads removal from experiment 3):

L1 latents: layer-1 edge heads at every latent query, path edges masked; alone and plus removal.
Answer heads: the answer position's edge-reading heads (both layers), path edges masked; alone and plus removal.
All routes: L1 latents and answer heads together; alone and plus removal.
L2 latents mask: layer-2 heads at the latent queries, path edges masked, alone. This is the mask version of the query-key removal; it should reproduce its null and calibrates masking against subtraction.
Off-path controls: every mask set applied to the matched off-path edges.
Query-key removal alone (the experiment 3 cell, rerun here on the same graphs as the reference).

Predictions, one line per story where a story forces one:

Identity story, through the mechanism the experiments have mapped: the thought's match to the path is one of at least two content routes. A mask that blocks one route alone leaves the answer near baseline (flips at the same-answer reference), as the removal did. Blocking a route together with the removal collapses the answer (flips beyond the reference, comparable to the random donor) if that route is the recovery path. Concretely: at least one of "L1 latents plus removal" and "answer heads plus removal" collapses, and "all routes plus removal" collapses on the largest share of graphs. Off-path masks do nothing beyond the reference in any combination.
Position story: refuted; no prediction.
Neither story: if "all routes plus removal" still leaves the answer at the reference rate, the recovery does not go through attention onto edges at all (candidates: the MLPs at the final positions, or content the final thought already carries before any edge is read), and the cell says so.
Calibration: "L2 latents mask" alone reproduces the query-key null (attention onto the path gone, flips at the reference). If it collapses where the removal did not, the removal was incomplete and the loose end is an artifact of the subtraction, which changes the reading of experiment 3.

Cutoffs, all named: edge-reading head 0.5 of attention on edge slots; flip and escape as everywhere; "collapses" means flips beyond the same-answer reference with the interval above zero.

Pilot outcome (seed 0, test graphs 400-409; results/seed0/masking_pilot.json; the first run's file is kept as masking_pilot_v1_l2_in_latents_set.json):

Head sets on seed 0 at the 0.5 cutoff: layer-1 heads 3, 4, 6, 7 at the latents; at the answer position layer-1 heads 2, 5, 6 and layer-2 heads 0, 1, 2, 5, 6, 7.
Masks alone do nothing: layer-1 latent heads, answer heads, and both together, on the path edges, medians -0.0, no flips on 10 of 10; off-path the same.
Masks on top of the query-key removal add nothing to it. Per graph, the removal alone and every "plus removal" cell land within a few points of each other (graph 400: -74.9 alone, -67.7 with layer 1 masked, -77.7 with the answer heads masked, -71.2 with all routes masked; graph 401: -96.5 in all four; graph 407: -0.4 in all four). Medians -7.9 (removal), -9.9, -8.6, -10.4; flips 3 of 10 in each; escape near 0. The off-path masks plus removal give the same numbers again (-10.6, -8.4, -10.7).
Calibration: the layer-2 mask on the path edges at the latents reproduces the removal cell almost exactly (median -9.3, the same three graphs flipped, per-graph values within 3 points). Masking and subtraction agree, so the removal was not incomplete.
Controls: reserialized and self-transplant exactly zero; random donor -25.1 with e 0.87; same-answer donor 2 of 10 at intermediates, none at all K. Reading, per the predictions: this is the "neither story" branch. Blocking the thought's layer-2 match, layer 1's edge reading at the latents, and the answer position's edge reading, all at once, leaves the answer exactly where the layer-2 removal alone leaves it. On 7 of 10 graphs that is within a few points of baseline; on 3 it is a flip that no added mask deepens. The recovery does not go through attention onto the path edges at any position. What remains: the final positions' MLPs, attention onto non-edge tokens (the candidates, the root), or content the recycled thoughts already carry before any edge is read at the last steps. What the pilot changed: one fix. The first run's head-set rule admitted every layer-2 head into the "latents" set (all eight read edges there), so the layer-1 cell masked layer 2 too and duplicated the calibration cell. The latents set is now layer 1 only, as the predictions say; the answer set is unchanged. Nothing else. n=100 outcome (both seeds; results/seed0/masking_n100.json, results/seed1/masking_n100.json). Head sets at the 0.5 cutoff: seed 0, layer-1 heads 3, 4, 6, 7 at the latents and layer-1 heads 2, 5, 6 plus layer-2 heads 0, 1, 2, 5, 6, 7 at the answer position; seed 1, layer-1 heads 1, 3, 5, 6, 7 at the latents and layer-1 head 0 plus layer-2 heads 2, 4 at the answer position.
Masks alone do nothing on either seed: layer-1 latent heads, answer heads, and both together, on the path edges, medians -0.0, flips 0.00, escapes at most 0.01; off-path masks the same.
Masks on top of the query-key removal add nothing to it. Paired per graph against the removal alone, flips beyond it are 0.00 [0.00, 0.00] for the layer-1 mask, -0.02 [-0.05, 0.00] for the answer heads and -0.01 [-0.03, 0.00] for all routes on seed 0; 0.00 [-0.03, +0.03] for all three on seed
1. The median per-graph difference in T against the removal alone is 0.0 in every cell on both seeds. The flip rates sit at the removal's own (17 to 19 percent on seed 0, 22 on seed 1), which is the same-answer reference rate (19 and 26). Off-path masks plus removal: identical numbers.
Calibration: the layer-2 mask on the path edges at the latents matches the removal (flips 0.15 vs 0.19 and 0.20 vs 0.22; median differences 0.0 and +0.2). Masking and subtraction agree; the removal was not incomplete.
Controls: reserialized and self-transplant exactly zero on all 200 runs; random donor -8.4 (e 0.94) and -37.7 (e 0.83); same-answer donor 19 and 26 percent at intermediates, none at all K; matched random directions 0.00. Conclusion, the "neither story" branch at n=100 on both seeds: with the thought's layer-2 match, layer 1's edge reading at the latents, and the answer position's edge reading all blocked on the answer-path edges, the answer stays where the layer-2 removal alone leaves it, which is at the fallback rate. The answer is not recovered through attention onto the path edges at any position or layer. Remaining candidates, not tested here: the MLPs at the final positions; attention onto non-edge tokens (the two candidates after [Q], the root); and content the recycled thoughts carry into the last steps before any edge is read there. The third fits experiment 4: restoring the final thought alone recovers the answer (0.98 to 1.00) and the intermediate latents matter only through the thought they emit. Whether to test these is the user's decision; each is an analysis-only step on the existing checkpoints.

Experiment 6: where the answer is recovered, part two (written 2026-09-11)

Experiment 5 excluded every attention route onto the path edges. Three candidates remain: the MLPs at the final positions; attention onto non-edge tokens (the two candidates after [Q], the root); and content the recycled thoughts already carry into the last steps. Three checks on existing data were run before the design was fixed (2026-09-11, on this Mac):

Target id below decoy id on 41.5 percent of test graphs (174 of 419), 42.2 percent of training graphs, 44 of the 100 masking recipients. A mild id cue against the target; not a shortcut the fallback can use.
Every decoy on both splits has at least one incoming edge (mean in-degree 2.15; target 1.54), so the requested split of the masking-run graphs by decoy in-edges has an empty group. Under the query-key removal the flipped graphs put 0.83 and 0.87 of the mass on the decoy: the fallback is a clean switch to the other candidate.
Per-step separation of target from decoy inside the thought (500 held-out training graphs, paper's inner-product readout, target minus decoy): the target scores higher on 0.46/0.42 of graphs at step 1, 0.63/0.57 at step 2, 0.92/0.88 at step 3, 1.00/1.00 at step 4 (seed 0 / seed 1); a paired probe reaches AUC 0.99/0.98 at step 3 and 1.00 at step 4. The thought separates the candidates one full step before the answer is emitted. The same comparison for the candidates' parents is flat (0.44 to 0.79).

Design. Recipients, controls and the query-key every-step removal as in experiment 5 (eager path). "Final positions" means the final latent (holding thought K) and the answer position. Cells, each alone and plus removal:

Candidate tokens: mask attention from the final positions onto the two candidate tokens (both layers). Control: mask onto the [Q] and [R] markers. Exploration: no prediction.
Root token: mask onto the root token. Same control. Exploration: no prediction.
Final-position MLPs: replace the MLP output at a final position, per layer, by its mean over 2,000 training graphs at the same role (final latent, answer position). Control: the mean plus Gaussian noise at the norm of the mean deviation. Cells: answer position (both layers), final latent (both layers), each layer alone at the answer position.
Thought carry-over: substitute thought K by the same-answer donor's thought K (same candidates, same correct answer) or by the random donor's; each alone and plus removal.
Decoy's incoming edges: mask the final positions' attention (both layers) onto the slots of every edge into the decoy. Control: the same count of the most attended edges into unreachable non-candidates.
Both candidates' incoming edges: 5 plus the target's parent edges.
Candidate tokens plus decoy edges: cell 1 and cell 5 together.

Predictions (position story refuted throughout; "fallback line" is about the switch to the decoy, which experiment 5 showed is the fallback):

3, MLPs. If the answer is computed at the last step from the edges, the answer-position MLPs collapse the answer alone (flips beyond the reference, comparable to the random donor). If the answer is carried in the thought, the final latent's MLPs matter more than the answer position's, and plus-removal exceeds alone at the latent, not at the answer position. The noise control moves less than the mean replacement either way.
4, thought carry-over, the primary cell (check 3 says the content is there). Identity story: under the removal, the same-answer donor's thought K keeps the answer (flips at or below the reference) while the random donor's breaks it, and the gap between them is the carried content. If both donor thoughts give the removal's own result, the final thought's content is not what is used and the recovery is computed at the end.
5, decoy's edges. Identity story: no change alone; no change beyond the removal on graphs the removal leaves correct, because their thought carries the target. Fallback line: on graphs that flip under the removal, masking the decoy's edges takes away the evidence the fallback reads, so the switch to the decoy weakens (p_decoy falls, mass moves to the target or escapes); if it does not, the fallback does not read the decoy's edges either. Control edges: nothing.
6, both candidates' edges. Identity story: alone, nothing (the thought carries the target); plus removal, no more than 5 plus removal. Fallback line: if the fallback compares the two candidates' evidence, removing both gives escape rather than a switch.
7, candidate tokens plus decoy edges. Identity story: whatever cell 1 does, adding the decoy's edges adds nothing on graphs whose thought carries the target. Fallback line: the decoy cannot be named without its token and cannot be supported without its edges, so escape replaces the switch on the flipped graphs.
Off-path and marker controls: nothing in any combination.
Overall: prediction 4 on the "carried" side and 3 on the "final latent" side is the identity story's account (the thought carries the answer into the last steps; the edge reading there is confirmation). The other pattern (answer-position MLP collapses alone, donor thoughts indifferent) would be a readout computed from the prompt at the very end.

Cutoffs, all named: flips and escapes as everywhere; "collapses" is flips beyond the same-answer reference with the interval above zero; "weakens" for the fallback line is mean p_decoy on the removal-flipped graphs falling by at least 0.25 (protects "the switch lost its support").

Pilot outcome, run 1 (seed 0, test graphs 400-409; results/seed0/recovery_pilot_v1_noise_control.json):

Cell 4, thought carry-over: under the removal, the same-answer donor's thought K restores the answer on 10 of 10 graphs, including the three the removal flipped (their p_target 0.97, p_decoy 0.03); the random donor's thought K destroys it (e 1.00 on all, mass on other nodes). Alone, the same-answer donor's thought K changes nothing (median +0.0) and the random donor's breaks (-75.3, e 1.00). The identity story's "carried" prediction, per graph without exception.
Cell 3, MLPs: replacing the answer position's MLP output by the training mean breaks alone (median -30.8, e 0.49; 5 of 10 escape), and the effect sits in layer 1 (-20.0, e 0.46) not layer 2 (-0.0). The final latent's MLPs do nothing alone or beyond the removal. The noise control broke more than the mean replacement (-34.1, e 0.79), the reverse of the prediction: the control was off-manifold (mean plus a random vector at the deviation norm is not an MLP output the model produces), so it cannot calibrate the mean replacement. Design fault, fixed below.
Cells 1, 2, 5, 6, 7 (candidate tokens, root token, decoy's edges, both candidates' edges, tokens plus decoy edges): nothing alone (medians +0.0, no flips on 10 of 10) and nothing beyond the removal (per graph within 8 points of the removal alone; controls identical). Masking the candidate tokens at the final positions does not stop the model naming the target.
Fallback line, on the three removal-flipped graphs (p_decoy 0.84 under the removal): the token and edge masks leave it at 0.78 to 0.85 (none weakened); the answer position's layer-1 MLP mean replacement drops it to 0.41 (escape 0.54); the same-answer donor's thought K drops it to 0.03 with p_target 0.97. The switch to the decoy is carried in thought K and read out by the answer position's layer-1 MLP; it does not read the decoy's edges or the candidate tokens.
Controls: reserialized and self-transplant exactly zero; random donor -25.1 (e 0.87); same-answer donor 2 of 10 at intermediates, none at all K; matched random directions +0.0. What the pilot changed: the MLP noise control is replaced by on-manifold donor controls, mirroring cell 4: the MLP output captured at the same role and layer from the same-answer donor's own run and from the random donor's own run. Amended prediction for cell 3 (written before run 2): if the answer position's layer-1 MLP output encodes the answer, the same-answer donor's output keeps the answer and the random donor's breaks it, with the mean in between; if that MLP is a generic on-manifold requirement, all three replacements behave alike. Nothing else changed.

Pilot outcome, run 2 (seed 0, same graphs; results/seed0/recovery_pilot.json): only the MLP control changed, and the on-manifold controls settle cell 3. At the answer position, layer 1: the same-answer donor's MLP output keeps the answer on 10 of 10 (median +0.0; on the three removal-flipped graphs, under the removal, p_target 0.98); the random donor's output destroys it (-49.3 alone, e 1.00 on all); the training mean sits between (-20.0, e 0.46). Layer 2 alone does nothing under any replacement. The final latent's MLPs do nothing under any replacement, alone or beyond the removal. Every other cell is unchanged from run 1. Amended prediction for cell 3 confirmed: the answer position's layer-1 MLP output encodes the answer (same-answer donor's keeps it, random donor's breaks it), and it does so whether or not the thought's match to the path was removed. Design frozen after run 2. n=100 outcome (both seeds; results/seed0/recovery_n100.json, results/seed1/recovery_n100.json). Removal-flipped graphs: 19 on seed 0, 22 on seed 1 (p_decoy under the removal 0.83 and 0.87).

Cell 4, thought carry-over: under the removal, the same-answer donor's thought K restores the answer on 19 of 19 and 22 of 22 removal-flipped graphs (p_target 0.96 and 1.00; p_decoy 0.04 and 0.00), and leaves every other graph at baseline (medians +0.0, no flips, no escapes, 98 of 98 on each seed). The random donor's thought K destroys the answer (e 1.00; escape 74 and 77 percent) alone and under the removal alike. Identity story's "carried" prediction, per graph without exception on 200 graphs.
Cell 3, MLPs at the answer position, layer 1: the same-answer donor's output keeps the answer on 98 of 98 (alone and under the removal; under the removal it restores 19 of 19 and 22 of 22 flipped graphs, p_target 0.98 and 1.00); the random donor's destroys it (e 1.00, escape 79 percent); the training mean sits between (escape 25 and 14 percent). Layer 2 at the answer position: mean alone -0.0, and under the removal the same-answer output lowers the flip rate below the removal's (0.14 vs 0.19; 0.08 vs 0.22) while the random output leaves it (0.20; 0.26). The final latent's MLPs: nothing under any replacement, alone or beyond the removal, on both seeds. Amended prediction confirmed: the answer position's layer-1 MLP output encodes the answer.
Cells 1, 2, 5, 6, 7: nothing alone (medians -0.0 to +0.0, flips 0.00, on both seeds) and nothing beyond the removal (flips beyond the reference -0.03 to +0.03, all intervals spanning zero; medians within 2.5 points of the removal alone). Masking the candidate tokens, the root, the decoy's edges, both candidates' edges, or tokens plus decoy edges at the final positions changes neither the correct answers nor the fallback.
Fallback line, on the removal-flipped graphs: the token and edge masks leave p_decoy at 0.83 to 0.90 (weakened on 0 to 5 percent); the answer position's layer-1 MLP mean replacement drops it to 0.45 and 0.49 with escape 0.41 and 0.27 (weakened 63 and 55 percent); the same-answer donor's thought K or layer-1 MLP output drops it to 0.00 to 0.04 with p_target 0.96 to 1.00 (weakened 95 to 100 percent). The switch to the decoy is carried in thought K and read out by the layer-1 MLP at the answer position; it does not read the decoy's edges or the candidate tokens.
Controls: reserialized and self-transplant exactly zero on all 200 runs; random donor breaks (e 0.94 and 0.83); same-answer donor 19 and 26 percent at intermediates, none at all K; matched random directions +0.0. Conclusion. After the thought's layer-2 match to the answer path is removed, what still produces the answer is content thought K already carries, read out by the answer position's layer-1 MLP; when the removal flips a graph, thought K carries the wrong candidate and the same readout names it. No attention onto edges or tokens at the final positions is involved in either case (experiments 5 and 6). Together with the per-step separation in check 3 (the target separated from the decoy in the thought by step 3, perfectly by step 4), this is the identity story's account carried through to the readout: by the last step the thought encodes which candidate wins, and the edge reading at the final positions is confirmation rather than computation. It also revises the paper's sentence "identity is bound late": the identity is present in the thought from step 3 on; the paper's final- step swap and final-step transplant acted on a thought that already carried the answer.

Experiment 6b: candidate-edge masks at every latent query, and the per-step carry-over (written 2026-09-11, before the run)

Two additions requested after experiment 6, predictions first.

Cells (each alone and plus the query-key every-step removal; controls as in experiment 6, plus the matched off-path edges for the masks):

Decoy's incoming edges masked at every latent query (both layers, all heads) and at the answer position. Elimination line: if the search rules out the decoy by reading its edges, the thought loses the winner and the fallback rises (flips beyond the reference above zero, p_decoy up on the removal-flipped graphs, or escape up). Label-cue line: nothing changes.
Both candidates' incoming edges masked at every latent query and the answer position. Elimination line: the thought loses both, escape rises. Label-cue line: nothing changes.
Per-step carry-over: the same-answer donor's thought at step j only, for j = K-1 and K-2 (thought K left to the model), alone and under the removal. Identity story: under the removal, restoration of the flipped graphs falls with distance from the end: near-complete at K-1 if the winner is already carried there (check 3: 0.92/0.88 separated at step 3), partial at K-2. If restoration at K-1 equals that at K, the carried answer is settled one step before the end; if it is near zero, thought K alone carries it. Random donor's thought at the same step as the breaking reference.

Checks run before this design (2026-09-11, on existing data and the trained models with thoughts free):

Test accuracy split by candidate id order, seed 0 / seed 1: target id below decoy id 161/174 = 0.925 / 165/174 = 0.948; target id above decoy id 235/245 = 0.959 / 0.959. A 1 to 3 point difference; the id order is not what the model relies on.
Winner probe from ids alone (candidate ids, the depth-k frontier ids, and their order; no graph structure), held-out training graphs 2500-2999: AUC 0.64 at every step (an order-only rule scores 0.60). The thought's own separation at step 3 is 0.99/0.98 and at step 4 1.00. The numbering predicts almost none of the step-3 separation; it is computed from the graph.

Pilot outcome (seed 0, test graphs 400-409; results/seed0/recovery_pilot.json; the pre-6b pilot file is kept as recovery_pilot_v2_before_6b.json):

Decoy's edges masked at every latent query and the answer position: alone, nothing (median +0.0, no flips); on top of the removal, the fallback rises: median -51.5, flips 5 of 10 (removal alone 3), and on the three removal-flipped graphs p_decoy goes from 0.84 to 0.95. The matched off-path edges plus removal reproduce the removal alone (-8.7, 3 flips). The elimination line, under the removal only: the decoy's incoming edges are read during the search and count against the decoy, but only matter once the thought's match to the path is gone.
Both candidates' edges masked at every latent query and the answer position: alone, 4 of 10 flip with escape 0 (median -1.5; the two K=3 graphs and two others); on top of the removal, 6 of 10 flip. With the evidence for both candidates removed the model still names a candidate, at about chance; escape does not rise. Neither prediction line as written: the fallback is a candidate-level choice, never an escape.
Per-step carry-over, under the removal: the same-answer donor's thought at K-1 restores 2 of the 3 removal-flipped graphs (p_target 0.71 on them) but disturbs 2 others (402: -6 to -34; 408: +11 to -74); at K-2 it restores none (p_target 0.00, escape 0.33) and the random donor's K-2 thought plus removal breaks (-37.0). The random donor's K-1 thought breaks (e 0.83). Consistent with check 3: the winner is nearly settled by step K-1 and not by K-2. What the pilot changed: nothing in the cells. One reading rule added: for the both-candidates mask the reference is a coin flip (flip rate near 0.5, escape near 0), not the same-answer donor's rate. The per-step cells are read split by K (the split machinery already carries K). n=100 outcome (both seeds; results/seed0/recovery_n100.json, results/seed1/recovery_n100.json; the experiment 6 files before 6b are kept as recovery_n100_v1_exp6.json):
Decoy's edges masked at every latent query and the answer position. Alone: flips 0.05 and 0.04 (below the reference; the decoy's edges are not needed when the thought carries the winner). On top of the removal: flips 0.46 and 0.40, that is +0.27 [+0.17, +0.37] and +0.18 [+0.10, +0.26] beyond the removal alone, with escape at 0.01 and 0.02 and the mass moving to the decoy (0.52 and 0.43 on average). The matched off-path edges plus removal reproduce the removal alone (-0.01 and 0.00 beyond it). The elimination line holds under the removal on both seeds: the decoy's incoming edges are read during the search and count against the decoy, and this only matters once the thought's match to the path is gone. On the graphs the removal already flipped, p_decoy stays where it was (0.81 and 0.83); the added flips come from graphs the removal had left correct.
Both candidates' edges masked at every latent query and the answer position. Alone: flips 0.33 and 0.28, +0.14 [+0.02, +0.27] and +0.03 [-0.09, +0.14] beyond the reference, escape 0.03 and 0.02; on top of the removal, flips 0.42 and 0.42, +0.23 and +0.20 beyond the removal. Read against a coin flip (the pilot's rule): with the evidence for both candidates removed the model still names a candidate, and gets it right more often than chance (58 to 72 percent), so some of the winner survives the loss of both candidates' edges; escape never rises above 0.03. Distribution of T under this mask alone (added 2026-09-11, since chance is the prediction): median 70.1 [37.7, 87.5] and 78.2 [60.6, 90.4], mean 56.6 [48.6, 64.7] and 62.8 [55.4, 70.1]; bimodal, with 34 and 25 graphs below 20, 43 and 48 above 80, and 23 to 27 percent between 20 and 80. Under the removal as well: median 36.6 and 62.0, mean 48.8 [40.7, 57.0] and 54.9 [46.8, 62.7]. So the mean sits at or a little above 50 while the per-graph values are near 0 or near 100: the model does not hedge; it commits to a candidate on each graph and is right on somewhat more than half. Chance in the sense of "a coin per graph" fits the mean, not the shape; the shape says the choice is still driven by something graph- specific that the two masks do not remove (for comparison, the decoy-edge mask alone has mean T 85 and 91 with 78 and 87 graphs above 80).
Per-step carry-over, under the removal, restoration of the removal-flipped graphs by the same-answer donor's thought at one step: at K, 19 of 19 and 22 of 22; at K-1, 9 of 19 (K=3: 7 of 14; K=4: 2 of 5) and 16 of 22 (K=3: 6 of 10; K=4: 10 of 12); at K-2, 7 of 19 and 10 of 22. The K-1 donor thought is not a clean intervention: alone it flips 0.19 and 0.26 of graphs (the same-answer donor's intermediate rate), and under the removal it flips 9 of 79 and 19 of 76 graphs the removal had left correct. Its net effect is zero beyond the reference (0.00 [0.00, 0.00]) on both seeds. So the K-1 thought carries the winner on most graphs (it restores about half to three quarters of the flipped ones and, in check 3, separates the candidates on 0.88 to 0.92 of graphs) but not reliably enough to be substituted across graphs; only thought K is. The random donor's K-1 thought breaks (e 0.90 and 0.82). Conclusion. Three additions to experiment 6's picture, replicated on both seeds. First, the decoy's incoming edges are part of the search: reading them lowers the decoy, but this is a secondary route that shows only when the thought's match to the path has been removed. Second, the fallback is always a candidate-level choice, never an escape, even with both candidates' edges hidden. Third, the carried winner is complete in thought K and largely but not reliably present in thought K-1; it is not present in K-2. The label cue does not enter: accuracy differs by 1 to 3 points across candidate id order and an ids-only probe reaches AUC 0.64 against the thought's 0.99 at step 3.

Amendment (2026-10-05): the last sentence above ("The label cue does not enter") is too strong. Under the both-candidates mask alone, over baseline-correct graphs, "target id below the decoy's" predicts a wrong answer with AUC 0.63, 0.71, 0.64, 0.59 on seeds 0 to 3 (gaps_reanalysis.json, key experiment10_zero_cost). With the evidence hidden, the id order goes with the model's choice; see experiment 10.

Earlier notes for this slot (checkpoints along training via train.py --save-every; three- and four-layer models) remain optional extras.

Experiment 7: the same measurements on fine-tuned GPT-2 (written 2026-09-16, before the pilot)

Design. Everything so far is on the two-layer model trained from scratch. Rizvi-Martel, Dilgren and Wiegreffe, and Aswal (lit/NOTES.md) report that COCONUT fine-tuned from pretrained GPT-2 barely uses its latents on ProsQA: removing them costs at most about one point, and causal tracing puts the effect at prompt positions, near zero at the thought positions. Four of our measurements run on that model, unchanged in design and cutoffs, on the same held-out graphs: the heads measurement (all twelve layers), the query-key removal with its calibration mask, the same-answer-donor carry-over cell, and the per-step winner probe. The question is whether the circuit mapped on the from-scratch model exists there at all. No new cell design.

Models. No official checkpoint exists (the authors declined twice, facebookresearch/coconut issues 3 and 37); two follow-up groups released theirs. gpt2_dilgren is connordilgren/gpt2-prosqa-coconut, file checkpoint_40 (Dilgren and Wiegreffe, arXiv 2604.04902; sha256 231dbdee73cd4654...), trained with the official repository at its current commit 27273cb8 and a configuration identical to the official args/prosqa_coconut.yaml: GPT-2 small, one thought per step, five epochs per stage, six latent stages, fifty epochs, learning rate 1e-4, effective batch 128. gpt2_aswal is darpanaswal/coconut-gpt2-prosqa, file checkpoint_best (Aswal et al., arXiv 2606.12689; sha256 6b607f88b167fe4b...). Both are raw state dicts of the official Coconut wrapper (150 keys, three tied copies of the 50260 x 768 embedding), loaded strictly with the tied-weight checks (src/phoenix/gpt2.py); files in ckpts/<run>/best.pt with ckpts/<run>/SOURCE.json, not committed. The literature's numbers for this model: Dilgren and Wiegreffe 98.0 on the original test set and no change with the latents removed; Aswal 98.0 with thoughts, 97.8 removed. Both are third-party checkpoints, so the regime check below is also the gate on each.

Format the model saw (official dataset.py). The question text, a newline, <|start-latent|>, six <|latent|> (always six at evaluation, padded to the last stage), <|end-latent|>, then "### Sally is a sterpus." and eos; greedy decoding, exact match on the text after the last "#". The vendor graphs carry the original natural-language question and answer: the vendor sets are the original ProsQA splits restricted to three- and four-step problems (14,785 / 419). A renderer that writes "{Name} is a {y}." for a capitalised source (a person; several per graph) and "Every {x} is a {y}." otherwise, joined by single spaces, then " Is {Root} a {c1} or {c2}?" in the displayed order, reproduces every stored question byte for byte (15,204 graphs; tests/test_gpt2_plumbing.py), once the sentence order is read off the question (the stored edge list is in another order). Held-out check (2026-09-16, results/gpt2_dilgren/evaluation.json, held_out_check): all 110 recipients and all 419 vendor test graphs are in the original test set and none is in the original training set.

Sets and measures for this model. Recipients as always (test 0-99 at n=100, 400-409 for the pilot), rendered in the model's format. The baseline serialization is the original sentence order (what the model sees at test); the reserialized control and the reorder counterfactual are pinned sentence permutations (the usual seeds). Donors are training graphs in their stored order; the same-answer donor is matched by target name, decoy name and solution length (one exists for every recipient); the random donor by solution length. K = 6 for every prompt; L = solution length (3 or 4). "Intermediates" = passes 0..4, "all" = 0..5, thought K = pass 5. The removal acts at passes 0..L-2, the passes with an answer-path edge to read; passes L-1..4 have none and are left alone (forced by the model's fixed K). Readout: the prompt ends with the common token prefix of the two answer strings, which on every recipient is the four-token frame "### Root is a"; T is read from the two next tokens (the names' first tokens), e = 1 - p_target - p_decoy, and the node set is the graph's concept-name tokens under that prefix (38 names, 32 distinct first tokens; a third name colliding with a candidate's token is counted per graph). GPT-2 parks attention on position 0, which is the first token of sentence 0: position 0 is excluded from slot 0 in every slot quantity (mass, keys, masks) and never masked, and each head's mass on it is reported as sink mass. Query classes added to the heads measurement: latent0..latent5 and search_latent (latents 0..L-2).

The route and the removal. From the run's own heads file (the pilot's for the pilot, the n=100 file for n=100) a head is in the latents route if its mean edge-slot mass at the search latents is at least 0.5 (experiment 5's cutoff, protecting "this head reads edges"); if no head reaches it, 0.2 (experiment 2's classification cutoff); if none reaches that, the eight heads with the most mass, labelled exploration. The rung is recorded in the results file. For each (layer, head) in the route the direction is built as in experiment 3: that layer's query matrix applied to the head's attention-weighted key over the path edge's tokens, scaled by the layer-norm gain and centered. The span of all of them is removed from the recycled thought at each pass 0..L-2 (norm preserved); the matched random control removes the same number of random orthonormal directions. For layers above the first this is the same first-order construction the two-layer cell made through layer 1, weaker the deeper the layer, so the recorded attention drop is the gate, and the calibration mask (the same heads masked at all six latent queries onto the path edges' tokens, and onto a count-matched off-path set) carries the exact route-blocking claim. Carry-over: the same-answer donor's and the random donor's thought K at pass 5, alone and after the removal, with the fallback line on removal-flipped graphs as in experiment 6.

Winner probe, defined once for both models (src/phoenix/winner_probe.py; check 3 of experiment 6 was computed ad hoc, so its numbers are recomputed by the same script and the table points to files). On the last 500 training graphs, per pass k: the fraction of graphs whose thought scores the target above the decoy in the input-embedding basis, the probe basis and the Jacobian basis (refit per GPT-2 model: the probe basis on train[:2500] with holdout train[-500:], smaller than the from-scratch fit and recorded; the Jacobian basis on train[500:2500], per token; src/phoenix/gpt2_bases.py), and a learned linear winner probe (multinomial logistic regression from the thought to the target's node token, fit on train[500:2500], L2 1e-3, LBFGS), scored as the AUC over the held-out candidate instances (target 1, decoy 0, score the candidate's logit; pairs whose candidates share a token are dropped and counted). The ids-only control stays from-scratch only: names carry no depth cue.

Regime check, first. Accuracy as run.py scores it (greedy, exact match after the last "#", with the key/value cache) on the original test set (500) and the vendor test set (419), under three prompts: six latents; the markers with no latent between them (the stage-0 format); no markers. Then, on the vendor test set at the readout position, two-candidate accuracy with the model's own thoughts and with every recycled thought replaced by the zero vector, and the change in T under zeroing, as a row against the from-scratch necessity cell zero/all (paper: -42.9 median). Gate: six-latent exact match near 97-98 on the 500. Hand check on three original test items (2026-09-16): the model writes "### Sally is a sterpus." with six latents and, with the markers and no latents, still goes straight to the answer without step sentences.

Predictions, two lines, written before the pilot:

The literature's "thoughts unused" claim: accuracy with the latents removed within about one point of the full model, and zeroing them costs little; no head set at the latents reaches the edge-reading cutoff, or if one does, removing its geometry and masking its route leave T unchanged with no flips beyond the same-answer reference; neither the same-answer donor's nor the random donor's final thought moves the answer (nothing is carried in thought K); the winner is as separable from the thought at pass 1 as at pass 6 (decided from the prompt before the latents run).
Our from-scratch account, if the circuit were present: removing or zeroing the latents costs tens of points; a head set at the latents follows edge content (content near 0.9, position near 0); the removal empties its attention onto the path and flips the answer only at the fallback rate; the same-answer donor's thought K restores the removal-flipped graphs and the random donor's destroys the answer (e near 1); the winner separates late, from pass L-1, not at pass 1.

Cutoffs, all named: flips and escapes as everywhere; the route rungs above; "weakened" for the fallback line as in experiment 6 (p_decoy down by at least 0.25 on removal-flipped graphs).

Second model. After the pilot passes on gpt2_dilgren, gpt2_aswal gets the same regime check, thought cache, bases, winner probe, and the n=100 heads and cells directly (no pilot: the design is frozen), as a second column of the table. Its route is picked by the same rungs from its own heads file.

Regime check outcome (gpt2_dilgren, 2026-09-16, results/gpt2_dilgren/evaluation.json; exact match as run.py scores it, 95 percent intervals over items):

Original test set (500): six latents 98.0 [96.6, 99.2]; the markers with no latent 98.0 [96.6, 99.2]; no markers 95.4 [93.6, 97.2]. Dilgren and Wiegreffe report 98.0 for this checkpoint; the paper's COCONUT number is 97.0. The model is in the literature's regime: removing the latents costs nothing with the markers kept and 2.6 points with the markers gone.
Vendor test set (419, our held-out graphs): six latents 98.1 [96.7, 99.3]; markers only 98.1 [96.7, 99.3]; no markers 95.7 [93.8, 97.6].
What "no latents" produced: with the markers the model goes straight to the answer (8.2 tokens, step sentences on 0 percent); without the markers it sometimes writes a fragment first (8.8 tokens, step sentences on 11 percent). So "unchanged without latents" here means the answer is produced directly from the prompt, not through textual chain of thought.
Two-candidate readout on the vendor test set (teacher-forced, at the first name token after "### Root is a"; the frame is four tokens, five on the 43 graphs whose root is Fae or Wren; on 3 graphs the two candidates share their first token and the readout is at the second; on 107 graphs a third name shares a candidate's first token, so that token's mass is not the candidate's alone): accuracy 98.1 with the model's own thoughts, 98.3 with every recycled thought replaced by the zero vector; median change in T under zeroing +0.0 [+0.0, +0.0], mean -0.0 [-0.2, +0.1], flips 0 percent, median e 0.000 (own thoughts 0.000). From-scratch zeroing at every step (results/seed0/necessity_n100.json, results/seed1/necessity_n100.json, cell zero/all): median change in T -25.4 and -19.1, flips 34 and 39 percent. The thoughts of the fine-tuned model are not necessary for its answer; the from-scratch model's are.
Held-out check: 110 of 110 recipients are in the original test set, 0 in the original training set; the whole vendor test set is inside the original test set. Gate passed (six-latent exact match on the 500 at the literature's number).

Regime check outcome, second model (gpt2_aswal, 2026-09-16, results/gpt2_aswal/evaluation.json): original test set six latents 98.0 [96.6, 99.2], markers only 98.2 [97.0, 99.2], no markers 97.8 [96.4, 99.0] (Aswal et al. report 98.0 with thoughts and 97.8 removed); vendor test set 97.9 [96.4, 99.0], 98.1 [96.9, 99.3], 97.6 [96.2, 98.8]. Step sentences on 0 percent of the markers-only outputs and 0 percent without markers. Two-candidate readout on the vendor test set: 97.9 with the model's own thoughts, 97.9 with every thought zeroed; median change in T +0.0 [+0.0, +0.0], mean -0.1, flips 0 percent, median e 0.000. Gate passed; the same regime as the first model.

Pilot outcome, heads (seed gpt2_dilgren, test graphs 400-409, results/gpt2_dilgren/heads_pilot.json; run before the regime check had finished, since it decides nothing about the model's validity):

The pre-registered route rule saturates instead of emptying: 84 of the 144 heads put at least 0.5 of their attention on edge sentences at the search latents (112 at 0.2). In a natural-language prompt the sentences are nearly the whole prompt, as the edge slots were in the symbol prompt, so mass alone says little; on the from-scratch model it identified the route only because mass and content coincided there (every layer-2 head had content 0.85 to 0.94).
None of these heads follows the edges when the sentences move. Among the fifteen heads with the most edge mass (layers 2 and 3, mass 0.97 to 1.00) the content scores are -0.08 to +0.07 and the position scores 0.43 to 0.99; the highest content score of any edge-reading head is 0.22 (layer 9 head 8, position 0.21). Per layer, the mean edge mass at the search latents runs 0.91 and 0.97 in layers 2 and 3, 0.5 to 0.6 in layers 4, 5, 8 and 9, and 0.15 to 0.46 elsewhere. Sink mass (position 0) is small at the latents in layers 1 to 5 (0.00 to 0.08) and 0.16 to 0.20 in layers 6 to 8 and 12.
Read against the predictions: the literature's line ("no head set follows edge content at the latents") holds on ten graphs; the from-scratch line (content near 0.9) does not.

What the pilot changed (heads). One addition, before the pilot cells ran. The route rule as written picks 84 position-following heads, and the removal's directions (a head's query matrix applied to an edge's key) only mean something for heads whose attention tracks edge content. The pre-registered rule stays the primary route (cells keep the from-scratch names), and a second, content-filtered route runs as exploration with cells prefixed "content/": heads that read edges (mass at least 0.5) and follow them by content (content score at least 0.5; the cutoff protects "this head follows the edge when it moves"); if none qualifies, the eight most content-following edge readers, labelled top8_content_exploration. Both routes get the removal, the random control, the calibration mask and the plus-removal carry-over cells; the standing controls and the alone cells are shared. Head sets, rungs and scores are recorded in the cells file.

Pilot outcome, cells (gpt2_dilgren, test graphs 400-409, results/gpt2_dilgren/cells_pilot.json; run after the regime check passed):

Baseline: T = 100.0 and e = 0.000 on all ten graphs; the readout is saturated, so a cell can only show a drop.
Routes: primary rung edge_head_cut_0.5 with 84 heads; content route rung content_cut_0.5 with 2 heads (L4H4, L4H5; mass 0.82, 0.51, content 0.74, 0.55, position 0.04, 0.26), whose attention onto the answer-path edge is 0.02 at the last search pass (they follow content, but not the path).
Every cell is exactly zero on every graph: median and mean change in T 0.0, no flips, no escapes, e 0.000 throughout, for the reserialized control, the random donor at the intermediates, the same-answer donor at the intermediates and at all K, the removal and its random control on both routes, the calibration masks (masked attention verified at 0.0) on both routes, and the final-thought carry-over from either donor alone and after the removal. Self-transplant exactly zero.
The removal does act on the route's attention, partially: the primary route's attention onto the answer-path edge at the last search pass falls from 2.32 to 1.72 (mean per-step drop 0.68; the random 84-direction control 2.32 to 2.33); the route's total edge mass 63.9 to 61.1. On the from-scratch model the same construction emptied the attention (1.79 to 0.31). The removed component is about 0.3 of the thought's norm per pass (84 of 768 directions), and the answer does not move; nor does it move under the matched random removal.
Read against the predictions: the literature's line holds on every cell of the pilot; the from-scratch line holds on none.

What the pilot changed (cells): nothing in the design. The n=100 runs the same two routes, each picked from the n=100 heads file by the recorded rungs.

n=100 outcome, gpt2_dilgren (test graphs 0-99; results/gpt2_dilgren/heads_n100.json, results/gpt2_dilgren/cells_n100.json, results/gpt2_dilgren/winner_probe.json, results/gpt2_dilgren/probe_basis_report.json):

Heads. 81 of 144 heads put at least 0.5 of their attention on edge sentences at the search latents (117 at 0.2). Among them the content score has median 0.08 and maximum 0.71; the position score has median 0.59. The three most content-following edge readers are L4H4 (content 0.71, mass 0.82, position 0.12), L4H5 (content 0.45, mass 0.52, position 0.30), L8H6 (content 0.40, mass 0.89, position 0.15). Layers 2 and 3 carry the most edge mass at the latents (0.90 and 0.97) with content +0.05 and +0.03 and position 0.89 and 0.83; content rises to +0.26 to +0.26 in layers 6 to 8 as position falls. Sink mass at the latents is 0.00 to 0.07 in layers 1 to 5 and 0.14 to 0.22 in layers 6 to 8 and 12. No head follows edge content the way every from-scratch layer-2 head does (0.85 to 0.94, four seeds).
Routes. Primary (edge_head_cut_0.5): 81 heads. Content route (content_cut_0.5): L4H4 alone, whose attention onto the answer-path edge is 0.01 at the last search pass.
Baseline: median T 100.0, e median 0.0001 (maximum 0.010); 2 of 100 graphs are answered wrongly at baseline (T below 50).
Every cell is zero on 100 graphs, both routes: medians 0.0 with intervals [0.0, 0.0], means within 0.1, flips 0.00, escapes 0.00, e 0.000: the removal and its random control, the calibration masks (masked attention verified 0.0 on every graph), the final-thought carry-over from either donor alone and after the removal, the random donor at the intermediates, and the same-answer donor at the intermediates and at all K. Self-transplant exactly zero. The only motion anywhere is the reserialized control's mean -1.7 [-5.6, +1.9] with flips 0.02: a second sentence order changes the answer on two graphs; nothing done to the thoughts does.
The removal's action on attention: the primary route's attention onto the answer-path edge at the last search pass 1.62 to 1.12 (mean per-step drop 0.69; total edge mass 63.7 to 56.3); random 81 directions 1.62 to 1.63. About 0.32 of the thought's norm is removed per pass. From scratch the same construction emptied the attention (1.79 to 0.31); here it takes a third off, and the answer does not depend on it.
Winner probe (last 500 training graphs, 32 classes, 3 pairs dropped for a shared first token): pass 1: input-embedding 0.73, probe basis 0.19, learned probe AUC 0.998 (n 497) pass 2: input-embedding 0.68, probe basis 0.30, learned probe AUC 0.980 (n 497) pass 3: input-embedding 0.66, probe basis 0.34, learned probe AUC 0.973 (n 497) pass 4: input-embedding 0.65, probe basis 0.35, learned probe AUC 0.973 (n 497) pass 5: input-embedding 0.65, probe basis 0.35, learned probe AUC 0.973 (n 497) pass 6: input-embedding 0.65, probe basis 0.35, learned probe AUC 0.972 (n 497) The winner is decodable from the first thought, the hidden state at <|start-latent|> after the question, at least as well as from the last; from scratch the same probe is at 0.67 and 0.71 at pass 1 and reaches 0.999 only at the final pass. The input-embedding readout is flat around 0.65 to 0.73 across passes (from scratch 0.46 rising to 0.99). Jacobian rows (basis fit on train[500:2500], 40 node tokens, all six passes; results/gpt2_dilgren/jlens_basis_report.json): separation of the target from the decoy in the Jacobian basis pass 1: 0.52, pass 2: 0.54, pass 3: 0.56, pass 4: 0.54, pass 5: 0.55, pass 6: 0.55 (from scratch, seed 0: 0.51 at pass 1 rising to 1.00 at pass 4). The Jacobian directions are diffuse and unrelated to the input embeddings: median concentration 0.33 to 0.33 across passes and median absolute cosine to the embedding 0.05 to 0.06 (from scratch, seed 0: concentration 0.49 to 0.93, cosine 0.41 to 0.78). The gradient of the answer logit with respect to a thought points nowhere consistent across graphs here; the from-scratch Jacobians were both concentrated and aligned with the identity readout.
Probe basis refit (step-1 thoughts, depth-1 membership, 38 names, fit train[500:2500], holdout the last 500): median holdout AUC 0.69, minimum 0.57. From scratch the same probe reached 0.998: the fine-tuned model's first thought barely encodes the frontier.
Read against the predictions: the literature's line holds on all four cells at n=100; the from-scratch line on none. The circuit mapped on the from-scratch model is absent from this model.

Winner probe on the from-scratch seeds, recomputed by the shared script (results/seed0/winner_probe.json, results/seed1/winner_probe.json; seed 0 / seed 1; last 500 training graphs; step 4 exists on the 252 four-step graphs): step 1: input-embedding 0.46/0.42, probe basis 0.44/0.41, Jacobian basis 0.51/0.45, learned probe AUC 0.67/0.71 (n 500) step 2: input-embedding 0.57/0.56, probe basis 0.33/0.34, Jacobian basis 0.55/0.49, learned probe AUC 0.81/0.78 (n 500) step 3: input-embedding 0.89/0.85, probe basis 0.52/0.61, Jacobian basis 0.94/0.88, learned probe AUC 0.94/0.91 (n 500) step 4: input-embedding 1.00/0.99, probe basis 0.56/0.81, Jacobian basis 1.00/1.00, learned probe AUC 1.00/1.00 (n 252) The pattern recorded in check 3 of experiment 6 holds (near chance at steps 1 and 2, high at step 3, complete at step 4); the earlier ad hoc numbers (0.92/0.88 at step 3, paired probe 0.99/0.98) came from a differently built probe and are superseded by these file-backed ones. The probe basis, fit on step-1 thoughts for depth-1 membership, does not separate the winner at later steps (0.33 to 0.61 on seed 0), which is expected of a basis fit for a different step and label; the Jacobian basis tracks the input-embedding readout.

n=100 outcome, gpt2_aswal (the second fine-tuned model; no pilot, the design frozen; results/gpt2_aswal/heads_n100.json.gz, results/gpt2_aswal/cells_n100.json, results/gpt2_aswal/winner_probe.json, results/gpt2_aswal/probe_basis_report.json):

Heads: 62 of 144 edge readers at the search latents; content median 0.06, maximum 0.66 (L2H7, mass 0.88, position 0.16); position median 0.72. Layers 2 and 3 again carry the most edge mass (0.89, 0.97) with content +0.08 and +0.02. Routes: primary 62 heads (edge_head_cut_0.5); content route L2H7 alone (content_cut_0.5), whose attention onto the answer-path edge is 0.01.
Baseline median T 100.0 (1 graph wrong), e median 0.0000.
Every cell zero on 100 graphs, both routes (medians 0.0, means within 0.1, flips 0.00, escapes 0.00), as on the first model. The reserialized control moves the mean -3.6 [-7.7, +0.4] with flips 0.06: a second sentence order changes the answer on six graphs; nothing done to the thoughts does.
Removal on attention: 1.24 to 0.98 onto the answer-path edge at the last search pass (62 directions; random 1.24 to 1.26); masked attention 0.0.
Winner probe: learned AUC 0.999 at pass 1, 0.989 at pass 3, 0.988 at pass 6; input-embedding separation 0.74 at pass 1, 0.66 at pass 6. Probe basis median holdout AUC 0.64. Jacobian basis (train[500:2500], 40 node tokens; results/gpt2_aswal/jlens_basis_report.json): separation pass 1: 0.58, pass 2: 0.56, pass 3: 0.55, pass 4: 0.54, pass 5: 0.54, pass 6: 0.54; concentration 0.27 to 0.29, cosine to the embedding 0.04 to 0.05. Chance level and diffuse, as on the first model.

Comparison table (built by scripts/exp7_table.py from the results files; seeds 2 and 3 ran only the headline cells, so their winner-probe and attention rows are empty; the carry-over row counts removal-flipped graphs that have a same-answer donor, which is why seeds 2 and 3 show 32 and 24 rather than the 33 and 25 of the replication entry):

Cell	seed0	seed1	seed2	seed3	gpt2_dilgren	gpt2_aswal
Accuracy, the model's own prompt	94.5%	95.5%	95.5%	95.2%	98.0% (original 500); 98.1% (vendor 419)	98.0% (original 500); 97.9% (vendor 419)
Accuracy with the latents removed	not applicable (K latents are the search)	not applicable (K latents are the search)	not applicable (K latents are the search)	not applicable (K latents are the search)	98.0% markers only, 95.4% no markers	98.2% markers only, 97.8% no markers
Every recycled thought zeroed	median dT -25.4, flips 34%	median dT -19.1, flips 39%	median dT -24.8, flips 40%	median dT -28.8, flips 43%	median dT +0.0, flips 0% (accuracy 98.1% to 98.3%)	median dT +0.0, flips 0% (accuracy 97.9% to 97.9%)
Heads reading edges at the latents (mass at least 0.5)	12 of 16 (all 8 of layer 2)	13 of 16 (all 8 of layer 2)	12 of 16 (all 8 of layer 2)	13 of 16 (all 8 of layer 2)	81 of 144	62 of 144
Do they follow edge content? (scores)	layer 2: content 0.85 to 0.90, position -0.02 to +0.02	layer 2: content 0.91 to 0.94, position -0.00 to +0.03	layer 2: content 0.90 to 0.92, position -0.00 to +0.02	layer 2: content 0.88 to 0.92, position -0.00 to +0.02	content median 0.08, max 0.71 (L4H4); position median 0.59	content median 0.06, max 0.66 (L2H7); position median 0.72
Removal: attention onto the answer-path edge, before to after	1.79 to 0.13 (8 directions)	2.54 to 0.13 (8 directions)	not run on this seed	not run on this seed	1.62 to 1.12 (81 directions)	1.24 to 0.98 (62 directions)
Removal: the answer	flips 19%, beyond reference +0.00 [-0.08, +0.08]	flips 22%, beyond reference -0.03 [-0.15, +0.09]	flips 33%, beyond reference +0.07 [-0.06, +0.20]	flips 25%, beyond reference -0.03 [-0.16, +0.09]	flips 0%, beyond reference +0.00 [+0.00, +0.00]	flips 0%, beyond reference +0.00 [+0.00, +0.00]
Calibration mask on the path edges	flips 15% (removal 19%)	flips 20% (removal 22%)	flips 33% (removal 33%)	flips 24% (removal 25%)	flips 0% (removal 0%)	flips 0% (removal 0%)
Final-thought carry-over	19 of 19 removal-flipped graphs restored; random donor escape 75%	22 of 22 removal-flipped graphs restored; random donor escape 75%	32 of 32 removal-flipped graphs restored; random donor escape 76%	23 of 24 removal-flipped graphs restored; random donor escape 74%	0 removal-flipped graphs (nothing to restore); random donor escape 0%	0 removal-flipped graphs (nothing to restore); random donor escape 0%
Content-filtered route (GPT-2 only, exploration)	-	-	-	-	content_cut_0.5: L4H4; every cell zero	content_cut_0.5: L2H7; every cell zero
Learned winner probe, held-out AUC	AUC 0.67 at pass 1, 0.94 at pass 3, 0.999 at pass 4	AUC 0.71 at pass 1, 0.91 at pass 3, 0.999 at pass 4	not run on this seed	not run on this seed	AUC 0.998 at pass 1, 0.973 at pass 3, 0.972 at pass 6	AUC 0.999 at pass 1, 0.989 at pass 3, 0.988 at pass 6
Winner separation, input-embedding basis	0.46 at pass 1, 1.00 at pass 4	0.42 at pass 1, 0.99 at pass 4	not run on this seed	not run on this seed	0.73 at pass 1, 0.65 at pass 6	0.74 at pass 1, 0.66 at pass 6
Winner separation, Jacobian basis	0.51 at pass 1, 1.00 at pass 4	0.45 at pass 1, 1.00 at pass 4	not run on this seed	not run on this seed	0.52 at pass 1, 0.55 at pass 6	0.58 at pass 1, 0.54 at pass 6
Frontier probe basis, median holdout AUC	0.997	0.992	-	-	0.691	0.643

Conclusion, experiment 7 (both fine-tuned models, n=100, every cell including the Jacobian-basis rows). The circuit mapped on the from-scratch model does not exist in COCONUT fine-tuned from GPT-2. Its latents are not necessary: removing them, zeroing every recycled thought, replacing them with a random graph's thoughts at every intermediate pass, or with a donor's final thought, leaves the answer where it was on every one of 100 held-out graphs, on both checkpoints. No head at the latents follows edge content (best content score 0.71 and 0.66 for one head each, medians 0.06 to 0.08; every from-scratch layer-2 head scores 0.85 to 0.94), so there is no query-key match of thought to edges to remove; the construction that emptied the from-scratch attention takes a third off here and the answer does not depend on it, nor on the route when it is masked outright. And the winner is decodable from the very first thought (AUC 0.998 and 0.999 at pass 1), before any latent pass has run, where the from-scratch probe is at 0.67 to 0.71 and reaches 0.999 only at the last pass. The literature's claim holds on every cell; the from-scratch account holds on none. What we mapped is a property of the from-scratch regime, where the thoughts are the search; in the fine-tuned regime the answer is read off the prompt and the thoughts are carried along unused. Both prediction lines were written before the pilot; no cell was changed after seeing a result other than the addition of the content-filtered route, which is as empty of effect as the primary one.

Experiment 8: the removal's flipped graphs, the carried winner, and a held-out check (written 2026-10-01, before any run)

Why. A review of the n=100 files, recomputed per graph in results/gaps_reanalysis.json (scripts/reanalysis_gaps.py), found that the query-key removal is not a null in the sense experiments 5 and 6 use. It flips 19, 22, 33 and 25 of 100 graphs on seeds 0 to 3 (almost always to the decoy: 18, 22, 30 and 20 of those flips put over half the mass on the decoy; 1, 0, 3 and 3 escape) while the matched random directions flip none. Its flipped graphs overlap the same-answer donor's only at chance on seeds 1 to 3 (6 of 22, 7 of 32, 7 of 24 shared; 5.6, 8.2, 6.6 expected) and above chance on seed 0 (10 of 19; 3.7 expected). The same test graphs flip across seeds (12 graphs on at least 3 of 4 seeds; 4.8 expected if independent). Baseline p_decoy predicts the removal's flips (AUC 0.72, 0.75, 0.72, 0.71) and not the donor's (0.54, 0.57, 0.49, 0.45); the target's parent's out-degree does too, weakly (0.67, 0.59, 0.63, 0.68). The donor's intermediate thoughts carry the shared winner (its K-1 thought alone flips 13 to 28 percent of removal-correct graphs, recovery_n100.json), so its rate is not a "broken search" reference. Four other points from the same review: the winner probe pools K=3 and K=4 at each absolute pass and is scored on training graphs; no mask cell blocks queries at edge-token positions, and target-token queries keep only 0.05 to 0.11 of their attention inside their own slot (heads_n100.json); no cell masks every route onto all path edges at once (layer 2 at the latents was covered by the removal, which takes one edge per step); 9, 4, 6 and 7 recipients are wrong at baseline and cannot flip. Four parts, all analysis on the existing checkpoints; the user launches every run. The reading rules are those of experiments 5 and 6 (flip, escape, "collapses"); "baseline-correct" means baseline T above 50 (protects "the baseline answer was the target").

8a. Winner margins per graph (src/phoenix/winner_margin.py; files results/<run>/winner_margin_<mode>.json; needs the per-K probe weights that winner_probe.py now saves). Per recipient, the removal and its random control are rebuilt as in experiment 5; three runs (unedited, removal, random) record every recycled thought as the model reads it. Two readouts give a winner margin, target minus decoy, on thought K-1 and thought K: the learned per-K probe (fit on training graphs) and the input-embedding readout. Read first, before any margin: the share of each readout direction inside the removed eight-direction span at each pass (squared projection; the random span's share, about 0.01, is the reference).

Share cutoff: a probe-direction share at or below 0.05 at pass K-2 protects "the removal does not aim at the winner readout". If the share is above 0.2 on the flipped graphs, the removal takes the winner out directly and the damage line below is the reading, whatever the margins say.
Two-route line (the match covers where the carried winner is weak): the unedited K-1 margin predicts the removal's flips with an AUC above baseline p_decoy's (0.72 to 0.75 on these graphs); the removal lowers the K-1 margin no more than the random control does (paired difference, interval covering zero); the K margin under the removal predicts the answer under the removal (AUC above 0.95).
Damage line (the removal corrupts the carried winner): the K-1 margin falls under the removal on the flipped graphs and not under the random control (interval of the difference below zero on the flipped graphs, covering zero on the unflipped).
Parent out-degree is reported again here (found on these graphs, AUC 0.59 to 0.68); its test is 8b.

8b. Held-out check (masking.py --mode heldout: test graphs 100-399, 148 with K=3 and 152 with K=4, untouched until now; cells reserialized, self-transplant, the removal, its random control and the layer-2 mask on path and off-path slots; no donors; file results/<run>/masking_heldout.json, read by scripts/reanalysis_gaps.py). Only the two predictors found on test graphs 0-99 are tested there, baseline p_decoy and the parent out-degree; nothing else is read from these graphs. Predictions, per seed: removal flip rate between 0.15 and 0.40, random control at most 0.02; AUC of baseline p_decoy for the removal's flip among baseline-correct graphs between 0.65 and 0.80; AUC of the parent out-degree between 0.55 and 0.75 with the interval above 0.5; the layer-2 mask's flipped set shared with the removal's above chance (hypergeometric P below 0.01). A predictor whose interval covers 0.5 on two or more seeds is dropped from the paper.

8c. Every mask at once, and the isolation mask (masking.py, pilot then n100 on all four seeds; the experiment 5 files are rerun and keep their names). New cells: all_masks/<path|offpath>/alone (the layer-1 latent heads, the answer heads and all eight layer-2 heads at the latents masked together on the slots, no removal); isolation/<path|offpath>/alone and /plus_removal (every query after a slot and outside it blocked from the slot's three tokens, both layers, all heads; the separator's copy of its own edge is kept). Root edges (depth 0 to 1) stay out of the path set, as in experiment 5; recorded here as a decision. Predictions:

Carried-winner line: all_masks flips at the layer-2 mask's level (paired flips beyond l2_latents_mask/path/alone with the interval covering zero); isolation adds nothing beyond all_masks (same); isolation plus removal equals the removal; off-path isolation flips at most 0.05.
Leak line (masked edge content reaches the latents through other edge tokens' layer-1 residuals): isolation on the path slots flips beyond the layer-2 mask with the interval above zero, and off-path isolation does not.

8d. Winner probe per K and on the test split (winner_probe.py: the pooled numbers are unchanged and a by_K block is added; --eval test writes winner_probe_test.json for the 419 test graphs; fit_probes.py --score-test writes probe_basis_test_report.json; fit_jlens.py --only-K 3 and 4 write jlens_basis_K<K>.pt, which the per-K Jacobian readout uses when present). Predictions: on the test split the learned probe's AUC at thought K-1 (K=4 at step 3; K=3 at step 2) is between 0.70 and 0.90 for both K, the final step above 0.99 for both K, and every test number within 0.05 of its training-holdout twin; the frontier probe's median test AUC is above 0.99. An AUC below 0.6 at K-1 means the K-1 donor thought's partial restoration in experiment 6b carried the frontier, not the winner. The sentence "largely present in K-1" is not used until these numbers exist.

Reporting (numbers only): scripts/exp7_table.py gains three rows (recipients wrong at baseline; the removal's flips over baseline-correct graphs; where the flips go), and common.SPLIT_COVARIATES gains parent_out_degree, decoy_in_degree and target_in_degree. The zero-cost split of the last-hop rewrite by the decoy's remaining in-degree (seeds 0 and 1, counterfactuals_n100.json) is in gaps_reanalysis.json and is a side result, not a headline.

Order, each after the user says so (all --device cpu):

.venv/bin/python scripts/reanalysis_gaps.py
# seeds 2 and 3 first need the two bases:
.venv/bin/python src/phoenix/fit_probes.py --run-name seed2 --device cpu
.venv/bin/python src/phoenix/fit_jlens.py --run-name seed2 --device cpu
# then per seed (seed0 shown):
.venv/bin/python src/phoenix/fit_jlens.py --run-name seed0 --device cpu --only-K 3
.venv/bin/python src/phoenix/fit_jlens.py --run-name seed0 --device cpu --only-K 4
.venv/bin/python src/phoenix/winner_probe.py --model symbol --run-name seed0 --device cpu
.venv/bin/python src/phoenix/winner_probe.py --model symbol --run-name seed0 --device cpu --eval test
.venv/bin/python src/phoenix/fit_probes.py --run-name seed0 --device cpu --score-test
.venv/bin/python src/phoenix/winner_margin.py --run-name seed0 --device cpu --mode pilot
.venv/bin/python src/phoenix/masking.py --run-name seed0 --device cpu --mode pilot
# pilots read against the predictions above, then on every seed:
.venv/bin/python src/phoenix/winner_margin.py --run-name seed0 --device cpu --mode n100
.venv/bin/python src/phoenix/masking.py --run-name seed0 --device cpu --mode n100
.venv/bin/python src/phoenix/masking.py --run-name seed0 --device cpu --mode heldout
.venv/bin/python scripts/reanalysis_gaps.py

Pilot outcome (seed 0, test graphs 400-409, 2026-10-01; files results/seed0/winner_margin_pilot_v1_before_orth.json, results/seed0/masking_pilot.json, results/seed0/winner_probe.json, results/seed0/winner_probe_test.json, results/seed0/probe_basis_test_report.json):

8a. The probe's winner direction holds a median 0.088 [0.061, 0.124] of its norm in the removed span at pass K-2 (0.04 to 0.17 per graph), nine times the random span's 0.008; the input-embedding direction 0.059. That is above the 0.05 line and below the 0.2 damage trigger. The removal lowers the probe's K-1 margin by 2.3 [0.6, 4.1] logits and the K margin by 5.9 [3.6, 7.8], on flipped and unflipped graphs alike (K-1: -2.9 and -2.0; K: -6.9 and -5.4); the random control lowers neither (+0.4, +0.2). The removal flips 3 of 10, the random control none. The K margin under the removal predicts the answer under the removal (AUC 0.86 [0.54, 1.00], n=10). Neither prediction line as written: the margin falls, so this is not a pure attention edit, but it falls on every graph, not on the flipped ones in particular. At n=10 nothing separates the predictors (AUC 0.67 to 0.86 with intervals from 0.2 to 1).
8c. All masks at once flips the layer-2 mask's three graphs and no other (per-graph dT within 7 points); isolation flips two of them (graph 406: -6.8 under isolation, -62.2 under the layer-2 mask), off-path isolation none, isolation plus removal the removal's three. No leak signature.
8d. Training holdout, per K: learned-probe AUC at K-1 is 0.842 (K=3) and 0.892 (K=4), final 0.999 for both; input-embedding separation at K-1 0.66 and 0.79 (the pooled step-3 number, 0.89, mixed K=3 finals in). Test split (419): K-1 AUC 0.719 and 0.765, final 0.976 and 0.955; separation at K-1 0.61 and 0.69. The K-1 prediction holds (0.70 to 0.90, barely for K=3); "final above 0.99" and "test within 0.05 of training" fail on the test split. The frontier probe's median test AUC is 0.995 (minimum 0.75); that prediction holds. What the pilot changed, with predictions written before the rerun:
8a gains one cell, removal_orth: the same eight-direction span at every pass with its component along the winner readout direction (the probe's target-minus-decoy direction at that pass; the input-embedding one when the probe has no row for a candidate) taken out before the removal, so the edit leaves that direction alone. Every condition now also records the layer-2 attention from each search query onto its answer-path edge, and the orthogonal removal's attention drop as a fraction of the removal's is the gate. Match line (the flips come from losing the match): the orthogonal removal keeps at least 0.8 of the removal's attention drop at the last step, flips the same graphs (paired difference of flip indicators against the removal with the interval covering zero, on at least 0.8 of the removal's flipped graphs) and changes the K-1 probe margin no more than the random control. Damage line (the flips come from the tenth of the winner direction the span carries): the attention drop is kept and the flips fall to the random control's level (paired difference interval below zero) with the K margin change shrinking toward zero. The share cutoff of 0.05 stays as the line that says the span is not independent of the winner readout; the orthogonal cell is what decides whether that matters.
8d: because the model answers about 5 percent of test graphs wrongly and a thought on such a graph carries the decoy, every readout is also reported over the graphs the model answers correctly (auc_correct_only, frac_target_higher_correct_only, from one extra forward per graph). The prediction "final above 0.99" is re-read on that subset; the "within 0.05" prediction stands as written and is already failed at K-1 on seed 0. Nothing else in 8b, 8c or 8d changes. Pilot outcome, 8a rerun with removal_orth (seed 0, test graphs 400-409; results/seed0/winner_margin_pilot.json): the orthogonal removal keeps the removal's attention drop at the last search step in full (2.09 to 0.15, against 0.14 under the removal; ratio of drops 1.00 [0.98, 1.00]), flips the same three graphs (paired difference 0.00 [0.00, 0.00]), and changes the K-1 probe margin by +0.1 [-1.5, +1.7], as the random control does (+0.4), where the removal changed it by -2.3. The winner direction's share in the orthogonal span is zero on every graph. Thought K still loses 5.4 [3.1, 7.3] logits of probe margin under the orthogonal removal (5.9 under the removal), so that loss follows from losing the match at the last step, not from the span clipping the winner direction. The match line, on all three counts; design frozen, n=100 next. n=100 outcome, 8a, seeds 0 and 1 (results/seed0/winner_margin_n100.json, results/seed1/winner_margin_n100.json; 9 and 4 recipients wrong at baseline, none skipped):
The removal flips 0.19 and 0.22 (0.21 and 0.23 over baseline-correct graphs); the orthogonal removal 0.16 and 0.20, the same graphs (16 of 19 and 20 of 22 shared; paired difference -0.03 [-0.07, 0.00] and -0.02 [-0.05, 0.00]); the random control 0.00. The orthogonal removal keeps the removal's attention drop at the last search step (1.79 to 0.13 against 0.16, and 2.54 to 0.13 against 0.15; ratio 0.99 [0.99, 1.00] and 1.00). Match line on both seeds: at most two or three of the flips come from the span clipping the winner direction.
The probe's winner direction holds a median 0.087 and 0.062 of its norm in the removed span (random span 0.009 and 0.008), the same on flipped and unflipped graphs (0.081 vs 0.088; 0.068 vs 0.067); the share does not predict the flip (AUC 0.46 and 0.49).
K-1 probe margin: the removal lowers it by 2.4 [1.8, 2.9] and 2.6 [1.9, 3.3]; the orthogonal removal by +0.3 [-0.3, +0.8] and -0.1 [-0.9, +0.7], within the random control's (-0.3 and -0.6). Thought K loses 4.9 [4.2, 5.6] and 6.2 [5.6, 6.9] logits under the removal and 4.3 and 5.8 under the orthogonal removal, more on the flipped graphs (7.7 and 8.7) than on the unflipped (4.3 and 5.5); the random control changes neither.
Which graphs flip: the unedited K-1 probe margin is -1.7 [-3.2, -0.3] and -0.9 [-2.8, +1.1] on the graphs the removal flips and +4.2 [2.9, 5.3] and +4.8 [3.5, 6.0] on the rest. It predicts the flip with AUC 0.80 [0.71, 0.88] and 0.80 [0.69, 0.91] (0.85 and 0.83 over baseline-correct graphs), above baseline p_decoy's 0.72 and 0.75 (0.81 and 0.79) on both seeds, with overlapping intervals; the parent out-degree gives 0.67 and 0.59; the input-embedding readout at K-1 0.60 and 0.68. Under the removal, the thought-K margin predicts the model's answer with AUC 0.95 [0.91, 0.98] and 0.94 [0.89, 0.98] (predicted above 0.95; at the line). Reading. The removal takes away the thought's match to the answer-path edges and nothing else that matters: a removal that spares the winner direction empties the attention just as completely and flips the same graphs. Without the match, thought K is produced with 5 to 6 logits less winner margin on every graph; the answer survives where thought K-1 already carried the winner and flips where it did not. The two routes of experiment 6's account are both real and are used on different graphs: the last-step match carries the graphs whose K-1 thought has not settled the winner, and the carried winner covers the loss of the match on the rest. Seeds 2 and 3 (results/seed{2,3}/winner_margin_n100.json; 6 and 7 wrong at baseline, none skipped):
The removal flips 0.33 and 0.25; the orthogonal removal 0.32 and 0.22, the same graphs (32 of 33 and 22 of 25 shared; paired -0.01 [-0.03, 0.00] and -0.03 [-0.07, 0.00]); random 0.00. Attention drop kept in full (ratio 1.00 on both). Probe direction's share in the removed span 0.041 and 0.067 (random 0.011 and 0.009), equal on flipped and unflipped graphs.
K-1 probe margin: removal -2.1 [-2.8, -1.4] and -3.7 [-4.4, -3.0]; orthogonal removal -0.3 [-0.9, +0.3] and -0.7 [-1.3, -0.03]; random -0.6 and -0.2. On seed 3 the orthogonal removal lowers the K-1 margin by about half a logit more than the random control, small beside the removal's 3.7. Thought K loses 6.6 and 6.2 logits under the removal (6.2 and 5.6 orthogonal), 9.0 and 8.6 on flipped graphs.
Which graphs flip: the unedited K-1 margin is lower on flipped graphs (1.9 vs 5.4; 1.3 vs 3.2) but predicts the flip with AUC 0.66 [0.56, 0.77] and 0.57 [0.44, 0.69] only, below baseline p_decoy's 0.72 and 0.71 (0.79 and 0.79 over baseline-correct graphs). The thought-K margin under the removal predicts the answer under the removal with AUC 0.93 and 0.95. Four-seed reading of 8a. The match line holds on all four seeds: sparing the winner direction leaves the attention drop and the flipped graphs unchanged, so the removal's flips come from losing the match, not from clipping the carried winner. The two-route reading by graph holds in direction on all four (lower K-1 margin on the flipped graphs) but the K-1 margin beats baseline p_decoy as a predictor on seeds 0 and 1 only; on seeds 2 and 3 it is weaker. The per-graph claim the paper can make is baseline p_decoy (four seeds, and held out on 300 more graphs each), not the K-1 margin.

n=100 outcome, 8c, four seeds (results/seed{0,1,2,3}/masking_n100.json; the experiment 5 files are kept as masking_n100_v1_exp5.json, and every shared cell reproduces them to the last digit on all four seeds):

Every mask at once (all_masks/path/alone) flips 0.15, 0.19, 0.31, 0.24 against the layer-2 mask's 0.15, 0.20, 0.33, 0.24 (paired difference +0.00 [0.00, 0.00], -0.01 [-0.03, 0.00], -0.02 [-0.05, 0.00], +0.00 [-0.03, +0.03]; 15, 19, 31, 23 graphs shared). Off-path 0.00, 0.00, 0.00, 0.01. Blocking layer 1 at the latents and the answer position's heads on top of layer 2 adds no flip on any seed.
Isolation (isolation/path/alone) flips 0.14, 0.21, 0.28, 0.27: against the layer-2 mask -0.01 [-0.03, 0.00], +0.01 [-0.02, +0.05], -0.05 [-0.09, -0.01], +0.03 [-0.01, +0.08]; against every mask at once -0.01, +0.02 [0.00, +0.05], -0.03, +0.03 [-0.01, +0.08]. Off-path isolation 0.00, 0.00, 0.01, 0.01. Isolation plus removal against the removal: +0.03 [-0.03, +0.09], +0.01 [-0.06, +0.08], +0.05 [-0.03, +0.13], +0.04 [-0.04, +0.11]. Reading: the carried-winner line on all four seeds. No interval for isolation beyond the layer-2 mask lies above zero (seed 2's lies below it: isolating the path slots from every reader flips fewer graphs than masking the latents' layer-2 reads alone); the leak line's signature is absent. Blocking every reader of the path edges, in both layers, leaves the answer exactly where blocking the layer-2 latent reads leaves it.

n=100 outcome, 8b, the held-out check, four seeds (results/seed{0,1,2,3}/masking_heldout.json, 300 test graphs 100-399 each; 12, 13, 11, 8 wrong at baseline; read in results/gaps_reanalysis.json, key heldout):

Removal flips 0.22 [0.18, 0.27], 0.29 [0.24, 0.34], 0.39 [0.33, 0.44], 0.27 [0.22, 0.32] (predicted 0.15 to 0.40; seed 2 at the edge); random directions 0.00 on all four; the layer-2 mask 0.15, 0.23, 0.30, 0.22, sharing 32 of 66, 49 of 86, 73 of 116, 49 of 81 removal-flipped graphs (10.1, 19.8, 34.4, 17.6 expected if independent; P below 1e-13 on each).
Baseline p_decoy predicts the removal's flip among baseline-correct graphs with AUC 0.81 [0.75, 0.86], 0.80 [0.74, 0.85], 0.68 [0.62, 0.74], 0.78 [0.72, 0.84] (predicted 0.65 to 0.80; seed 0 a point above the upper line; every interval above 0.5). Holds.
Parent out-degree: 0.54 [0.46, 0.62], 0.53 [0.45, 0.60], 0.46 [0.40, 0.53], 0.50 [0.43, 0.58]. Every interval covers 0.5; by the rule written above the predictor is dropped. The 0.59 to 0.68 on test graphs 0-99 was noise. Nothing else was read from graphs 100-399.

Seeds 2 and 3, 8d (results/seed{2,3}/winner_probe.json, winner_probe_test.json, probe_basis_test_report.json): training holdout at K-1 0.742 and 0.748 (K=3), 0.864 and 0.821 (K=4), final 0.997 to 1.000. Test split at K-1 0.703 and 0.640 (K=3), 0.801 and 0.755 (K=4); over correctly answered graphs 0.724 and 0.654, 0.816 and 0.775. Final step on test 0.966 and 0.987 (K=3), 0.980 and 0.985 (K=4); over correct graphs 0.981 to 0.997. Frontier probe test median 0.990 and 0.996. Jacobian basis fit per K, at K-1: training 0.52 and 0.61 (K=3), 0.65 and 0.66 (K=4); test 0.46 and 0.54, 0.62 and 0.59; final step 1.00 and 0.99 to 1.00 on training, 0.95 to 0.99 on test. The K-1 prediction (0.70 to 0.90 on test) holds for K=4 on both and for K=3 on seed 2 at the line; it fails for K=3 on seed 3 (0.64). Across four seeds, K-1 on test: K=3 0.64 to 0.72, K=4 0.76 to 0.80.

n=100 outcome, 8d, seeds 0 and 1 (results/seed{0,1}/winner_probe.json, winner_probe_test.json, probe_basis_test_report.json; per-K Jacobian bases jlens_basis_K{3,4}.pt on both seeds; seeds 2 and 3 pending):

Learned probe, per K, training holdout (last 500 training graphs, 499 to 500 answered correctly): at thought K-1, 0.842 and 0.778 (K=3, step 2), 0.892 and 0.876 (K=4, step 3); final step 0.999 and 0.998 for both K. The pooled step-3 number (0.941, 0.910) mixed K=3 finals in.
Learned probe, test split (419 graphs; 398 and 402 answered correctly): at K-1, 0.719 and 0.707 (K=3), 0.765 and 0.781 (K=4); over the correctly answered graphs 0.739 and 0.726, 0.809 and 0.811. Final step 0.976 and 0.973 (K=3), 0.955 and 0.976 (K=4); over correct graphs 0.986 and 0.990, 0.984 and 0.996.
Input-embedding separation at K-1: training 0.66 and 0.62 (K=3), 0.79 and 0.70 (K=4); test 0.61 and 0.57, 0.69 and 0.65. Jacobian basis fit per K, at K-1: training 0.67 and 0.51 (K=3), 0.77 and 0.67 (K=4); test 0.56 and 0.54, 0.70 and 0.66; final step 1.00 on training, 0.93 to 0.96 on test (0.97 to 1.00 over correct graphs).
Frontier probe on the test split: median AUC 0.995 and 0.992, minimum 0.75 and 0.79. Against the predictions: K-1 between 0.70 and 0.90 on the test split holds on both seeds for both K (K=3 by a hair). "Final above 0.99" fails on the test split (0.955 to 0.976) and holds over correctly answered graphs on seed 1 only (seed 0: 0.984 and 0.986). "Test within 0.05 of training" fails at K-1 on both seeds (gaps of 0.10 to 0.12, also over correct graphs), so the probe fit on training graphs partly fits those graphs, not only the winner. The frontier probe's line holds. For the paper: the winner is complete in thought K (0.998 to 0.999 held-out training, 0.98 to 1.00 over correctly answered test graphs) and partial in thought K-1 (0.71 to 0.81 on test graphs, 0.78 to 0.89 on training graphs); the old sentence "largely present in K-1" overstated it.

Experiment 9: the restore test, and four zero-cost reads (written 2026-10-02, before the restore runs)

Zero-cost reads on existing files (numbers in results/gaps_reanalysis.json, keys substitutes_p_decoy_auc, last_step_vs_every_step, decoy_edges_among_removal_correct; item 5 in winner_margin_n100.json):

Baseline p_decoy as a predictor of flips under the substitutes (necessity_n100.json, baseline-correct graphs): AUC 0.46 to 0.72 across zero, noise, mean thought, random donor, thought removed and the same-answer donor, at all passes and at the intermediates; one or two cells per seed have an interval above 0.5, never the same cell on every seed. The removal: 0.79 to 0.85 on all four. The link is specific to the removal (and the layer-2 mask).
Last-step-only vs every-step removal (counterfactuals_n100.json, seeds 0 and 1): 17 vs 19 and 15 vs 22 flips, 12 and 11 shared (3.2 and 3.3 expected). Most of the effect is the last step; the earlier steps add 7 and 11 graphs.
Graphs the removal leaves correct (baseline-correct), the decoy's incoming edges also masked at every latent and the answer position (recovery_n100.json): 30 of 72, 20 of 74, 4 of 61, 27 of 68 flip (0.42, 0.27, 0.07, 0.40); matched control edges 1, 1, 0, 0; the same mask at the final positions only 1, 4, 0, 3.
Thought K against the model's own answer, all 100 graphs including the baseline-wrong ones: the probe margin's sign agrees with the answer on 98, 97, 95, 98 of 99 (AUC 0.98 to 1.00) unedited and on 86, 85, 83, 87 of 99 (AUC 0.93 to 0.95) under the removal. The same readout on the whole test split is auc_own_answer in winner_probe_test.json.

9a. The restore test (src/phoenix/restore.py; files results/<run>/restore_<mode>.json). The every-step removal is run, and at every latent query part of layer 2's attention output is put back from the unedited run: the per-head sum over chosen keys of weight x value, before c_proj. Edge tokens precede the latents, so their keys and values are the same in both runs and putting the part back undoes exactly the change in how much those keys were read. Restores: the path-edge tokens (masking.py's path set: every shortest-path edge from depth 1 on); the matched off-path edge tokens (control); every edge token; the whole output (edges, earlier thoughts, question tokens). The path restore on the unedited run must be exactly zero.

Measure: the rescue share, the fraction of the removal's flipped graphs that a restore brings back to unflipped, with a bootstrap interval, per seed and pooled over the four seeds (about 100 flipped graphs). Also the new flips a restore creates among graphs the removal left alone. No pass/fail cutoff.

Reading ranges for the path restore, applied to the interval and not the point alone:

90 percent or more: the lost read of the path edges is the whole story.
50 to 90 percent: it is most of the story; the rest is located by the other restores.
10 to 50 percent: it is part of the story.
Below 10 percent: the flips do not come through the read of the path edges. How the ranges were chosen: round fractions fixed before any restore ran; the 90 and 10 percent lines leave room for the two or three graphs per seed (about 10 percent of the flipped ones) that experiment 8's orthogonal removal already showed flip through another route.

The other restores (paired over the removal's flipped graphs): edges_all minus path is what undoing the attention pushed onto other edges adds; all minus edges_all is what undoing changed reads of earlier thoughts and question tokens adds; offpath is the control (expected rescue near zero); whatever all leaves unrescued acts outside layer 2's attention at the latents (the thought's own residual, layer 1, the MLPs). Prediction from experiment 8 (the orthogonal removal flips the same graphs, so the winner is not the target): the path restore lands in the top or second range, all rescues at least as much as path, and offpath rescues under 10 percent.

Pilot outcome (seed 0, test graphs 400-409, results/seed0/restore_pilot.json): the removal flips 3 graphs (400, 401, 403; dT -74.9, -96.5, -81.8); the path restore brings all three back (-0.1, -6.6, -0.6), as do every-edge and whole-output restores. The off-path restore leaves 400 and 401 flipped and brings 403 back (-7.1): one rescue in three, so the control is not clean on that graph. The path restore on the unedited run is zero on all ten. Graph 408 (baseline T 81) moves by -27.5 under the every-edge restore and -23.2 under the whole-output restore without flipping. What the pilot changed: nothing. n=100 outcome, four seeds (results/seed{0,1,2,3}/restore_n100.json; pooled in results/gaps_reanalysis.json, key experiment9_restore_pooled). The removal cell equals masking_n100.json on 100 of 100 graphs on every seed; the path restore on the unedited run is exactly zero everywhere.

Restore	seed 0 (19 flipped)	seed 1 (22)	seed 2 (33)	seed 3 (25)	pooled (99)
path	0.89 [0.74, 1.00]	0.95 [0.86, 1.00]	0.85 [0.73, 0.97]	0.80 [0.64, 0.96]	0.87 [0.80, 0.93]
off-path (control)	0.16 [0.00, 0.32]	0.14 [0.00, 0.27]	0.06 [0.00, 0.15]	0.16 [0.04, 0.32]	0.12 [0.06, 0.19]
every edge	1.00	1.00	1.00	1.00	1.00
whole output	1.00	1.00	1.00	1.00	1.00

Rescue share = fraction of the removal's flipped graphs not flipped under the restore. Flipped graphs surviving the path restore: 2, 1, 5, 5 (13 pooled). New flips among graphs the removal left unflipped: none under the path, every-edge and whole-output restores; 2, 3, 1, 1 under the off-path restore. Paired, pooled: every edge minus path +0.13 [0.07, 0.20]; whole output minus every edge 0.00; path minus off-path +0.75 [0.66, 0.84].

Reading, by the ranges written above. The path restore's pooled interval, 0.80 to 0.93, lies in the 50-90 range and crosses the 90 line: the lost read of the path edges is most of the story and may be all but a few graphs. The rest is located exactly: putting back every edge's part rescues every flipped graph on every seed, and putting back the reads of earlier thoughts and question tokens adds nothing. So the 13 graphs the path restore leaves flipped flip because the removal pushed layer-2 attention at the latents onto other edges, and nothing outside layer 2's reads of the edges is needed to explain any flip. The off-path restore (0.12) is not a clean zero for the same reason: on some graphs the matched off-path edges are where the pushed attention went. The prediction (path in the top or second range, whole output at least as high, off-path under 10 percent) holds except the off-path line, which is 0.06 to 0.16 per seed. For the paper: the removal's effect is the change in which edges layer 2 reads at the latents, about seven parts losing the path read to one part reading the wrong edges instead.

Item 5 on the whole test split (winner_probe_test.json, auc_own_answer, 419 graphs including the 17 to 21 the model answers wrongly): at the final thought the probe margin predicts the model's own answer with AUC 0.978, 0.990, 0.974, 0.945 (K=3) and 0.984, 0.988, 0.994, 0.993 (K=4); its sign agrees with the answer on 192 to 198 of 201 and 199 to 206 of 209 graphs.

Experiment 10: how much of the isolation leftover each source carries (written 2026-10-05, before any run)

Why. Under 8c's isolation mask (every query outside a path slot blocked from it, both layers, all heads) 0.82, 0.77, 0.69, 0.71 of baseline-correct graphs stay right on seeds 0 to 3 (masking_n100.json; recomputed in gaps_reanalysis.json, key experiment10_zero_cost). Two known contributors, neither measured as a share of that leftover: the decoy's incoming edges (6b; seeds 0, 1, 3, not 2) and the label-id order. Under 6b's both-candidates mask alone, over baseline-correct graphs, "target id below the decoy's" predicts a wrong answer with AUC 0.63, 0.71, 0.64, 0.59 (same key). This contradicts 6b's sentence "the label cue does not enter the trained model's choice"; see the amendment line at the end of the 6b block. No training; the existing checkpoints only.

What the task structure forces. On test graphs 0-99 and 400-409 both candidates are sinks (no outgoing edge on any graph), the decoy is never reachable, and every edge precedes the question line. So a candidate's label appears only in its incoming-edge slots and its question token. With the path slots and every incoming edge of both candidates isolated, no position after them can read any token that pairs a candidate with another node: the graph no longer says which candidate is reachable. What remains is the question tokens' ids, the ids of the nodes still visible, and the candidates' order in the question line. The forced prediction is that survival falls to what those non-graph cues allow. The id cue cannot be manipulated on-manifold: under this mask an edge-list-only label swap touches only unreadable slots, and renaming a candidate to a fresh id breaks the breadth-first numbering (v1 label-swap problem). So the numbering read below is correlational.

On 78 of 100 graphs the target's only incoming edges are its depth-(K-1) parent edges, already in the path set; there cell 1 adds only the decoy's incoming edges, which gives the decoy's edges their own share without a second cell. The other 22 mix the decoy's edges and the target's extra incoming edges and are read pooled only.

Cells (masking.py --part leftover, file masking_leftover_<mode>.json):

isolation/path/alone: 8c's cell, rerun; must equal masking_n100.json on every graph (pairs the leftover with cell 1).
isolation/path_cand/alone (cell 1): path slots plus every incoming edge of both candidates (all depths), isolated in both layers, all heads.
isolation/offpath_matched/alone (control): the same number of slots per graph, the most attended edges (summed over the latent queries) with a reachable source at depth 1 or more, outside the path, not into a candidate; a shortfall is recorded per graph.
invariance: cell 1 with the two candidate labels swapped inside cell 1's slots only. Answer logits must equal cell 1's (max difference below 1e-6, the self-transplant tolerance): checks that the mask is complete, on which the forced prediction rests. Also a random-weights test.
Standing: reserialized, self_transplant. No plus-removal variant, no donors. Every wrong answer is classified as a switch (T at or below 50, e below 0.5) or an escape (e at or above 0.5). Cells 1 and 8c report T as in 6b (median, mean, 20-point histogram, share between 20 and 80). The split covariates gain target_id_below_decoy; target_first is read too.

Measure, over baseline-correct graphs (baseline T above 50), paired bootstrap over graphs, per seed and pooled over the four seeds (reanalysis_gaps.py, key experiment10):

s_iso, s_1: share right (T above 50) under 8c isolation and under cell 1.
Leftover = s_iso - 0.5. The 0.5 is what a model with no evidence that still names a candidate gets (6b: escape never above 0.03).
Edge share E = (s_iso - s_1) / leftover.
Within cell 1, s_above and s_below are the shares right when the target's id is above / below the decoy's; s_bal is their average.
Numbering share N = (s_1 - s_bal) / leftover; unexplained U = (s_bal - 0.5) / leftover. E + N + U = 1.
N has a ceiling: it equals (share of target-above graphs - 0.5) x (s_above
- s_below), and the target is above on 51-53 of 91-96 baseline-correct graphs, so even pure use of the id order gives N of at most about 0.2-0.3. The numbering verdict therefore rests on U and on G.
G: share of cell-1 answers naming the higher-id candidate (1 = follows the id order, 0.5 = does not use it). Correlational.
E on the 78-graph subset (decoy's edges only) and on the 22, pooled.

Reading ranges, applied to the interval, crossings reported as crossings:

E at or above 0.9: the edges explain it. 0.5-0.9: most of it. 0.1-0.5: part of it. Below 0.1: none of it.
The numbering cue goes with it: E below 0.9, U's interval reaches 0.1 or below, and G's interval is above 0.5.
Unexplained: U's interval above 0.1.
How the lines were chosen: the leftover is 19 to 30 graphs per seed, so 0.1 of it is 2 to 3 graphs, the size of the escapes under 8c isolation (1 to 3 per seed); 0.5 separates most from part, as in experiment 9.
Guard on the 0.5 reference: if escapes under cell 1 exceed 0.05 of baseline-correct graphs, the decomposition is not read and the cell is reported as an escape result.

Predictions:

Forced by the structure: s_1 between 0.45 and 0.65 on every seed; E at or above 0.5 on every seed, pooled interval reaching 0.9; U at or below 0.1; escapes at most 0.05; T bimodal as under 6b's both-candidates mask.
From the files: G between 0.6 and 0.8; the gap s_above - s_below larger than under 8c isolation (0.14, 0.13, 0.35, 0.38 there; same key).
Decoy's edges on the 78-graph subset: a positive share on seeds 0, 1, 3, smaller on seed 2.
Controls: off-path matched isolation flips at most 0.05; self-transplant exactly zero; invariance passes on every graph.

What each outcome means:

E at or above 0.9, G near 0.5: the candidates' incoming edges carry the whole leftover; the id order does nothing once they are hidden.
E 0.5-0.9, U at or below 0.1, G above 0.5: the edges carry most of it and the rest goes with the id order (a correlation only).
U above 0.1 with invariance passing: the model picks the target with no readable edge; the remaining candidate is a finer id cue (candidate id against the visible nodes' ids; the 6b ids-only probe, AUC 0.64, never saved, would need a refit).
Invariance fails: the mask leaks; nothing else is read until fixed.
E below 0.1: the leftover never ran through the edges; it is id-driven already under 8c.

Order: zero-cost reads to gaps_reanalysis.json; code and tests; pilot on seed 0, test graphs 400-409 (3 target-below graphs: checks plumbing, the exact rerun match, invariance and the rough survival level, not the id split); n=100 on all four seeds; reanalysis.

.venv/bin/python src/phoenix/masking.py --run-name seed0 --device cpu --mode pilot --part leftover
.venv/bin/python src/phoenix/masking.py --run-name seed0 --device cpu --mode n100 --part leftover   # and seed1-3
.venv/bin/python scripts/reanalysis_gaps.py

Pilot outcome (seed 0, test graphs 400-409, results/seed0/masking_leftover_pilot.json): the 8c cell reproduces masking_pilot.json to the last digit; invariance holds exactly (maximum logit difference 0.0 on all ten); the matched off-path isolation flips none; no escapes. Right under 8c isolation 8 of 10, under cell 1 7 of 10 (403 and 408 lost; 401 gained: wrong under path isolation, T 1.5, right once its single decoy edge is also hidden, T 81.6, so hiding the decoy's edge can also help the target when the path is gone). Too few graphs for the id split (3 target-below). What the pilot changed: nothing.

n=100 outcome, four seeds (results/seed{0,1,2,3}/masking_leftover_n100.json; every number below is in results/gaps_reanalysis.json, key experiment10, per seed and pooled; baseline-correct graphs 91, 96, 94, 93, pooled 374). Checks: the 8c cell equals masking_n100.json on every graph (maximum difference 0.0); invariance holds exactly on all 400 graphs (maximum logit difference 0.0); the matched off-path control keeps 1.00, 0.98, 0.99, 0.99 right; it is short of slots on 9 graphs per seed (1 to 5 slots).

	seed 0	seed 1	seed 2	seed 3	pooled
right, 8c isolation (s_iso)	0.82 [0.75, 0.90]	0.77 [0.69, 0.85]	0.69 [0.60, 0.79]	0.71 [0.61, 0.80]	0.75 [0.70, 0.79]
right, cell 1 (s_1)	0.52 [0.42, 0.62]	0.60 [0.50, 0.70]	0.56 [0.47, 0.66]	0.55 [0.44, 0.65]	0.56 [0.51, 0.61]
lost / gained by cell 1	31 / 3	23 / 7	15 / 3	20 / 5	89 / 18
edge share E	0.95 [0.65, 1.29]	0.62 [0.26, 1.00]	0.67 [0.27, 1.29]	0.77 [0.34, 1.37]	0.76 [0.59, 0.96]
numbering share N	0.06 [-0.03, 0.19]	0.06 [-0.06, 0.20]	0.08 [-0.10, 0.26]	0.04 [-0.07, 0.19]	0.06 [0.01, 0.12]
unexplained U	-0.01 [-0.35, 0.30]	0.33 [-0.03, 0.67]	0.25 [-0.35, 0.65]	0.19 [-0.41, 0.62]	0.18 [-0.01, 0.35]
G (answer = higher id)	0.64 [0.54, 0.74]	0.66 [0.56, 0.75]	0.65 [0.55, 0.74]	0.59 [0.49, 0.69]	0.63 [0.59, 0.68]
cell 1 right, target id above / below	0.63 / 0.36	0.74 / 0.44	0.69 / 0.40	0.63 / 0.45	0.67 / 0.42
id-order gap, 8c / cell 1	0.14 / 0.28	0.13 / 0.29	0.35 / 0.29	0.38 / 0.18	0.25 / 0.26
escapes among cell-1 wrong	1 of 44	1 of 38	1 of 41	4 of 42	7 of 165

Decoy's edges alone (graphs whose target has no incoming edge outside its parents; 73, 78, 75, 74 baseline-correct): cell 1 lowers survival by 0.33 [0.22, 0.45], 0.19 [0.08, 0.31], 0.11 [0.03, 0.20], 0.16 [0.05, 0.27]; E 0.98, 0.65, 0.55, 0.71; pooled 0.75 [0.57, 0.95] (300 graphs). The other 74 pooled graphs: E 0.86 [0.25, 1.71].
T under cell 1, baseline-correct, pooled: 105 graphs below 20, 156 above 80, 0.30 [0.25, 0.34] between 20 and 80 (8c isolation: 0.11 to 0.25 per seed over all graphs). Mostly committed, less so than under 8c.
Question order: right 0.59 [0.52, 0.65] with the target first, 0.52 [0.45, 0.60] second; intervals overlap. Against the predictions: s_1 in 0.45-0.65 on every seed, holds. E at or above 0.5 on every seed (point), holds, with wide per-seed intervals; the pooled interval reaches 0.9 (0.96), holds. Escapes at most 0.05, holds (seed 3 at 0.04). G in 0.6-0.8, holds pooled and on seeds 0-2; seed 3 is 0.59 with the interval touching 0.5. U at or below 0.1: not decided, pooled 0.18 with an interval from -0.01 to 0.35. The id-order gap larger under cell 1 than under 8c: holds on seeds 0 and 1, fails on seeds 2 and 3 and pooled (0.25 vs 0.26). Decoy's edges positive on seeds 0, 1, 3 and smaller on seed 2: positive on all four, seed 2 the smallest (0.11, interval above zero), unlike 6b where seed 2 showed nothing under the removal. Controls hold. Reading, by the ranges written above. The candidates' incoming edges carry most of the leftover: E 0.76 [0.59, 0.96] pooled, in the 0.5-0.9 range and crossing the 0.9 line. Most of that is the decoy's own incoming edges: on the 300 graphs where cell 1 adds only those, E is 0.75. E is a net figure: 89 graphs lost and 18 gained (gained graphs, like pilot graph 401, go wrong under path isolation and right once the decoy's edges are hidden too). With every candidate edge hidden, the answer goes with the id order (G 0.63 [0.59, 0.68]; 0.67 right when the target's id is the higher one, 0.42 when lower), a correlation only. Because the target's id is the higher one on only 208 of 374 graphs, that order can account for N = 0.06 of the leftover at most at this gap; what neither the edges nor the id order account for is U = 0.18 [-0.01, 0.35], which crosses the 0.1 line, so "unexplained" is neither shown nor excluded. Since invariance holds exactly, that remainder reads no edge; the only candidates left are the question tokens' ids against the visible nodes' ids (a finer id cue) or noise. The id-order gap is already 0.25 under 8c isolation and does not grow under cell 1: the id order goes with the choice once the path is hidden, before the candidate edges are. For the paper: with the path edges isolated, about three quarters of the graphs still right above chance are right because of the candidates' incoming edges, mainly the decoy's; with those hidden too, survival is 0.56, the level of a "higher id wins" rule (0.556 on these graphs), and the choice goes with the id order.

Experiment 11: the INLP (amnesic probing) arm on this repo's seeds (written 2026-10-06, before any run)

Why. The paper cites the v1 INLP arm (RRR repository, DECISIONS #014 and its addenda), which ran on the v1 checkpoint only; this repo never ran it. Same recipe, seeds 0 and 1, analysis only.

Fit (fit_inlp.py --k-max 40 --lbfgs-iters 50 --tag _full, the v1 "full" recipe): per node, the step-1 depth-1 probe (logistic, L2 1e-3, last 500 training graphs held out, at least 50 positives) is refit on the residual and its direction projected out until a fresh probe's held-out AUC is at or below 0.55 (protects "no linear signal left": about chance) or 40 directions are removed. Files results/<run>/inlp_basis_full.pt, inlp_basis_report_full.json.

Arm (inlp_arm.py, file results/<run>/inlp_arm_<mode>.json): the recipients and edit of this repo's subtraction cell (test graphs whose target has a unique depth-1 ancestor; that ancestor's identity removed from the step-1 thought, norm preserved), under four deletions: the probe direction (must reproduce baseline_subtraction_n100.json, subtract_answer/probe, to the last digit); the first INLP direction; the full INLP span; the same number of random orthonormal directions (rank-matched control, Elazar et al.). Standing controls reserialized and self-transplant.

What the structure forces. At step 1 the winner is not yet in the thought (learned probe AUC 0.67 and 0.71, winner_probe.json), and the step-1 thought is read by layer 2 to match the next edges, a match that experiment 3 showed can be removed outright with the answer kept at the fallback rate. So removing one node's identity from the first thought should not move the answer. Predictions: median k at or near the cap of 40 on most nodes (v1: 40) with the residual AUC after the last direction between 0.5 and 0.7 (v1: 0.50 to 0.67); full INLP removal median change in T within 1 point of zero and flips no more than the rank-matched control's plus 0.05 (v1: -0.0, T after 99.99, random 0.0); the first INLP direction equals the probe cell. If the INLP removal flips beyond the random control with the interval above zero, the identity in the first thought is causal after all and the paper's null was a basis problem.

Order: fit on seeds 0 and 1; pilot of the arm on seed 0, test graphs 400-409; n=100 on both seeds.

Fit outcome (results/seed{0,1}/inlp_basis_report_full.json, about 8.5 minutes per seed on CPU): 23 nodes each; median k 40 (the cap) on both seeds, 13 and 12 nodes at the cap. Pilot outcome (seed 0, test graphs 400-409, 9 with a unique ancestor; results/seed0/inlp_arm_pilot.json): the probe cell reproduces baseline_subtraction_pilot.json to 5e-5 in T (that file was made on MPS, this run on CPU; the baselines differ by the same amount). The probe and the first INLP direction do nothing, as in experiment 0. The full INLP span does: graphs 400, 402 and 403 fall by 80, 83 and 99 points (mass on the decoy 0.79, 0.42 with escape 0.49, 0.99), 405 and 406 by 3 and 2; the rank-matched random span moves nothing (at most 3 points, on a low-confidence graph). Against the prediction ("within 1 point of zero, flips at the random control's level"): fails on the pilot, 3 of 9 flips against 0. Unlike v1. What the pilot changed: one record added, no cell changed. A rank-matched random span removes far less of the thought than INLP directions, which follow the data's high-variance directions, so each row now records the share of the step-1 thought's squared norm inside the INLP span, the random span and the probe direction (removed_share). The reading of the n=100 INLP cell is made against that: if INLP flips beyond the random control, the question is whether it is the identity or the size of the removal, and the removed shares are the first evidence on it. n=100 outcome, seeds 0 and 1 (results/seed{0,1}/inlp_arm_n100.json; 68 graphs each with a unique ancestor; numbers in paper/numbers_table.md, macros inlp*):

Fit: median k 40 on both seeds; residual AUC of the last fresh probe 0.48 to 0.67 (seed 0) and 0.44 to 0.71 (seed 1) over the 23 nodes.
Probe direction and first INLP direction: median change -0.00, flips 0.03 and 0.04, as in experiment 0 (the probe cell equals baseline_subtraction_n100.json to 4e-4 in T).
Full INLP span: median change -0.06 [-0.55, -0.00] and -0.28 [-0.76, -0.06], median T after 99.4 on both; flips 0.15 [0.07, 0.24] and 0.19 [0.10, 0.29], to the decoy (mean p_decoy on flipped graphs 0.83 and 0.80), escape 0.01. Rank-matched random span: median 0.00, flips 0.00 on both seeds. INLP minus random, paired: +0.15 [+0.07, +0.24] and +0.19 [+0.10, +0.29]. Against the same-answer reference (0.19 and 0.28): -0.04 [-0.16, +0.06] and -0.09 [-0.22, +0.04].
Removed share of the step-1 thought's squared norm, median: INLP 0.19 and 0.20, random 0.05 and 0.05, probe 0.05 and 0.04; on the flipped graphs the INLP share is 0.24 and 0.25 against 0.19 on the rest. Against the predictions: median k at the cap, holds; residual AUC 0.5 to 0.7, holds on seed 0, seed 1 reaches 0.71; median change within 1 point, holds; flips no more than the random control plus 0.05, fails on both seeds. The line written for that case ("the identity in the first thought is causal after all and the paper's null was a basis problem") is met as written: INLP flips beyond its rank-matched control with the interval above zero. Two things qualify it. The flips stay at the fallback rate (no excess over the same-answer reference) and go to the decoy, the signature of a broken search, not of a redirected one. And the rank-matched control is not size-matched: the INLP span holds about 3.6 times the random span's share of the thought, and flipped graphs lost more. These cells do not separate "the node's identity is used" from "removing a fifth of the first thought breaks the search"; a random span matched in removed norm (from the data's high-variance directions) would. Not run. v1's INLP null (median -0.0, no flip reported) does not replicate as a flip count on these seeds; it does as a median.

Size-matched control (written 2026-10-06, before the run; user's request). Cell subtract_answer/inlp_random_sizematched, same graphs, same edit (pass 0, norm preserved), same rank k. Per graph, with t the step-1 thought and s the INLP span's share of its squared norm: one direction u = sqrt(s) t/|t| + sqrt(1-s) r, r a random unit vector orthogonal to t; the other k-1 directions random and orthogonal to both t and r. The span then holds exactly the share s of t's squared norm (recorded per graph as removed_share.random_sizematched and checked against s), so the edit moves the thought by the same angle as the INLP removal, in a random direction. The rank-matched control stays. Reading, written now:

Size line (the flips come from how much of the thought is removed): the size-matched control flips about as often as INLP, paired difference INLP minus size-matched with the interval covering zero.
Identity line (the flips come from what INLP removes): the size-matched control flips at the rank-matched control's level (near 0), and INLP minus size-matched has the interval above zero.
In between (both intervals exclude zero): part of each; reported as the two paired differences. Reported side by side for INLP, rank-matched and size-matched: flips, flips beyond the same-answer reference with intervals, escapes. The same columns are added for the path-deletion cells (experiment 12). Pilot on seed 0, test graphs 400-409 (checks the share match and that every other cell reproduces inlp_arm_pilot.json), then n=100 on seeds 0 and 1; the n=100 files are rewritten with every earlier cell unchanged. Pilot outcome (seed 0, test graphs 400-409, results/seed0/inlp_arm_pilot.json): every earlier cell reproduces the previous pilot file exactly (maximum difference in T 0.0); the size-matched span holds the INLP span's share to four decimals on all nine graphs (0.12 to 0.36; rank-matched 0.03 to 0.07). On the three graphs INLP flips (400, 402, 403: -80, -83, -99) the size-matched control moves T by 0.0 to -0.1. What the pilot changed: nothing. n=100 outcome, seeds 0 and 1 (results/seed{0,1}/inlp_arm_n100.json, 68 graphs each; every earlier cell reproduces the previous n=100 file exactly; the size-matched span's share equals the INLP span's to 4e-7 on every graph; macros amn*):

seed	cell	flips	beyond the same-answer reference	escapes
0	INLP span	0.15 [0.07, 0.24]	-0.04 [-0.16, +0.06]	0.01 [0.00, 0.04]
0	rank-matched random	0.00 [0.00, 0.00]	-0.19 [-0.30, -0.10]	0.00
0	size-matched random	0.00 [0.00, 0.00]	-0.19 [-0.30, -0.10]	0.00
1	INLP span	0.19 [0.10, 0.29]	-0.09 [-0.22, +0.04]	0.01 [0.00, 0.04]
1	rank-matched random	0.00 [0.00, 0.00]	-0.28 [-0.40, -0.18]	0.00
1	size-matched random	0.00 [0.00, 0.00]	-0.28 [-0.40, -0.18]	0.00

Same-answer reference 0.19 and 0.28. Paired, INLP minus size-matched: +0.15 [+0.07, +0.24] and +0.19 [+0.10, +0.29]; size-matched minus rank-matched 0.00 on both. Reading, by the lines written above: the identity line. Moving the step-1 thought by the same angle in a random direction changes nothing on any graph; moving it out of the answer-branch node's INLP span flips 15 to 19 percent, to the decoy. So the size of the removal is not the cause, and the v1 conclusion "the identity in the first thought is causally inert" does not hold once the whole linearly decodable identity subspace is removed. What it does when removed is break the search on some graphs, at the fallback rate (no excess over the same-answer reference on either seed), not redirect it.

Path deletion (experiment 12), the same columns (path_deletion_n100.json):

seed	cell	flips	beyond the same-answer reference	escapes
0	input embedding	0.06 [0.02, 0.11]	-0.13 [-0.21, -0.05]	0.01 [0.00, 0.03]
0	probe	0.04 [0.01, 0.08]	-0.15 [-0.23, -0.08]	0.01 [0.00, 0.03]
0	random, rank-matched	0.00	-0.19 [-0.28, -0.12]	0.00
1	input embedding	0.12 [0.06, 0.19]	-0.13 [-0.23, -0.04]	0.01 [0.00, 0.03]
1	probe	0.05 [0.01, 0.10]	-0.20 [-0.31, -0.11]	0.00
1	random, rank-matched	0.00	-0.26 [-0.35, -0.17]	0.00

Same-answer reference 0.19 and 0.26. The path-deletion control is rank-matched only; a size-matched one was not asked for here and was not run.

Experiment 12: whole answer-path deletion at every intermediate step (written 2026-10-06, before any run)

Why. The paper cites v1's cell perstep_sub_path_nolast (RRR, explore_perstep_*.json, v1 checkpoint, training graphs 0-99); this repo never ran it. Ported (path_deletion.py, file results/<run>/path_deletion_<mode>.json) on test graphs 0-99 like the rest of the paper's cells (user's decision, 2026-10-06), seeds 0 and 1.

Cell. At each intermediate pass k = 0..K-2 the thought loses the span of the identity directions of every node on a shortest path to the target at depth k+1 (ProsQA's neighbor_k, checked equal to the on-path nodes on all 356 steps of test graphs 0-99), norm preserved. Bases: input embeddings; the step-1 probe basis (nodes without a direction left out and counted). Control: as many random orthonormal directions at the same passes. Standing controls reserialized, self-transplant, same-answer donor, random donor.

What the structure forces. The answer is not read from the intermediate thoughts' identity directions: experiment 0's subtractions are null in both bases, and experiment 3 showed that even removing the thought's layer-2 match to the path at every step flips only at the fallback rate. Removing the identity span of all path nodes at every intermediate step is a larger edit than either, so a fallback-rate effect is possible; a redirection is not forced by either story. Predictions: median change in T within 1 point of zero in both bases (v1: -0.01 embedding, -0.0 probe); flips in each basis not above the same-answer reference by more than 0.05 (interval covering zero); the random control the same. A basis whose flips exceed the reference with the interval above zero would mean the identity of the path nodes is read at the intermediate steps after all.

Order: pilot on seed 0, test graphs 400-409; n=100 on seeds 0 and 1.

Pilot outcome (seed 0, test graphs 400-409, results/seed0/path_deletion_pilot.json): input-embedding basis, 2 of 10 flip (401: T 0.3; 402: T 1.6, both to the decoy), median -0.2; probe basis 0 flips (401 falls 31 points); random control nothing. The same-answer reference flips 2 of 10. Read at n=100 against that reference. What the pilot changed: nothing. n=100 outcome, seeds 0 and 1 (results/seed{0,1}/path_deletion_n100.json; macros dTpathNolast*, pathNolast*):

Input-embedding basis: median change -0.01 [-0.05, -0.00] and -0.06 [-0.35, -0.01]; flips 0.06 [0.02, 0.11] and 0.12 [0.06, 0.19], escape 0.01; against the same-answer reference (0.19 and 0.26) -0.13 [-0.21, -0.05] and -0.13 [-0.23, -0.04].
Probe basis: median -0.00 and -0.02; flips 0.04 and 0.05; against the reference -0.15 and -0.20, intervals below zero.
Random control: median 0.00, flips 0.00 on both seeds. Paired against it: embedding +0.06 [+0.02, +0.11] and +0.12 [+0.06, +0.19]; probe +0.04 [+0.01, +0.08] and +0.05 [+0.01, +0.10]. Against the predictions: median within 1 point of zero, holds in both bases on both seeds; flips not above the same-answer reference by more than 0.05, holds (below it, intervals under zero); the random control the same, fails by a few graphs (4 to 12 flips against none). Reading: deleting the identity span of every answer-path node at every intermediate step leaves the answer in place on 88 to 96 percent of graphs, below the fallback rate a same-answer donor's thoughts produce; v1's null (-0.01, -0.0) replicates on the median.

Next run

Device on this Mac: cpu (measured 2026-09-06: 25 ms per batch-one forward on CPU vs 178 ms on MPS, and MPS memory grew 750 MB in 60 forwards; the seed-1 chain on MPS was killed for memory at 14,000 of 14,785 probe graphs). Seed 0's experiment-0 files were produced on MPS before this was known; the two paths differ at rounding level only (tests/test_fast_equivalence.py).

.venv/bin/python src/phoenix/evaluate.py --run-name seed0 --device cpu --serialization-seed 0
.venv/bin/python src/phoenix/fit_probes.py --run-name seed0 --device cpu
.venv/bin/python src/phoenix/fit_jlens.py --run-name seed0 --device cpu
.venv/bin/python src/phoenix/baseline.py --run-name seed0 --device cpu --mode pilot

Experiment 7 (GPT-2), in order, all on --device cpu:

.venv/bin/python src/phoenix/gpt2_eval.py --run-name gpt2_dilgren --device cpu
.venv/bin/python src/phoenix/gpt2_cells.py --run-name gpt2_dilgren --device cpu --mode pilot --part heads
.venv/bin/python src/phoenix/gpt2_cells.py --run-name gpt2_dilgren --device cpu --mode pilot --part cells
.venv/bin/python src/phoenix/winner_probe.py --model gpt2 --run-name gpt2_dilgren --device cpu
.venv/bin/python src/phoenix/gpt2_bases.py --run-name gpt2_dilgren --device cpu --part probe
.venv/bin/python src/phoenix/gpt2_bases.py --run-name gpt2_dilgren --device cpu --part jlens
.venv/bin/python src/phoenix/winner_probe.py --model symbol --run-name seed0 --device cpu

then --mode n100 for heads and cells, then the same chain on gpt2_aswal.

Then, each after the user says so and after its predictions above are re-read: necessity.py, heads.py, counterfactuals.py, tracing.py, cache_patch.py, each with --mode pilot, then --mode n100 once on both seeds. Result files: results/<run>/<driver>_<mode>.json.

Code map

src/phoenix/prompts.py: explicit prompt renderer (byte-identical to the vendor builder), slot layout, counterfactual constructors (reorder, unreachable-only reorder, rename, decoy swap, rewrite at depth, label swap), per-graph covariates.
src/phoenix/attn_hooks.py: AttnHooks (record attention, overwrite cached keys/values by layer/head/position, mask query-key pairs; needs attn_eager=True on the forward) and ResidualHooks (record or overwrite the residual stream by level and position).
src/phoenix/measure.py: one forward on explicit token ids, answer split, thought capture and injection helpers.
src/phoenix/stats.py: bootstrap intervals, cell summaries, the split analysis.
src/phoenix/sets.py: graph sets, pinned seeds, checkpoint check, result paths.
src/phoenix/fast_coconut.py: attn_eager flag (default off; bit-exact path untouched).
src/phoenix/train.py: --save-every periodic checkpoints.
Drivers: baseline.py, necessity.py, heads.py, counterfactuals.py, tracing.py, cache_patch.py, masking.py (experiment 5), recovery.py (experiments 6 and 6b); qk_cells.py holds the query-key cells; relabel.py the amendment relabelers; evaluate.py, fit_probes.py, fit_jlens.py the experiment-0 fits.
GPT-2 path (experiment 7): nl.py (natural-language prompts, layout from tokenizer offsets, donors by name), gpt2.py (loader with the tied-weight checks, cached greedy generation), gpt2_eval.py (regime check), gpt2_cells.py (heads on all layers; the removal, calibration mask and carry-over cells), winner_probe.py (both model kinds), gpt2_bases.py (probe and Jacobian bases per GPT-2 run). Shared edits, all backward compatible: heads.slot_mass sums any slot tuple, heads.run scores the reordered run with its own layout, measure.answer_split(node_ids=).
Tests: tests/test_fast_equivalence.py (bit-exact, plus hooks-off and eager-path checks), tests/test_prompts.py, tests/test_attn_hooks.py, tests/test_minimal_pairs.py, tests/test_drivers_smoke.py (every driver on random weights), tests/test_relabel.py, tests/test_gpt2_plumbing.py (the GPT-2 path on a tiny random model).

Navigation Menu

Repository navigation

NOTES.md

NOTES.md

Notes

Amendments

Status

Established (from paper/v1.pdf, treated as data)

Open: what does a mid-search thought encode?

Working rules

Sets and measures

Experiment 0: setup and baseline

Experiment 1: necessity, done properly

Experiment 2: which heads look by position and which by content

Experiment 3: the counterfactual set (thought fixed, one thing changed)

Experiment 4: where the effect lives

Experiment 5: where the answer is recovered (masking), written 2026-09-10

Experiment 6: where the answer is recovered, part two (written 2026-09-11)

Experiment 6b: candidate-edge masks at every latent query, and the per-step carry-over (written 2026-09-11, before the run)

Experiment 7: the same measurements on fine-tuned GPT-2 (written 2026-09-16, before the pilot)

Experiment 8: the removal's flipped graphs, the carried winner, and a held-out check (written 2026-10-01, before any run)

Experiment 9: the restore test, and four zero-cost reads (written 2026-10-02, before the restore runs)

Experiment 10: how much of the isolation leftover each source carries (written 2026-10-05, before any run)

Experiment 11: the INLP (amnesic probing) arm on this repo's seeds (written 2026-10-06, before any run)

Experiment 12: whole answer-path deletion at every intermediate step (written 2026-10-06, before any run)

Next run

Code map

FilesExpand file tree

NOTES.md

Latest commit

History

NOTES.md

File metadata and controls

Notes

Amendments

Status

Established (from paper/v1.pdf, treated as data)

Open: what does a mid-search thought encode?

Working rules

Sets and measures

Experiment 0: setup and baseline

Experiment 1: necessity, done properly

Experiment 2: which heads look by position and which by content

Experiment 3: the counterfactual set (thought fixed, one thing changed)

Experiment 4: where the effect lives

Experiment 5: where the answer is recovered (masking), written 2026-09-10

Experiment 6: where the answer is recovered, part two (written 2026-09-11)

Experiment 6b: candidate-edge masks at every latent query, and the per-step carry-over (written 2026-09-11, before the run)

Experiment 7: the same measurements on fine-tuned GPT-2 (written 2026-09-16, before the pilot)

Experiment 8: the removal's flipped graphs, the carried winner, and a held-out check (written 2026-10-01, before any run)

Experiment 9: the restore test, and four zero-cost reads (written 2026-10-02, before the restore runs)

Experiment 10: how much of the isolation leftover each source carries (written 2026-10-05, before any run)

Experiment 11: the INLP (amnesic probing) arm on this repo's seeds (written 2026-10-06, before any run)

Experiment 12: whole answer-path deletion at every intermediate step (written 2026-10-06, before any run)

Next run

Code map