Code cells inside ANY directive never execute in the browser (exercise, solution, note)

[mmcky]

short_path cannot be completed by a reader on the live site. The cell that loads the graph never executes, so every later cell raises NameError. This predates any recent change.

What a reader sees

Click through the cells in order and the final one fails:

NameError: name 'graph_file_data' is not defined
Cell In[7], line 1
----> 1 Q = map_graph_to_distance_matrix(graph_file_data)

The execution counter is the tell: In[7] for a cell at document position #8 on a 9-cell page. Exactly one preceding cell never ran.

Why

myst build nests code cells inside directives, and thebe only wires up cells at the top-level block layer. From the published short-path.json:

cell	ancestry	runs?
import numpy as np	['root','block']	✅
from numpy import inf	['root','block']	✅
nodes = range(7)	['root','block']	✅
the graph data cell	['root','block','exercise','block']	❌
num_nodes = 100 … and the rest	['root','block']	✅

On main two cells are nested there — import requests and graph_file_data = str(graph_file_response.content, 'utf-8') — so the data is never fetched and never assigned.

The part that makes this subtle: a second bug is currently masking a worse version of the first

short_path.md:480 is a bare ```{solution} used as a closer. mystmd rejects it:

mystDirectiveError  line 480
"required argument not provided for directive: solution"

The opener at :381 is {solution} too, where lecture-python-intro — and every other repo carrying this lecture — uses the gated pair {solution-start} / {solution-end}.

Because that directive is unbalanced, the solution node in the AST is empty (its only child is admonitionTitle) and all of the solution's content spills out into top-level blocks. That accident is load-bearing: the spilled cells are the ones a reader can actually run.

Fixing the directive to the gated form — the obvious correct change — was tried on a branch and rebuilt. It removes the error, but the solution's five cells then correctly nest as ['block','solution','block'], i.e. the fix would likely take this lecture from one dead cell to six. So the two bugs must be addressed together, and the directive must not be "tidied up" on its own.

What to decide

1. Is nesting genuinely what stops execution? Strongly implied by the correlation, but not yet isolated — worth confirming, and worth raising upstream with mystmd/thebe if gated directives are meant to keep code cells runnable. The gated -start/-end forms exist precisely so executable content can live at the root level, so this may be a genuine upstream defect rather than intended behaviour.
2. If it is, the graph data cell needs to sit outside {exercise-start} … {exercise-end}, which is a small content decision — the exercise prose says "The text below describes a weighted directed graph" and the solution says "reads in the graph data above", so moving it needs a sentence adjusted.
3. Only then fix the {solution} directive.

Scope

This is independent of the dataset migration. #63 removes this lecture's cross-repo fetch of lecture-python-intro's committed graph.txt and is neutral on the problem here: it goes from two nested dead cells to one, and does not make the lecture runnable. That is why it is not being merged as a fix for this.

Worth checking whether other lectures here put code cells inside {exercise} blocks — this may not be the only one.

[mmcky]

Probe results — measured, and the scope is wider than this issue assumed

Run on the #65 preview. One page, seven cells, six positions:

EXECUTED : ['A_toplevel_setup', 'C_toplevel_after_magic']
MISSING  : ['D_in_gated_exercise', 'E_in_gated_solution', 'F_in_note']
%%file worked at top level: True

Cell	Position	Ran?
A	top level	✅
B	top level, %%file probe_b.txt	✅ — printed Writing probe_b.txt
C	top level, reads B's file back	✅ — exists? True, contents = hello-from-B
D	gated {exercise-start}	❌
E	gated {solution-start}	❌
F	plain {note} admonition	❌

Three findings

1. This is not about exercises. Any directive nesting strands a cell. {note} fails exactly like {exercise} and {solution}. This issue's title and framing are too narrow — the rule is that thebe only wires up cells at the top-level block layer, and any containing directive puts a cell below it.

2. The cells are not visibly disabled. D and F both render with run controls (▶ ⊖) in the page. Clicking produces no output and no error. That is the whole reason this bug survived so long: a stranded cell is indistinguishable from a cell that ran and printed nothing, which is the normal case for an assignment. Any future diagnosis here should use an explicit marker-and-report pattern rather than reading outputs.

3. %%file works fine. B and C prove the magic fires at top level and writes to a filesystem a later open() can read. Earlier in this investigation I claimed %%file does not fire under thebe-lite — that was wrong, and it was wrong because the cell I tested it in was nested and never ran. Retracted.

This also confirms the trap in the second half of this issue, empirically

E — a correctly gated {solution-start} — does not run. In short_path.md the solution's five cells do run, but only because :480's malformed ```{solution} closer leaves the directive unbalanced and spills its content to root level.

So fixing that directive on its own would take short_path from one stranded cell to six. That was previously an inference from a rebuilt AST; it is now measured. Do not tidy up the {solution} directive without moving the cells out first.

What the fix looks like now

For short_path: the graph data cell has to sit at top level, outside {exercise-start} … {exercise-end}. Since %%file is confirmed working, it can be the %%file graph.txt cell that every sibling repo uses — which is what #63 already contains. That PR stays blocked only on the cell being moved out of the exercise block.

The prose needs one adjustment: "The text below describes a weighted directed graph" and the solution's "reads in the graph data above" both point at the data, so moving the cell outside the exercise changes what "below" refers to.

Worth sweeping the other lectures here for code cells inside any directive — on this evidence every one of them is dead in the browser.

Upstream

This looks like a genuine mystmd/thebe defect rather than a lecture error, and it is now reproducible in six lines. The gated -start/-end forms exist precisely so executable content can live at the root level, and myst-ext-exercise implements exercise-start as alias: ["exercise"] with the body pushed into children — so the gate never hoists anything. Worth reporting with this probe attached.

[mmcky]

Root cause located, and the audit is done

The audit: short_path is the only affected page

Every one of the 42 published pages was fetched and its AST walked for executable cells with any non-root ancestor — not a guessed list of container types, but every ancestor type that wraps an executable cell anywhere on the site:

42 published pages, all kind: Notebook
EVERY ancestor type wrapping an executable cell, site-wide:  {'exercise': 2}
Only page affected: short-path

So the exposure is exactly 2 cells on 1 page — import requests and graph_file_data = str(...), both inside {exercise-start}. No other lecture puts an executable cell inside any directive, and no page is missing a kernelspec. Much narrower than this issue first assumed.

The bug is in @myst-theme/jupyter, not in myst-ext-exercise

@myst-theme/jupyter/dist/execute/utils.js:

export function executableNodesFromBlock(block) {
    if (!block || block.type !== 'block') return;
    let target = block;
    if (block.children?.length === 1 && block.children[0].type === 'container') target = block.children[0];
    if (target.children?.length >= 2 && target.children[0].type === 'code') {
        return { codeCell: target.children[0], output: target.children[1] };
    }
}

notebook.cells = mdast.children.map((block) => { ... executableNodesFromBlock(block) ... });

Two things combine:

1. notebook.cells = mdast.children.map(...) walks only the root's direct children. No recursion.
2. executableNodesFromBlock requires the code node to be children[0] of a top-level block — with exactly one special case, a lone container (that is the figure case).

A cell inside an exercise sits at root > block > exercise > block > code. The outer block's children[0] is the exercise node, so the check fails and the cell is never added to notebook.cells — never registered with the kernel. It still renders with run controls, because rendering and execution-registration are separate paths. Hence a visible ▶ that does nothing.

myst-ext-exercise is behaving correctly. It nests the body under the exercise node, which is semantically right — an exercise should contain its content. Changing that would be the wrong fix and would break cross-referencing and numbering. The gated -start/-end form is a parsing convenience for content that cannot be indented; it was never a hoist, and exercise-start is literally alias: ["exercise"] in the directive spec.

Where a fix goes

Layer	Repo	QuantEcon fork?
the exercise directive (correct as-is)	jupyter-book/mystmd	yes — QuantEcon/mystmd, 45 commits ahead
the execution-registration bug	jupyter-book/myst-theme	no fork
the kernel engine	jupyter-book/thebe	no fork

So this cannot be fixed in QuantEcon/mystmd — wrong package, and lecture-wasm builds with the public mystmd@1.9.1 from npm anyway, not the fork.

But QuantEcon can ship a fix immediately without forking anything. QuantEcon/quantecon-theme-src depends on @myst-theme/jupyter: ^1.3.0 and already uses patch-package — "postinstall": "patch-package --patch-dir patches", with two existing patches for @jupyter-widgets/controls and jupyterlab-plotly. A third patch making notebookFromMdast walk nested blocks would land in the theme build straight away, and the same diff is the upstream PR to jupyter-book/myst-theme.

The shape of the fix: recurse into container children when collecting cells, so any block containing a code node is registered regardless of depth. Care needed on ordering — cells must keep document order so execution sequence is preserved.

Sequencing for this repo

None of this blocks short_path. Moving its data cell to top level fixes the lecture today with no dependency on an upstream release, and #63 already carries the %%file form that every sibling repo uses. The theme fix is the general cure; the cell move is the local one.

[mmcky]

Filed the theme-side fix as QuantEcon/quantecon-theme.mystmd#117 — that is where @myst-theme/jupyter is consumed, and that repo already runs patch-package, so a fix can ship there before the upstream PR to jupyter-book/myst-theme lands.

(Note quantecon-theme-src and quantecon-theme.mystmd are the same repo — the former is a rename redirect.)

That leaves two independent tracks:

General cure — QuantEcon/quantecon-theme.mystmd#117: make cell collection recurse so any nested block > code is registered with the kernel, preserving document order.

Local fix — this issue: move short_path's data cell out of {exercise-start} so it sits at top level. This works today with no dependency on a theme release, and #63 already carries the %%file graph.txt form that every sibling repo uses. It needs one prose adjustment, since "The text below describes a weighted directed graph" and the solution's "reads in the graph data above" both point at where the cell currently sits.

Doing the local fix does not make the theme fix unnecessary: any author adding a code cell inside an admonition or exercise hits the same silent failure, and ci.yml cannot see it.

[mmcky]

Resolved for short_path; the general bug remains

#63 merged and deployed. The live site now reports 9 cells, 0 stranded — short_path is completable in the browser for the first time.

The fix was to move the data cell below {exercise-end} so it is registered with the kernel. That works because every other cell on the page was already top level, which is itself an accident: short_path.md:480's bare ```{solution} closer leaves that directive unbalanced and spills its content to root level.

So this issue's second half is still live, and still a trap. Correcting that {solution} directive to the gated pair — an obviously right-looking tidy-up — would re-nest its five cells and take the page from zero stranded cells back to five. Probe cell E in #65 measured exactly that: a correctly gated {solution-start} cell does not run.

Do not fix :480 until QuantEcon/quantecon-theme.mystmd#117 lands. A comment in short_path.md records why the data cell sits outside the exercise so it is not moved back.

What is left here

The general defect: any code cell inside any directive is never registered with the kernel. Root cause and the fix route are in QuantEcon/quantecon-theme.mystmd#117 — @myst-theme/jupyter's notebookFromMdast maps only the root's direct children, and that repo already runs patch-package, so it can ship before the upstream PR to jupyter-book/myst-theme.

An audit of all 42 published pages found no other affected page, so nothing else in this repo is waiting on it. The exposure is future lectures: anyone putting a code cell inside an admonition or exercise gets a page that looks correct, builds green, and cannot be run — ci.yml runs myst build --html with no execution and can never see it.

Suggest keeping this open as the consumer-side tracker until #117 is fixed, then re-running the probe as the regression test.