Skip to content

fix(serialization): find the function's symtable child by type, not by count (Python 3.14) - #698

Merged
JadenFiotto-Kaufman merged 1 commit into
ndif-team:0.8from
Hotragn:fix/symtable-annotation-table
Aug 26, 2026
Merged

fix(serialization): find the function's symtable child by type, not by count (Python 3.14)#698
JadenFiotto-Kaufman merged 1 commit into
ndif-team:0.8from
Hotragn:fix/symtable-annotation-table

Conversation

@Hotragn

@Hotragn Hotragn commented Aug 24, 2026

Copy link
Copy Markdown

Rebased onto 0.8 as requested in #696. This one is a bug I hit because I moved over — it is specific to 0.8 and not present on dev.

Summary

_function_referenced_names filters a def's serialized payload down to the names its body actually needs from outside it. Its own docstring gives the reason:

a helper doing with open(path) as f: would drag in whatever f a notebook cell left lying around, and a closed file can't be pickled at all ("Cannot pickle closed files")

It located the function's symtable entry by counting children:

children = top.get_children()
if len(children) != 1:
    return _referenced_names(source)   # block rule — captures locals
return globals_of(children[0])

A single module-level def does not produce a single child table on every Python. Under PEP 649 (3.14+) it also gets an __annotate__ table — whether or not the def is annotated:

>>> import symtable
>>> [(c.get_name(), c.get_type()) for c in symtable.symtable(
...     "def f(x):\n    return g(x)\n", "<s>", "exec").get_children()]
[('__annotate__', 'annotation'), ('f', 'function')]

So on 3.14 the guard is true for every def, the fallback runs every time, and the filter is inert — reinstating exactly the capture it exists to stop. PEP 695 generics add a type_parameters table for the same reason. Note children[0] is also the wrong table by then, so dropping the guard alone would not have been enough.

Why CI didn't catch it

The two tests that already cover this — test_function_locals_are_not_shipped and test_function_globals_still_shipfail on 3.14 today:

FAILED tests/test_serialization.py::TestScopeFiltering::test_function_locals_are_not_shipped
E   AssertionError: assert 'f' not in {'f': 'a-closed-file', 'json': 'json-mod', 'open': 'builtin'}

FAILED tests/test_serialization.py::TestScopeFiltering::test_function_globals_still_ship
E   Left contains 2 more items: {'x': 'stale', 'xs': 'stale'}

.github/workflows/python-app.yml runs Python 3.12, where PEP 649 is not active, so nothing surfaced it. pyproject.toml lists 3.14 as supported (Programming Language :: Python :: 3.14), so this is in scope rather than a future concern.

Fix

Select the one child whose get_type() is "function". That covers def and lambda, and is stable across versions — it does not care how many non-function tables a given Python decides to emit.

Tests

  • test_scope_filter_survives_extra_symtable_children — parametrised over plain / annotated / defaulted def and a lambda. x is the parameter, so it is local no matter what; if it ships, the filter was bypassed. (The lambda case passes without the fix — lambdas still yield one child — which is why it is worth pinning alongside the others.)
  • test_function_local_shadowing_an_unpicklable_global — the original symptom end to end: a closed file handle in the enclosing scope, shadowed by a with ... as f: local. Capturing it doesn't merely bloat the payload; serializing raises.

Result

Without the fix, 6 of 15 TestScopeFiltering tests fail on 3.14. With it, the full CPU suite is green:

856 passed, 7 skipped, 27 warnings

0.8 @ 1f974f0, Python 3.14.3, transformers 5.15.1, torch 2.9.1, CPU. Also confirmed the fix is a no-op on the 3.12 path (one function table either way).

…y count

`_function_referenced_names` filters a `def`'s payload down to the names its
body actually needs from outside it, so a name the function only ever binds
does not drag in the enclosing scope's same-named object. Its own docstring
gives the failure it exists to prevent:

    a helper doing ``with open(path) as f:`` would drag in whatever ``f`` a
    notebook cell left lying around, and a closed file can't be pickled at
    all ("Cannot pickle closed files")

It located the function's symtable entry by counting children:

    children = top.get_children()
    if len(children) != 1:
        return _referenced_names(source)   # block rule -- captures locals
    return globals_of(children[0])

A single module-level `def` does not produce a single child table on every
Python. Under PEP 649 (3.14+) it also gets an `__annotate__` table of type
`annotation`, whether or not the `def` is annotated:

    >>> [(c.get_name(), c.get_type()) for c in symtable.symtable(
    ...     "def f(x):\n    return g(x)\n", "<s>", "exec").get_children()]
    [('__annotate__', 'annotation'), ('f', 'function')]

So on 3.14 the guard is true for *every* `def`, the fallback runs every time,
and the filter is inert -- reinstating exactly the capture it was written to
stop. PEP 695 generics add a `type_parameters` table for the same reason.
`children[0]` was also the wrong table by then, so removing the guard alone
would not have been enough.

Select the one child whose `get_type()` is `"function"` instead, which covers
`def` and `lambda` and is stable across versions.

`pyproject.toml` lists 3.14 as supported, and the two tests that already
covered this (`test_function_locals_are_not_shipped`,
`test_function_globals_still_ship`) fail on it today -- CI runs 3.12, so
nothing surfaced it.

Adds `test_scope_filter_survives_extra_symtable_children`, parametrised over
plain / annotated / defaulted `def`s and a lambda, asserting the parameter
never ships; and `test_function_local_shadowing_an_unpicklable_global`, which
pins the original symptom end to end by putting a closed file handle in the
enclosing scope.

Without the fix, 6 of the 15 TestScopeFiltering tests fail on 3.14. With it,
the full CPU suite is green: 856 passed, 7 skipped.
@JadenFiotto-Kaufman

Copy link
Copy Markdown
Member

Perfect. Thank you @Hotragn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants