diff --git a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md index 399484804b..d543ea5012 100644 --- a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md +++ b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md @@ -409,9 +409,26 @@ dictionaries, call keywords, owner-member results and declared scalar returns are parsed with AST. Imported enum aliases resolve only to the registered owner, including one unrenamed re-export hop through a tracked module (a second hop, a renamed re-export or a rebinding stays unknown); shadowed -names, reassignments and unresolved calls remain unknown. Conditional -results exclude the condition's literals. TypeScript object writes, assignments -and declared returns use the repository's TypeScript parser rather than regex. +names and unbindable calls remain unknown. Conditional +results exclude the condition's literals. Three further local forms are bound, +each only under a stated condition: a call to an undecorated, non-generator, +plainly-defined top-level `def` **of the same module** resolves to the union of +that function's own returns, with arguments never bound to parameters, so a +returned parameter stays unknown and the result is independent of the call site; +a local written more than once resolves to the union of the writes that +textually precede the read, and only when every store of that name is a plain +`name = expression` **and** no write shares an enclosing loop with the read; +and a container mutated only through direct literal-key subscript writes keeps +its untouched keys, with a written key carrying the union of its initializer and +every write. Anything outside those conditions — a decorator, `async def`, a +generator, recursion, an imported or attribute call, a back edge that carries a +later write to the read, a `with`, `except`, walrus, augmented, unpacking, +`global` or `del` rebinding, an alias, a method call, a computed, negative or +deeper store, an escape into a call, or an unknown `**` spread — leaves the site +unknown rather than admitting a value. TypeScript object writes, assignments +and declared returns use the repository's TypeScript parser rather than regex, +and report the same blocker vocabulary as the Python scanner, so one residue +taxonomy covers both runtimes. Neither parser executes inspected source. These are syntactic result witnesses, not a proof of reachability or whole-program data flow. @@ -833,6 +850,7 @@ on the next full-tree scan; genuine shared-contract changes still need review. | Historical committed snapshots could become stale across merges | Replay the scanner over the first parent and the merge of the last twenty `upstream/main` merge commits | 8 of 20 merges change at least one carrier | Historical cost motivating Q9; current checks compute the combined tree without a committed snapshot | | The formal model cannot silently lose a proof obligation | Remove an invariant, role, relation, candidate decision, or proof-boundary category from `formal_model` | The drift smoke fails on the exact formal-model shape | The model is a finite contract and proof ledger; it does not prove the listed properties by itself | | An obligation cannot claim a domain nobody counts | `uv run --extra test python -m pytest tests/architecture/test_semantic_vocabulary_drift.py -k domain` | Dropping `domain`, inflating `verified` or `registered`, inventing a selector, claiming an unanchored selector or an out-of-stage evidence bound, and an advisory invariant claiming verified members each fail closed | The sizes are derived from the registry, so the check grounds the declared domain in registry data; it does not prove the obligation over that domain | +| Bounded binding forms cannot be loosened into false evidence | `uv run --extra test python -m pytest tests/architecture/test_semantic_producer_binding.py` | pass; every recognized form has a negative twin — a decorated, `async`, generator, rebound, imported or recursive callee, an unordered store, an aliased or escaped container, and an unknown `**` spread each keep the site unresolved | Fixture repository; a site the scan cannot bind stays unresolved with its recorded reason, never dead | | F1/F2 quantify over exactly what the producer check walks | Same test module: compare `check_producers`' predicate with the declared F1/F2 domain | The vocabularies with `producers` are exactly the `kernel` tier, 6 of 26; the other 20 are all `cross_runtime` | The scan reach bounds the claim further and is reported, not pinned | Known limits, stated so the check is not over-trusted: @@ -1124,6 +1142,117 @@ introduce a competing target state. ## Appendix A: Execution ledger (non-normative) +### 2026-09-17 — B2: three bounded binding forms, and the residue that stays unresolved + +- **Trigger:** [#4447](https://github.com/huangruiteng/loopx/issues/4447) recorded + the B2 residue as "34 total minus the 15 unprovable by design". Re-measured on + `9003577f9` the total is **41**, and the breakdown is across every scanned + vocabulary rather than `effective_action` alone: `annotation_only=5, + argument_name_only=10, attribute_read=2, call_result=11, other=1, + typescript_dynamic=8, unstable_local=4`. The issue's number was stale; this + entry records the measured split. +- **Delivered:** three bounded local forms in `python_production`, each with + positive and negative fixtures in + `tests/architecture/test_semantic_producer_binding.py`. + 1. **Same-module call results.** A call to an undecorated, non-generator, + plainly-defined top-level `def` of the same module resolves to the union of + that function's own returns. Arguments are never bound to parameters, so a + returned parameter stays unknown and the answer does not depend on the call + site; it is memoised per module scan. A decorator, `async def`, a generator, + a second top-level binding of the name, an imported or attribute call, a + local rebinding and recursion all keep the `call_result` blocker. + 2. **Ordered rebinding of a local.** A local written more than once resolves to + the union of the writes that textually precede the read, and only when every + store of that name is a plain `name = expression` and no write shares an + enclosing loop with the read. `with`, `except`, walrus, augmented, + unpacking, `global` and `del` rebindings are not ordered by this scan and + erase the local. + + Textual position is execution order only where no back edge crosses it. A + write later in a loop body reaches the read at the top of the next + iteration, so the preceding-writes filter dropped a live value and reported + a closed value set that was not closed: a producer emitting an unregistered + value on every iteration after the first read as fully resolved, which is + the one failure mode that turns an unknown into wrong evidence rather than + into a smaller residue. Four shapes are pinned as regressions — a `for` + back edge, a `while` back edge, a write carried by an outer loop, and a + `finally` that rebinds — beside two positives that keep the ordering this + form was built for. + 3. **Key-precise container writes.** A local container mutated only through + direct literal-key subscript writes keeps its untouched keys, and a written + key carries the union of its initializer and every write. An alias, a method + call, a computed or deeper store, a `del`, a negative index, or passing the + container to any call still discards the container. A negative index names + the same slot as a non-negative one whose number depends on the container's + length, so recording it against the key `-1` left a read of `table[0]` + looking at an initializer the write had already replaced. A `**` spread of statically + known dict literals is flattened, so an optional spread no longer hides a + sibling key; an unknown spread still makes every key dynamic. + + The TypeScript parser gains the two sound forms the Python scanner already had + (`||` and `??` arms, a transparent `String(x)`, and `undefined` read as no + value) and, more importantly, reports the **same blocker vocabulary**: the + single `typescript_dynamic` catch-all is replaced by `attribute_read`, + `call_result`, `unstable_local` and `dynamic_key`, with `typescript_dynamic` + kept only as the fallback for a form it cannot classify. An owner-member + result (`enum_result`) now carries its reason too; an unlabelled unknown was + invisible in the report breakdown. +- **Result:** unresolved sites **41 → 40**, split + `annotation_only=5, argument_name_only=10, attribute_read=7, call_result=14, + other=1, unstable_local=3`. The back-edge and negative-index rules were added + after that measurement and left every number in it unchanged, so no site on + the tree was resolving through the unsound path: the generality had bought + nothing that the soundness fix takes away. +- **Three non-blocking review findings closed afterwards, none of which moves a + number.** `_MODULE_FUNCTIONS` was keyed on `id(tree)`, so its correctness + depended on `_TREES` never evicting; give that cache a bound and a reused id + would hand back another file's functions, binding a call to the wrong callee + with no symptom. It is keyed by path and text hash now, as `_TREES` is. + `_is_generator` used `ast.walk`, which descends into nested scopes, so a plain + function that merely defined a generator inside itself read as a generator and + lost its binding -- safe in direction, but it withheld evidence this slice + exists to make actionable. And `blockerFor` labelled an object literal, an + array literal and a template expression `dynamic_key`, which the shared + vocabulary defines as a computed or non-literal subscript; Python answers + `other` for the same shapes, so both runtimes now agree. The residue stays at + 40 sites with the same split. All eight TypeScript sites are reclassified (five + `attribute_read`, three `call_result`); none was resolvable, so that part is a + taxonomy, not a shrink. The one site that closes is + `driver.py::build_loopx_turn_plan:500`, which needed all three forms and the + spread flattening at once. Evidence improves further than the count shows: + unresolved rows carrying at least one known value go **2 → 7**. Registry + values, budgets and the producer site list are unchanged, and no site becomes + newly visible or unregistered. +- **Deliberately not bound, with the reason recorded:** + - `annotation_only` (5) — all five are bare `effective_action: str` field + declarations carrying **no value node at all**. Unprovable by design; + the issue's classification is confirmed. + - `argument_name_only` (10) — confirmed unprovable by design, with one + sharpening: these are unprovable as a *production role*, not unresolvable as + an expression. Four of the ten now carry a fully resolved value set and are + still correctly unresolved, because the callee (`_execution_obligation` and + its peers) reads the field rather than emitting it. The honest way to shrink + this bucket is a registry `call_producers` declaration naming a reviewed + output builder — a data edit a reviewer sees — never a scanner change. + Counting any field-named keyword as production would make the obligation + tautological, which Section 5 forbids. + - `attribute_read` (7), `call_result` (14), `unstable_local` (3) and `other` + (1) — every remaining site bottoms out in one of four things outside this + scan's bound: a read off a caller-supplied mapping or object + (`decision.get("effective_action")`, `run_decision.effective_action`), a call + into another module, a returned parameter, or a method chain. Binding any of + them needs cross-module or object-field resolution, a separate bounded form + with its own blast radius; it is not attempted here. **A site this scan + cannot bind stays `unresolved` with its recorded reason — it is never + treated as dead.** +- **Cost:** the producer scan runs on every pull request touching `loopx/`. Over + the 319 Python files it reaches and the 5 producer vocabularies, best of three + runs on one tree: **9.20 s → 7.17 s**. The deepened scan is net faster because + it now reuses the memoised parse and the module-function table across + vocabularies instead of re-parsing once per scan. +- **Effect on normative design:** Section 5's bounded producer model names the + three forms and the shared blocker taxonomy; no invariant or milestone changes. + ### 2026-09-17 — B5: consumer roles for the vocabularies that declare a slot Non-normative; advisory evidence only, and B5 is an optional item of #4447. No @@ -1515,6 +1644,7 @@ result on the current tree; what changes is what the invariants claim. | 2026-09-16 | Q9: compute the full inventory on demand; retire the committed census | Implementation for [maintainer feedback](https://github.com/huangruiteng/loopx/pull/4360#issuecomment-5692062394); PR review pending | Committed snapshot with post-merge regeneration; diff-only scan rejected | 1, I6, 3, 5, 9, 10, 12 | | 2026-09-16 | B2: bind one unrenamed re-export hop in the Python producer scanner | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2; PR review pending | Require every consumer to import the owner module (fragile; failed silently in M2); unbounded multi-hop resolution rejected | 5, Appendix A | | 2026-09-16 | B1 rename invariance: add the name-keyed divergence advisory; state the limit it does not close | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B1; PR review pending | Keying the budget on value sets (rejected: `CONFIDENCE_LEVELS` and `EDGE_CASE_COMPLEXITIES` share `high/low/medium` with different meanings); a committed name ledger (rejected at M0: Q9 retired the committed census). The advisory lists surviving forks by name; it was first described as catching a one-sided rename, which measurement disproved, so both mirrors state the limit as it behaves | 9 | +| 2026-09-17 | B2: bind same-module call results, ordered local rebinding and key-precise container writes; reclassify the TypeScript residue rather than shrink it | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2; PR review pending | Bind cross-module calls and object fields (rejected: a separate bounded form with its own blast radius, not this slice); count a field-named keyword as production (rejected: it makes the obligation tautological, Section 5); leave `typescript_dynamic` as one catch-all (rejected: eight sites shared one reason, so the residue was not actionable); bind the callee's parameters to the call-site arguments (rejected: the answer would depend on the caller and could not be memoised, and a wrong binding would invent evidence) | 5, 9, Appendix A | | 2026-09-17 | B5 (optional): report consumer roles per site only for vocabularies that declare `literal_scan.field`, and state the unknown share instead of classifying everything | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B5; PR review pending | Fall back to the vocabulary id when no slot is declared (rejected on review: it analysed 25 of 26 vocabularies against a field name nobody declared, and every row downstream inherited the guess; they are now named under `missing_slot_identity` and not analysed); register consumers in the registry (rejected: the tracking issue forbids blanket consumer registration, and a declared list is a claim rather than evidence); make the report a merge gate (rejected: F3 is the advisory lane, and a 90.2% unknown share cannot gate anything); report only the sites the grammar resolves (rejected: the tables would read as complete, so an unrecognized mention and a computed-key read are printed rows with reasons) | 9, Appendix A, Appendix B, Appendix C | | 2026-09-17 | B0: state schema validation, implementation stage, evidence status and blocking behaviour separately for I2/I11-I14 and the enforcement lanes; require each formal invariant id exactly once | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B0; PR review pending | Rename the `blocking_next` lane to match its behaviour (rejected: the lane name is the milestone that owns the check, and renaming it would lose that and collapse the two readings the other way); add a `blocks_today` boolean to `formal_model` (rejected: it would be one more declared field a reader could mistake for a measurement, and the fact is a property of the smoke's `main()`, which no registry edit can change); leave the lane gloss and note the gap in the ledger only (rejected: the gloss is the sentence a reviewer quotes) | 2, 5, 11, Appendix A, Appendix B | | 2026-09-17 | Bound F1/F2 to the kernel tier and the scan reach, restate F4 as scope enumeration completeness, and give every obligation a derived `domain` | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447); **kernel-maintainer approval required, not yet given** | Leave the unconditional statements and record the gap in prose only (rejected: the statement was stronger than `validate_production`'s own docstring); restate F4 as per-context value-set disjointness (rejected: refuted by the repo's own data, since `scope_declarations` exists to permit legitimate same-name reuse); widen the scan so the unconditional claim becomes true (rejected: a separate change with its own risk) | 5, 9, Appendix B, Appendix C | diff --git a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md index 94192e59d8..6f1e9f21f6 100644 --- a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md +++ b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md @@ -316,9 +316,21 @@ M0.5 之后新增或删除值时；`cross_module` 只在晋升后（Q8）。持 Python 通过 AST 解析字段赋值（含下标、属性及带注解赋值）、字典、调用关键字、 owner 成员结果及声明函数的标量返回。导入枚举的别名只解析到已登记 owner，含经由一个被跟踪模块的一跳未改名再导出（第二跳、改名再导出或重新绑定保持 unknown）； -被遮蔽的名字、重复赋值及未解析调用仍为 unknown。条件表达式只检查结果分支， -排除条件中的字面量。TypeScript 的对象写入、赋值及声明返回使用仓库的 TypeScript -解析器。两个解析器都不执行被检查源码。这些是句法结果证据，不是可达性或全程序 +被遮蔽的名字与无法绑定的调用仍为 unknown。条件表达式只检查结果分支， +排除条件中的字面量。另有三条局部形式被绑定，且各自只在明确条件下成立：对 +**同模块内**一个未被装饰、非生成器、以普通 `def` 定义的顶层函数的调用，解析为 +该函数自身全部 return 的并集，实参从不绑定到形参，因此返回形参仍为 unknown， +结果与调用点无关；被写入多次的局部变量解析为文本上位于该读取之前的那些写入的 +并集，且仅当该名字的每一次 store 都是普通的 `name = expression`、**并且**没有 +任何写入与该读取处在同一个循环内时成立；只经由直接字面量键下标写入被改动的容器 +保留其未被触碰的键，被写入的键携带其初始化值与每一次写入的并集。凡落在上述条件 +之外的——装饰器、`async def`、生成器、递归、导入调用或属性调用，把靠后的写入带回 +读取处的回边，`with`、`except`、海象、增量赋值、解包、`global` 或 `del` 造成的 +重绑定，别名、方法调用、计算键、负索引或更深层的 store、逃逸进调用，以及未知的 +`**` 展开——都让该位点保持 unknown，而不是采信一个取值。 +TypeScript 的对象写入、赋值及声明返回使用仓库的 TypeScript +解析器，并报告与 Python 扫描器相同的阻塞原因词汇，因此同一套残量分类覆盖两个 +运行时。两个解析器都不执行被检查源码。这些是句法结果证据，不是可达性或全程序 数据流证明。 `uv run python examples/semantic-vocabulary-drift-smoke.py --report` 列出未解析的生产 @@ -676,6 +688,7 @@ owner 符号集合的组：`EffectiveAction` 与 `EFFECTIVE_ACTIONS` 是同一 | 历史上的已提交清单会因上游合并而过期 | 对 `upstream/main` 最近二十个合并提交，在第一父提交与合并结果之间重放扫描器 | 20 次合并中 8 次至少改变一个载体 | Q9 的历史动机；当前检查直接计算合并后的全树，不再依赖提交快照 | | 形式模型不能静默丢失证明义务 | 从 `formal_model` 删除不变量、角色、候选决策、关系或证明边界分类 | 漂移 smoke 针对形式模型结构失败 | 该模型是有限契约和证明账本，本身不等于这些性质已经被证明 | | 义务不能声称一个无人清点的值域 | `uv run --extra test python -m pytest tests/architecture/test_semantic_vocabulary_drift.py -k domain` | 删掉 `domain`、调大 `verified` 或 `registered`、自造 selector、使用未钉住的 selector 或跨阶段的证据边界、以及 advisory 不变量声称已验证成员，逐项失败关闭 | 规模从注册表推导，因此该检查把声明值域接地到注册表数据；它不证明该义务在那个值域上成立 | +| 有界绑定形式不能被放宽成假证据 | `uv run --extra test python -m pytest tests/architecture/test_semantic_producer_binding.py` | 通过；每条被识别的形式都有反例孪生——被装饰的、`async`、生成器、被重绑定的、导入的或递归的被调方，无序 store，被别名或逃逸的容器，以及未知 `**` 展开，都让该位点保持未解析 | 夹具仓库；本扫描无法绑定的位点保持未解析并带上被记录的原因，绝不当作 dead | | F1/F2 恰好量化 producer 检查真正走到的集合 | 同一测试模块：将 `check_producers` 的谓词与 F1/F2 声明的值域对比 | 声明了 `producers` 的词表恰好是 `kernel` 层，26 中的 6；其余 20 个全部是 `cross_runtime` | 扫描范围进一步约束该声明，它被上报而不被钉住 | 已知边界，写明是为了不让这个检查被过度信任： @@ -907,6 +920,87 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 ## 附录 A：执行账本（非规范） +### 2026-09-17 — B2：三条有界绑定形式，以及仍然保持未解析的残量 + +- **起因：**[#4447](https://github.com/huangruiteng/loopx/issues/4447) 把 B2 残量 + 记为“34 个减去 15 个设计上不可证的”。在 `9003577f9` 上重新测量，总数是 **41**， + 且分布覆盖全部被扫描词表，而不只是 `effective_action`：`annotation_only=5, + argument_name_only=10, attribute_read=2, call_result=11, other=1, + typescript_dynamic=8, unstable_local=4`。issue 里的数字已经过期；本条记录实测分布。 +- **交付：**在 `python_production` 中新增三条有界局部形式，每条都在 + `tests/architecture/test_semantic_producer_binding.py` 里配有正例与反例。 + 1. **同模块调用结果。**对同模块内一个未被装饰、非生成器、以普通 `def` 定义的 + 顶层函数的调用，解析为该函数自身全部 return 的并集。实参从不绑定到形参， + 因此返回形参仍然未知，且结果与调用点无关，可按模块扫描记忆化。装饰器、 + `async def`、生成器、该名字的第二个顶层绑定、导入调用与属性调用、局部重绑定 + 以及递归，都继续保留 `call_result` 阻塞原因。 + 2. **局部变量的有序重绑定。**被写入多次的局部变量，解析为文本上位于该读取之前 + 的那些写入的并集，且仅当该名字的每一次 store 都是普通的 + `name = expression`、并且没有任何写入与该读取处在同一个循环内时才成立。 + `with`、`except`、海象、增量赋值、解包、`global` 与 `del` 造成的重绑定不被 + 本扫描定序，会直接抹掉该局部变量。 + + 只有在没有回边横跨的位置上，文本先后才等于执行先后。循环体里靠后的一次写入 + 会在下一轮迭代抵达位于顶部的读取，于是「取文本在前的写入」这条过滤会丢掉一个 + 活的取值，并报出一个并不封闭的取值集合：一个从第二轮迭代起就发出未注册取值的 + 生产者会被读作「完全解析」。这是唯一一种把未知变成错误证据、而不是变成更小残量 + 的失效方式。四种形态已被钉为回归——`for` 回边、`while` 回边、由外层循环携带的 + 写入、以及在 `finally` 中重绑定——旁边另有两条正例，守住这条形式本来要支持的定序。 + 3. **按键精确的容器写入。**只经由直接字面量键下标写入被改动的局部容器，保留其 + 未被触碰的键；被写入的键则携带其初始化值与每一次写入的并集。别名、方法调用、 + 计算键、负索引或更深层的 store、`del`，以及把容器作为任何调用的实参传出，仍然 + 丢弃整个容器。负索引指向的槽位，其序号取决于容器长度，与某个非负索引是同一个 + 槽；把它按键 `-1` 记录下来，会让对 `table[0]` 的读取仍然看到那次写入早已替换掉 + 的初始化值。对静态可知的 dict 字面量的 `**` 展开会被摊平，因此可选展开 + 不再遮蔽同级键；未知展开仍使所有键变为动态。 + + TypeScript 解析器补上了 Python 扫描器早已具备的两条可靠形式（`||` 与 `??` 的 + 分支、透明的 `String(x)`，以及把 `undefined` 读作无值），更重要的是开始报告 + **同一套阻塞原因词汇**：单一的 `typescript_dynamic` 兜底被 + `attribute_read`、`call_result`、`unstable_local` 与 `dynamic_key` 取代， + `typescript_dynamic` 只保留为无法进一步归类时的兜底。owner 成员结果 + （`enum_result`）现在也携带原因；未打标签的未知在报告分布里是不可见的。 +- **结果：**未解析位点 **41 → 40**，分布为 `annotation_only=5, + argument_name_only=10, attribute_read=7, call_result=14, other=1, + unstable_local=3`。八个 TypeScript 位点全部被重新归类（五个 `attribute_read`、 + 三个 `call_result`）；它们没有一个是可解析的，因此这部分是分类学修正，不是缩减。 + 唯一被关闭的位点是 `driver.py::build_loopx_turn_plan:500`，它同时需要三条形式加上 + 展开摊平。证据的改善超过数字所显示的：携带至少一个已知值的未解析行从 **2 → 7**。 + 注册表取值、预算与 producer 位点清单均未变化，也没有任何位点变为新可见或未注册。 + 回边规则与负索引规则是在这次测量之后补上的，补上后这组数字一个都没有变化：树上 + 没有任何一个位点是靠那条不可靠的路径解析出来的——那份「通用性」并没有换来任何 + 被这次可靠性修复夺走的东西。 +- **随后收掉三条非阻塞的评审发现，没有一条移动任何数字。**`_MODULE_FUNCTIONS` + 以 `id(tree)` 为缓存键，其正确性依赖「`_TREES` 永不淘汰」这个远处不变量；一旦 + 给那个缓存加上上限，被复用的 id 就可能返回另一个文件的函数表，把调用静默绑到 + 错的被调用者。现在按路径与文本哈希做键，与 `_TREES` 一致。`_is_generator` 用 + `ast.walk`，而它会下降进嵌套作用域，于是一个只是在内部定义了生成器、自己并不 + yield 的普通函数被当成生成器而丢掉绑定——方向是安全的，但它扣住了本切片正要变 + 得可行动的证据。`blockerFor` 把对象字面量、数组字面量与模板表达式标成 + `dynamic_key`，而共用词汇把该标签定义为「计算键或非字面量下标」；Python 对同样 + 的形状给出 `other`，现在两个运行时一致。残量仍为 40 个位点，分布不变。 +- **有意不绑定，并记录原因：** + - `annotation_only`（5）——五个全部是裸的 `effective_action: str` 字段声明， + **根本没有值节点**。设计上不可证；issue 的归类得到确认。 + - `argument_name_only`（10）——确认设计上不可证，但需要一点锐化：它们不可证的是 + *生产角色*，而不是表达式不可解析。十个里现在有四个已经携带完整解析出的取值集合， + 并且仍然被正确地判为未解析，因为被调方（`_execution_obligation` 及其同类）是在 + 读取该字段而不是产出它。缩减这一类的诚实做法是在注册表 `call_producers` 中声明 + 一个经过评审的输出构造器——一次评审者看得见的数据编辑——而绝不是改扫描器。把任何 + 与字段同名的关键字算作生产，会使该义务变成同义反复，这是第 5 节所禁止的。 + - `attribute_read`（7）、`call_result`（14）、`unstable_local`（3）与 `other` + （1）——剩下的每个位点最终都落在本扫描边界之外的四件事之一：读取调用方提供的 + 映射或对象（`decision.get("effective_action")`、`run_decision.effective_action`）、 + 跨模块调用、返回形参，或方法链。绑定其中任何一种都需要跨模块或对象字段解析， + 那是另一条有自己影响面的有界形式，本切片不做。**本扫描无法绑定的位点，保持 + `unresolved` 并带上它被记录的原因——绝不当作 dead。** +- **成本：**producer 扫描在每个触及 `loopx/` 的 PR 上都会运行。在它覆盖的 319 个 + Python 文件与 5 个 producer 词表上，同一棵树三次取最优：**9.20 s → 7.17 s**。 + 加深后的扫描净变快，因为它现在跨词表复用记忆化的语法树与模块函数表，而不再每次 + 扫描重新解析。 +- **对规范设计的影响：**第 5 节的有界 producer 模型写明这三条形式与共享的阻塞原因 + 分类；不变量与里程碑均无变化。 + ### 2026-09-17 — B5：对声明了槽位的词表报告消费者角色 非规范性；仅为参考证据，且 B5 是 #4447 的可选项。当前源码树上没有任何检查的 @@ -1211,6 +1305,7 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 | 2026-09-16 | Q9：全树按需计算；移除已提交结构清单 | 根据[维护者反馈](https://github.com/huangruiteng/loopx/pull/4360#issuecomment-5692062394)实现，PR 评审待完成 | 取代合并后补再生成；拒绝只扫描 diff | 1、I6、3、5、9、10、12 | | 2026-09-16 | B2：Python producer 扫描器绑定一跳未改名再导出 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2；PR 评审待完成 | 要求每个消费者都从 owner 模块导入（脆弱；M2 中已静默失效）；拒绝无界多跳解析 | 5、附录 A | | 2026-09-16 | B1 改名不变性：新增按名字归组的分歧报告；写明它未闭合的边界 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B1；PR 评审待完成 | 把预算改按值集归组（否决：`CONFIDENCE_LEVELS` 与 `EDGE_CASE_COMPLEXITIES` 共享 `high/low/medium` 而含义不同）；提交名字账本（M0 否决：Q9 已退役提交式清单）。该报告列出仍然存在的分叉；初稿称它能抓住单侧改名，实测证否，故两份镜像按真实行为写明边界 | 9 | +| 2026-09-17 | B2：绑定同模块调用结果、局部变量有序重绑定与按键精确的容器写入；对 TypeScript 残量做重新归类而非缩减 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2；PR 评审待完成 | 绑定跨模块调用与对象字段（拒绝：那是另一条有自己影响面的有界形式，不属于本切片）；把与字段同名的关键字算作生产（拒绝：会使该义务变成同义反复，见第 5 节）；保留 `typescript_dynamic` 作为单一兜底（拒绝：八个位点共用一个原因，残量无法被行动）；把被调方形参绑定到调用点实参（拒绝：结果会依赖调用方而无法记忆化，且一次错误绑定会凭空造出证据） | 5、9、附录 A | | 2026-09-17 | B5（可选项）：只对声明了 `literal_scan.field` 的词表按位点报告消费者角色，并写明未知占比，而不是把一切都分类 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B5；PR 评审待完成 | 没有声明槽位时回退到词表 id（评审中否决：这把 26 个词表中的 25 个按一个谁也没声明过的字段名去分析，下游每一行都继承了该猜测；它们现在列在 `missing_slot_identity` 下，不做分析）；在注册表中登记消费者（否决：跟踪 issue 明令禁止全量消费者登记，且一份声明清单是主张而非证据）；把该报告做成合并闸门（否决：F3 属于参考层级，90.2% 的未知占比也不足以闸住任何东西）；只报告语法能解析的位点（否决：这会让表格读起来像是完整的，因此未识别的提及与计算键读取都作为带原因的行打印出来） | 9、附录 A、附录 B、附录 C | | 2026-09-17 | B0：为 I2/I11-I14 与各强制层级分别陈述 schema 校验、实施阶段、证据状态与阻断行为；要求每个形式不变量 ID 恰好出现一次 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B0；PR 评审待完成 | 把 `blocking_next` 层级改名以匹配其行为（否决：层级名字表示拥有该检查的里程碑，改名会丢掉这层含义，并从另一个方向把两种读法重新合并）；在 `formal_model` 中加一个 `blocks_today` 布尔字段（否决：那只会多出一个可被读者误当作度量的声明字段，而该事实是 smoke `main()` 的性质，任何注册表修改都改不了它）；保留原注解、只在账本里记一笔缺口（否决：评审者引用的正是那句注解） | 2、5、11、附录 A、附录 B | | 2026-09-17 | 将 F1/F2 限定在 kernel 层与扫描范围，把 F4 重述为作用域枚举完备性，并给每条义务加上可推导的 `domain` | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447)；**需要内核维护者批准，尚未获得** | 保留无条件表述、只在正文记一笔缺口（否决：该表述比 `validate_production` 自己的 docstring 还强）；把 F4 重述为各上下文值集互斥（否决：会被仓库自身数据推翻，`scope_declarations` 恰恰就是为了允许合理的同名复用）；扒宽扫描让无条件声明成立（否决：那是自带风险的另一个变更） | 5、9、附录 B、附录 C | diff --git a/loopx/semantics/production.py b/loopx/semantics/production.py index 0b12cbaa61..97eedc2c7a 100644 --- a/loopx/semantics/production.py +++ b/loopx/semantics/production.py @@ -126,8 +126,10 @@ def _typescript_scan( and isinstance(error.get('line'), int) and error['line'] > 0): raise ValueError(f"{error['path']}:{error['line']}: invalid TypeScript source; repair syntax before semantic scanning") raise ValueError('TypeScript production parser failed; run npm ci --ignore-scripts and check the Node runtime') + # The parser names the same blocker vocabulary as the Python scanner; + # ``typescript_dynamic`` stays the fallback for a form it cannot classify. rows.extend(Production(r['site'], r['line'], r['form'], frozenset(r['values']), r['unresolved'], - 'typescript_dynamic' if r['unresolved'] else None) + (r.get('blocker') or 'typescript_dynamic') if r['unresolved'] else None) for r in json.loads(completed.stdout)) return rows diff --git a/loopx/semantics/python_production.py b/loopx/semantics/python_production.py index 3bcf1be656..6a3d0a0af5 100644 --- a/loopx/semantics/python_production.py +++ b/loopx/semantics/python_production.py @@ -9,6 +9,7 @@ import ast from collections import Counter from dataclasses import dataclass +from types import SimpleNamespace from typing import Mapping, TypeVar from .inventory import SourceFile @@ -25,13 +26,15 @@ class Production: """Why the unknown portion stayed unknown; ``None`` when fully resolved. ``argument_name_only`` a field-named keyword argument, which never proves an - output role. ``unstable_local`` a parameter, reassignment or shadowed name. - ``call_result`` the value comes back from a call. ``dynamic_key`` a computed - or non-literal subscript. ``serialized_value`` a string where an enum object - was required. ``annotation_only`` a bare annotation that declares the field - without a value. ``attribute_read`` an attribute of an unresolved object. - ``typescript_dynamic`` the TypeScript scanner could not resolve the write. - ``other`` anything else; it keeps the site visible. + output role. ``unstable_local`` a parameter, an unordered rebinding or a + shadowed name. ``call_result`` the value comes back from a call this scan + cannot bind. ``dynamic_key`` a computed or non-literal subscript. + ``serialized_value`` a string where an enum object was required. + ``annotation_only`` a bare annotation that declares the field without a + value. ``attribute_read`` an attribute of an unresolved object. + ``typescript_dynamic`` a TypeScript form the parser cannot classify further; + every other label is shared by both runtimes, so one residue taxonomy + covers them. ``other`` anything else; it keeps the site visible. """ @@ -227,6 +230,94 @@ def _qualified_bindings(source: SourceFile, tree: ast.Module, owners: Mapping[st return bindings +def _is_generator(node: ast.FunctionDef) -> bool: + """Whether this ``def`` itself yields, not whether it contains one that does. + + ``ast.walk`` descends into nested functions and lambdas, so a plain function + that merely defines a generator inside itself read as a generator and lost + its binding. The direction was safe -- the call kept ``call_result`` -- but + it withheld evidence this slice exists to make actionable, and it did not + match what the docstring says is excluded. A nested scope owns its own + yields, so the walk stops at one. + """ + pending: list[ast.AST] = list(node.body) + while pending: + current = pending.pop() + if isinstance(current, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)): + continue + if isinstance(current, (ast.Yield, ast.YieldFrom)): + return True + pending.extend(ast.iter_child_nodes(current)) + return False + + +# Keyed the same way ``_TREES`` is, by path and text hash, because the table is +# derived from the module body alone and is the same for every vocabulary +# scanned over that file. Keying it on ``id(tree)`` made its correctness depend +# on ``_TREES`` never evicting: give that cache a bound and a reused id would +# hand back another file's functions, binding a call to the wrong callee with +# no symptom. +_MODULE_FUNCTIONS: dict[tuple[str, int], dict[str, ast.FunctionDef]] = {} + + +def _module_functions(source: SourceFile, tree: ast.Module) -> dict[str, ast.FunctionDef]: + """Top-level plain ``def``s a same-module call may be bound to. + + A decorator can replace the returned object, ``async def`` hands back a + coroutine rather than the value, and a generator yields instead of + returning, so none of those is a recognized producer. Any second top-level + binding of the name -- a redefinition, class, import, assignment or + ``del`` -- leaves the name unproven and the call keeps ``call_result``. + """ + key = (source.path, hash(source.text)) + known = _MODULE_FUNCTIONS.get(key) + if known is not None: + return known + bound: Counter[str] = Counter() + defined: dict[str, ast.FunctionDef] = {} + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + bound[node.name] += 1 + if isinstance(node, ast.FunctionDef): + defined[node.name] = node + continue + for child in ast.walk(node): + if isinstance(child, ast.Name) and isinstance(child.ctx, (ast.Store, ast.Del)): + bound[child.id] += 1 + elif isinstance(child, (ast.Import, ast.ImportFrom)): + for alias in child.names: + bound[alias.asname or alias.name.split('.')[0]] += 1 + elif isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + bound[child.name] += 1 + functions = {name: node for name, node in defined.items() + if bound[name] == 1 and not node.decorator_list and not _is_generator(node)} + _MODULE_FUNCTIONS[key] = functions + return functions + + +def _index_value(node: ast.AST) -> str | int | None: + if isinstance(node, ast.Constant) and type(node.value) in (str, int): + return node.value + if (isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub) + and isinstance(node.operand, ast.Constant) and type(node.operand.value) is int): + return -node.operand.value + return None + + +def _union(values: list[ast.AST]) -> ast.AST: + """Fold several definitions of one local into a finite selection node. + + The scan reports syntactic result possibilities, so a name written more than + once carries the union of the writes that precede the read. The synthetic + test is never inspected; only the arms are resolved. + """ + node = values[0] + for other in values[1:]: + test = ast.copy_location(ast.Constant(value=True), other) + node = ast.copy_location(ast.IfExp(test=test, body=node, orelse=other), other) + return node + + def scan_python_production( source: SourceFile, *, @@ -244,15 +335,41 @@ def scan_python_production( ``modules`` additionally lets one unrenamed re-export hop through a tracked module bind the owner. Longer chains and renamed re-exports stay unknown. Local aliases and complete branch selections resolve only at output sites. - General reassignment and parameter shadowing become unknown. Explicit call - metadata names only reviewed builder arguments; arbitrary calls are consumers. - Nested function returns belong to that function, not a registered enclosure. + Explicit call metadata names only reviewed builder arguments; arbitrary calls + are consumers. Nested function returns belong to that function, not a + registered enclosure. + + Three bounded local forms are recognized beyond a single straight-line + binding. A local written more than once resolves to the union of the writes + that textually precede the read, provided every store of that name is a + plain ``name = expression`` (loop, ``with``, ``except``, walrus, augmented, + unpacking, ``global`` and ``del`` rebindings are not ordered by this scan and + stay unknown) **and** no write shares an enclosing loop with the read. + Textual position is execution order only where no back edge crosses it: a + write later in a loop body reaches the read at the top of the next + iteration, so such a name is not a finite selection and stays unknown. A + local container mutated only through direct literal-key subscript writes + keeps its untouched keys, and a written key carries the union of its + initializer and every write; an alias, a method call, a deeper or computed + store, a negative index (which names a slot whose number depends on the + container's length), or passing the container to any call still discards it. + A call to an undecorated, non-generator, plainly-defined top-level function + of the same module resolves to the union of that function's own returns. + Arguments are never bound to parameters, so a returned parameter stays + unknown and the result is independent of the call site; recursion, imported + and attribute calls keep the ``call_result`` blocker. """ - tree = ast.parse(source.text, filename=source.path) + # The scan never mutates the tree, so one parse per file serves every + # vocabulary; synthetic selection nodes are built fresh, never spliced in. + tree = _parsed(source) bindings = _qualified_bindings(source, tree, enums, modules) call_arguments = call_arguments or {} calls = _qualified_bindings(source, tree, call_arguments, modules) return_paths = return_paths or {} + module_functions = _module_functions(source, tree) + call_memo: dict[tuple[str, str], tuple[frozenset[str], tuple[str, ...]]] = {} + environments: dict[tuple[int, str], SimpleNamespace] = {} + resolving: set[str] = set() result: list[Production] = [] @@ -264,19 +381,32 @@ def matches(node: ast.AST) -> bool: return (isinstance(node, ast.Subscript) and isinstance(node.slice, ast.Constant) and node.slice.value == field) - def scan_scope(body: list[ast.stmt], scope: str, parameters: set[str]) -> None: + def environment(body: list[ast.stmt], scope: str, parameters: set[str], + call_shadows: frozenset[str] = frozenset(), + own: frozenset[str] = frozenset()) -> SimpleNamespace: + """Build one scope's bounded local view and its resolvers, once.""" + cached = environments.get((id(body), scope)) + if cached is not None: + return cached nodes: list[ast.AST] = [] nested: list[ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef] = [] + # Which loops enclose each node. A write inside a loop reaches a read in + # the same loop through the back edge, so their textual order says + # nothing about which value the read sees. + enclosing_loops: dict[int, frozenset[int]] = {} - def collect(node: ast.AST) -> None: + def collect(node: ast.AST, loops: frozenset[int] = frozenset()) -> None: if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): nested.append(node) return if isinstance(node, ast.Lambda): return nodes.append(node) + enclosing_loops[id(node)] = loops + if isinstance(node, (ast.For, ast.AsyncFor, ast.While)): + loops = loops | {id(node)} for child in ast.iter_child_nodes(node): - collect(child) + collect(child, loops) for statement in body: collect(statement) assigned = Counter(n.id for n in nodes if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)) @@ -293,50 +423,61 @@ def collect(node: ast.AST) -> None: exception_targets = {n.name for n in nodes if isinstance(n, ast.ExceptHandler) and n.name} deleted = {n.id for n in nodes if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Del)} shadows = set(assigned) | parameters | nested_names | imported | exception_targets | deleted + # A same-module call binds to a top-level ``def``, so that definition is + # not itself a shadow; only a rebinding inside this scope or an + # enclosing one takes the name away from the module function. + # ``parameters`` also carries every enclosing owner shadow, including the + # module's own top-level definitions, so only this scope's real bindings + # may take a name away from the module function it would otherwise name. + rebinds = set(assigned) | set(own) | imported | exception_targets | deleted + if scope != '': + rebinds |= nested_names + calls_shadowed = frozenset(call_shadows) | rebinds local_bindings = {k: v for k, v in bindings.items() if k not in shadows} local_calls = {k: v for k, v in calls.items() if k not in shadows} - single_values = {} - for node in nodes: - if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name): - target = node.targets[0].id - if assigned[target] == 1 and target not in parameters: - single_values[target] = node.value - elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name): - target = node.target.id - if assigned[target] == 1 and target not in parameters and node.value is not None: - single_values[target] = node.value - - def conditional_values(node: ast.If) -> dict[str, tuple[ast.AST, int]]: - # A complete if/elif/else defining a local in each arm is one finite - # selection. Partial branches, loops and general reassignments stay - # unknown; no assignment is itself an enum production site. - def arm(statements: list[ast.stmt]) -> dict[str, tuple[ast.AST, int]]: - if len(statements) == 1 and isinstance(statements[0], ast.If): - return conditional_values(statements[0]) - definitions = {} - for statement in statements: - if (isinstance(statement, ast.Assign) and len(statement.targets) == 1 - and isinstance(statement.targets[0], ast.Name)): - name = statement.targets[0].id - definitions[name] = (statement.value, definitions.get(name, (None, 0))[1] + 1) - return {name: item for name, item in definitions.items() if item[1] == 1} - left, right = arm(node.body), arm(node.orelse) - return {name: (ast.copy_location(ast.IfExp(test=node.test, body=left[name][0], - orelse=right[name][0]), node), left[name][1] + right[name][1]) - for name in left.keys() & right.keys()} + plain: dict[str, list[ast.AST]] = {} + for node in nodes: + target = value = None + if (isinstance(node, ast.Assign) and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name)): + target, value = node.targets[0].id, node.value + elif (isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) + and node.value is not None): + target, value = node.target.id, node.value + if target is not None: + plain.setdefault(target, []).append(value) + declared = {name for node in nodes if isinstance(node, (ast.Global, ast.Nonlocal)) + for name in node.names} + # Every store of the name must be one of those plain writes, so a value + # this scan cannot order never masquerades as a finite selection. + definitions = {name: values for name, values in plain.items() + if assigned[name] == len(values) and name not in parameters + and name not in declared and name not in deleted} + + # Resolve only local containers that have not been mutated through an + # unrecognized path or escaped. A direct literal-key subscript write is + # recorded against that key; anything else invalidates every alias, + # rather than turning a stale initializer into false scalar evidence. + written: dict[str, dict[str | int, list[ast.AST]]] = {} + recorded: set[int] = set() for node in nodes: - if isinstance(node, ast.If): - for name, (value, count) in conditional_values(node).items(): - if assigned[name] == count and name not in parameters: - single_values[name] = value - - # Resolve only local containers that have not been mutated or escaped. - # A subscript write through an alias invalidates every alias, rather - # than turning a stale initializer into false scalar output evidence. - containers = {name for name, value in single_values.items() - if isinstance(value, (ast.List, ast.Dict, ast.Set))} - aliases = [(name, value.id) for name, value in single_values.items() if isinstance(value, ast.Name)] + if isinstance(node, ast.Assign) and len(node.targets) == 1: + store = node.targets[0] + # A negative index names the same slot as a non-negative one + # whose number depends on the container's length, so it cannot + # be recorded against a key. Leaving it unrecorded sends the + # container down the existing invalidation path below. + if (isinstance(store, ast.Subscript) and isinstance(store.value, ast.Name) + and not isinstance(store.slice, ast.Slice) + and (key := _index_value(store.slice)) is not None + and not (type(key) is int and key < 0)): + recorded.add(id(store)) + written.setdefault(store.value.id, {}).setdefault(key, []).append(node.value) + containers = {name for name, values in definitions.items() + if any(isinstance(value, (ast.List, ast.Dict, ast.Set)) for value in values)} + aliases = [(name, value.id) for name, values in definitions.items() + for value in values if isinstance(value, ast.Name)] unsafe: set[str] = set() def root_name(node: ast.AST) -> str | None: @@ -346,6 +487,8 @@ def root_name(node: ast.AST) -> str | None: for node in nodes: if isinstance(node, (ast.Attribute, ast.Subscript)) and isinstance(node.ctx, (ast.Store, ast.Del)): + if id(node) in recorded: + continue if name := root_name(node): unsafe.add(name) elif isinstance(node, ast.Call): @@ -354,6 +497,9 @@ def root_name(node: ast.AST) -> str | None: for argument in [*node.args, *(kw.value for kw in node.keywords)]: if isinstance(argument, ast.Name): unsafe.add(argument.id) + # A second name for the same container would let a write land outside + # this key map, so an aliased container keeps no key-precise evidence. + unsafe.update(name for name in written if name in {n for pair in aliases for n in pair}) for group in (containers, unsafe): changed = True while changed: @@ -363,46 +509,106 @@ def root_name(node: ast.AST) -> str | None: group.update((left, right)) changed = len(group) != before for name in containers & unsafe: - single_values.pop(name, None) + definitions.pop(name, None) + for name in unsafe: + written.pop(name, None) + + blockers: list[str] = [] + + def blocked(label: str) -> bool: + blockers.append(label) + return True def bound(node: ast.AST | None, seen: frozenset[str]) -> tuple[ast.AST | None, frozenset[str]]: - while isinstance(node, ast.Name) and node.id in single_values and node.id not in seen: - definition = single_values[node.id] - if (definition.lineno, definition.col_offset) >= (node.lineno, node.col_offset): + while isinstance(node, ast.Name) and node.id in definitions and node.id not in seen: + writes = definitions[node.id] + # Textual position is execution order only where no back edge + # crosses it. When a second write shares a loop with the read, + # the next iteration sees that write and the preceding-writes + # filter would drop a live value, so the name is not a finite + # selection and stays unknown. + if len(writes) > 1 and any( + enclosing_loops.get(id(value), frozenset()) + & enclosing_loops.get(id(node), frozenset()) + for value in writes + ): + break + values = [value for value in writes + if (value.lineno, value.col_offset) < (node.lineno, node.col_offset)] + if not values: break seen = seen | {node.id} - node = definition + node = values[0] if len(values) == 1 else _union(values) return node, seen - def index_value(node: ast.AST) -> str | int | None: - if isinstance(node, ast.Constant) and type(node.value) in (str, int): - return node.value - if (isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub) - and isinstance(node.operand, ast.Constant) and type(node.operand.value) is int): - return -node.operand.value - return None - - def lookup(container: ast.AST | None, key: str | int | None) -> tuple[list[ast.AST], bool]: + def flatten(container: ast.Dict, seen: frozenset[str], + depth: int = 0) -> tuple[list[tuple[str | int, ast.AST]], bool] | None: + """Expand ``**`` spreads of statically known dict literals, in write order. + + A spread whose operand is not a finite selection of dict literals + with literal keys could overwrite any key, so the whole lookup falls + back to the unknown-key answer instead of trusting a literal entry. + """ + if depth > 4: + return None + pairs: list[tuple[str | int, ast.AST]] = [] + spread = False + for key, value in zip(container.keys, container.values, strict=True): + if key is not None: + index = _index_value(key) + if index is None: + return None + pairs.append((index, value)) + continue + spread = True + arms = [value] + while arms: + arm, visited = bound(arms.pop(), seen) + if isinstance(arm, ast.IfExp): + arms.extend((arm.body, arm.orelse)) + continue + if not isinstance(arm, ast.Dict): + return None + inner = flatten(arm, visited, depth + 1) + if inner is None: + return None + pairs.extend(inner[0]) + return pairs, spread + + def lookup(container: ast.AST | None, key: str | int | None, + seen: frozenset[str] = frozenset(), absent_ok: bool = False) -> tuple[list[ast.AST], bool]: + # ``absent_ok`` says a recorded write already supplies this key, so an + # initializer that does not carry it is not an unknown boundary. if isinstance(container, (ast.Tuple, ast.List)): if type(key) is int: - return ([container.elts[key]], False) if -len(container.elts) <= key < len(container.elts) else ([], blocked('dynamic_key')) + if -len(container.elts) <= key < len(container.elts): + return [container.elts[key]], False + return [], (False if absent_ok else blocked('dynamic_key')) if key is not None: return [], blocked('dynamic_key') return list(container.elts), blocked('dynamic_key') if isinstance(container, ast.Dict): - keys = [index_value(k) if k is not None else None for k in container.keys] - if key is not None and all(k is not None for k in keys): - # Python dict construction keeps the last duplicate key. - found = [v for k, v in zip(keys, container.values, strict=True) if k == key] - return ([found[-1]], False) if found else ([], blocked('dynamic_key')) - return list(container.values), blocked('dynamic_key') + expanded = flatten(container, seen) + if expanded is None: + return list(container.values), blocked('dynamic_key') + pairs, spread = expanded + if key is None: + return [value for _, value in pairs], blocked('dynamic_key') + found = [value for index, value in pairs if index == key] + if not found: + return [], (False if absent_ok else blocked('dynamic_key')) + # Python dict construction keeps the last duplicate key; an + # optional spread makes each contributor a live possibility. + return (found if spread else [found[-1]]), False return [], blocked('unstable_local' if isinstance(container, ast.Name) else 'other') - blockers: list[str] = [] - - def blocked(label: str) -> bool: - blockers.append(label) - return True + def element(root: str | None, container: ast.AST | None, key: str | int | None, + seen: frozenset[str] = frozenset()) -> tuple[list[ast.AST], bool]: + updates = written.get(root or '') + extra = [] if not updates else (updates.get(key, []) if key is not None + else [v for values in updates.values() for v in values]) + choices, unknown = lookup(container, key, seen, absent_ok=bool(extra)) + return [*choices, *extra], unknown def resolve(node: ast.AST | None, seen: frozenset[str] = frozenset(), *, enum_only: bool = False) -> tuple[set[str], bool]: node, seen = bound(node, seen) @@ -414,8 +620,9 @@ def resolve(node: ast.AST | None, seen: frozenset[str] = frozenset(), *, enum_on if isinstance(node.slice, ast.Slice) or (isinstance(node.slice, ast.Constant) and type(node.slice.value) not in (str, int)): return set(), blocked('dynamic_key') + root = node.value.id if isinstance(node.value, ast.Name) else None container, visited = bound(node.value, seen) - choices, unknown = lookup(container, index_value(node.slice)) + choices, unknown = element(root, container, _index_value(node.slice), visited) known: set[str] = set() for value in choices: part, unresolved = resolve(value, visited, enum_only=enum_only) @@ -442,13 +649,25 @@ def resolve(node: ast.AST | None, seen: frozenset[str] = frozenset(), *, enum_on return enum_object_value(node.value, seen) return set(), blocked('attribute_read') if isinstance(node, ast.Call): - return set(), blocked('call_result') + hit = same_module_call(node, 'enum' if enum_only else 'value') + return hit if hit is not None else (set(), blocked('call_result')) if isinstance(node, ast.Name): return set(), blocked('unstable_local') if node is None: return set(), blocked('other') return set(), blocked('other') + def same_module_call(node: ast.Call, mode: str) -> tuple[set[str], bool] | None: + if not isinstance(node.func, ast.Name) or node.func.id in calls_shadowed: + return None + hit = call_values(node.func.id, mode) + if hit is None: + return None + values, reasons = hit + for reason in reasons: + blocked(reason) + return set(values), bool(reasons) + def enum_object_value(node: ast.AST, seen: frozenset[str]) -> tuple[set[str], bool]: node, seen = bound(node, seen) if isinstance(node, ast.IfExp): @@ -456,6 +675,13 @@ def enum_object_value(node: ast.AST, seen: frozenset[str]) -> tuple[set[str], bo return set().union(*(v for v, _ in parts)), any(u for _, u in parts) if isinstance(node, ast.Attribute) and isinstance(node.value, ast.Name) and node.value.id in local_bindings: return resolve(node, seen, enum_only=True) + if isinstance(node, ast.Call): + # Only a same-module function that returns owner members as enum + # objects has another ``.value``; a function handing back + # ``Action.RUN.value`` already returns the serialized string. + hit = same_module_call(node, 'object') + if hit is not None and hit[0]: + return hit # A serialized string (including Action.RUN.value) is not an enum # object with another .value attribute. return set(), blocked('serialized_value') @@ -463,15 +689,76 @@ def enum_object_value(node: ast.AST, seen: frozenset[str]) -> tuple[set[str], bo def returned(node: ast.AST | None, path: tuple[str | int, ...], seen: frozenset[str] = frozenset()) -> tuple[set[str], bool]: if not path: return resolve(node, seen) + root = node.id if isinstance(node, ast.Name) else None node, seen = bound(node, seen) - choices, unknown = lookup(node, path[0]) + choices, unknown = element(root, node, path[0], seen) parts = [returned(value, path[1:], seen) for value in choices] return set().union(*(v for v, _ in parts)), unknown or any(u for _, u in parts) + built = SimpleNamespace(nodes=nodes, nested=nested, shadows=shadows, blockers=blockers, + calls_shadowed=calls_shadowed, local_calls=local_calls, + resolve=resolve, returned=returned, enum_object=enum_object_value) + environments[(id(body), scope)] = built + return built + + def call_values(name: str, mode: str) -> tuple[frozenset[str], tuple[str, ...]] | None: + """Union of one same-module function's own returns, or ``None``. + + ``mode`` asks what the caller needs of each return: its written value, + only owner members (``enum``), or the member behind an enum object + (``object``). The last is not the same question as the first: a function + returning ``Action.RUN.value`` hands back a string that has no further + ``.value``, so it answers ``object`` with nothing. + + Call arguments are never bound to parameters, so a returned parameter + stays unknown and the answer does not depend on the call site; it is + memoised per module scan. A call reached from inside its own callee + chain keeps the ``call_result`` blocker instead of unrolling recursion. + A bare ``return`` or a fall-through yields ``None``, which is not a + vocabulary value: it contributes nothing and blocks nothing. + """ + target = module_functions.get(name) + if target is None or name in resolving: + return None + key = (name, mode) + if key not in call_memo: + resolving.add(name) + try: + args = target.args + params = {a.arg for a in (*args.posonlyargs, *args.args, *args.kwonlyargs)} + params.update(a.arg for a in (args.vararg, args.kwarg) if a) + env = environment(target.body, name, params | module_env.shadows, + module_env.calls_shadowed, frozenset(params)) + values: set[str] = set() + unknown = False + start = len(env.blockers) + for node in env.nodes: + if isinstance(node, ast.Return) and node.value is not None: + part, missing = (env.enum_object(node.value, frozenset()) if mode == 'object' + else env.resolve(node.value, enum_only=mode == 'enum')) + values |= part + unknown |= missing + first = env.blockers[start] if len(env.blockers) > start else 'call_result' + del env.blockers[start:] + call_memo[key] = (frozenset(values), (first,) if unknown else ()) + finally: + resolving.discard(name) + return call_memo[key] + + def scan_scope(body: list[ast.stmt], scope: str, parameters: set[str], + call_shadows: frozenset[str] = frozenset(), own: frozenset[str] = frozenset(), + env: SimpleNamespace | None = None) -> None: + env = env if env is not None else environment(body, scope, parameters, call_shadows, own) + blockers = env.blockers + def record(node: ast.AST | None, form: str, location: ast.AST) -> None: - blockers.clear() - values, unknown = (returned(node, return_paths.get(scope, ())) if form == 'return' else resolve(node)) - blocker = blockers[0] if blockers else None + # Reasons are read back by position: one scope's environment is + # shared with same-module call resolution, which may nest inside. + start = len(blockers) + values, unknown = (env.returned(node, return_paths.get(scope, ())) + if form == 'return' else env.resolve(node)) + blocker = blockers[start] if len(blockers) > start else None + del blockers[start:] if node is None and form == 'assignment': # A bare annotation declares the field; there is no value to resolve. blocker = 'annotation_only' @@ -481,7 +768,7 @@ def record(node: ast.AST | None, form: str, location: ast.AST) -> None: result.append(Production(f'{source.path}::{scope}', location.lineno, form, frozenset(values), unknown, blocker if unknown else None)) - for node in nodes: + for node in env.nodes: if isinstance(node, ast.Assign): if field and any(matches(t) for t in node.targets): record(node.value, 'assignment', node) @@ -492,7 +779,7 @@ def record(node: ast.AST | None, form: str, location: ast.AST) -> None: if isinstance(key, ast.Constant) and key.value == field: record(value, 'dict', node) elif isinstance(node, ast.Call): - output_arguments = local_calls.get(node.func.id, {}) if isinstance(node.func, ast.Name) else {} + output_arguments = env.local_calls.get(node.func.id, {}) if isinstance(node.func, ast.Name) else {} for kw in node.keywords: if kw.arg in output_arguments: record(kw.value, 'call_argument', node) @@ -505,10 +792,16 @@ def record(node: ast.AST | None, form: str, location: ast.AST) -> None: if scope in return_functions: record(node.value, 'return', node) else: - values, unknown = resolve(node.value, enum_only=True) + start = len(blockers) + values, unknown = env.resolve(node.value, enum_only=True) + # An owner-member result keeps its reason too; an unlabelled + # unknown would be invisible in the report breakdown. + reason = blockers[start] if len(blockers) > start else None + del blockers[start:] if values: - result.append(Production(f'{source.path}::{scope}', node.lineno, 'enum_result', frozenset(values), unknown)) - for child in nested: + result.append(Production(f'{source.path}::{scope}', node.lineno, 'enum_result', + frozenset(values), unknown, reason if unknown else None)) + for child in env.nested: name = child.name if scope == '' else f'{scope}.{child.name}' params: set[str] = set() if not isinstance(child, ast.ClassDef): @@ -516,7 +809,8 @@ def record(node: ast.AST | None, form: str, location: ast.AST) -> None: params = {a.arg for a in (*args.posonlyargs, *args.args, *args.kwonlyargs)} params.update(a.arg for a in (args.vararg, args.kwarg) if a) # A nested closure might shadow an owner in any enclosing scope. - scan_scope(child.body, name, params | shadows) + scan_scope(child.body, name, params | env.shadows, env.calls_shadowed, frozenset(params)) - scan_scope(tree.body, '', set()) + module_env = environment(tree.body, '', set()) + scan_scope(tree.body, '', set(), frozenset(), frozenset(), module_env) return sorted(set(result), key=lambda row: (row.site, row.line, row.form, sorted(row.values))) diff --git a/scripts/semantic_production_scan.mjs b/scripts/semantic_production_scan.mjs index d1a6b0efec..b82c8f1775 100644 --- a/scripts/semantic_production_scan.mjs +++ b/scripts/semantic_production_scan.mjs @@ -16,19 +16,48 @@ for (const source of request.sources) { const field = request.field; const returns = new Set((request.return_functions ?? []).filter(x => x.startsWith(`${source.path}::`))); const unwrap = node => { - while (node && (ts.isParenthesizedExpression(node) || ts.isAsExpression(node) || ts.isSatisfiesExpression(node))) node = node.expression; + while (node && (ts.isParenthesizedExpression(node) || ts.isAsExpression(node) || + ts.isSatisfiesExpression(node) || ts.isNonNullExpression(node))) node = node.expression; return node; }; + // Say why a write stayed unknown using the same labels as the Python scanner, + // so one residue taxonomy covers both runtimes instead of a single catch-all. + const blockerFor = node => { + if (!node) return 'other'; + if (ts.isPropertyAccessExpression(node) || ts.isElementAccessExpression(node)) return 'attribute_read'; + if (ts.isCallExpression(node) || ts.isNewExpression(node) || ts.isAwaitExpression(node)) return 'call_result'; + if (ts.isIdentifier(node)) return 'unstable_local'; + // `other`, not `dynamic_key`: the shared vocabulary defines `dynamic_key` + // as a computed or non-literal subscript, and none of these is one. Python + // answers `other` for the same shapes -- a dict literal or an f-string + // where a scalar was required -- so the two runtimes agree on the label. + if (ts.isObjectLiteralExpression(node) || ts.isArrayLiteralExpression(node) || + ts.isTemplateExpression(node)) return 'other'; + return 'typescript_dynamic'; + }; + const merge = parts => ({ + values: [...new Set(parts.flatMap(part => part.values))].sort(), + unresolved: parts.some(part => part.unresolved), + blocker: parts.find(part => part.unresolved)?.blocker, + }); const values = expression => { const node = unwrap(expression); - if (!node) return {values: [], unresolved: true}; + if (!node) return {values: [], unresolved: true, blocker: 'other'}; if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) return {values: node.text ? [node.text] : [], unresolved: false}; if (node.kind === ts.SyntaxKind.NullKeyword) return {values: [], unresolved: false}; - if (ts.isConditionalExpression(node)) { - const left = values(node.whenTrue), right = values(node.whenFalse); - return {values: [...new Set([...left.values, ...right.values])].sort(), unresolved: left.unresolved || right.unresolved}; + // ``undefined`` carries no value and is not an unknown, matching the way + // the Python scanner treats an explicit ``None``. + if (ts.isIdentifier(node) && node.text === 'undefined') return {values: [], unresolved: false}; + if (ts.isConditionalExpression(node)) return merge([values(node.whenTrue), values(node.whenFalse)]); + // ``a || b`` and ``a ?? b`` are a finite selection, exactly like the + // Python scanner's BoolOp arms; ``String(x)`` is a transparent wrapper. + if (ts.isBinaryExpression(node) && [ts.SyntaxKind.BarBarToken, + ts.SyntaxKind.QuestionQuestionToken].includes(node.operatorToken.kind)) { + return merge([values(node.left), values(node.right)]); } - return {values: [], unresolved: true}; + if (ts.isCallExpression(node) && ts.isIdentifier(node.expression) && + node.expression.text === 'String' && node.arguments.length === 1) return values(node.arguments[0]); + return {values: [], unresolved: true, blocker: blockerFor(node)}; }; const staticName = expression => { const node = unwrap(expression); diff --git a/tests/architecture/test_semantic_producer_binding.py b/tests/architecture/test_semantic_producer_binding.py new file mode 100644 index 0000000000..1529bea8ef --- /dev/null +++ b/tests/architecture/test_semantic_producer_binding.py @@ -0,0 +1,454 @@ +"""Counterexamples for the bounded binding forms the producer scan recognizes. + +Each form here narrows an ``unresolved`` blocker, so every one needs a negative +twin: a site the scan must keep unresolved rather than call dead. Shrinking the +residue by loosening the rule is the failure this file is meant to catch. +""" +from __future__ import annotations + +import ast +import time + +import pytest + +from loopx.semantics.inventory import SourceFile +from loopx.semantics.python_production import scan_python_production + +OWNER = 'loopx/quota/owner.py::Action' +ENUMS = {OWNER: {'RUN': 'run', 'WAIT': 'wait'}} +CONSUMER = 'loopx/quota/client.py' + + +def scan(text, *, returns=(), paths=None, calls=None, field='action', enums=ENUMS): + return scan_python_production(SourceFile(CONSUMER, '.py', text), field=field, enums=enums, + return_functions=frozenset(returns), return_paths=paths, + call_arguments=calls) + + +def known(rows): + return set().union(*(row.values for row in rows if row.form != 'keyword_unproved')) + + +def blockers(rows): + return {row.blocker for row in rows if row.unresolved} + + +def at(rows, scope): + """Only the consumer's own rows; a helper's own returns are its evidence.""" + return [row for row in rows if row.site == f'{CONSUMER}::{scope}'] + + +# --- same-module call results ------------------------------------------------- + + +def test_same_module_call_resolves_to_the_callee_own_returns(): + rows = scan('def pick(flag):\n return "run" if flag else "wait"\n' + 'def emit():\n return {"action": pick(True)}\n') + assert known(rows) == {'run', 'wait'} + assert not any(row.unresolved for row in rows) + + +def test_same_module_call_keeps_the_callee_unknown_portion(): + """A bound call reports the callee's own reason, not a blanket call_result.""" + rows = scan('def pick(flag):\n return "run" if flag else dynamic()\n' + 'def emit():\n return {"action": pick(True)}\n') + assert known(rows) == {'run'} + assert blockers(rows) == {'call_result'} + + +def test_call_arguments_are_never_bound_to_callee_parameters(): + """A returned parameter stays unknown however literal the argument is.""" + rows = scan('def pick(choice):\n return choice\n' + 'def emit():\n return {"action": pick("run")}\n') + assert known(rows) == set() + assert blockers(rows) == {'unstable_local'} + + +def test_enum_object_returned_by_a_same_module_call_supplies_value(): + rows = scan('from .owner import Action\ndef pick():\n return Action.RUN\n' + 'def emit():\n choice = pick()\n return {"action": choice.value}\n') + assert known(rows) == {'run'} + assert not any(row.unresolved for row in rows) + + +def test_serialized_result_is_not_an_enum_object_with_a_value_attribute(): + rows = at(scan('from .owner import Action\ndef pick():\n return Action.RUN.value\n' + 'def emit():\n choice = pick()\n return {"action": choice.value}\n'), 'emit') + assert known(rows) == set() + assert blockers(rows) == {'serialized_value'} + + +@pytest.mark.parametrize('definition, reason', [ + # A decorator can replace the returned object entirely. + ('@wrap\ndef pick():\n return "run"\n', 'call_result'), + # await of a coroutine is a different expression; a bare call is not the value. + ('async def pick():\n return "run"\n', 'call_result'), + # A generator yields; the call returns the iterator, not a member. + ('def pick():\n yield "run"\n', 'call_result'), + # A second top-level binding takes the name away from the definition. + ('def pick():\n return "run"\npick = other\n', 'call_result'), + ('def pick():\n return "run"\nfrom .elsewhere import pick\n', 'call_result'), +]) +def test_unbindable_definitions_keep_the_site_unresolved(definition, reason): + rows = scan(definition + 'def emit():\n return {"action": pick()}\n') + assert known(rows) == set() + assert blockers(rows) == {reason} + + +def test_imported_and_attribute_calls_are_not_same_module_definitions(): + rows = scan('from .elsewhere import pick\nimport helper\n' + 'def emit():\n return {"action": pick(), "other": helper.pick()}\n') + assert known(rows) == set() + assert blockers(rows) == {'call_result'} + + +def test_locally_rebound_name_does_not_borrow_the_module_definition(): + rows = scan('def pick():\n return "run"\n' + 'def emit(pick):\n return {"action": pick()}\n') + assert known(rows) == set() + assert blockers(rows) == {'call_result'} + + +def test_recursive_call_chain_terminates_and_stays_unresolved(): + rows = scan('def left():\n return right()\ndef right():\n return left()\n' + 'def emit():\n return {"action": left()}\n') + assert known(rows) == set() + assert blockers(rows) == {'call_result'} + + +def test_a_callee_that_only_raises_produces_no_value_and_no_blocker(): + rows = scan('def pick():\n raise ValueError("no route")\n' + 'def emit():\n return {"action": pick()}\n') + assert known(rows) == set() + assert not any(row.unresolved for row in rows) + + +# --- ordered rebinding of a local -------------------------------------------- + + +def test_rebound_local_unions_the_writes_that_precede_the_read(): + rows = scan('from .owner import Action\ndef emit(flag):\n choice = Action.RUN.value\n' + ' if flag:\n choice = Action.WAIT.value\n return {"action": choice}\n') + assert known(rows) == {'run', 'wait'} + assert not any(row.unresolved for row in rows) + + +def test_a_write_after_the_read_is_not_a_definition_for_that_read(): + rows = scan('from .owner import Action\ndef emit(flag):\n choice = Action.RUN.value\n' + ' packet = {"action": choice}\n choice = Action.WAIT.value\n return packet\n') + assert known(rows) == {'run'} + assert not any(row.unresolved for row in rows) + + +def test_a_rebound_local_keeps_its_unknown_arm_visible(): + rows = scan('from .owner import Action\ndef emit(flag):\n choice = Action.RUN.value\n' + ' if flag:\n choice = dynamic()\n return {"action": choice}\n') + assert known(rows) == {'run'} + assert blockers(rows) == {'call_result'} + + +@pytest.mark.parametrize('rebind', [ + 'for choice in options: pass', + 'with opened() as choice: pass', + 'choice += suffix', + 'choice, extra = pair()', + 'del choice', + 'print(choice := dynamic())', +]) +def test_stores_this_scan_cannot_order_leave_the_local_unknown(rebind): + """Only a plain ``name = expression`` is an ordered write; nothing else is.""" + rows = scan('from .owner import Action\ndef emit():\n choice = Action.RUN.value\n ' + + rebind + '\n return {"action": choice}\n') + assert known(rows) == set() + assert blockers(rows) == {'unstable_local'} + + +def test_a_global_declaration_takes_the_name_out_of_this_scope(): + rows = scan('from .owner import Action\ndef emit():\n global choice\n' + ' choice = Action.RUN.value\n return {"action": choice}\n') + assert known(rows) == set() + assert blockers(rows) == {'unstable_local'} + + +# --- key-precise container writes -------------------------------------------- + + +def test_untouched_keys_survive_a_literal_key_write(): + rows = scan('from .owner import Action\ndef emit(flag):\n' + ' packet = {"action": Action.RUN.value, "note": ""}\n' + ' if flag:\n packet["note"] = "changed"\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run'} + assert not any(row.unresolved for row in rows) + + +def test_a_written_key_carries_every_contributing_write(): + rows = scan('from .owner import Action\ndef emit(flag):\n' + ' packet = {"action": Action.RUN.value}\n' + ' if flag:\n packet["action"] = Action.WAIT.value\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run', 'wait'} + assert not any(row.unresolved for row in rows) + + +def test_a_key_supplied_only_by_a_write_is_not_an_unknown_boundary(): + rows = scan('from .owner import Action\ndef emit():\n packet = {}\n' + ' packet["action"] = Action.RUN.value\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run'} + assert not any(row.unresolved for row in rows) + + +def test_an_unresolved_write_to_the_read_key_stays_unresolved(): + rows = scan('from .owner import Action\ndef emit(flag):\n' + ' packet = {"action": Action.RUN.value}\n' + ' if flag:\n packet["action"] = dynamic()\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run'} + assert blockers(rows) == {'call_result'} + + +@pytest.mark.parametrize('mutation', [ + # A second name could carry a write this key map never sees. + 'alias = packet\n alias["action"] = dynamic()', + # A computed key could land on any key at all. + 'packet[key()] = dynamic()', + # A deeper store is not a write to a key of this container. + 'packet["action"]["kind"] = dynamic()', + # A method or an escape can rewrite the whole container. + 'packet.update(other)', + 'consume(packet)', + 'del packet["action"]', +]) +def test_writes_outside_the_recognized_form_discard_the_container(mutation): + rows = [row for row in scan('from .owner import Action\ndef emit():\n' + ' packet = {"action": Action.RUN.value}\n ' + mutation + '\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) if row.form == 'return'] + assert known(rows) == set() + assert rows and all(row.unresolved for row in rows) + + +# --- dict literal spreads ----------------------------------------------------- + + +def test_a_spread_of_known_literals_does_not_hide_a_sibling_key(): + rows = scan('from .owner import Action\ndef emit(flag):\n' + ' packet = {"action": Action.RUN.value, **({"note": "x"} if flag else {})}\n' + ' return packet\n', returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run'} + assert not any(row.unresolved for row in rows) + + +def test_a_spread_that_may_carry_the_key_keeps_both_possibilities(): + rows = scan('from .owner import Action\ndef emit(flag):\n' + ' packet = {"action": Action.RUN.value, **({"action": Action.WAIT.value} if flag else {})}\n' + ' return packet\n', returns=['emit'], paths={'emit': ('action',)}) + assert known(rows) == {'run', 'wait'} + assert not any(row.unresolved for row in rows) + + +@pytest.mark.parametrize('spread', ['**overrides', '**dict(overrides)', '**{key(): "x"}']) +def test_an_unknown_spread_could_overwrite_any_key(spread): + rows = scan('from .owner import Action\ndef emit(overrides):\n' + ' packet = {"action": Action.RUN.value, ' + spread + '}\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert blockers(rows) == {'dynamic_key'} + + +# --- the residue stays honest ------------------------------------------------- + + +def test_an_unbound_site_is_unresolved_and_never_silently_dropped(): + """No recognized producer means unresolved; it never means dead.""" + rows = scan('def emit(payload):\n return {"action": payload.get("action")}\n') + assert len(rows) == 1 + assert rows[0].unresolved and rows[0].blocker == 'call_result' + assert rows[0].values == frozenset() + + +def test_every_recognized_form_still_labels_what_it_could_not_bind(): + rows = scan('from .owner import Action\ndef pick(flag):\n return Action.RUN.value if flag else late()\n' + 'def emit(flag):\n packet = {"action": pick(flag)}\n' + ' packet["note"] = dynamic()\n return packet\n', + returns=['emit'], paths={'emit': ('action',)}) + assert known(at(rows, 'emit')) == {'run'} + assert all(row.blocker for row in rows if row.unresolved) + + +def test_deep_call_chains_stay_bounded(): + """The memo and the recursion guard keep a long chain linear, not explosive.""" + depth = 60 + text = 'def step0():\n return "run"\n' + text += ''.join(f'def step{i}():\n return step{i - 1}() or step{i - 1}()\n' + for i in range(1, depth)) + text += f'def emit():\n return {{"action": step{depth - 1}()}}\n' + started = time.perf_counter() + rows = scan(text) + assert known(rows) == {'run'} + assert time.perf_counter() - started < 5.0 + + +def test_synthetic_selection_nodes_never_enter_the_shared_tree(): + """Rebinding folds a union for the read only; the parsed tree is untouched.""" + text = ('from .owner import Action\ndef emit(flag):\n choice = Action.RUN.value\n' + ' if flag:\n choice = Action.WAIT.value\n return {"action": choice}\n') + source = SourceFile(CONSUMER, '.py', text) + first = scan_python_production(source, field='action', enums=ENUMS) + before = ast.dump(ast.parse(text)) + second = scan_python_production(source, field='action', enums=ENUMS) + assert first == second + assert ast.dump(ast.parse(text)) == before + + +# A write that textually follows a read still reaches it when a loop carries +# control back. The preceding-writes filter reads file position as execution +# order, so without these the scan reports the value that happens to appear +# first and marks the site fully resolved -- the one failure mode that turns an +# unknown into wrong evidence. F1 would then pass over a producer that emits an +# unregistered value on every iteration after the first. +@pytest.mark.parametrize('text', [ + # for-loop back edge: iteration 2 emits 'leaked' + 'def build(rows):\n' + ' chosen = "run"\n' + ' for row in rows:\n' + ' emit({"action": chosen})\n' + ' chosen = "leaked"\n', + # while-loop back edge + 'def build(rows):\n' + ' chosen = "run"\n' + ' while rows:\n' + ' emit({"action": chosen})\n' + ' chosen = "leaked"\n' + ' rows = rows[1:]\n', + # the carrying write sits in the outer loop, the read in the inner one + 'def build(rows):\n' + ' chosen = "run"\n' + ' for row in rows:\n' + ' for inner in row:\n' + ' emit({"action": chosen})\n' + ' chosen = "leaked"\n', + # finally runs after the read and feeds the next iteration + 'def build(rows):\n' + ' chosen = "run"\n' + ' for row in rows:\n' + ' try:\n' + ' emit({"action": chosen})\n' + ' finally:\n' + ' chosen = "leaked"\n', +]) +def test_a_loop_back_edge_leaves_the_local_unordered(text): + rows = [row for row in scan(text) if row.form == 'dict'] + assert rows, 'the dict write must still be observed' + assert all(row.unresolved for row in rows), ( + 'a write the back edge carries past the read makes the writes unorderable; ' + 'reporting only the textually earlier value states a closed value set that is not closed' + ) + assert blockers(rows) == {'unstable_local'} + assert 'leaked' not in known(rows) + + +def test_a_straight_line_rebinding_still_resolves(): + """The back-edge rule must not retract the ordering it was built for. + + Without a loop the preceding-writes filter is execution order, so a local + written twice before the read is still a finite selection. + """ + rows = [row for row in scan( + 'def build(flag):\n' + ' chosen = "run"\n' + ' if flag:\n' + ' chosen = "wait"\n' + ' return {"action": chosen}\n', + ) if row.form == 'dict'] + assert known(rows) == {'run', 'wait'} and not any(row.unresolved for row in rows) + + +def test_a_single_write_inside_a_loop_is_still_its_only_value(): + """One plain store is the only value a read can see, back edge or not.""" + rows = [row for row in scan( + 'def build(rows):\n' + ' for row in rows:\n' + ' chosen = "run"\n' + ' emit({"action": chosen})\n', + ) if row.form == 'dict'] + assert known(rows) == {'run'} and not any(row.unresolved for row in rows) + + +def test_a_negative_index_store_discards_the_container(): + """``table[-1]`` names a slot whose number depends on the length. + + Recording it against the key ``-1`` leaves a read of ``table[0]`` looking at + the untouched initializer, so a one-element list reports the value the write + replaced and calls the site resolved. + """ + rows = [row for row in scan( + 'def build():\n' + ' table = ["run"]\n' + ' table[-1] = "leaked"\n' + ' return {"action": table[0]}\n', + ) if row.form == 'dict'] + assert rows and all(row.unresolved for row in rows) + assert 'run' not in known(rows), 'the initializer was overwritten by the negative store' + + +def test_a_non_negative_index_store_still_carries_its_key(): + """The negative-index rule must not discard the key map it was built on. + + A written key carries the union of its initializer and the write, which is + this scan's documented answer: it reports syntactic possibilities, not the + one value a flow-sensitive reading would pick. What matters here is that the + site stays resolved and the write is visible, both of which the discard path + would have taken away. + """ + rows = [row for row in scan( + 'def build():\n' + ' table = ["run"]\n' + ' table[0] = "wait"\n' + ' return {"action": table[0]}\n', + ) if row.form == 'dict'] + assert known(rows) == {'run', 'wait'} and not any(row.unresolved for row in rows) + + +def test_a_module_scope_back_edge_is_unordered_too(): + """The carve-out must hold outside a function body. + + The scope table is built per scope, so a module-level loop is a separate + path through the same rule and needs its own negative fixture. + """ + rows = [row for row in scan( + 'chosen = "run"\n' + 'for row in rows:\n' + ' emit({"action": chosen})\n' + ' chosen = "leaked"\n', + ) if row.form == 'dict'] + assert rows and all(row.unresolved for row in rows) + assert blockers(rows) == {'unstable_local'} + + +def test_a_nested_generator_does_not_make_its_enclosing_function_one(): + """``ast.walk`` descends into nested defs; a containing function is not a generator. + + The direction was safe -- the call kept ``call_result`` -- but it withheld + evidence this slice exists to make actionable. + """ + rows = at(scan( + 'from loopx.quota.owner import Action\n' + 'def choose():\n' + ' def stream():\n' + ' yield 1\n' + ' return Action.RUN.value\n' + 'def emit():\n' + ' return {"action": choose()}\n', + ), 'emit') + assert known(rows) == {'run'} and not any(row.unresolved for row in rows) + + +def test_a_function_that_yields_itself_is_still_not_bound(): + rows = at(scan( + 'from loopx.quota.owner import Action\n' + 'def choose():\n' + ' yield Action.RUN.value\n' + 'def emit():\n' + ' return {"action": choose()}\n', + ), 'emit') + assert blockers(rows) == {'call_result'} diff --git a/tests/architecture/test_semantic_python_production.py b/tests/architecture/test_semantic_python_production.py index faf6299705..59e41a7d0d 100644 --- a/tests/architecture/test_semantic_python_production.py +++ b/tests/architecture/test_semantic_python_production.py @@ -58,7 +58,14 @@ def test_single_local_variable_and_reassignment_boundary(): rows = scan('def emit(flag):\n code = "run" if flag else "wait"\n return code\n', returns=['emit']) assert known(rows) == {'run', 'wait'} assert not any(r.unresolved for r in rows) + # An ordered rebinding is a finite selection, so the literal arm is real + # evidence; the unknown arm still keeps the row unresolved. + # tests/architecture/test_semantic_producer_binding.py covers the form. rows = scan('def emit(flag):\n code = "run"\n if flag:\n code = dynamic()\n return code\n', returns=['emit']) + assert known(rows) == {'run'} + assert rows[0].unresolved + # A store this scan cannot order erases the local altogether. + rows = scan('def emit(codes):\n code = "run"\n for code in codes:\n pass\n return code\n', returns=['emit']) assert known(rows) == set() assert rows[0].unresolved