Skip to content

feat(capability): add auditable external evidence lifecycle - #4813

Open
huangruiteng wants to merge 12 commits into
mainfrom
codex/external-evidence-capability
Open

huangruiteng wants to merge 12 commits into
mainfrom
codex/external-evidence-capability

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a typed TypeScript owner for the external_evidence_research_v0 request, discovery, plan, receipt-observation, admission, and retirement lifecycle
  • unify host methods such as external-research and connector providers behind one auditable capability boundary
  • add thin Python CLI adapters for loopx external-evidence discover|plan|receipt|admit|retire, plus capability catalog, bilingual RFC, and roadmap registration
  • keep provider execution outside Core; connector-registry presence is inventory only, and a caller-presented receipt is not provider execution attestation
  • bind receipts to a canonical content-addressed plan_id, and bind retirement to the complete normalized admission identity
  • package the TypeScript capability owner in the Python wheel so installed managed runtimes execute the same contract as source checkouts

Product delivery

  • delivered: CLI and managed-Turn Effect handlers use the same typed discovery, planning, receipt-observation, admission, and retirement reducers
  • delivered: semantic mutation of request, provider readiness, execution envelope, disposition, source set, or downstream projection fails closed before an audit claim or retirement decision
  • companion work remains: frontend and Lark projection/readback, plus one real host-method and one connector end-to-end acceptance
  • raw source bodies, credentials, and provider-private state are not persisted in the public record; admission grants no downstream action authority

Validation

  • npm run typecheck:control-plane
  • node --no-warnings --experimental-strip-types --test tests/control_plane_ts/external_evidence_research.test.ts (13 passed)
  • uv run --extra test python -m pytest -q tests/capabilities/test_external_evidence_cli.py tests/control_plane/test_effect_runtime_integration.py::test_runtime_decode_change_rotates_identity_and_starts_replacement (7 passed)
  • uv run --extra test npm run test:control-plane (2202 passed, 18 skipped, 0 failed)
  • uv run --extra test ruff check loopx/capabilities/external_research tests/capabilities/test_external_evidence_cli.py
  • uv run --extra test ruff format --check loopx/capabilities/external_research tests/capabilities/test_external_evidence_cli.py
  • uv run --extra test loopx canary premerge --from-git-diff (19 selected checks passed, 0 manual holds)
  • built and installed the wheel in an isolated environment; external_evidence.ts is present, the Effect semantic probe passed, and doctor --deep reported typescript_effect_runtime_ready=ready
  • git diff --check origin/main...HEAD

Running the control-plane suite outside the source-checkout uv environment resolves subprocesses to the system Python 3.9 and fails on the repository's existing @dataclass(slots=True) usage. The required uv run --extra test source environment uses Python 3.13 and passes the full suite.

@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Windows CI root cause was the managed Effect runtime package boundary, not the contract logic: the runtime staging test copies only loopx/control_plane/**, while the new handler imported a TypeScript owner from loopx/capabilities/**, so the staged child exited before readiness.\n\nFixed at exact head f732a8ce67b76e9aed5e93d128cec16e620bd746 by moving the semantic TS owner into loopx/control_plane/capabilities/external_evidence.ts; the Python capability directory remains the thin CLI/product adapter.\n\nValidated locally:\n- control-plane typecheck passed\n- external-evidence TS suite: 6 passed\n- exact failing runtime-rotation test plus CLI suite: 4 passed\n- diff check passed\n\nFresh GitHub checks are running.

@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Added the missing typed discovery step at exact head ce7d7827de94966a3c8909d925e86a2f3a25ac4a.

What changed:

  • TypeScript now owns external_evidence.discover, normalizes the same provider contract used by planning, and reports empty|inventory_only|ready plus method/connector and readiness counts.
  • The discovery truth contract explicitly keeps registry presence / supported separate from readiness and states that discovery observes neither provider execution nor evidence coverage.
  • CLI now exposes loopx external-evidence discover; bare --connector-registry reads the existing default registry, while observed method/provider inventory may be supplied separately.
  • Managed Turn uses the same Effect handler. Bilingual RFC, catalog, and capability docs now match discover→select→execute→receipt→admit/reject→projection→retire.

Validation:

  • npm run typecheck:control-plane
  • node --no-warnings --experimental-strip-types --test tests/control_plane_ts/external_evidence_research.test.ts (8 passed)
  • uv run --extra test ruff check loopx/capabilities/external_research tests/capabilities/test_external_evidence_cli.py
  • uv run --extra test python -m pytest -q tests/capabilities/test_external_evidence_cli.py tests/control_plane/test_effect_runtime_integration.py::test_runtime_decode_change_rotates_identity_and_starts_replacement (6 passed)
  • source CLI readback: default connector registry projects 17 inventory-only providers, 0 ready, and no execution/evidence-coverage claim.

Product surfaces: CLI and managed Turn changed in this slice. Frontend/Lark remain a companion because there is still no persisted external-evidence configuration/readback owner for those surfaces; adding form state now would create a second source of truth. They should consume this same discovery/plan/admission projection once that owner exists.

@huangruiteng
huangruiteng force-pushed the codex/external-evidence-capability branch from ce7d782 to a99a8db Compare September 20, 2026 15:55
@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main and force-updated exact head to a99a8dbbf55f1638f5d49f6bdb683c428fd13d8f.

Post-rebase validation:

  • npm run typecheck:control-plane
  • TypeScript external-evidence suite: 8 passed
  • Ruff passed
  • Python capability + Effect runtime packaging checks: 6 passed
  • git diff --check origin/main...HEAD

The contract remains unchanged: discovery reports inventory/readiness only and does not claim provider execution or evidence coverage. Fresh CI is running.

@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Added the provider-execution observation boundary at exact head 7cd3dc5ad373adbc4fc19b3e04ccd3b08b2956ee.

This keeps execution with the existing method/connector owner while adding the shared TypeScript external_evidence.receipt reducer and CLI external-evidence receipt. It binds the observed receipt to the exact ready plan and explicitly reports: execution observed, whether evidence was produced, no evidence-coverage claim, no automatic admission/promotion, and raw-source fallback allowed.

Validation:

  • control-plane typecheck passed
  • external-evidence TypeScript suite: 10 passed
  • Python CLI + runtime packaging checks: 7 passed
  • Ruff check/format and diff check passed

This consumes the parent calibration constraint for todo_16320b86fa93: the capability stays experimental/fail-open and cannot auto-promote before holdout evidence.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

Reviewed exact head: 7cd3dc5ad373adbc4fc19b3e04ccd3b08b2956ee

动机

这个 PR 要把分散的 host research method、connector inventory 和下游证据消费统一成一个可审计的 discover → plan → execute → receipt → admit/reject → retire 生命周期。方向是对的:它明确区分“注册表里存在”“当前 ready”“已经执行”“证据覆盖”“被父调用方接纳”这些过去容易混在一起的语义,并且不把 provider 执行权、凭据或后续动作权限收进 Core。

当前 blocker 不在方向,而在这个生命周期最核心的承诺——receipt/admission 必须绑定到“原先的 exact ready plan”。现有实现只比较调用方重新提交的可变 plan 对象里的几个字符串,没有证明它仍然是最初生成的 plan。

改动思路

TypeScript 的 external_evidence.ts 负责 request/provider 规范化、ready provider 选择、receipt/admission 校验和 retirement 投影;Python external_research/cli.py 只读取有界 JSON 并通过 Effect runtime 调用同一 typed owner。connector registry 被保留为 inventory,provider 真正执行仍由原 method/connector owner 完成;这符合现有所有权边界。

正向路径是完整且内聚的。但负向路径有断层:planExternalEvidenceRequest 会为规范化 request 计算 request_id,之后 normalizeExecutionReceipt 却不重新规范化 request、也不校验完整 plan digest。调用方可以修改 objective/decision/constraints 或 provider 状态,只要保留旧 request_id 和 provider 字段,receipt 仍会被认为“绑定 exact plan”。最小修复应是对完整规范化 ready plan 生成并校验 plan_id,或在后续 reducer 中完整重建并核验所有 plan 不变量。

具体改动

  • 新增 7 个 v0 schema 与五个 CLI/Effect 方法,覆盖 discovery、planning、execution receipt、admission、retirement。
  • 新增 built-in capability catalog、双语 RFC/README 和 roadmap 登记。
  • 新增 10 个 TypeScript 单测和 Python CLI/runtime 包装测试;现有 stale receipt identity、file provenance、subset admission 与 retirement coverage 都有覆盖。

关键代码讲解

  1. planExternalEvidenceRequest 正确地把规范化 request hash 成 request_id,并只选择 declared && installed && enabled && ready 的 provider。
  2. normalizeExecutionReceipt 是后续审计边界,但它直接信任传入 plan.request.request_idplan.selected_provider,没有验证 plan 自身是否被改写。
  3. recordExternalEvidenceExecution 在这个弱校验后输出 provider_execution_observed=true,因此问题不是少一个防御性字段,而是公开 truth contract 可能说错话。
  4. handle_external_evidence_command 把文件里的 plan/receipt 直接交给 reducer;CLI 文件边界使篡改场景成为真实公共入口,而不只是内部函数误用。

对主干的风险

P1:可变 plan 可以绕过 exact-plan 绑定

我在 exact head 上先生成合法 plan,随后只把 plan.request.objective 从原值改为另一语义,保留旧 request_id,再提交匹配该 id 的 succeeded receipt。实际输出仍是:

status=succeeded, provider_execution_observed=true

这会让后续 admission/retirement 产生看似一致的 digest/id,但这些 id 证明的是调用方当下递交的对象组合,不是原始 ready plan 未被篡改。现有测试只改 receipt 的 request/provider id,因而全部通过但没有命中这个反例。

最低修复:

  • 对完整规范化 ready plan(request、selected provider、execution envelope 等)生成不可歧义的 plan_id
  • receipt/admission 必须重新验证该 plan 或校验 plan_id,不能只比较对象内自报字段;
  • 增加 objective、decision、constraints、provider readiness、execution envelope 的逐字段 mutation regression。

验证结果:control-plane typecheck 通过;10 个 TS 测试通过;7 个 Python CLI/runtime 测试通过;Ruff check/format 与 git diff --check 通过。按 capability packet 的 wait_for_ci=false,未轮询 GitHub CI。失败证据来自真实 exact-head TypeScript reducer,不依赖 mock。

语义与 CI 对齐

RFC 的“exact ready plan binding”和运行时当前能保证的“与传入 plan 对象字段相等”不一致。新增 mutation regression 后,应让旧实现失败、修复 head 通过,才能把这个 v0 audit vocabulary 视为成立。

我的整体评价

能力归属、provider 权限边界、inventory/readiness 分离以及 Python→TypeScript owner 的架构选择都合理,1694 行也围绕一个完整 preview 生命周期展开,不是无关拼装;future-facing pass 最值得做的是抽出一个唯一 canonical plan decoder/digest verifier,让 plan、receipt、admission 共用同一不变量,而不是再增加第二套校验。

但 exact-plan 绑定是这项能力存在的核心价值,当前反例会直接让审计语义失真,所以本 head 需要修改后复审。修复后仍应把一个真实 host method 与一个 connector 的端到端 acceptance 保留为明确 companion evidence,不能用 reducer 单测替代。

English verdict: REQUEST_CHANGES - head 7cd3dc5 accepts a semantically mutated plan while still reporting succeeded observed execution; typecheck, 10 TS tests, 7 Python tests, Ruff and diff checks pass, but canonical exact-plan integrity needs a plan digest plus mutation coverage.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
@huangruiteng
huangruiteng force-pushed the codex/external-evidence-capability branch from 258255e to b34f14a Compare September 20, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant