Recover validated Todo creation after lost responses

docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-24-validation-publication-recovery.md

@@ -0,0 +1,20 @@
+# Validated Todo publication recovery
+
+Source: #5007. This closes a local create/revision delivery gap in the native
+mutation path: provider commit could precede private validator publication, so a
+lost response stranded the entire Goal's Markdown projection.
+
+The host now durably prepares immutable, digest-addressed private declarations
+before dispatch. Only the canonical Todo's digest selects authority; prepared
+but rejected content is inert. Projection and completion share this reader,
+with strict legacy sidecar compatibility. TS remains the single owner of Todo
+admission, CAS and operation receipts. `todo add --operation-id` provides exact
+create recovery without count-derived identities or a second Python journal.
+Historical retries cannot revert the current validator.
+
+File/SQLite regression cases cover lost responses, abrupt process exit with
+public CLI recovery, later edits/revisions, corrupt selected declarations and
+preparation failure before commit. These are isolated synthetic stores, not a
+live Goal migration or a PostgreSQL publication test. The change leaves
+provider selection, D1–D3 qualification and cross-host distribution of private
+commands unchanged. See the [caller contract](../../../../reference/canonical-todo-completion-update.md).

docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-24-validation-publication-recovery.zh-CN.md

@@ -0,0 +1,15 @@
+# 已验证 Todo 的声明发布恢复
+
+来源：#5007。此交付修复原生创建/验证器修订路径的一个缺口：provider 已提交，
+私有验证声明尚未保存时丢失响应，会使整个 Goal 的 Markdown 投影无法重建。
+
+宿主现在先持久保存按摘要寻址的不可变私有声明，再发送请求。只有权威 Todo 选择的
+摘要才有效，被拒绝请求留下的内容不授予任何权威。投影和完成验证共享同一读取器，
+并严格兼容旧 sidecar。Todo 准入、CAS 和操作回执仍由 TS 单独持有；
+`todo add --operation-id` 复用该回执实现精确恢复，不再依赖当前 Todo 数量生成身份，
+也不增加 Python 提交日志。历史创建重试不能回滚当前验证器。
+
+隔离的 File/SQLite 合成用例覆盖响应丢失、进程突然退出后的公共 CLI 恢复、后续编辑与
+验证器修订、被选声明损坏，以及提交前保存失败。这不等于活动 Goal 迁移或 PostgreSQL
+私有声明发布验证；provider 默认值、D1–D3 和私有命令跨主机分发保持原边界。
+操作方法见[调用方合同](../../../../reference/canonical-todo-completion-update.md)。

docs/reference/canonical-todo-completion-update.md

@@ -33,9 +33,9 @@ loopx todo update --goal-id example --todo-id todo_observation \
 
 The TypeScript transaction compares the current declaration digest, commits the
 new digest, monotonic revision and public-safe audit receipt under one provider
-CAS, and rejects terminal, archived or stale edits. The Python boundary stores
-the private command declaration only after provider success and verifies its
-readback. Reuse the same operation id, expected revision and replacement after
+CAS, and rejects terminal, archived or stale edits. The Python boundary durably prepares digest-addressed private command content
+before the provider can reference it. Canonical readback selects that exact
+digest; a lost response does not leave projection waiting for a sidecar. Reuse the same operation id, expected revision and replacement after
 a lost response; a different intent requires a new operation id and a fresh
 read. Validator replacement cannot be combined with another Todo edit.
 
@@ -52,7 +52,8 @@ introduced.
 读取当前 provider revision，再把新命令作为独立的 reviewed edit 提交。TypeScript
 事务在同一次 provider CAS 中核对旧声明摘要，并提交新摘要、单调递增的 revision 和
 公开安全的审计回执；已完成、已归档或基于旧 revision 的修改会被拒绝。Python 边界
-只在 provider 成功后保存私有命令声明，并校验读回结果。丢失响应时复用相同的
+在 provider 提交前持久保存按摘要寻址的私有命令声明；权威读回只选择匹配的摘要，
+因此丢失响应不再阻塞投影。丢失响应时复用相同的
 operation id、expected revision 和替换内容；新的意图必须使用新的 operation id 并
 重新读取。验证器修改不能和其他 Todo 编辑合并提交。
 
@@ -210,3 +211,42 @@ projection delivery before downgrading. Older binaries reject request v3 and
 cannot recover this operation through the old update route. Existing durable
 Todo/lease records, historical receipts and permanent import/export obligations
 are not removed by this change; never revive stale Markdown as authority.
+
+
+## Retrying canonical Todo creation
+
+For an already promoted File/SQLite Goal, provide a stable caller operation id:
+
+```sh
+loopx todo add --goal-id example --role agent --claimed-by agent-a \
+  --text 'Validate the artifact' --operation-id artifact-create-1 \
+  --validation-command-json '["python3","-m","pytest","-q","tests/test_artifact.py"]'
+loopx todo receipt --goal-id example --operation-id artifact-create-1
+```
+
+Retry the same `todo add` intent with the same id after a lost response. The
+TypeScript receipt recovers the original Todo even if its text or validator
+has since changed. Changing the intent under the same id is rejected. An
+omitted id is generated and returned on success or ambiguous timeout; callers
+that must survive process termination should choose the id before dispatch.
+Legacy Markdown creation rejects this option instead of pretending to provide
+canonical idempotency.
+
+Validation content is prepared privately before create/revision dispatch. Its
+presence alone never activates a validator: the authoritative Todo selects its
+exact digest. Corrupt selected content fails closed. Legacy per-Todo sidecars
+remain readable when no digest-addressed content exists. Rejected requests may
+leave unreferenced private content; this change introduces no automatic deletion
+of declarations that historical receipts may still reference. An old create
+retry cannot replace the current canonical validator. This repairs local
+publication recovery, not cross-host distribution of private validation commands.
+
+对已晋升的 File/SQLite Goal，调用方可在 `todo add` 传入稳定的
+`--operation-id`。响应丢失后用同一编号和同一意图重试，TS 回执返回原 Todo，
+不会因 Todo 后来改名、完成或修订验证器而重复创建。相同编号搭配不同意图会被拒绝。
+省略编号时会自动生成并在成功或不确定超时错误中返回；需要应对进程终止的调用方
+应在发送前自行确定编号。旧 Markdown 路径不支持此参数。
+
+私有声明先持久保存，权威摘要再引用它；没有被权威 Todo 引用的内容不会成为验证要求。
+被选中内容损坏时仍拒绝执行。旧 sidecar 可继续读取，历史创建回执不能回滚新验证器。
+此改动不提供私有验证命令的跨主机分发，也不会自动清理未引用内容。

loopx/cli_commands/todo.py

@@ -305,6 +305,7 @@ def handle_todo_command(
                 runtime_root_arg=runtime_root_arg,
             )
             payload = add_goal_todo(
+                operation_id=args.operation_id,
                 registry_path=registry_path,
                 runtime_root_arg=runtime_root_arg,
                 goal_id=args.goal_id,

loopx/cli_commands/todo_argument_validation.py

@@ -514,8 +514,8 @@ def validate_todo_archive_completed_options(args: argparse.Namespace) -> None:
 
 
 def validate_shared_todo_options(args: argparse.Namespace) -> None:
-    if getattr(args, "operation_id", None) and args.todo_command != "receipt":
-        raise ValueError("--operation-id is supported only by todo receipt")
+    if getattr(args, "operation_id", None) and args.todo_command not in {"receipt", "add"}:
+        raise ValueError("--operation-id is supported only by todo receipt and canonical todo add")
     if args.result_file and args.todo_command != "complete":
         raise ValueError("--result-file is supported only by todo complete")
     agent_id_allowed_for_user_authoring = (

loopx/cli_commands/todo_registration.py

@@ -57,7 +57,7 @@ def register_todo_command(
     todo_parser.add_argument("--todo-id", help="Structured todo id from status/quota, such as todo_ab12cd34ef56.")
     todo_parser.add_argument(
         "--operation-id",
-        help="For todo receipt, read the exact historical canonical operation after an ambiguous response; this does not grant a retry or lease.",
+        help="For canonical todo add, reuse this identity with unchanged intent after an ambiguous response. For todo receipt, read the exact historical operation; a receipt grants no lease.",
     )
     todo_parser.add_argument(
         "--update-operation-id",

loopx/control_plane/todos/completion_validation.py

@@ -451,14 +451,15 @@ def resolve_private_completion_validation_declaration(
         raise ValueError(
             "canonical Todo requires completion validation but omits its digest"
         )
-    # A missing sidecar is an availability case: the digest-bound Markdown or
-    # event projection may rehydrate it below. A present sidecar that fails its
-    # identity or digest checks is corruption/tamper evidence and deliberately
-    # raises instead of falling back, so a second source cannot mask the fault.
+    # The canonical digest selects immutable private content, including after
+    # a lost write response. Older installs may still use a per-Todo sidecar.
+    # Corruption in the selected source fails closed; only absence permits
+    # digest-checked rehydration from a materialized projection below.
     declaration = read_completion_validation_declaration(
         runtime_root=runtime_root,
         goal_id=goal_id,
         todo_id=todo_id,
+        expected_digest=expected,
     )
     if declaration is None:
         source = _materialized_todo_item(

loopx/control_plane/todos/completion_validation_store.py

@@ -9,6 +9,8 @@
 from typing import Any
 
 from ...registry import atomic_write_json, read_json
+from ...file_lock import exclusive_cross_runtime_file_lock
+from .active_state_editing import fsync_state_directory
 from .completion_validation_projection import (
     completion_validation_declaration,
     completion_validation_declaration_sha256,
@@ -47,7 +49,9 @@ def persist_completion_validation_declaration(
     normalized = completion_validation_declaration(dict(declaration))
     if normalized is None:
         raise ValueError("completion validation declaration is empty")
-    digest = completion_validation_declaration_sha256(normalized)
+    digest = prepare_completion_validation_declaration(
+        runtime_root=runtime_root, goal_id=goal_id, declaration=normalized
+    )
     path = completion_validation_declaration_path(
         runtime_root=runtime_root,
         goal_id=goal_id,
@@ -68,14 +72,64 @@ def persist_completion_validation_declaration(
     return digest
 
 
+def prepare_completion_validation_declaration(
+    *, runtime_root: Path, goal_id: str, declaration: Mapping[str, Any],
+) -> str:
+    """Persist immutable private content before a canonical digest can reference it.
+
+    A prepared blob grants no Todo authority. Only a canonical record selecting
+    its exact digest can consume it. Rejected creates may leave unreferenced blobs.
+    """
+    normalized = completion_validation_declaration(dict(declaration))
+    if normalized is None:
+        raise ValueError("completion validation declaration is empty")
+    digest = completion_validation_declaration_sha256(normalized)
+    path = completion_validation_declaration_path(
+        runtime_root=runtime_root, goal_id=goal_id, todo_id="blobs",
+    ).parent / "blobs" / f"{digest}.json"
+    payload = {"schema_version": "loopx_todo_validation_blob_v0", "goal_id": goal_id,
+               "declaration_sha256": digest, "declaration": normalized}
+    with exclusive_cross_runtime_file_lock(path, operation="prepare_validation_declaration"):
+        if path.exists():
+            if read_json(path) != payload:
+                raise ValueError("prepared validation declaration digest mismatch")
+        else:
+            atomic_write_json(path, payload)
+        # Also re-establish directory durability on an idempotent retry.
+        fsync_state_directory(path)
+    return digest
+
+
+def _read_prepared_declaration(path: Path, goal_id: str, digest: str) -> dict[str, Any] | None:
+    if not re.fullmatch(r"[a-f0-9]{64}", digest):
+        raise ValueError("canonical validation digest must be SHA-256")
+    try:
+        value = read_json(path.parent / "blobs" / f"{digest}.json")
+    except FileNotFoundError:
+        return None
+    if not isinstance(value, Mapping) or not isinstance(value.get("declaration"), dict):
+        raise ValueError("prepared validation declaration is malformed")
+    declaration = completion_validation_declaration(value["declaration"])
+    if (value.get("schema_version") != "loopx_todo_validation_blob_v0"
+            or value.get("goal_id") != goal_id or value.get("declaration_sha256") != digest
+            or declaration is None or completion_validation_declaration_sha256(declaration) != digest):
+        raise ValueError("prepared validation declaration digest or identity mismatch")
+    return declaration
+
+
 def read_completion_validation_declaration(
-    *, runtime_root: Path, goal_id: str, todo_id: str
+    *, runtime_root: Path, goal_id: str, todo_id: str,
+    expected_digest: str | None = None,
 ) -> dict[str, Any] | None:
     path = completion_validation_declaration_path(
         runtime_root=runtime_root,
         goal_id=goal_id,
         todo_id=todo_id,
     )
+    if expected_digest is not None:
+        prepared = _read_prepared_declaration(path, goal_id, expected_digest)
+        if prepared is not None:
+            return prepared
     try:
         value = read_json(path)
     except FileNotFoundError:
@@ -96,6 +150,8 @@ def read_completion_validation_declaration(
     digest = completion_validation_declaration_sha256(normalized)
     if value.get("declaration_sha256") != digest:
         raise ValueError("completion validation declaration store digest mismatch")
+    if expected_digest is not None and digest != expected_digest:
+        raise ValueError("private completion validation declaration does not match canonical Todo digest")
     return normalized
 
 
@@ -114,6 +170,7 @@ def load_completion_validation_declarations(
             runtime_root=runtime_root,
             goal_id=goal_id,
             todo_id=todo_id,
+            expected_digest=str(todo.get("completion_validation_sha256") or ""),
         )
         if declaration is not None:
             loaded[todo_id] = declaration
@@ -125,5 +182,6 @@ def load_completion_validation_declarations(
     "completion_validation_declaration_path",
     "load_completion_validation_declarations",
     "persist_completion_validation_declaration",
+    "prepare_completion_validation_declaration",
     "read_completion_validation_declaration",
 ]