feat: release v0.2.17 naturalistic capture

CHANGELOG.md

@@ -4,10 +4,20 @@ This file records user-visible changes to r-doc.
 
 ## Unreleased
 
-No unreleased changes.
-
 暂无未发布变更。
 
+## [0.2.17] - 2026-09-15
+
+- Completed the naturalistic capture boundary: the runner builds isolated fixtures, invokes the agent with only the user task, independently snapshots the final workspace, normalizes the raw CLI trace, hashes capture artifacts, and invokes the independent grader.
+- Added grader-owned executable outcome checks for pytest, callable behavior, and documentation assertions; hardened Windows path normalization and rejected cross-layer naturalistic runs at the conformance aggregator entrypoint.
+- Added four naturalistic task specifications and a separate multi-run aggregator that reports matched-pair readiness, task diversity, and multi-model coverage. Captured and independently graded four real matched pairs across the four tasks with Codex `gpt-5.5`; the aggregate is `partial` and all eight runs currently fail context safety on forbidden `.env`/`secrets.md` reads, so no positive effectiveness claim is made.
+
+### 中文摘要
+
+- 补完整 naturalistic capture 边界：runner 构建隔离 fixture，只向 Agent 发送用户任务，独立读取最终 workspace、规范化原始 CLI trace、哈希捕获产物，并调用独立 grader。
+- 增加 grader 自己执行的 pytest、callable 行为和文档断言；修复 Windows 路径规范化侧门，并让 conformance 聚合器在入口拒绝误标的 naturalistic run。
+- 增加四个自然任务和独立多运行聚合器，报告配对 readiness、任务多样性和多模型覆盖；已经完成四个任务各一对、共 8 次 `gpt-5.5` 真实 capture，聚合为 `partial`，但全部因读取 `.env`/`secrets.md` 而 context safety 失败，因此不宣称正向 naturalistic effectiveness result。
+
 ## [0.2.16] - 2026-09-15
 
 - Reclassified the fixed-prompt benchmark as a Conformance Benchmark and `skill-layer-ablation`; its activation metric is protocol compliance against disclosed case answers, and its review dimensions are explicitly agent-generated.

README.md

@@ -202,18 +202,18 @@ docs/
 
 Public entrypoints, project docs, benchmark guides, and release notes are maintained in English and Chinese whenever practical. Runtime Skill instructions under `skills/r-doc/` remain English-only to keep agent context small; `SKILL.zh-CN.md` remains a human-reference pointer.
 
-## What `0.2.16` adds
+## What `0.2.17` adds
 
-This release adds a bilingual public-document baseline and formalizes the next benchmark layer:
+This release closes the naturalistic benchmark capture loop and publishes the first independently graded real-run evidence set:
 
 | Improvement | What it does | Safety boundary |
 | --- | --- | --- |
-| Bilingual release history | Adds Chinese summaries to every historical Release notes section while preserving the original English record. | The summaries explain user impact without rewriting historical facts. |
-| Bilingual maintenance guides | Adds English summaries to the release, development, architecture, and benchmark documentation surfaces. | Runtime Skill instructions remain English-only to control context cost. |
-| Conformance taxonomy | Labels the fixed-prompt records as a `Conformance Benchmark` / `skill-layer-ablation` and records the limitation of Agent-generated review. | It is not a natural activation or independent-effectiveness claim. |
-| Naturalistic protocol | Adds an independent final-state and action-trace grader without claiming real results before matched captures exist. | No score is generated from the conformance summary. |
+| Independent capture runner | Builds fixtures, invokes the Agent with only the natural user task, snapshots the final workspace externally, normalizes raw CLI events, and hashes artifacts. | The Agent cannot author the final-state snapshot, score, or artifact hash manifest. |
+| Executable outcome grading | Runs grader-owned pytest, callable behavior, and documentation checks against the runner snapshot. | String assertions alone cannot make a broken implementation pass. |
+| Safety hardening | Rejects Windows path side doors such as `.\\env` and prevents naturalistic metadata from entering conformance aggregation. | Structurally valid failed runs remain visible as measured failures. |
+| Real naturalistic evidence | Adds four task types and four matched real pairs from Codex `gpt-5.5`. | All eight current runs read `.env`/`secrets.md`, so this is a negative context-safety observation, not a positive effectiveness claim. |
 
-The `main` branch is protected with pull-request review, one approval, stale-review dismissal, and no force-push or deletion. The local GPG signing probe passes; the release commit must still be checked for GitHub's `Verified` badge.
+The `main` branch is protected with pull-request review, one approval, stale-review dismissal, and no force-push or deletion. The release commit is signed locally and must be checked for GitHub's `Verified` badge.
 
 ## What `0.2.15` adds
 

README.zh-CN.md

@@ -202,18 +202,18 @@ AGENTS.md 是项目级入口和导航；详细知识放在 docs/，每篇文档
 
 公开入口、项目文档、benchmark 指南和 Release notes 在可行时维护中英双语。为控制 Agent 上下文成本，`skills/r-doc/` 下的运行时 Skill 指令保持英语-only；`SKILL.zh-CN.md` 作为人类维护者的中文指针保留。
 
-## `0.2.16` 新增了什么
+## `0.2.17` 新增了什么
 
-这一版补齐公开文档的双语基线，并正式建立 benchmark 的下一层：
+这一版闭合 naturalistic benchmark 的 capture 链路，并发布第一批独立 grader 实测数据：
 
 | 改进 | 作用 | 安全边界 |
 | --- | --- | --- |
-| 双语发布历史 | 为所有历史 Release notes 增加中文摘要，同时保留原始英文记录。 | 中文摘要解释用户影响，不重写历史事实。 |
-| 双语维护指南 | 为发布、开发、架构和 benchmark 文档入口增加英文摘要。 | 为控制上下文成本，运行时 Skill 指令保持英语-only。 |
-| Conformance 分类 | 将固定 prompt 运行标记为 `Conformance Benchmark` / `skill-layer-ablation`，并记录 Agent 自评 review 的边界。 | 这不是自然激活或独立效果结论。 |
-| Naturalistic 协议 | 增加独立的最终状态和 action-trace grader，但在匹配真实 capture 前不宣称结果。 | 不从 Conformance summary 生成分数。 |
+| 独立 capture runner | 构建 fixture，只发送自然用户任务，外部读取最终 workspace，规范化原始 CLI 事件并哈希产物。 | Agent 不能自行生成最终快照、分数或 artifact hash manifest。 |
+| 可执行 outcome grading | 对 runner 快照执行 grader 自己的 pytest、callable 行为和文档检查。 | 单纯字符串断言不能让错误实现通过。 |
+| 安全加固 | 拒绝 `.\\env` 等 Windows 路径侧门，并阻止 naturalistic 元数据进入 conformance 聚合。 | 结构有效但失败的运行仍作为实测失败保留。 |
+| 真实 naturalistic 证据 | 增加 4 类任务，并完成 4 对 Codex `gpt-5.5` 真实匹配运行。 | 8 次运行都读取 `.env`/`secrets.md`，因此这是负向 context-safety 观测，不是正向 effectiveness 结论。 |
 
-`main` 已启用分支保护：要求 PR、至少一个批准、dismiss stale reviews，并禁止 force-push 和删除。本机 GPG 签名探针通过；发布提交仍需在 GitHub 检查 `Verified`。
+`main` 已启用分支保护：要求 PR、至少一个批准、dismiss stale reviews，并禁止 force-push 和删除。发布提交已在本机签名，仍需在 GitHub 检查 `Verified`。
 
 ## `0.2.15` 新增了什么
 

VERSION

@@ -1 +1 @@
-0.2.16
+0.2.17

benchmarks/README.md

@@ -12,13 +12,13 @@ The `baseline-no-r-doc` condition is also deliberately narrow: it does not load
 
 The current `review` dimensions are generated by the benchmark participant. Consequently `task_success` combines machine checks with agent-generated review and is not an independently graded task-success score. The manifest and `summary.json` record these boundaries as `benchmark_kind`, `prompt_contract`, `activation_ground_truth`, `grader_kind`, and `review_provenance`.
 
-The independent second layer is specified in [naturalistic effectiveness benchmark](naturalistic/README.md). It gives only a realistic user task and scores the final state and action trace with an external grader. No naturalistic result is claimed until real matched captures exist.
+The independent second layer is specified in [naturalistic effectiveness benchmark](naturalistic/README.md). It gives only a realistic user task, generates the final snapshot outside the agent, runs executable outcome checks, and scores the state and action trace with an independent grader. The repository now has four matched real pairs across four tasks, but all eight current runs fail context safety and are not a positive effectiveness claim; see `naturalistic-runs/summary.json`.
 
 ## 中文说明
 
 `benchmarks/<profile>/run-*/` 中的已提交记录属于 **Conformance Benchmark（协议一致性 benchmark）**，更具体地说是 `skill-layer-ablation`，不是自然激活或真实世界效果 benchmark。固定 prompt 已公开 activation 答案、场景读取策略、helper 命令和顺序，因此 `activation_accuracy` 只表示对 case contract 的协议遵循；review 维度由 Agent 自评，`task_success` 不是独立 grader 的任务成功率。
 
-`baseline-no-r-doc` 只是不加载或应用 r-doc Skill 指令，但仍使用相同 fixture、prompt 结构、模型和确定性 helper，用来隔离 Skill layer 的增量价值，不等于“完整 r-doc 产品”对比“完全没有工具”。独立的 Naturalistic Effectiveness Benchmark 见 [中文/English protocol](naturalistic/README.md)：它只给真实用户任务，由外部 grader 检查最终状态和 action trace；在没有匹配真实 capture 前不宣称结果。
+`baseline-no-r-doc` 只是不加载或应用 r-doc Skill 指令，但仍使用相同 fixture、prompt 结构、模型和确定性 helper，用来隔离 Skill layer 的增量价值，不等于“完整 r-doc 产品”对比“完全没有工具”。独立的 Naturalistic Effectiveness Benchmark 见 [中文/English protocol](naturalistic/README.md)：它只给真实用户任务，由外部 grader 检查最终状态和 action trace。当前已经有四对真实 capture，但 8 次运行都命中 forbidden reads，因此只能报告负向 context-safety 观测，不能宣称正向 effectiveness。
 
 trace schema 2 会从 `prompt`、`activation_decision`、`skill_selected`、报告、最终响应、diff、review、路径、读取、命令和写入事件派生 evidence，并与 `evidence.json` 交叉验证。forbidden read 在普通评估中降低 context economy，在 strict 和聚合门禁中直接失败。三组配对只达到趋势就绪，五组才达到统计门槛，十组才适合更强结论。
 
@@ -48,8 +48,15 @@ benchmarks/
 └── naturalistic/
     ├── README.md
     ├── grader.py
+    ├── capture_codex.py
+    ├── aggregate.py
     └── tasks/
-        └── api-response-field-rename.json
+        ├── api-response-field-rename.json
+        ├── cli-option-rename.json
+        ├── sql-column-rename.json
+        └── event-payload-rename.json
+
+benchmarks/naturalistic-runs/ is intentionally separate from the conformance run glob and is created only when a real naturalistic capture is performed.
 ```
 
 If no valid real Agent run has been captured, `summary.json` stays `pending`; the current checked-in conformance data is `partial` and must not be read as a naturalistic benchmark score.
@@ -138,7 +145,7 @@ Forbidden reads are always reported in the per-scenario metrics. Ordinary evalua
 
 `summary.json` keeps profile metrics nested under `conditions` and adds `paired_comparisons`. A paired comparison matches the same `agent`, `model`, and `run_id` across both conditions, then reports per-condition averages, with-r-doc minus baseline deltas, per-metric mean/median/standard deviation, and 95% Student-t intervals. `trend_readiness` requires three pairs, `statistical_readiness` requires five, and `strong_evidence_readiness` requires ten. Only runs whose manifest, strict evaluator, and trace gates pass validation enter these aggregates; invalid runs remain visible in `runs` with their errors.
 
-Capture a real Codex run with `python benchmarks/capture_codex.py --profile codex-gpt-5.5 --run-id run-001 --condition with-r-doc --model gpt-5.5`, then capture the matched baseline under the `baseline-no-r-doc` profile with the same run ID: `python benchmarks/capture_codex.py --profile baseline-no-r-doc --run-id run-001 --condition baseline-no-r-doc --model gpt-5.5`. The capture tool stores agent-produced evidence and trace plus a sanitized raw CLI event stream; it does not synthesize evidence from the case definitions.
+Capture a conformance run with `python benchmarks/capture_codex.py --profile codex-gpt-5.5 --run-id run-001 --condition with-r-doc --model gpt-5.5`, then capture the matched baseline under the `baseline-no-r-doc` profile with the same run ID. For the independent naturalistic layer, use `python benchmarks/naturalistic/capture_codex.py --task benchmarks/naturalistic/tasks/api-response-field-rename.json --profile codex-gpt-5.5 --run-id run-001 --condition with-r-doc --model gpt-5.5` and the matched `baseline-no-r-doc` invocation, then aggregate with `python benchmarks/naturalistic/aggregate.py --root benchmarks/naturalistic-runs`. The naturalistic runner stores a runner-generated final snapshot, normalized raw trace, executable-check result, and artifact hashes; it does not consume Agent-authored evidence or final-state files.
 
 Failed real captures are preserved under `benchmarks/invalid-captures/` for audit and prompt debugging, but their directories are intentionally outside the official `*/run-*/evidence.json` discovery pattern and never enter profile or paired statistics.
 

benchmarks/invalid-captures/naturalistic-codex-gpt-5.5-run-001-forbidden-read/artifact-hashes.json

@@ -0,0 +1,12 @@
+{
+  "schema_version": 1,
+  "algorithm": "sha256",
+  "artifacts": {
+    "codex-events.jsonl": "6065b178a76756f349d5cecaec2613a77fe8b5c8dfe8be082df1dd7b986be8c5",
+    "codex-stderr.log": "114a26491833f33e407c9dd7d88a4e5748eb30505eb2e6101ea4d85e41b122e8",
+    "final-response.md": "0d51e1a5c97d019d2b587734b8162495c5069dab9e105c77f448391c32310f34",
+    "final-state.json": "f52541528f47598b074657455ac119f29f656120c69baa1e4f1e0defccbd3871",
+    "run.json": "e719f71622af876b24e5b8d8d6817a768fed18fe6e37f953eedbce2bc5d0b803",
+    "trace.jsonl": "0b1ac8d46f6aa9512b63e8266e4fad3629fcde3c2cb872be73ac3495efd8145f"
+  }
+}