diff --git a/README.md b/README.md index 55fb895..74951e4 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,7 @@ Rule of thumb: if you build and operate your own agent in production, use a trac ## Features - **Automatic session health** — Opens with factual failure, repetition, follow-up and last-recorded call-state summaries. Missing/running/unknown results have evidence links; no human labels or model calls required. Manual notes and transfers are opt-in and never hide automatic facts. [Scope and offline checks](docs/diagnostics.md#automatic-session-health). +- **Codex background-process evidence** — Connect explicit `exec_command` process IDs to later `write_stdin` results, with launch/poll/exit source links. Ambiguous IDs or polling sequences stay unknown; process completion never rewrites historical tool-call states or proves a task passed. [Association limits](docs/diagnostics.md#codex-background-process-evidence). - **Modification/check chronology** — Distinguish checks before an edit, checks overlapping it and later outcomes. A passed earlier check or a successful output pipeline is not post-change validation; ambiguous command fragments remain unknown. [Recognition and coverage limits](docs/diagnostics.md#modification-and-verification-chronology). - **Per-turn ledger** — In the session summary, from two user turns on: one row per user turn with wall-clock time, tokens (input + output + cache) and cost, bars scaled to the session maximum, tool-call counts inline (error counts in the row tooltip), click to jump. Answers "why did this take 40 minutes / cost $3" without reading the transcript. diff --git a/README.zh-CN.md b/README.zh-CN.md index b08baae..f7e32fe 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -55,6 +55,7 @@ LangSmith、Langfuse 这类观测平台面向的是*你自己写的* agent：接 ## 功能特性 - **自动会话体检** — 默认自动整理失败、重复操作、后续候选和调用最后记录状态；执行中、未知及未记录结果可追溯证据。不依赖人工标注或模型调用，笔记与迁移改为可选，不影响自动事实展示。[口径与离线验证](docs/diagnostics.md#自动体检无需人工标注)。 +- **Codex 后台进程证据** — 用明确进程 ID 关联启动、`write_stdin` 轮询和退出结果，可逐步跳转原始证据。ID 重用、轮询交叠和冲突保持未知；不改写历史工具状态，不把进程退出当作任务通过。[关联边界](docs/diagnostics.md#codex-后台进程证据)。 - **修改—检查时序** — 区分修改前成功的检查、与修改重叠的检查及后续最新结果；不把先前通过或管道整体成功当作修改后的验证。复杂命令片段执行状态保持未知。[识别边界](docs/diagnostics.md#修改与验证的先后顺序)。 - **有证据的失败事件（React UI）** — 将同一调用所在用户轮次、同工具、完整同参数的待复查失败分组，重复最多的操作优先展示；可跳转首末及每条原始证据。同参成功切断分组，缺少参数不合并。执行成功采用明确零退出码或 OMP 原生完成证据；事件不等于根因或任务失败。本地规则，无 LLM。[合成演示与判定边界](docs/diagnostics.md#中文使用指南)。 diff --git a/claims.json b/claims.json index 7e81c20..f2716fd 100644 --- a/claims.json +++ b/claims.json @@ -102,16 +102,16 @@ }, { "id": "test-count", - "claim": "290 tests pass on Node's built-in test runner, the count docs/ROADMAP.md records for `npm test`.", - "value": "290", - "metric": "passing node:test cases (# tests 290 / # pass 290 / # fail 0)", - "method": "npm test → node --test test/*.test.js, run in the claims job after npm ci, and the TAP summary is asserted. The roadmap sentence ('290 tests on Node's built-in runner (`npm test`, 2026-09-23)') is verified by the run, not read back from the prose.", + "claim": "313 tests pass on Node's built-in test runner, the count docs/ROADMAP.md records for `npm test`.", + "value": "313", + "metric": "passing node:test cases (# tests 313 / # pass 313 / # fail 0)", + "method": "npm test → node --test test/*.test.js, run in the claims job after npm ci, and the TAP summary is asserted. The roadmap sentence ('313 tests on Node's built-in runner (`npm test`, 2026-09-23)') is verified by the run, not read back from the prose.", "repro": "npm test 2>&1 | grep -E '^# (tests|pass|fail)'", "evidence": "docs/ROADMAP.md", "as_of": "2026-09-13", "check": { "cmd": "npm test 2>&1 | grep -E '^# (tests|pass|fail)'", - "expect": { "contains": ["# tests 290", "# pass 290", "# fail 0"] }, + "expect": { "contains": ["# tests 313", "# pass 313", "# fail 0"] }, "timeout": 120 } }, @@ -238,7 +238,7 @@ "check": { "cmd": "node scripts/claims-receipts.mjs tests-node-only", "expect": { - "equals": "20 files in test/ · 14 distinct requires: 11 node builtins, 3 relative, 0 third-party" + "equals": "21 files in test/ · 15 distinct requires: 11 node builtins, 4 relative, 0 third-party" }, "timeout": 60 } diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index ecc2f68..92abafa 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -8,7 +8,7 @@ - **Session browser** with tool-call inspection, trace/waterfall view, spawn tracking and message timeline - **Prompt tooling** — extraction (noise filtered), template clustering with outcome attribution, Claude-powered rewrites, and a prompt library that installs entries as native slash commands - **Global search** across all platforms, insights dashboard, incremental session backup -- **React + Vite frontend** served by an Express backend; 290 tests on Node's built-in runner (`npm test`, 2026-09-23), CI on Node 22 +- **React + Vite frontend** served by an Express backend; 313 tests on Node's built-in runner (`npm test`, 2026-09-23), CI on Node 22 - **Evidence-backed failure events and local review** with full-result invalidation, evidence navigation and narrow-screen session layout ## Current priorities @@ -21,7 +21,7 @@ External grounding: official guidance emphasizes [executable verification](https | --- | --- | --- | | P0 | Make the new workflow immediately testable | A demo-only entry opens a clearly synthetic case: 7 pending records in 2 events, all evidence accessible, local review does not rewrite automatic results. Preserve the existing default demo and samples. | | P1 | Current-session review portability implemented | Preview-only import and explicit plaintext download; exact identity/evidence matching, no overwrites, bounded schema and partial-failure reporting. Validate with synthetic migration and publish after CI; whole-history backup and path remapping remain out of scope. | -| P2 | Execution/verification evidence, not another statistics dashboard | Automatic health and modification/check chronology: distinguish before/overlap/after and latest outcomes, show unsupported command coverage, keep optional notes independent. Deterministic transformations and frozen-log regressions verify association rules, not file coverage, human time saved or universal accuracy. Next investigate explicit per-step check outcomes and task/child linkage before relaxing ambiguous shell assumptions. | +| P2 | Execution/verification evidence, not another statistics dashboard | Automatic health and modification/check chronology distinguish before/overlap/after and unknown outcomes. Codex process IDs now connect launch/poll/exit evidence without rewriting history or original check start. Next investigate explicit task/child linkage and per-step outcomes; reject ambiguous associations rather than relaxing shell assumptions. Deterministic tests and frozen-log checks are not file coverage, human time saved or universal accuracy. | | P3 | Make releases reproducible for contributors | Keep clean-install tests, generated fixtures, documentation claims and release/package verification aligned. Add browser regression automation when it can run deterministically without personal logs. | No launch dates or star-count targets are promised. Progress is gated on these observable outcomes. Physical-device/keyboard coverage and complex Trace/analytics layouts remain separate work, not implied by the session-screen checks. diff --git a/docs/diagnostics-verification.md b/docs/diagnostics-verification.md index 95f4a83..16469de 100644 --- a/docs/diagnostics-verification.md +++ b/docs/diagnostics-verification.md @@ -118,3 +118,15 @@ Browser checks used synthetic fixtures only: - Call/result jumps resolve to actual source records without moving outer page navigation. At 390px the chronology panel has no horizontal overflow; static demo uses zero backend API requests. Use `node scripts/demo-verification.cjs` to replay the temporal example and `node scripts/demo-follow-up.cjs` for missing-result updates. The public hosted demo exposes the earlier-test/later-edit example under automatic health. Physical-device keyboards and every complex view are not certified. Existing lint findings (91 warnings / 159 infos) and bundle-size warnings remain. + +## Codex background-process acceptance + +This increment associates explicit Codex process IDs from launch result headers with later `write_stdin.session_id` calls. It does not rewrite the historical per-call state, failure counts, recovery or human review data. Process outcomes are a separate evidence chain; the original launch index is retained when chronology uses a final polling result. + +The full suite has **313 passing tests**. The **23 process-evidence tests** cover success/failure/running/missing results, output-text false positives, malformed/contradictory headers, noncanonical identifiers, duplicate calls/results, reused process IDs, overlap, post-terminal polls, input-sending calls, source references, qualified tool names and multiple launches in one message. Forty fixed transformations verify terminal-result append without inferring automatic recovery. The generated hosted sample is asserted equal to the real Codex parser output. + +The frozen real corpus was re-read under content hashes: all previous `diagnoseSession` and `summarizeSessionHealth` outputs match v1.21.0 exactly. In its Codex subset, 60 launch headers and 53 polling calls yielded **53 uniquely linked polls**, **48 successful process exits**, **1 failed exit**, and **11 launches whose last evidence is running**. Every process association and terminal result was checked against the recorded process ID and header boundary. No real commands, process IDs or detailed logs are published. These results do not establish live runtime state, test coverage, general accuracy or time savings. + +Synthetic browser checks validate the process summary, unlinked poll count, source-result navigation, a test launched before an edit remaining overlapping, a later failed check remaining visible and live completion changing terminal process count from 2 to 3 without changing historical failure events. At 390px the process panel has no horizontal overflow. `node scripts/demo-process-evidence.cjs` reproduces the example locally without executing transcript commands. + +Unknown/conflicting chains are not silently certified; callers who need live job control or cross-session task association still need stronger runtime evidence. Input is represented as a boolean in the process summary, though original call evidence remains accessible. Existing large-bundle and lint findings remain unchanged. diff --git a/docs/diagnostics.md b/docs/diagnostics.md index 522c093..7d18636 100644 --- a/docs/diagnostics.md +++ b/docs/diagnostics.md @@ -24,6 +24,30 @@ Try `node scripts/demo-follow-up.cjs`: the fixture includes running, unknown and ![Synthetic automatic session health](../screenshots/automatic-health.png) +## Codex background-process evidence + +The **Codex 后台进程证据** section links a recorded `exec_command` launch header (`Process running with session ID ...`) to later `write_stdin` calls with that exact numeric `session_id`. It shows launch and polling source records, whether input was sent, and an unambiguous terminal exit result where available. + +This is a separate unit from per-tool-call state: the historical launch still returned a running/unknown record, while the process may have finished through a later poll. Polling is not a second launch, and none of these links changes failure events, automatic retry recovery or saved review fingerprints. + +- Only supported Codex wrapper headers **before** `Output:` / `Final output:` are used, on exact `exec_command` / `write_stdin` tool names or their `functions.`-qualified forms. Matching text printed in stdout or read from a file is not process metadata. +- Launch and poll call IDs must be unique in the currently viewed transcript. Process IDs must be nonnegative safe integers; string IDs and missing launch history are not guessed. +- Reused process IDs, duplicate launch/poll records, mismatched returned IDs, missing or invalid polling results, overlapping polls or polls after a terminal result make completion ambiguous. The original records remain accessible, but no final result is adopted from the conflicting chain. +- A last running record is not live process monitoring. Input-sending polls are marked without showing input text in the process summary. The raw call remains available if you choose to inspect it. +- A recognized verification command can use its linked final process result, but retains its **original launch position**. A check launched before an edit and finished afterward still overlaps the edit; it is not post-edit verification. Input-fed processes and compound shell fragments keep an unknown check outcome even when the process exits zero. + +Use the hosted demo's **Codex** tab and select the `[Synthetic demo] Background process evidence` session. It contains two launches: one finishes through a poll, one has only a running record, plus an unlinked poll. The previous default demo remains unchanged. + +For live updates from a source checkout: + +```sh +node scripts/demo-process-evidence.cjs +``` + +Select the Codex synthetic session. Three launches initially include a successful exit, a failed exit and a pending process; one poll has no matching launch. Enter `c` to append the missing completion and watch the process evidence update, or `q` to stop and clean up. No logged commands are executed. + +![Synthetic Codex process evidence](../screenshots/codex-process-evidence.png) + ## Modification and verification chronology The **修改—检查时序** section asks a narrower, evidence-based question than “did the task pass?”: @@ -208,6 +232,7 @@ node --test test/review-transfer.test.js node --test test/follow-up-evidence.test.js node --test test/session-health.test.js node --test test/verification-chronology.test.js +node --test test/codex-process-evidence.test.js npm test npm run build:ui npm run lint @@ -287,6 +312,16 @@ The local frozen regression set contained 30 sessions and 9,076 tool results. Gr 本轮的 373 个真实冻结事件只使用**内存中的合成测试标记**检验隔离与失效，没有替你判断真实事件，也没有把这些测试标记写成真实复核。完整证据见 [本机复核验收](diagnostics-verification.md)。 +## Codex 后台进程证据 + +展开“Codex 后台进程证据”，可以沿着 `exec_command` 的包装头进程 ID，找到同一会话内 `write_stdin.session_id` 对应的轮询与最终退出结果。启动、每次轮询和结果均可跳转。 + +它与“每条工具调用的最后记录状态”分开统计：历史启动结果不被改写，轮询也不是另一次启动。重复进程 ID、重复调用/结果、返回 ID 不符、交叠轮询、终止后继续轮询或缺少可靠结果时，完成状态保持未知。只认包装头，不把 stdout 中的示例文本当证据，不跨会话猜测。 + +若这个进程运行的是可识别检查，时序视图可采用唯一关联的终止结果，但仍保留原始启动位置：修改前启动、修改后才返回的测试仍是“重叠”，不算修改后验证。发送过输入或执行复合命令的检查结果仍保持未知；整个进程退出零码不代表任务通过。 + +在线 Demo 的 Codex 页有明确标记的合成案例；源码可运行 `node scripts/demo-process-evidence.cjs`，输入 `c` 为待完成进程追加结果，`q` 退出。当前只支持上述 Codex 输出契约，不宣称 OMP/Claude 后台任务链也已覆盖。 + ## 后续相关操作 失败事件下若有候选，可展开“后续相关操作”，查看匹配依据、五类状态、调用/结果位置、参数摘要及原始结果。**它不自动关闭事件，不代表任务已通过。** diff --git a/docs/releases/v1.22.0.md b/docs/releases/v1.22.0.md new file mode 100644 index 0000000..10ae5bc --- /dev/null +++ b/docs/releases/v1.22.0.md @@ -0,0 +1,25 @@ +# v1.22.0 — Trace Codex background completion to its source + +Codex can return from `exec_command` while a process is still running, then deliver its exit result through `write_stdin`. AgentXRay now links that lifecycle using explicit recorded process IDs rather than leaving the launch isolated from its later result. + +## New + +- **Launch → poll → exit evidence:** source links for the original call, initial result, every associated poll and unique final exit result. +- **Ambiguity stays visible:** reused IDs, duplicate records, overlapping polls, mismatched IDs or polls after terminal output do not certify completion. Unknown/unlinked polls are counted. +- **Verification timing keeps the original start:** a test launched before a modification remains overlapping even if its exit result arrives after the modification. Later polling is not a new test run. +- **No historical rewrites:** per-call health, failure events, recovery decisions and review fingerprints are unchanged. Process outcomes are shown separately. Input-fed and compound commands cannot become clean test successes from an exit-zero wrapper. +- **Synthetic demos:** select the new Codex process-evidence sample in the hosted demo, or run `node scripts/demo-process-evidence.cjs` from a checkout for live completion updates. + +## Validation + +313 Node tests pass, including 23 process-evidence tests and 40 deterministic terminal transformations. The prior frozen 15-session/2,024-result regression preserves all existing health and diagnostic output. Its Codex subset links 53 polls across 60 starts, finding 48 successful exits, 1 failed exit and 11 starts with only a running record. These are association results, not task correctness or live process monitoring. + +Browser checks cover result navigation, failed later checks, pre-edit launch overlap, unlinked polling, live completion and narrow screens. Normal/demo builds, lint and claims checks pass with existing lint findings unchanged. + +Only the recognized Codex wrapper contract is supported. Quoted/output text is not metadata; no cross-session or OMP/Claude lifecycle guessing, no automatic commands, no new runtime dependencies or uploads. + +[Usage and limits](https://github.com/alloevil/AgentXRay/blob/master/docs/diagnostics.md#codex-background-process-evidence) · [Verification receipt](https://github.com/alloevil/AgentXRay/blob/master/docs/diagnostics-verification.md) + +```sh +npx @alloevil/agent-xray@1.22.0 +``` diff --git a/frontend/demo/sample-logs/codex/2026/09/24/rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl b/frontend/demo/sample-logs/codex/2026/09/24/rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl new file mode 100644 index 0000000..12018d1 --- /dev/null +++ b/frontend/demo/sample-logs/codex/2026/09/24/rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl @@ -0,0 +1,13 @@ +{"type":"session_meta","timestamp":"2026-09-24T08:00:00.000Z","payload":{"id":"01990000-0000-7000-8000-000000000199","cwd":"/demo/process-evidence","timestamp":"2026-09-24T08:00:00.000Z"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:01.000Z","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"[Synthetic demo] Background process evidence: startup, polling and exit. No real commands were executed."}]}} +{"type":"response_item","timestamp":"2026-09-24T08:00:02.000Z","payload":{"type":"function_call","call_id":"demo-process-launch","name":"exec_command","arguments":"{\"cmd\":\"npm test\",\"workdir\":\"/demo/process-evidence\"}"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:03.000Z","payload":{"type":"function_call_output","call_id":"demo-process-launch","output":"Chunk ID: synthetic\nWall time: 1 seconds\nProcess running with session ID 42\nFinal output:\nSynthetic test started; no exit result yet."}} +{"type":"response_item","timestamp":"2026-09-24T08:00:04.000Z","payload":{"type":"function_call","call_id":"demo-process-edit","name":"edit","arguments":"{\"path\":\"/demo/process-evidence/config.ts\",\"cwd\":\"/demo/process-evidence\",\"oldText\":\"before\",\"newText\":\"after\"}"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:05.000Z","payload":{"type":"function_call_output","call_id":"demo-process-edit","output":"Synthetic edit completed while the earlier test process was still running."}} +{"type":"response_item","timestamp":"2026-09-24T08:00:06.000Z","payload":{"type":"function_call","call_id":"demo-process-poll","name":"write_stdin","arguments":"{\"session_id\":42,\"chars\":\"\"}"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:07.000Z","payload":{"type":"function_call_output","call_id":"demo-process-poll","output":"Chunk ID: synthetic\nWall time: 1 seconds\nProcess exited with code 0\nFinal output:\nSynthetic test process finished. Its launch preceded the edit, so this is not a post-edit test."}} +{"type":"response_item","timestamp":"2026-09-24T08:00:08.000Z","payload":{"type":"function_call","call_id":"demo-process-pending","name":"exec_command","arguments":"{\"cmd\":\"pytest\",\"workdir\":\"/demo/process-evidence\"}"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:09.000Z","payload":{"type":"function_call_output","call_id":"demo-process-pending","output":"Chunk ID: synthetic\nWall time: 1 seconds\nProcess running with session ID 43\nFinal output:\nSynthetic second process has no subsequent completion in this static sample."}} +{"type":"response_item","timestamp":"2026-09-24T08:00:10.000Z","payload":{"type":"function_call","call_id":"demo-process-unlinked","name":"write_stdin","arguments":"{\"session_id\":99,\"chars\":\"\"}"}} +{"type":"response_item","timestamp":"2026-09-24T08:00:11.000Z","payload":{"type":"function_call_output","call_id":"demo-process-unlinked","output":"Chunk ID: synthetic\nWall time: 1 seconds\nProcess exited with code 0\nFinal output:\nSynthetic orphan poll: this transcript has no matching launch."}} +{"type":"response_item","timestamp":"2026-09-24T08:00:12.000Z","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"Synthetic exercise: expand Codex process evidence to trace process 42 to its final poll. Process 43 remains last-recorded running, and the poll for process 99 has no launch. Process exit is not proof of task correctness; no live results are appended here."}]}} diff --git a/frontend/src/demo/fixtures.json b/frontend/src/demo/fixtures.json index ccba545..308d952 100644 --- a/frontend/src/demo/fixtures.json +++ b/frontend/src/demo/fixtures.json @@ -67,6 +67,33 @@ } ], "codex": [ + { + "id": "01990000-0000-7000-8000-000000000199", + "timestamp": "2026-09-24T08:00:00.000Z", + "lastActivity": "2026-09-24T08:00:12.000Z", + "messageCount": 2, + "userCount": 1, + "assistantCount": 1, + "toolCallCount": 5, + "toolResultCount": 5, + "topTools": [ + { + "name": "exec_command", + "count": 2 + }, + { + "name": "write_stdin", + "count": 2 + }, + { + "name": "edit", + "count": 1 + } + ], + "firstUserMessage": "[Synthetic demo] Background process evidence: startup, polling and exit. No real commands were executed.", + "cwd": "/demo/process-evidence", + "file": "rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl" + }, { "id": "01991234-aaaa-7000-8000-syntheticdemo1", "timestamp": "2026-08-13T14:02:00.000Z", @@ -797,6 +824,208 @@ } ] }, + "codex/01990000-0000-7000-8000-000000000199": { + "session": { + "id": "01990000-0000-7000-8000-000000000199", + "cwd": "/demo/process-evidence", + "timestamp": "2026-09-24T08:00:00.000Z", + "version": null, + "model": null + }, + "messages": [ + { + "id": null, + "timestamp": "2026-09-24T08:00:01.000Z", + "role": "user", + "content": [ + { + "type": "text", + "text": "[Synthetic demo] Background process evidence: startup, polling and exit. No real commands were executed." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": null, + "toolName": null, + "details": null, + "isError": false + }, + { + "id": "demo-process-launch", + "timestamp": "2026-09-24T08:00:02.000Z", + "role": "toolCall", + "content": [], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-launch", + "toolName": "exec_command", + "details": "{\"cmd\":\"npm test\",\"workdir\":\"/demo/process-evidence\"}", + "isError": false + }, + { + "id": "demo-process-launch", + "timestamp": "2026-09-24T08:00:03.000Z", + "role": "toolResult", + "content": [ + { + "type": "text", + "text": "Chunk ID: synthetic\nWall time: 1 seconds\nProcess running with session ID 42\nFinal output:\nSynthetic test started; no exit result yet." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-launch", + "toolName": null, + "details": null, + "isError": false + }, + { + "id": "demo-process-edit", + "timestamp": "2026-09-24T08:00:04.000Z", + "role": "toolCall", + "content": [], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-edit", + "toolName": "edit", + "details": "{\"path\":\"/demo/process-evidence/config.ts\",\"cwd\":\"/demo/process-evidence\",\"oldText\":\"before\",\"newText\":\"after\"}", + "isError": false + }, + { + "id": "demo-process-edit", + "timestamp": "2026-09-24T08:00:05.000Z", + "role": "toolResult", + "content": [ + { + "type": "text", + "text": "Synthetic edit completed while the earlier test process was still running." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-edit", + "toolName": null, + "details": null, + "isError": false + }, + { + "id": "demo-process-poll", + "timestamp": "2026-09-24T08:00:06.000Z", + "role": "toolCall", + "content": [], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-poll", + "toolName": "write_stdin", + "details": "{\"session_id\":42,\"chars\":\"\"}", + "isError": false + }, + { + "id": "demo-process-poll", + "timestamp": "2026-09-24T08:00:07.000Z", + "role": "toolResult", + "content": [ + { + "type": "text", + "text": "Chunk ID: synthetic\nWall time: 1 seconds\nProcess exited with code 0\nFinal output:\nSynthetic test process finished. Its launch preceded the edit, so this is not a post-edit test." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-poll", + "toolName": null, + "details": null, + "isError": false + }, + { + "id": "demo-process-pending", + "timestamp": "2026-09-24T08:00:08.000Z", + "role": "toolCall", + "content": [], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-pending", + "toolName": "exec_command", + "details": "{\"cmd\":\"pytest\",\"workdir\":\"/demo/process-evidence\"}", + "isError": false + }, + { + "id": "demo-process-pending", + "timestamp": "2026-09-24T08:00:09.000Z", + "role": "toolResult", + "content": [ + { + "type": "text", + "text": "Chunk ID: synthetic\nWall time: 1 seconds\nProcess running with session ID 43\nFinal output:\nSynthetic second process has no subsequent completion in this static sample." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-pending", + "toolName": null, + "details": null, + "isError": false + }, + { + "id": "demo-process-unlinked", + "timestamp": "2026-09-24T08:00:10.000Z", + "role": "toolCall", + "content": [], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-unlinked", + "toolName": "write_stdin", + "details": "{\"session_id\":99,\"chars\":\"\"}", + "isError": false + }, + { + "id": "demo-process-unlinked", + "timestamp": "2026-09-24T08:00:11.000Z", + "role": "toolResult", + "content": [ + { + "type": "text", + "text": "Chunk ID: synthetic\nWall time: 1 seconds\nProcess exited with code 0\nFinal output:\nSynthetic orphan poll: this transcript has no matching launch." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": "demo-process-unlinked", + "toolName": null, + "details": null, + "isError": false + }, + { + "id": null, + "timestamp": "2026-09-24T08:00:12.000Z", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "Synthetic exercise: expand Codex process evidence to trace process 42 to its final poll. Process 43 remains last-recorded running, and the poll for process 99 has no launch. Process exit is not proof of task correctness; no live results are appended here." + } + ], + "usage": null, + "model": null, + "provider": null, + "toolCallId": null, + "toolName": null, + "details": null, + "isError": false + } + ] + }, "codex/01991234-aaaa-7000-8000-syntheticdemo1": { "session": { "id": "01991234-aaaa-7000-8000-syntheticdemo1", @@ -2317,9 +2546,9 @@ ] }, "codex": { - "totalSessions": 1, - "totalMessages": 3, - "totalToolCalls": 5, + "totalSessions": 2, + "totalMessages": 5, + "totalToolCalls": 10, "errorRate": 0, "totalCost": 0, "tokenUsage": { @@ -2342,6 +2571,27 @@ "errorRate": 0, "avgDurationMs": 0 }, + { + "name": "exec_command", + "calls": 2, + "errors": 0, + "errorRate": 0, + "avgDurationMs": 0 + }, + { + "name": "write_stdin", + "calls": 2, + "errors": 0, + "errorRate": 0, + "avgDurationMs": 0 + }, + { + "name": "edit", + "calls": 1, + "errors": 0, + "errorRate": 0, + "avgDurationMs": 0 + }, { "name": "tool", "calls": 0, @@ -2566,8 +2816,8 @@ }, "codex": { "platform": "codex", - "totalSessions": 1, - "totalPrompts": 1, + "totalSessions": 2, + "totalPrompts": 2, "groups": [ { "directory": "/demo/orders-service", @@ -2590,6 +2840,28 @@ ], "sessionCount": 1, "promptCount": 1 + }, + { + "directory": "/demo/process-evidence", + "sessions": [ + { + "id": "01990000-0000-7000-8000-000000000199", + "file": "rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl", + "timestamp": "2026-09-24T08:00:00.000Z", + "lastActivity": "2026-09-24T08:00:12.000Z", + "slug": null, + "title": null, + "promptCount": 1, + "prompts": [ + { + "text": "[Synthetic demo] Background process evidence: startup, polling and exit. No real commands were executed.", + "timestamp": "2026-09-24T08:00:01.000Z" + } + ] + } + ], + "sessionCount": 1, + "promptCount": 1 } ] }, diff --git a/frontend/src/views/sessions/CodexProcesses.tsx b/frontend/src/views/sessions/CodexProcesses.tsx new file mode 100644 index 0000000..b0668ef --- /dev/null +++ b/frontend/src/views/sessions/CodexProcesses.tsx @@ -0,0 +1,70 @@ +import { useState } from 'react'; +import type { analyzeCodexProcesses, CodexProcessEvidence } from './diagnostics'; +import type { SessionMessage } from '@/api/types'; +import { messageAnchorId } from './lib'; + +const STATES = { success: '进程退出：成功', failure: '进程退出：失败', running: '最后记录为运行中', unknown: '完成状态未知', 'no-result': '未记录返回结果' }; +const ISSUES: Record = { + 'ambiguous-launch': '启动调用或其结果不唯一', 'reused-process-id': '进程标识在本会话中重复使用', + 'ambiguous-poll': '轮询调用标识不唯一', 'overlapping-polls': '轮询调用交叠，结果顺序不明确', + 'poll-after-terminal': '终止结果后仍有轮询', 'unconfirmed-poll-result': '轮询结果缺失或包装头不可信', + 'mismatched-process-id': '返回的进程标识与调用参数不符', +}; + +function ResultLink({ message, onJump }: { message: SessionMessage | null; onJump: (id: string) => void }) { + const anchor = message && messageAnchorId(message); + return anchor ? : 没有可定位结果; +} + +export function ProcessEvidence({ process, onJump }: { process: CodexProcessEvidence; onJump: (id: string) => void }) { + return ( +
+ 进程 {process.processId} · {STATES[process.state]} · {process.polls.length} 次关联轮询 +

+ exec_command 启动消息 #{process.launchIndex + 1} → 返回 #{process.launchResultIndex + 1}； + 通过包装头 Process running with session ID 与 write_stdin.session_id 精确匹配。 + {process.exitCode !== null ? ` 最终退出码 ${process.exitCode}。` : ''} + {process.inputObserved ? ' 过程中发送过输入；不展示输入内容，也不视为未干预的验证。' : ''} + 仅当前日志的关联，不是实时进程监控或任务验收。 +

+ {process.issues.length ?

{process.issues.map((issue) => ISSUES[issue] || issue).join('；')}。不采用这些结果确认进程完成。

: null} +
+ {process.launchCallId ? : null} + +
+
    + {process.polls.map((poll) => ( +
  1. +

    轮询调用 #{poll.callIndex + 1}{poll.resultIndex !== null ? ` → 结果 #${poll.resultIndex + 1}` : ''} · {STATES[poll.state]}

    +

    {poll.exitCode !== null ? `包装头退出码 ${poll.exitCode} · ` : ''}{poll.hasInput ? '调用发送了输入' : '没有发送输入'}

    +
    + {poll.toolCallId ? : null} + +
    +
    2. + ))} +
+ {process.finalMessage ?
唯一关联的最终结果
: null} +
+ ); +} + +export function CodexProcesses({ report, onJump }: { report: ReturnType; onJump: (id: string) => void }) { + const [visible, setVisible] = useState(5); + if (!report.processes.length && !report.pollCalls) return null; + const terminal = report.processes.filter((process) => process.state === 'success' || process.state === 'failure').length; + return ( +
+ Codex 后台进程证据 · {report.processes.length} 次启动 · {terminal} 次明确退出 +

+ {report.linkedPolls} / {report.pollCalls} 次 write_stdin 关联到启动，{report.unlinkedPolls} 次未关联，{report.ambiguousCalls} 次调用标识不唯一。 + 与“每条工具调用的最后结果”分开统计，不改写历史失败或自动恢复。同一进程 ID 重用、交叠轮询或冲突结果保持未知。 +

+
+ {report.processes.slice(0, visible).map((process, index) => )} +
+ {visible < report.processes.length ? : null} +
+ ); +} diff --git a/frontend/src/views/sessions/SessionDiagnostics.tsx b/frontend/src/views/sessions/SessionDiagnostics.tsx index 71f36f0..9a50770 100644 --- a/frontend/src/views/sessions/SessionDiagnostics.tsx +++ b/frontend/src/views/sessions/SessionDiagnostics.tsx @@ -1,12 +1,13 @@ import { useMemo, useState } from 'react'; import type { SessionMessage } from '@/api/types'; -import { diagnoseSession, summarizeSessionHealth, analyzeVerificationChronology, type FailureDiagnostic, type FailureEvent } from './diagnostics'; +import { diagnoseSession, summarizeSessionHealth, analyzeVerificationChronology, analyzeCodexProcesses, type FailureDiagnostic, type FailureEvent } from './diagnostics'; import { formatDate, messageAnchorId } from './lib'; import { DiagnosticReview, useEventReviews } from './DiagnosticReview'; import { ReviewTransferPanel } from './ReviewTransferPanel'; import { RelatedOperations } from './RelatedOperations'; import { SessionHealth } from './SessionHealth'; import { VerificationChronology } from './VerificationChronology'; +import { CodexProcesses } from './CodexProcesses'; import { REVIEW_LABELS, reviewState, type EventReview, type ReviewState, type ReviewStatus } from './diagnostic-reviews'; const REASONS = { @@ -115,6 +116,7 @@ export function SessionDiagnostics({ messages, onScrollToMessage, reviewScope }: const report = useMemo(() => diagnoseSession(messages), [messages]); const health = useMemo(() => summarizeSessionHealth(messages, report), [messages, report]); const chronology = useMemo(() => analyzeVerificationChronology(messages), [messages]); + const processes = useMemo(() => analyzeCodexProcesses(messages), [messages]); const [manual, setManual] = useState(false); const [visibleCount, setVisibleCount] = useState(PAGE_SIZE); const [filter, setFilter] = useState('unreviewed'); @@ -134,6 +136,7 @@ export function SessionDiagnostics({ messages, onScrollToMessage, reviewScope }: 自动整理已加载日志中的事实，优先展示重复操作并保留证据。不评价任务成败，不把缺少结果等同于正在运行。

+
如何分组与判定 diff --git a/frontend/src/views/sessions/VerificationChronology.tsx b/frontend/src/views/sessions/VerificationChronology.tsx index a351e6f..d316256 100644 --- a/frontend/src/views/sessions/VerificationChronology.tsx +++ b/frontend/src/views/sessions/VerificationChronology.tsx @@ -1,6 +1,7 @@ import { useState } from 'react'; import type { analyzeVerificationChronology, ChronologyCheck, ExecutionEvidence } from './diagnostics'; import { messageAnchorId } from './lib'; +import { ProcessEvidence } from './CodexProcesses'; const STATES = { success: '成功返回', failure: '失败', running: '记录为执行中', cancelled: '取消 / 停止', unknown: '状态未知', 'no-result': '未记录结果' }; @@ -22,6 +23,7 @@ function OperationEvidence({ operation, onJump }: { operation: ExecutionEvidence {callAnchor ? : null} {anchor ? : null} + {operation.processEvidence ? : null} ); } diff --git a/frontend/src/views/sessions/diagnostics.ts b/frontend/src/views/sessions/diagnostics.ts index 6d97f0f..b30b0bb 100644 --- a/frontend/src/views/sessions/diagnostics.ts +++ b/frontend/src/views/sessions/diagnostics.ts @@ -381,6 +381,132 @@ export function summarizeSessionHealth(messages: SessionMessage[], report = diag }; } +export interface ProcessPoll { + callMessage: SessionMessage; + callIndex: number; + toolCallId: string | null; + resultMessage: SessionMessage | null; + resultIndex: number | null; + state: 'success' | 'failure' | 'running' | 'unknown' | 'no-result'; + exitCode: number | null; + hasInput: boolean; +} + +export interface CodexProcessEvidence { + processId: number; + launchMessage: SessionMessage; + launchIndex: number; + launchCallId: string | null; + launchResult: SessionMessage; + launchResultIndex: number; + polls: ProcessPoll[]; + state: 'success' | 'failure' | 'running' | 'unknown'; + exitCode: number | null; + finalMessage: SessionMessage | null; + finalIndex: number | null; + inputObserved: boolean; + issues: string[]; +} + +function codexTool(name: string): string { + return name.startsWith('functions.') ? name.slice('functions.'.length) : name; +} + +function codexEnvelope(message: SessionMessage): { processId: number | null; exitCode: number | null } | null { + if (message.ompOutcome || message.isError) return null; + const text = resultText(message); + if (!/^(?:Chunk ID: [^\n]+\n)?Wall time: [\d.]+ seconds\n/.test(text)) return null; + const boundary = text.search(/^(?:Final output|Output):\s*$/m); + if (boundary < 0) return null; + const status = text.slice(0, boundary).split('\n').filter((line) => /^(?:Process (?:running|exited)|Exit code:)/.test(line)); + if (status.length !== 1) return null; + const running = status[0].match(/^Process running with session ID (0|[1-9]\d*)$/); + const terminal = status[0].match(/^(?:Process exited with code |Exit code: )(-?\d+)$/); + if (running && Number.isSafeInteger(Number(running[1]))) { + if (message.details?.exitCode != null || message.details?.exit_code != null) return null; + return { processId: Number(running[1]), exitCode: null }; + } + if (terminal && Number.isSafeInteger(Number(terminal[1]))) { + const code = Number(terminal[1]); + const recordedCode = message.details?.exitCode ?? message.details?.exit_code; + if (recordedCode != null && recordedCode !== code) return null; + return { processId: null, exitCode: code }; + } + return null; +} + +export function analyzeCodexProcesses(messages: SessionMessage[]) { + const { calls, byId } = collectCallEvidence(messages); + const resultSets = new Map(); + messages.forEach((message, index) => { + if (message.role !== 'toolResult' || !message.toolCallId) return; + const rows = resultSets.get(message.toolCallId) || []; + rows.push({ message, index }); resultSets.set(message.toolCallId, rows); + }); + const processes: CodexProcessEvidence[] = []; + const byProcess = new Map(); + const unique = (entry: CallEvidence) => !!entry.id && byId.get(entry.id)?.length === 1; + for (const entry of calls) { + if (codexTool(entry.call.name) !== 'exec_command') continue; + const results = entry.id ? resultSets.get(entry.id) || [] : []; + const start = results.find((row) => row.index > entry.call.index && codexEnvelope(row.message)?.processId != null); + if (!start) continue; + const processId = codexEnvelope(start.message)!.processId!; + const issues = !unique(entry) || results.length !== 1 ? ['ambiguous-launch'] : []; + const process: CodexProcessEvidence = { processId, launchMessage: entry.message, launchIndex: entry.call.index, + launchCallId: entry.id, launchResult: start.message, launchResultIndex: start.index, polls: [], + state: 'running', exitCode: null, finalMessage: null, finalIndex: null, inputObserved: false, issues }; + processes.push(process); + const group = byProcess.get(processId) || []; group.push(process); byProcess.set(processId, group); + } + for (const group of byProcess.values()) if (group.length > 1) for (const process of group) process.issues.push('reused-process-id'); + let pollCalls = 0; + let linkedPolls = 0; + let ambiguousCalls = 0; + for (const entry of calls) { + if (codexTool(entry.call.name) !== 'write_stdin') continue; + pollCalls++; + const args = argumentObject(entry.call.args); + const id = args?.session_id; + const group = typeof id === 'number' && Number.isSafeInteger(id) && id >= 0 ? byProcess.get(id) : undefined; + if (!unique(entry)) { + ambiguousCalls++; + if (group?.length === 1) group[0].issues.push('ambiguous-poll'); + continue; + } + if (group?.length !== 1 || group[0].issues.includes('ambiguous-launch') || entry.call.index <= group[0].launchResultIndex) continue; + const process = group[0]; + const rows = resultSets.get(entry.id!) || []; + const result = rows.length === 1 && rows[0].index > entry.call.index ? rows[0] : undefined; + const parsed = result ? codexEnvelope(result.message) : null; + const hasInput = args?.chars != null && args.chars !== ''; + const poll: ProcessPoll = { callMessage: entry.message, callIndex: entry.call.index, toolCallId: entry.id, + resultMessage: result?.message ?? null, resultIndex: result?.index ?? null, hasInput, + state: !rows.length ? 'no-result' : !parsed ? 'unknown' : parsed.exitCode !== null ? parsed.exitCode === 0 ? 'success' : 'failure' : 'running', + exitCode: parsed?.exitCode ?? null }; + const previous = process.polls[process.polls.length - 1]; + if (previous && (previous.resultIndex === null || previous.resultIndex >= entry.call.index)) process.issues.push('overlapping-polls'); + if (process.finalMessage) process.issues.push('poll-after-terminal'); + if (!result || !parsed) process.issues.push('unconfirmed-poll-result'); + if (parsed?.processId != null && parsed.processId !== process.processId) process.issues.push('mismatched-process-id'); + process.polls.push(poll); linkedPolls++; + process.inputObserved ||= hasInput; + if (parsed?.exitCode != null) { + process.finalMessage = result!.message; + process.finalIndex = result!.index; + process.exitCode = parsed.exitCode; + process.state = parsed.exitCode === 0 ? 'success' : 'failure'; + } + } + for (const process of processes) { + process.issues = [...new Set(process.issues)]; + if (process.issues.length) { + process.state = 'unknown'; process.exitCode = null; process.finalMessage = null; process.finalIndex = null; + } + } + return { processes, pollCalls, linkedPolls, unlinkedPolls: pollCalls - linkedPolls, ambiguousCalls }; +} + export interface ExecutionEvidence { callMessage: SessionMessage; callIndex: number; @@ -397,6 +523,7 @@ export interface ExecutionEvidence { basis?: 'runner-command' | 'script-name'; commandMode?: 'direct' | 'directory-prefix' | 'compound-fragment'; fragments?: string[]; + processEvidence?: CodexProcessEvidence; } export interface ChronologyCheck { @@ -481,7 +608,7 @@ function verificationCommand(call: RecordedCall): { command: string; basis: 'runner-command' | 'script-name'; commandMode: 'direct' | 'directory-prefix' | 'compound-fragment'; fragments: string[]; explicitDirectory: string | null; usesInitialDirectory: boolean; } | null { - if (!['bash', 'Bash', 'shell', 'exec', 'exec_command', 'run_shell_command', 'execute_command', 'terminal'].includes(call.name)) return null; + if (!['bash', 'Bash', 'shell', 'exec', 'exec_command', 'run_shell_command', 'execute_command', 'terminal'].includes(codexTool(call.name))) return null; const args = argumentObject(call.args); if (!args) return null; const values = ['command', 'cmd'].filter((key) => key in args).map((key) => args[key]); @@ -527,6 +654,7 @@ function executionEvidence(entry: CallEvidence, ambiguous: boolean): ExecutionEv export function analyzeVerificationChronology(messages: SessionMessage[]) { const { calls, byId } = collectCallEvidence(messages); + const processByLaunch = new Map(analyzeCodexProcesses(messages).processes.map((process) => [process.launchCallId, process])); const modificationResults: ExecutionEvidence[] = []; const checks: ExecutionEvidence[] = []; let unclassifiedShellCalls = 0; @@ -547,6 +675,15 @@ export function analyzeVerificationChronology(messages: SessionMessage[]) { const recognized = verificationCommand(entry.call); if (recognized) { const evidence = executionEvidence(entry, ambiguous); + const process = processByLaunch.get(entry.id); + if (process && process.launchIndex === entry.call.index) { + evidence.processEvidence = process; + evidence.state = process.inputObserved ? 'unknown' : process.state; + evidence.resultMessage = process.finalMessage; + evidence.resultIndex = process.finalIndex; + evidence.evidence = `通过 exec_command 包装头进程标识 ${process.processId} 与 write_stdin.session_id 关联。${process.inputObserved ? '过程中发送过输入，不能视为未干预的检查结果。' : ''}\n` + + (process.finalMessage ? `最终进程退出码 ${process.exitCode}\n${resultText(process.finalMessage).slice(0, 400)}` : `未得到唯一可信的最终退出结果；${process.issues.join(', ') || '最后记录为执行中'}。`); + } const { explicitDirectory, usesInitialDirectory, ...command } = recognized; if (recognized.commandMode === 'compound-fragment') { checks.push({ ...evidence, ...command, state: evidence.state === 'no-result' ? 'no-result' : 'unknown', diff --git a/intent.md b/intent.md index c972fec..d8fb2c9 100644 --- a/intent.md +++ b/intent.md @@ -99,7 +99,18 @@ Real-log coverage refinement: the initial direct-only classifier recognized zero Validation: 290 tests pass, including 200 fixed health transformations and 120 temporal transformations without human labels. The frozen 15-session/2,024-result set retains exact v1.20 diagnostic output. It yields 59 check-containing calls, all with unknown fragment outcome; no check-coverage or task-success claim is made. Explicit edit/write tools return successfully 240 times; 112 have later check-containing calls and 128 do not. Browser checks cover default zero review-storage reads, optional-note preservation, live missing-result resolution, before/overlap/after chronology, source navigation and 390px rendering. Prepare the combined increment as v1.21.0 through the authorized protected PR/release path after final gates; keep the continuing project goal active, and do not claim real productivity or adoption from this release. -## Continuing non-goals +## Explicit Codex process evidence (2026-09-24) + +- Real frozen Codex logs contain 60 exec_command background-start envelopes and 53 write_stdin calls. Improve the missing cross-call lifecycle evidence instead of adding generic metrics. No SDK, new API or execution is needed. +- Parse only recognized Codex wrapper headers before Output/Final output, on exec_command/write_stdin (including functions.* names). Do not interpret matching text in stdout or outputs of read tools. Require an unambiguous numeric process ID and unique call IDs within the loaded transcript; no cross-session/child joins. +- Link write_stdin session_id to a single recorded launcher only after that launch result. Preserve every polling call/result, whether input was sent, and terminal exit code. A reused process ID, overlapping polls, mismatching wrapper ID, malformed/error result or conflicting terminal sequence cannot certify completion. +- Show process lifecycle separately from per-tool-call last states: polling is not another launch. Never rewrite historical call results, existing failure/event/recovery rules or human review fingerprints. A process's last running observation is not a live status query. +- Verification chronology may use a uniquely associated terminal result, but must retain the original launch index/time. A check started before a modification and finished through a later poll remains overlapping, not post-change verification. Compound shell fragments still have unknown check outcomes. Input sent to a running process weakens check certainty; it is not silently treated as a clean test run. +- Add deterministic counterexamples and real frozen-log relationship checks before publication. Keep detailed process IDs, log text and commands private; public examples/screenshots are synthetic. Whole-process exit code is not subcommand coverage or task correctness. + +Validation results: the frozen corpus has 60 Codex starts and 53 uniquely linked polls: 48 successful process exits, 1 failed exit and 11 last-recorded running starts. All prior per-call health and failure/candidate/recovery outputs remain identical. The implementation adds 23 tests including 40 fixed terminal transformations and generated-demo/raw-parser parity. Synthetic browser checks cover terminal-result jumps, original-start overlap semantics, unlinked poll counts, live completion and 390px layout. Prepare v1.22.0 through the previously authorized protected PR/release workflow after final checks; no private evidence or logs are published. + +## Continuing boundaries - No new platform, dependency, model call, account, telemetry, cloud log storage or automatic command execution. - No automatic human judgments of real sessions, relaxed argument matching or fabricated accuracy/productivity/adoption claims. diff --git a/package-lock.json b/package-lock.json index 4f596fa..a1e0ba6 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@alloevil/agent-xray", - "version": "1.21.0", + "version": "1.22.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@alloevil/agent-xray", - "version": "1.21.0", + "version": "1.22.0", "license": "MIT", "dependencies": { "express": "^4.21.2" diff --git a/package.json b/package.json index e8df681..84a9d3d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@alloevil/agent-xray", - "version": "1.21.0", + "version": "1.22.0", "description": "Web dashboard for viewing AI agent session logs — supports OpenClaw, Codex, Claude Code, Hermes, OMP, DeepSeek Harness, and Gemini CLI", "main": "server.js", "bin": { diff --git a/screenshots/codex-process-evidence.png b/screenshots/codex-process-evidence.png new file mode 100644 index 0000000..28cd774 Binary files /dev/null and b/screenshots/codex-process-evidence.png differ diff --git a/scripts/demo-process-evidence.cjs b/scripts/demo-process-evidence.cjs new file mode 100644 index 0000000..de6b858 --- /dev/null +++ b/scripts/demo-process-evidence.cjs @@ -0,0 +1,86 @@ +const fs = require('node:fs/promises'); +const path = require('node:path'); +const readline = require('node:readline'); +const { startServer } = require('../test/helpers'); + +async function main() { + const server = await startServer(); + const id = '01990000-0000-7000-8000-000000000199'; + const directory = path.join(server.home, '.codex/sessions/2026/09/24'); + const file = path.join(directory, `rollout-2026-09-24T08-00-00-${id}.jsonl`); + let tick = 0; + const stamp = () => new Date(Date.UTC(2026, 8, 24, 8) + tick++ * 1000).toISOString(); + const record = (payload) => ({ type: 'response_item', timestamp: stamp(), payload }); + const call = (callId, name, args) => + record({ type: 'function_call', call_id: callId, name, arguments: JSON.stringify(args) }); + const result = (callId, status) => + record({ + type: 'function_call_output', + call_id: callId, + output: `Chunk ID: synthetic\nWall time: 1 seconds\n${status}\nFinal output:\nSynthetic result only; no command was executed.`, + }); + const records = [ + { type: 'session_meta', timestamp: stamp(), payload: { id, cwd: '/synthetic', timestamp: stamp() } }, + record({ + type: 'message', + role: 'user', + content: [{ type: 'input_text', text: '[Synthetic] 后台进程：启动不是完成，轮询给出退出证据' }], + }), + call('launch-pass', 'exec_command', { cmd: 'npm test', workdir: '/synthetic' }), + result('launch-pass', 'Process running with session ID 42'), + call('edit', 'edit', { path: '/synthetic/config.ts', cwd: '/synthetic' }), + result('edit', 'Process exited with code 0'), + call('poll-pass', 'write_stdin', { session_id: 42, chars: '' }), + result('poll-pass', 'Process exited with code 0'), + call('launch-pending', 'exec_command', { cmd: 'pytest', workdir: '/synthetic' }), + result('launch-pending', 'Process running with session ID 43'), + call('launch-fail', 'exec_command', { cmd: 'npm run lint', workdir: '/synthetic' }), + result('launch-fail', 'Process running with session ID 44'), + call('poll-fail', 'write_stdin', { session_id: 44, chars: '' }), + result('poll-fail', 'Process exited with code 2'), + call('unlinked-poll', 'write_stdin', { session_id: 99, chars: '' }), + result('unlinked-poll', 'Process exited with code 0'), + ]; + try { + await fs.mkdir(directory, { recursive: true }); + await fs.writeFile(file, `${records.map((item) => JSON.stringify(item)).join('\n')}\n`); + } catch (error) { + await server.stop(); + throw error; + } + console.log(`Synthetic process evidence demo: ${server.base}`); + console.log('选择 Codex 中的 [Synthetic] 会话：3 次后台启动，2 次明确退出，1 次未关联轮询。'); + console.log('第一项测试启动早于修改，最终退出仍算重叠。输入 c 为第二个进程追加退出结果；q 或 Ctrl-C 清理。'); + const input = readline.createInterface({ input: process.stdin }); + let completed = false; + const stop = async () => { + input.close(); + await server.stop(); + process.exit(0); + }; + input.on('line', async (line) => { + if (line.trim() === 'q') return stop(); + if (line.trim() !== 'c' || completed) return; + completed = true; + try { + const extra = [ + call('poll-pending', 'write_stdin', { session_id: 43, chars: '' }), + result('poll-pending', 'Process exited with code 0'), + ]; + await fs.appendFile(file, `${extra.map((item) => JSON.stringify(item)).join('\n')}\n`); + console.log( + 'PASS: appended process completion; third terminal process is visible without changing launch history.' + ); + } catch (error) { + console.error(error.message); + await stop(); + } + }); + process.on('SIGINT', stop); + process.on('SIGTERM', stop); +} + +main().catch((error) => { + console.error(error.message); + process.exitCode = 1; +}); diff --git a/test/codex-process-evidence.test.js b/test/codex-process-evidence.test.js new file mode 100644 index 0000000..c5b2d2c --- /dev/null +++ b/test/codex-process-evidence.test.js @@ -0,0 +1,317 @@ +const { test, before } = require('node:test'); +const assert = require('node:assert/strict'); +const { readFileSync } = require('node:fs'); +const path = require('node:path'); +const { stripTypeScriptTypes } = require('node:module'); + +let api; +before(async () => { + const source = readFileSync(path.join(__dirname, '../frontend/src/views/sessions/diagnostics.ts'), 'utf8'); + api = await import(`data:text/javascript;base64,${Buffer.from(stripTypeScriptTypes(source)).toString('base64')}`); +}); +const call = (id, name = 'exec_command', args = { cmd: 'npm test', workdir: '/synthetic' }) => ({ + id, + toolCallId: id, + toolName: name, + details: args, + role: 'toolCall', + content: null, +}); +const poll = (id, session = 42, chars = '') => + call(id, 'write_stdin', { session_id: session, chars, yield_time_ms: 1000 }); +const output = (id, header, text = 'Synthetic output', extra = {}) => ({ + id: `${id}-result`, + toolCallId: id, + toolName: null, + role: 'toolResult', + isError: false, + details: null, + content: [ + { + type: 'text', + text: `Chunk ID: synthetic\nWall time: 1 seconds\n${header}\nOriginal token count: 4\nFinal output:\n${text}`, + }, + ], + ...extra, +}); +const running = (id, session = 42) => output(id, `Process running with session ID ${session}`); +const exited = (id, code = 0, extra = {}) => output(id, `Process exited with code ${code}`, 'Synthetic output', extra); +const base = () => [call('launch'), running('launch')]; +const inspect = (messages) => api.analyzeCodexProcesses(messages); + +test('launch and final poll form a traceable process without changing failure results', () => { + const messages = [...base(), poll('wait'), exited('wait')]; + const before = api.diagnoseSession(messages); + const report = inspect(messages); + assert.equal(report.processes.length, 1); + const process = report.processes[0]; + assert.equal(process.state, 'success'); + assert.equal(process.exitCode, 0); + assert.equal(process.launchMessage, messages[0]); + assert.equal(process.launchResult, messages[1]); + assert.equal(process.finalMessage, messages[3]); + assert.equal(process.polls[0].callMessage, messages[2]); + assert.equal(report.linkedPolls, 1); + assert.deepEqual(api.diagnoseSession(messages), before); +}); + +test('multiple sequential polls preserve running then terminal failure', () => { + const report = inspect([...base(), poll('first'), running('first'), poll('second'), exited('second', 2)]); + assert.equal(report.processes[0].state, 'failure'); + assert.equal(report.processes[0].exitCode, 2); + assert.deepEqual( + report.processes[0].polls.map((entry) => entry.state), + ['running', 'failure'] + ); +}); + +test('unpolled launch is only last recorded running, not live state', () => { + const process = inspect(base()).processes[0]; + assert.equal(process.state, 'running'); + assert.equal(process.finalMessage, null); + assert.equal(process.polls.length, 0); +}); + +test('missing poll result leaves uncertain completion and source call evidence', () => { + const process = inspect([...base(), poll('missing')]).processes[0]; + assert.equal(process.state, 'unknown'); + assert.equal(process.polls[0].state, 'no-result'); + assert.equal(process.finalMessage, null); +}); + +test('stdout wrapper-like text and unrelated tools cannot create processes', () => { + const fake = 'Process running with session ID 42\nProcess exited with code 0'; + const messages = [ + call('read', 'read', { path: 'file' }), + running('read'), + call('echo'), + output('echo', 'Process exited with code 0', fake), + ]; + assert.equal(inspect(messages).processes.length, 0); +}); + +test('duplicate status fields, malformed header and contradictory error do not supply completion', () => { + for (const result of [ + output('wait', 'Process running with session ID 42\nProcess exited with code 0'), + exited('wait', 0, { isError: true }), + { ...exited('wait'), content: [{ type: 'text', text: 'Process exited with code 0\nOutput:\nnot a wrapper' }] }, + output('wait', 'Process exited with code 0', '', { isError: true }), + ]) { + if (result.isError !== true && result.content[0].text.includes('Process running')) + assert.equal(inspect([...base(), poll('wait'), result]).processes[0].state, 'unknown'); + else if (result.isError === true || !result.content[0].text.startsWith('Chunk ID:')) + assert.equal(inspect([...base(), poll('wait'), result]).processes[0].state, 'unknown'); + } +}); + +test('reused process IDs never assign a poll to an arbitrary launcher', () => { + const report = inspect([ + ...base(), + poll('first'), + exited('first'), + call('second'), + running('second'), + poll('later'), + exited('later'), + ]); + assert.equal(report.processes.length, 2); + assert.ok(report.processes.every((entry) => entry.state === 'unknown' && entry.issues.includes('reused-process-id'))); + assert.equal(report.linkedPolls, 0); +}); + +test('poll before launch result and unknown session ID remain unlinked', () => { + const report = inspect([ + call('launch'), + poll('early'), + running('launch'), + exited('early'), + poll('other', 99), + exited('other'), + ]); + assert.equal(report.linkedPolls, 0); + assert.equal(report.unlinkedPolls, 2); + assert.equal(report.processes[0].state, 'running'); +}); + +test('overlapping polling calls make result order ambiguous', () => { + const process = inspect([...base(), poll('one'), poll('two'), exited('two'), running('one')]).processes[0]; + assert.equal(process.state, 'unknown'); + assert.ok(process.issues.includes('overlapping-polls')); +}); + +test('wrapper running ID mismatch invalidates process completion', () => { + const process = inspect([...base(), poll('one'), running('one', 99), poll('two'), exited('two')]).processes[0]; + assert.equal(process.state, 'unknown'); + assert.ok(process.issues.includes('mismatched-process-id')); +}); + +test('polls after terminal result do not silently replace conflicting history', () => { + const process = inspect([...base(), poll('one'), exited('one'), poll('two'), exited('two', 1)]).processes[0]; + assert.equal(process.state, 'unknown'); + assert.ok(process.issues.includes('poll-after-terminal')); +}); + +test('input is recorded without persisting or displaying its actual contents', () => { + const process = inspect([...base(), poll('input', 42, '\u0003'), exited('input', 130)]).processes[0]; + assert.equal(process.inputObserved, true); + assert.equal(process.state, 'failure'); + assert.equal(process.polls[0].hasInput, true); +}); + +test('only canonical safe numeric IDs and unique tool call IDs are linked', () => { + for (const id of ['42', '0042', -1, 4.2, 'anything', null]) { + const report = inspect([...base(), poll('wait', id), exited('wait')]); + assert.equal(report.linkedPolls, 0); + } + const duplicate = inspect([...base(), poll('dup'), exited('dup'), poll('dup'), exited('dup')]); + assert.equal(duplicate.linkedPolls, 0); + assert.equal(duplicate.ambiguousCalls, 2); +}); + +test('functions-qualified tools are accepted but OMP and arbitrary tool namespaces are not', () => { + const messages = [ + call('launch', 'functions.exec_command'), + running('launch'), + call('wait', 'functions.write_stdin', { session_id: 42 }), + exited('wait'), + ]; + assert.equal(inspect(messages).processes[0].state, 'success'); + messages[0].toolName = 'other.exec_command'; + assert.equal(inspect(messages).processes.length, 0); + assert.equal( + inspect([call('launch'), { ...running('launch'), ompOutcome: { state: 'running', evidence: [], warnings: [] } }]) + .processes.length, + 0 + ); +}); + +test('poll completion can refine a direct check but preserves original start for overlap', () => { + const edit = call('edit', 'edit', { path: '/synthetic/a', cwd: '/synthetic' }); + const changed = { ...exited('edit'), isError: false, details: { exitCode: 0 } }; + const messages = [...base(), edit, changed, poll('wait'), exited('wait')]; + const report = api.analyzeVerificationChronology(messages); + const check = report.checks[0]; + assert.equal(check.state, 'success'); + assert.equal(check.callIndex, 0); + assert.equal(check.resultIndex, 5); + assert.equal(check.processEvidence.processId, 42); + assert.equal(report.modifications[0].overlappingChecks.length, 1); + assert.equal(report.modifications[0].laterChecks.length, 0); +}); + +test('compound check or input-fed process cannot certify a passing check', () => { + for (const [command, chars] of [ + ['npm test | tail -20', ''], + ['npm test', 'answer\n'], + ]) { + const messages = [ + call('launch', 'exec_command', { cmd: command }), + running('launch'), + poll('wait', 42, chars), + exited('wait'), + ]; + const report = api.analyzeVerificationChronology(messages); + assert.equal(report.checks[0].state, 'unknown'); + } +}); + +test('per-call health and historical diagnostic counts remain unchanged', () => { + const messages = [...base(), poll('wait'), exited('wait')]; + const before = JSON.stringify(messages); + const health = api.summarizeSessionHealth(messages); + assert.equal(health.callStates.unknown, 1); + assert.equal(health.callStates.success, 1); + const process = inspect(messages).processes[0]; + assert.equal(process.state, 'success'); + assert.equal(JSON.stringify(messages), before); +}); + +test('duplicate poll results and contradictory structured exit codes do not certify completion', () => { + for (const tail of [ + [poll('wait'), exited('wait'), exited('wait')], + [poll('wait'), exited('wait', 0, { details: { exitCode: 1 } })], + ]) { + const process = inspect([...base(), ...tail]).processes[0]; + assert.equal(process.state, 'unknown'); + assert.equal(process.finalMessage, null); + } +}); + +test('incomplete launch wrapper and huge IDs are not linkable process evidence', () => { + for (const message of [ + output('launch', 'Process running with session ID 99999999999999999999'), + output('launch', 'Process running with session ID 0042'), + output('launch', 'Process running with session ID 42', '', { details: { exitCode: 0 } }), + ]) { + assert.equal(inspect([call('launch'), message]).processes.length, 0); + } +}); + +test('poll completion before modification is valid prior-check evidence; later launch is later', () => { + const edit = call('edit', 'edit', { path: '/synthetic/a', cwd: '/synthetic' }); + const mutation = { ...exited('edit'), details: { exitCode: 0 } }; + const prior = api.analyzeVerificationChronology([...base(), poll('wait'), exited('wait'), edit, mutation]); + assert.equal(prior.modifications[0].priorSuccess.operation.processEvidence.processId, 42); + const later = api.analyzeVerificationChronology([edit, mutation, ...base(), poll('wait'), exited('wait')]); + assert.equal(later.modifications[0].latestLater.operation.state, 'success'); +}); + +test('incrementally appended completion changes process evidence, not history or recovery', (context) => { + let cases = 0; + for (let processId = 1; processId <= 20; processId++) { + const messages = [call('launch'), running('launch', processId)]; + const before = inspect(messages); + for (const code of [0, 1]) { + const after = inspect([...messages, poll('wait', processId), exited('wait', code)]); + assert.equal(after.processes[0].state, code === 0 ? 'success' : 'failure'); + assert.equal(before.processes[0].state, 'running'); + assert.equal(api.diagnoseSession([...messages, poll('wait', processId), exited('wait', code)]).recoveredCount, 0); + cases++; + } + } + context.diagnostic(`Process terminal transformations: ${cases}/40 passed; no automatic recovery inferred.`); +}); + +test('multiple launches in one assistant record retain separate process evidence', () => { + const embedded = { + role: 'assistant', + content: [ + { type: 'toolCall', id: 'first', name: 'exec_command', arguments: { cmd: 'npm test' } }, + { type: 'toolCall', id: 'second', name: 'exec_command', arguments: { cmd: 'pytest' } }, + ], + }; + const messages = [ + embedded, + running('first', 42), + running('second', 43), + poll('wait-first', 42), + exited('wait-first'), + poll('wait-second', 43), + exited('wait-second', 1), + ]; + const report = api.analyzeVerificationChronology(messages); + assert.equal(report.checks[0].processEvidence.processId, 42); + assert.equal(report.checks[0].state, 'success'); + assert.equal(report.checks[1].processEvidence.processId, 43); + assert.equal(report.checks[1].state, 'failure'); +}); + +test('hosted process sample matches real parser output and preserves overlap boundaries', async () => { + const { parseCodexSessionFile } = require('../lib/platforms/codex'); + const filename = path.join( + __dirname, + '../frontend/demo/sample-logs/codex/2026/09/24/rollout-2026-09-24T08-00-00-01990000-0000-7000-8000-000000000199.jsonl' + ); + const detail = await parseCodexSessionFile(filename); + const fixtures = JSON.parse(readFileSync(path.join(__dirname, '../frontend/src/demo/fixtures.json'), 'utf8')); + assert.deepEqual(fixtures.details['codex/01990000-0000-7000-8000-000000000199']?.messages, detail.messages); + const processes = inspect(detail.messages); + assert.equal(processes.processes.length, 2); + assert.equal(processes.processes[0].state, 'success'); + assert.equal(processes.processes[1].state, 'running'); + assert.equal(processes.linkedPolls, 1); + assert.equal(processes.unlinkedPolls, 1); + const chronology = api.analyzeVerificationChronology(detail.messages); + assert.equal(chronology.modifications[0].overlappingChecks.length, 1); + assert.equal(chronology.modifications[0].latestLater.operation.state, 'running'); +});