diff --git a/.devin/wiki.json b/.devin/wiki.json new file mode 100644 index 0000000..059f823 --- /dev/null +++ b/.devin/wiki.json @@ -0,0 +1,41 @@ +{ + "repo_notes": [ + { + "content": "Document the current source tree and the active README, docs/RUNTIME.md, docs/BUYER.md, and platform READMEs. This is one retail brand's own store, not a multi-merchant marketplace. ShopMate owns the Python API and buyer/merchant/analysis agents; CityBuddy is a separate repository providing Java Auth/Commerce services. Do not conflate CityBuddy agent-service or its engineering Web console with this repository's current agent host and merchant workspace. Trace every architecture arrow to actual code. The host uses a SELECT-only MySQL reporting account; analysis_runner.py passes complete bounded tables and code to analysis_sandbox.py. The Python sandbox has no network, database connection, or credentials. SQLite/WAL holds single-instance host conversations, recovery intents, and memory; Java/MySQL owns identity and transactions. Native flash-sale reservation/status calls go directly to Commerce, while simulated-payment orchestration goes through the ShopMate host. Android uses native Jetpack Compose and iOS native SwiftUI; KMP shares business/protocol rules, not the UI. Historical demo, transport and evaluation records retain their original measured revisions: Next.js references there are historical, not the current React/Vite workspace. Distinguish raw quoted prompts from documentation instructions. Do not invent performance, production capacity, multi-instance support, or authorization effectiveness from unactivated trials." + } + ], + "pages": [ + { + "title": "Overview and architecture", + "purpose": "Explain the product, one-store business scope, native clients, Python host, external CityBuddy Auth/Commerce, and exact data/analysis boundaries. Use README.md, docs/RUNTIME.md, and actual routes." + }, + { + "title": "Local setup and repository guide", + "purpose": "Document current run/build/check commands from active runtime and platform READMEs; explain directory ownership, product site, and contribution workflow." + }, + { + "title": "Native clients and shared Kotlin core", + "purpose": "Describe Android Compose and iOS SwiftUI, shared SSE decoding/reduction/recovery rules, native lifecycle/network/storage, streaming rendering and reading position. Cover android/, ios/, shared/ and relevant tests." + }, + { + "title": "Buyer shopping and recovery", + "purpose": "Trace ordinary shopping, conversation ownership, user/OBO identities, request keys, quote confirmation, lost-response recovery, refunds and direct flash-sale calls using buyer routes, clients, sessions, and docs/BUYER.md." + }, + { + "title": "Merchant workspace and controlled execution", + "purpose": "Describe current React/Vite web/ UI, proposal previews, operator approval, Java receipts and concurrent ordinary/agent operations. Trace actual direct-user and delegated paths without treating all confirmations as the same token type." + }, + { + "title": "Agent analysis, sandbox, memory and budgets", + "purpose": "Cover buyer/merchant/analysis factories, bounded shared model/tool budgets, read-only host SQL, complete-table handoff to the no-network Python sandbox, skills, editable memory and model transport." + }, + { + "title": "Tests and measured evaluations", + "purpose": "Explain native/host/integration checks and retail acceptance. Keep retail-v1, retail-v2 and historical assessments separate by full measured SHA and denominator; report failures, provider errors, timing boundaries and non-activated ownership trials accurately." + }, + { + "title": "Product showcase and attribution", + "purpose": "Explain site/ as an independently built GitHub Pages showcase, local full-stack demonstration, source/asset provenance and vendored licensing. Do not present site interactions as a public live transaction backend." + } + ] +} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1a95ed3..3008ede 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,7 +11,7 @@ uv sync --frozen npm --prefix web ci ``` -The [runtime guide](docs/RUNTIME.md#本地运行) covers the sibling CityBuddy checkout, Java 21, Docker Compose, and local service configuration. Native prerequisites and build commands are in the [Android](android/README.md) and [iOS](ios/README.md) guides. The [product site](site/README.md) can be previewed independently. +The [runtime guide](docs/RUNTIME.md#run-locally) covers the sibling CityBuddy checkout, Java 21, Docker Compose, and local service configuration. Native prerequisites and build commands are in the [Android](android/README.md) and [iOS](ios/README.md) guides. The [product site](site/README.md) can be previewed independently. ## Check your changes @@ -19,7 +19,7 @@ Run checks for the areas you change: | Area | Checks | |---|---| -| Python host | `uv run ruff check src tests scripts integration_tests`, `uv run ruff format --check src tests scripts integration_tests`, and the application/runtime pytest suite in the [runtime guide](docs/RUNTIME.md#检查与历史记录) | +| Python host | `uv run ruff check src tests scripts integration_tests`, `uv run ruff format --check src tests scripts integration_tests`, and the application/runtime pytest suite in the [runtime guide](docs/RUNTIME.md#checks-and-historical-records) | | Merchant Web | `npm --prefix web run typecheck`, `npm --prefix web test`, `npm --prefix web run build` | | Android / shared Kotlin | `cd android && ./gradlew --no-daemon :shared:jvmTest :app:assembleDebug :app:testDebugUnitTest :app:lintDebug` | | iOS | Follow the [build and Simulator test commands](ios/README.md) for the affected behavior; the CI build-for-testing step builds test bundles but does not run them | @@ -27,7 +27,7 @@ Run checks for the areas you change: [GitHub Actions](.github/workflows/ci.yml) runs the Python, Web, Android, and Apple build checks on pull requests. Preserve existing tests; add regression coverage when a behavior change needs it. -Real Java/database integration tests and real-model evaluations are separate from the checks above. `uv run pytest integration_tests -q` changes demo business data: use the [fixture reset and serial-run instructions](docs/RUNTIME.md#检查与历史记录). Read a recorded evaluation's setup before running it; model calls use the configured provider and may incur cost. +Real Java/database integration tests and real-model evaluations are separate from the checks above. `uv run pytest integration_tests -q` changes demo business data: use the [fixture reset and serial-run instructions](docs/RUNTIME.md#checks-and-historical-records). Read a recorded evaluation's setup before running it; model calls use the configured provider and may incur cost. ## Submit a pull request diff --git a/README.md b/README.md index a4870bd..cf1973f 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,24 @@ -[![ShopMate · Native commerce and agents](docs/assets/cover.png)](https://chantso.github.io/shopmate/) +

+ + ShopMate + +

-# ShopMate +

ShopMate

-**English** · [简体中文](README.zh-CN.md) +

+ Explore the product ↗ +

+ +

Native shopping apps and commerce agents. From choosing to confirming to acting.

-[![CI](https://github.com/ChanTso/shopmate/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ChanTso/shopmate/actions/workflows/ci.yml) +

+ CI +

-**Native shopping apps and commerce agents. A continuous path from choosing to confirming to acting.** +**English** · [简体中文](README.zh-CN.md) -**[Explore the product ↗](https://chantso.github.io/shopmate/)** · [Android](android/README.md) · [iOS](ios/README.md) · [Run locally](docs/RUNTIME.md#本地运行) · [Retail evaluation](evals/records/retail-v2-20260907/README.md) · [Contributing](CONTRIBUTING.md) +[Android](android/README.md) · [iOS](ios/README.md) · [Run locally](docs/RUNTIME.md#run-locally) · [Retail evaluation](evals/records/retail-v2-20260907/README.md) · [Contributing](CONTRIBUTING.md) Android / iOS buyer apps and a React merchant workspace for one retail brand. Shoppers describe what they need, compare products, and confirm transactions. Operators turn business data into proposals, review changes, and approve execution. [CityBuddy](https://github.com/ChanTso/citybuddy) provides the transaction and identity backend. @@ -33,19 +43,28 @@ The product site presents native application footage and interaction demonstrati ## System boundaries ```mermaid -flowchart LR - App[Android / iOS] --> Host[ShopMate API] +flowchart TB + App[Android / iOS] --> Host[ShopMate API and agents] Web[React merchant workspace] --> Host - Host --> Agents[Buyer / Merchant agents] - Agents --> Analysis[Read-only SQL / Python sandbox] + App -->|Flash-sale reservations and status| Commerce[CityBuddy Commerce] + Host -->|Login and scoped delegation| Auth[CityBuddy Auth] + Host -->|Scoped tools and user actions| Commerce Host --> State[(SQLite: conversations and recovery)] - Host -->|Scoped tools / user confirmation| Java[CityBuddy: Auth / Commerce] - App -->|Flash sales| Java - Java --> DB[(MySQL: business state)] - Analysis -->|Read-only business views| DB + Auth --> DB[(MySQL: identity and transactions)] + Commerce --> DB +``` + +Merchant analysis uses a separate data path. The host queries reporting views with a read-only account, then passes complete, bounded tables to the network-isolated Python container. + +```mermaid +flowchart LR + SQL[Host SQL analysis] -->|SELECT-only| Views[(MySQL reporting views)] + SQL -->|Complete table and code| Python[Python container: no network] ``` -ShopMate currently runs as a single-instance host. SQLite with WAL stores conversations, intents, and preferences; MySQL stores identities, products, orders, and transaction receipts. See the [runtime guide](docs/RUNTIME.md#身份对话与持久状态) for ownership and deployment constraints. +The Python container has no database connection or credentials. Identity, conversation ownership, and business authorization remain enforced by their owning services. + +ShopMate currently runs as a single-instance host. SQLite with WAL stores conversations, intents, and preferences; MySQL stores identities, products, orders, and transaction receipts. See the [runtime guide](docs/RUNTIME.md#identity-conversations-and-persistent-state) for ownership and deployment constraints. ## Validation and results @@ -59,7 +78,7 @@ The [retail evaluation](evals/records/retail-v2-20260907/README.md) records **18 ## Run locally -Prerequisites: a sibling CityBuddy checkout, Java 21, Python 3.11+, Node.js 24, uv, and Docker Compose. Complete the [initial backend setup](docs/RUNTIME.md#本地运行), then run: +Prerequisites: a sibling CityBuddy checkout, Java 21, Python 3.11+, Node.js 24, uv, and Docker Compose. Complete the [initial backend setup](docs/RUNTIME.md#run-locally), then run: ```sh uv sync --frozen @@ -80,6 +99,6 @@ Open the merchant workspace at **http://127.0.0.1:8101/**. Build the buyer apps | [`integration_tests/`](integration_tests/) · [`evals/`](evals/) | Business-boundary tests and real-model evaluations | | [`site/`](site/) | Independently built GitHub Pages product site | -ShopMate reuses the retail cores and Messages runtime from [commerce-agents](vendor/commerce-agents/README.md), adding native clients, business tools, identity, persistent state, and transaction integration. Upstream [Apache-2.0 licensing](vendor/commerce-agents/LICENSE) and [image credits](web/public/products/IMAGE-CREDITS.md) are preserved. The cover uses the [same native demo footage as the product site](site/README.md). +ShopMate reuses the retail cores and Messages runtime from [commerce-agents](vendor/commerce-agents/README.md), adding native clients, business tools, identity, persistent state, and transaction integration. Upstream [Apache-2.0 licensing](vendor/commerce-agents/LICENSE) and [image credits](web/public/products/IMAGE-CREDITS.md) are preserved. The [product site notes](site/README.md) describe how its native footage and interaction demonstrations were made. [Contributing](CONTRIBUTING.md) · [Apache-2.0 license](LICENSE) diff --git a/README.zh-CN.md b/README.zh-CN.md index c8b1a6c..19e3e6a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1,14 +1,24 @@ -[![ShopMate · 选购有灵感,经营有把握](docs/assets/cover.png)](https://chantso.github.io/shopmate/) +

+ + ShopMate + +

-# ShopMate +

ShopMate

-[English](README.md) · **简体中文** +

+ Explore the product ↗ +

+ +

原生购物客户端与经营 Agent,让选择、确认与执行成为连续的体验。

-[![CI](https://github.com/ChanTso/shopmate/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/ChanTso/shopmate/actions/workflows/ci.yml) +

+ CI +

-**原生购物客户端 × 经营 Agent,让选择、确认与执行成为连续的体验。** +[English](README.md) · **简体中文** -**[浏览产品官网 ↗](https://chantso.github.io/shopmate/)** · [Android](android/README.md) · [iOS](ios/README.md) · [本地运行](docs/RUNTIME.md#本地运行) · [业务验收](evals/records/retail-v2-20260907/README.md) · [贡献指南](CONTRIBUTING.md) +[Android](android/README.md) · [iOS](ios/README.md) · [本地运行](docs/RUNTIME.md#run-locally) · [业务验收](evals/records/retail-v2-20260907/README.md) · [贡献指南](CONTRIBUTING.md) 面向同一零售品牌的 Android / iOS 买家 App 与 React 商家工作台。买家说出需求、比较商品、确认交易;运营人员从经营数据出发,准备方案、核对变更并批准执行。[CityBuddy](https://github.com/ChanTso/citybuddy) 提供实际交易与身份后端。 @@ -33,19 +43,28 @@ ## 系统边界 ```mermaid -flowchart LR - App[Android / iOS] --> Host[ShopMate API] +flowchart TB + App[Android / iOS] --> Host[ShopMate API 与 Agent] Web[React 商家工作台] --> Host - Host --> Agents[买家 / 经营 Agent] - Agents --> Analysis[只读 SQL / Python 沙箱] - Host --> State[(SQLite · 对话与恢复)] - Host -->|受限工具 / 人工确认| Java[CityBuddy · Auth / Commerce] - App -->|秒杀| Java - Java --> DB[(MySQL · 业务状态)] - Analysis -->|只读经营视图| DB + App -->|秒杀预约与状态查询| Commerce[CityBuddy Commerce] + Host -->|登录与受限委托交换| Auth[CityBuddy Auth] + Host -->|受限工具与用户操作| Commerce + Host --> State[(SQLite: 对话与恢复)] + Auth --> DB[(MySQL: 身份与交易)] + Commerce --> DB +``` + +经营分析有单独的数据通路:宿主以只读账号查询视图,再把完整、有界的数据表交给无网络 Python 容器。 + +```mermaid +flowchart LR + SQL[宿主 SQL 分析] -->|SELECT-only| Views[(MySQL 经营视图)] + SQL -->|完整数据表与代码| Python[Python 容器: 无网络] ``` -ShopMate 当前为单实例宿主,SQLite 使用 WAL 保存对话、意图和偏好;MySQL 保存身份、商品、订单和交易回执。两者职责与运行约束见[工程指南](docs/RUNTIME.md#身份对话与持久状态)。 +Python 容器没有数据库连接或凭证;账户、对话归属与业务授权仍由各自服务校验。 + +ShopMate 当前为单实例宿主,SQLite 使用 WAL 保存对话、意图和偏好;MySQL 保存身份、商品、订单和交易回执。两者职责与运行约束见[工程指南](docs/RUNTIME.md#identity-conversations-and-persistent-state)。 ## 验证与结果 @@ -57,7 +76,7 @@ ShopMate 当前为单实例宿主,SQLite 使用 WAL 保存对话、意图和 ## 本地运行 -需要同级 CityBuddy 仓库、Java 21、Python 3.11+、Node.js 24、uv 与 Docker Compose。完成[首次后端准备](docs/RUNTIME.md#本地运行)后: +需要同级 CityBuddy 仓库、Java 21、Python 3.11+、Node.js 24、uv 与 Docker Compose。完成[首次后端准备](docs/RUNTIME.md#run-locally)后: ```sh uv sync --frozen @@ -78,6 +97,6 @@ uv run uvicorn shopmate.app:create_app --factory --host 127.0.0.1 --port 8101 | [`integration_tests/`](integration_tests/) · [`evals/`](evals/) | 业务边界测试与真实模型验收 | | [`site/`](site/) | 独立构建的 GitHub Pages 产品官网 | -复用 [commerce-agents](vendor/commerce-agents/README.md) 的零售核心与 Messages 运行时,扩展原生客户端、业务工具、身份、持久状态与实际交易接入。保留上游 [Apache-2.0 许可](vendor/commerce-agents/LICENSE)及[图片来源](web/public/products/IMAGE-CREDITS.md);封面使用[官网中相同的原生演示画面](site/README.md)。 +复用 [commerce-agents](vendor/commerce-agents/README.md) 的零售核心与 Messages 运行时,扩展原生客户端、业务工具、身份、持久状态与实际交易接入。保留上游 [Apache-2.0 许可](vendor/commerce-agents/LICENSE)及[图片来源](web/public/products/IMAGE-CREDITS.md);[官网说明](site/README.md)记录了原生画面与交互演示的制作方式。 [贡献指南](CONTRIBUTING.md) · [Apache-2.0 许可](LICENSE) diff --git a/docs/BUYER.md b/docs/BUYER.md index b69e0a0..487dac3 100644 --- a/docs/BUYER.md +++ b/docs/BUYER.md @@ -1,37 +1,39 @@ -# 买家 Android App +# Buyer Android app -买家正式入口为原生 Kotlin/Jetpack Compose App;商家使用 React/Vite Web。它们是同一家官方商店的顾客和员工入口,共用 ShopMate API,交易与身份由 CityBuddy 提供。旧 `/buyer` 浏览器页面已经退役。 +The buyer entry point is a native Kotlin/Jetpack Compose app; merchants use the React/Vite Web app. These are the customer and staff interfaces for the same official store. They share the ShopMate API, with transactions and identity provided by CityBuddy. The old `/buyer` browser page has been retired. -## 启动与登录 +## Start and sign in -按[项目 README](../README.md#本地运行)启动服务,再按 [Android 构建说明](../android/README.md)安装 App。Android 模拟器连接 `http://10.0.2.2:8101`;登录页面可以设置 API 地址。 +Start the services using the [runtime guide](RUNTIME.md#run-locally), then install the app using the [Android build instructions](../android/README.md). The Android emulator connects to `http://10.0.2.2:8101`; the sign-in screen allows the API address to be configured. -| 演示账号 | 本地密码文件 | +| Demo account | Local password file | | --- | --- | | `shopmate-retail-buyer` | `.run/buyer_1_password` | | `shopmate-retail-buyer-2` | `.run/buyer_2_password` | -登录后即可购物,不必创建对话。首次发送消息时才创建买家对话。手机只保存以 Keystore 加密的令牌、当前对话标识和按服务地址/主体隔离的待恢复请求;密码、模型凭证不进入手机存储。 +Shopping is available after sign-in without creating a conversation. The first message creates the buyer conversation. The phone stores only the Keystore-encrypted token, current conversation identifier, and pending recovery requests scoped by service address and subject; passwords and model credentials are not stored on the phone. -## 建议体验顺序 +## Suggested walkthrough -1. 浏览首页和分类,搜索商品、打开详情并选择具体规格。现有目录为 ACME 演示商品,名称主要为英文;用途、预算和搭配需求可用中文问助手。 -2. 问“帮我比较两款咖啡机”,查看流式工具进度和比较卡;也可按预算规划组合、读取政策或咨询履约。推荐与描述中的价格不作为付款报价。 -3. 把在售 SKU 加入购物车,调整数量,核对版本、库存和整数分报价,再明确确认创建订单。报价变化需要重新读取并确认。 -4. 在结账记录中确认模拟付款。订单、付款、履约分别呈现,不从付款成功推断已发货。 -5. 在本人订单或助手准备的退款卡中核对金额与有效期,再确认提交退款申请。`REQUESTED` 表示申请受理,不表示到账。 -6. 从“我的”查看资料、记忆和操作记录;记忆支持纠正、单条忘记和明确确认后全部清除。清除记忆不删除订单或购物车。 +The quoted Chinese prompt and UI labels below are retained verbatim from the demo. -配送估算按实际 SKU 数量读取,独立展示币种、费用和时间。当前商品付款不收配送费,不将咨询估算描述成已购买的配送服务。 +1. Browse the home page and categories, search for products, open details, and select a specific variant. The ACME demo catalog primarily uses English names; the assistant also accepts Chinese questions about uses, budgets, and combinations. +2. Ask “帮我比较两款咖啡机” and inspect streaming tool progress and the comparison card. You can also plan a bundle within a budget, read policies, or ask about fulfillment. Prices in recommendations and descriptions are not payment quotes. +3. Add available SKUs to the cart, adjust quantities, review the version, stock, and quote in integer minor units, then explicitly confirm order creation. A changed quote must be read and confirmed again. +4. Confirm simulated payment from the checkout record. Order, payment, and fulfillment states are displayed separately; successful payment does not imply shipment. +5. Review the amount and expiry in your own order or an assistant-prepared refund card, then confirm the refund request. `REQUESTED` means the request was accepted, not that funds arrived. +6. Open “我的” to view your profile, memory, and operation history. Memory supports corrections, forgetting individual entries, and clearing all entries after explicit confirmation. Clearing memory does not delete orders or the cart. -## 中断和恢复 +Delivery estimates use actual SKU quantities and separately display currency, cost, and timing. Product payments currently do not charge delivery fees; a consultation estimate is not a purchased delivery service. -界面重建时 ViewModel 保留当前任务;宽窗口可同时展示购物和助手,窄窗口使用独立助手页。停止生成会取消 HTTP 流,不撤销已经提交的购物车、订单或退款申请。进程重启从服务端读取已保存的对话,不假定后台持续生成。 +## Interruption and recovery -幂等写入先在手机保存原 key 和请求体。如果未收到结果,操作记录保留“待核对”;恢复先查询原回执,用户确认继续后才重试原意图,不更换 key。付款沿原 checkout,退款确认沿原 pending action 回放。聊天切换不改变命令归属。 +The ViewModel retains the current task across view reconstruction. Wide windows can show shopping and the assistant together; narrow windows use a separate assistant page. Stopping generation cancels the HTTP stream without undoing committed cart changes, orders, or refund requests. After a process restart, saved conversations are read from the server; continued background generation is not assumed. -## 数据边界 +Before an idempotent write, the phone saves the original key and request body. If the response is missing, the operation remains marked “待核对” (verbatim UI label: pending verification). Recovery first queries the original receipt; only after the user confirms continuation does it retry the original intent with the same key. Payment follows the original checkout, and refund confirmation replays the original pending action. Switching chats does not change command ownership. -订单、价格、库存、付款与审批的权威来源是 Java/MySQL。ShopMate SQLite 保存单实例的对话、记忆、运行状态和恢复记录,不承担秒杀订单库职责。两个买家及商家身份的对话、命令和记忆相互隔离。 +## Data boundaries -后端历史吞吐不代表手机帧率、并发 Agent 容量或生产用户规模。Android 的构建、设备测试与实际交互记录独立于历史模型质量评测。 +Java/MySQL is authoritative for orders, prices, stock, payment, and approvals. ShopMate SQLite stores single-instance conversations, memory, runtime state, and recovery records; it is not the seckill order database. Conversations, commands, and memory are isolated between both buyers and the merchant identity. + +Historical backend throughput does not establish phone frame rates, concurrent agent capacity, or production user scale. Android builds, device tests, and interaction records are separate from historical model-quality evaluations. diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index a0180bb..9d05502 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -1,56 +1,58 @@ -# 运行与工程指南 +# Runtime and engineering guide -[返回项目首页](../README.md)。以下命令均从所注明的仓库根目录运行。 +[Back to the project overview](../README.md). Run each command from the repository root indicated below. -面向单一零售品牌官方商店的应用与 Agent 工作台。CityBuddy 提供同一家店的交易与身份后端;ShopMate 负责买家与操作员的应用入口、Agent 和对话。一个商品目录、一支运营团队、多位顾客,不包含多商户入驻。商家端支持经营分析、商品与库存、订单问题和审批执行;买家端支持推荐与比较、购物车、本人订单及由用户确认的结账、模拟付款和退款申请。 +An application and agent workspace for one retail brand's official store. CityBuddy provides the store's transaction and identity backend; ShopMate provides buyer and operator interfaces, agents, and conversations. The scope is one catalog, one operations team, and multiple customers, without multi-merchant onboarding. Merchants can analyze performance, manage products and stock, investigate order issues, and approve changes. Buyers can get recommendations and comparisons, manage their cart and own orders, and explicitly confirm checkout, simulated payment, and refund requests. -项目复用 [commerce-agents](../vendor/commerce-agents/README.md) 的商家与购物核心、Messages 运行时和零售页面组件;业务工具、身份、持久会话及实际写入接入 CityBuddy。原 [Apache-2.0 许可证](../vendor/commerce-agents/LICENSE)、版权声明和[图片来源](../web/public/products/IMAGE-CREDITS.md)保留。 +The project reuses [commerce-agents](../vendor/commerce-agents/README.md)' merchant and shopping cores, Messages runtime, and retail UI components. Business tools, identity, persistent conversations, and actual writes connect to CityBuddy. The original [Apache-2.0 license](../vendor/commerce-agents/LICENSE), copyright notices, and [image credits](../web/public/products/IMAGE-CREDITS.md) are preserved. -## 产品展示 +## Product showcase -[静态产品官网](../site/README.md)位于 `site/`,与商家 React 工作台 `web/` 分开。官网使用实际客户端画面与滚动交互,不依赖在线模型或交易服务;完整业务演示在本地运行。Android 与 SwiftUI 买家客户端共享 KMP 协议与恢复核心,构建说明分别见 `android/README.md`、`ios/README.md`。 +The [static product site](../site/README.md) lives in `site/`, separately from the merchant React workspace in `web/`. It uses real client images and scrolling interactions without requiring an online model or transaction service; the complete business demo runs locally. Android and SwiftUI buyer clients share a KMP protocol and recovery core. See `android/README.md` and `ios/README.md` for their build instructions. -## 当前能力 +## Current capabilities -- **经营分析**:主 Agent 组织查询与追问,复杂计算交给分析子 Agent;它通过受限 SQL 取数,也可在独立 Python 容器内计算完整查询结果。成交额来自成功付款的历史订单,流量和广告归因有独立的观察期间与来源;缺失数据不填零。 -- **经营首页与订单**:四项核心指标、可切换日趋势和三类待办使用同一报告口径;近期订单读取当前全店标准单与秒杀单,不受历史报告截止限制。按 SKU 子单展示成交时的金额,订单、付款、退款和履约状态分别保留。 -- **商品与运营**:服务端分页浏览商品系列和单品,详情展开真实 SKU、规格、当前价格、库存、内容和成本观察;库存预警及订单问题提供对应分析入口。 -- **五类草案**:支持 `LISTING_UPDATE`、`PRICE_UPDATE`、`INVENTORY_ACTION`、`PROMOTION`、`CAMPAIGN`。涉及商品的操作展开后至多 25 个 SKU;卡片分别显示金额、数量、开关和文字差异。 -- **操作员审批**:模型可读取、建案和取消未执行方案,不能批准。批准按钮使用登录操作员的直接身份;Java 核对快照、版本和业务条件,在同一事务内保存实际变更、草案回执及适用的商品 Outbox。冲突整批拒绝,重复批准返回原结果。 -- **恢复与停止**:会话和草案引用保存在 SQLite,业务终态以 Java 为准。刷新后重新登录可恢复记录;“停止生成”中断当前请求,不撤销已保存的草案或已执行的变更。审批结果独立保存在业务回执中,模型生成不阻止普通购物与操作员审批;真实版本冲突和未知写入仍需核对。 +- **Business analysis:** the main agent organizes queries and follow-ups; an analysis subagent handles complex calculations through restricted SQL and an independent Python container that receives complete query results. Revenue uses successfully paid historical orders. Traffic and advertising attribution have their own observation periods and sources; missing data is not filled with zero. +- **Merchant overview and orders:** four core metrics, selectable daily trends, and three task categories share the same reporting definitions. Recent orders read current standard and seckill orders across the store, independently of the historical report cutoff. Amounts reflect the original sale at SKU suborder level; order, payment, refund, and fulfillment states remain separate. +- **Products and operations:** server-side pagination covers product families and individual items. Details expose real SKUs, variants, current prices, stock, content, and cost observations; inventory alerts and order issues lead to relevant analysis. +- **Five draft types:** `LISTING_UPDATE`, `PRICE_UPDATE`, `INVENTORY_ACTION`, `PROMOTION`, and `CAMPAIGN`. Product operations expand to at most 25 SKUs. Cards distinguish changes to amounts, quantities, switches, and text. +- **Operator approval:** the model can read, propose, and cancel unexecuted plans, but cannot approve them. Approval uses the signed-in operator's direct identity. Java checks snapshots, versions, and business conditions, then atomically saves the change, draft receipt, and applicable product Outbox events. A conflict rejects the entire batch; repeated approval returns the original result. +- **Recovery and stopping:** SQLite stores conversations and draft references; Java remains authoritative for business outcomes. Refreshing and signing in again restores records. “停止生成” (verbatim UI label: stop generation) interrupts the request without undoing saved drafts or applied changes. Approval results remain in business receipts. Model generation does not block ordinary shopping or operator approval; actual version conflicts and uncertain writes still require verification. -促销批准会立即修改商品实际售价,**经营窗口结束后不会自动恢复价格**。开始前批准返回 `promotion_not_started` 并保留待批准状态;过期未执行方案被拒绝。营销活动创建或更新的是本站计划、受众、文案和预算,不代表向外部广告平台投放,也不改写既有支出或收入观察。 +Approving a promotion immediately changes the actual sale price; **the price does not automatically revert when the promotion window ends**. Approval before the window returns `promotion_not_started` and leaves the draft pending; an expired, unexecuted plan is rejected. Campaigns create or update local plans, audiences, copy, and budgets. They do not publish to external advertising platforms or overwrite existing spend or revenue observations. ```mermaid flowchart LR - Buyer[买家购物助手] --> Host[ShopMate API / 身份隔离的会话与记忆] - Merchant[商家经营工作台] --> Host - Host --> Shopping[购物 Agent] - Host --> Trading[经营 Agent] - Trading --> Analysis[只读 SQL 分析子 Agent] - Analysis --> Views[受限经营视图] - Analysis --> Sandbox[独立 Python 沙箱] - Shopping -->|买家 OBO / 工具权限| Java[CityBuddy 业务接口] - Trading -->|商家 OBO / 工具权限| Java - Host -->|用户确认 / 操作员批准| Java - Java --> Transaction[身份与版本校验 / 交易 / 回执 / Outbox] + Buyer[Buyer shopping assistant] --> Host[ShopMate API / identity-scoped conversations and memory] + Merchant[Merchant workspace] --> Host + Host --> Shopping[Shopping agent] + Host --> Trading[Merchant agent] + Trading --> Analysis[Read-only SQL analysis subagent] + Analysis --> Views[Restricted business views] + Analysis --> Sandbox[Independent Python sandbox] + Shopping -->|Buyer OBO / tool scopes| Java[CityBuddy business APIs] + Trading -->|Merchant OBO / tool scopes| Java + Host -->|User confirmation / operator approval| Java + Java --> Transaction[Identity and version checks / transactions / receipts / Outbox] ``` -商家入口为 React/Vite Web `/`,买家入口为 [Kotlin/Compose Android App](../android/README.md);买家登录、人工确认、停止恢复与记忆管理见[买家使用说明](../docs/BUYER.md)。旧 `/buyer` Web 页面已退役;旧客服入口与重复模型循环已撤下,Java 的授权、退款确认及回执机制继续复用。两个角色都可调用有来源的网页搜索;经营分析可调用独立 Python 沙箱。[完整零售验收](../evals/records/retail-v1-20260907/README.md)记录真实业务任务、页面操作、记忆、并发与中断恢复;业务成绩和边界检查分别报告。 +The merchant entry point is React/Vite Web at `/`; the buyer entry point is the [Kotlin/Compose Android app](../android/README.md). See the [buyer guide](BUYER.md) for sign-in, explicit confirmation, stopping, recovery, and memory management. The old `/buyer` page, support-agent entry point, and duplicate model loop have been retired; Java authorization, refund confirmation, and receipt mechanisms remain in use. Both roles can use web search with sources; business analysis can use an independent Python sandbox. The [complete retail acceptance record](../evals/records/retail-v1-20260907/README.md) covers real tasks, UI interactions, memory, concurrency, and interruption recovery, reporting business outcomes separately from boundary checks. -## 身份、对话与持久状态 + +## Identity, conversations, and persistent state -普通购物和经营接口只要求对应角色的 `Authorization: Bearer`,不要求聊天 ID。服务端按主体和角色保存内部授权绑定,再按 Java 端点交换精确 scope 的 OBO;部分 UI 购物操作也走此受限代理。模型没有付款、退款确认或操作员批准工具。 +Ordinary shopping and merchant APIs require the appropriate role's `Authorization: Bearer`, without a chat ID. The server stores internal authorization bindings by subject and role, then exchanges an OBO token with the exact scope required by each Java endpoint; some shopping UI operations use this restricted proxy too. The model has no payment, refund-confirmation, or operator-approval tool. -聊天通过 `POST /api/{buyer|merchant}/conversations` 创建,列表和恢复分别使用 `GET /conversations`、`GET /conversations/{id}`,流式调用为 `POST /conversations/{id}/chat`。原 `/session`、`/sessions`、`/chat` 保留为历史协议兼容接口,正式客户端不使用它们。命令与结账/退款记录按主体读取,旧操作保留原 key、请求体和授权绑定,换聊天不会变成新的业务意图。 +Create chat through `POST /api/{buyer|merchant}/conversations`, list and restore through `GET /conversations` and `GET /conversations/{id}`, and stream through `POST /conversations/{id}/chat`. The old `/session`, `/sessions`, and `/chat` routes remain for historical protocol compatibility; official clients do not use them. Commands and checkout/refund records are read by subject. Existing operations retain their original key, body, and authorization binding; changing chats does not create a new business intent. -当前为单进程、单实例 Python 服务;SQLite 存储对话、意图、恢复记录和记忆,使用 WAL,必须保存在持久目录,不能随容器重建丢弃。`state_path` 可配置,夹具重置先用 SQLite backup 保存原库;Java/MySQL 是交易权威来源。不得直接以多个 Uvicorn workers 扩容。默认最多 8 个活跃聊天任务、每用户 2 个,同一对话串行;超额返回 429,普通业务请求不占模型任务名额。这些是任务上限配置,不是容量测量。 +The Python service currently runs as one process and one instance. SQLite with WAL stores conversations, intents, recovery records, and memory in a persistent directory that must survive container replacement. `state_path` is configurable; fixture reset backs up the database through SQLite backup first. Java/MySQL is authoritative for transactions. Do not scale by simply adding Uvicorn workers. Defaults allow at most 8 active chat tasks and 2 per user, with one task per conversation; excess requests receive 429. Ordinary business requests do not consume model-task slots. These are configured limits, not measured capacity. -## 本地运行 + +## Run locally -需要同级 [CityBuddy](https://github.com/ChanTso/citybuddy) 仓库、Java 21、Python 3.11+、Node.js 24、uv 和 Docker Compose。CityBuddy 至少包含 [PR #159](https://github.com/ChanTso/citybuddy/pull/159)(`2eb42634f082c0ddf93639f902db38009381d337`),提供零售/营销迁移、商家操作、全店近期订单读取和 FAQ 发布 CLI。 +Prerequisites are a sibling [CityBuddy](https://github.com/ChanTso/citybuddy) checkout, Java 21, Python 3.11+, Node.js 24, uv, and Docker Compose. CityBuddy must include at least [PR #159](https://github.com/ChanTso/citybuddy/pull/159) (`2eb42634f082c0ddf93639f902db38009381d337`), which supplies retail/campaign migrations, merchant operations, store-wide recent orders, and the FAQ publication CLI. -首次准备 Java 服务: +Initial Java service setup: ```sh cd ../citybuddy @@ -65,37 +67,38 @@ npm --prefix web run build uv run uvicorn shopmate.app:create_app --factory --host 127.0.0.1 --port 8101 ``` -`up` 要求 ShopMate API 已停止;首次初始化统一零售夹具,已有该版本数据时保留当前业务变更。构建后由 Python 同源提供商家 Web,访问 `http://127.0.0.1:8101/`,不需要另起 Next/Node 服务。开发 Web 时另开终端 `npm --prefix web run dev`,3100 端口将 API 请求代理到 8101。 +`up` requires the ShopMate API to be stopped. It initializes the unified retail fixture on first use and preserves business changes when that data version already exists. After the build, Python serves the merchant Web app from `http://127.0.0.1:8101/`; no separate Next/Node service is needed. For Web development, run `npm --prefix web run dev` in another terminal; port 3100 proxies API requests to 8101. -操作员账号为 `shopmate-fixture-operator`,本地生成密码保存在忽略的 `.run/operator_password`。Bearer 只保留在页面内存,刷新后重新登录。Android 构建与安装见 [android/README.md](../android/README.md),模拟器连接 `http://10.0.2.2:8101`。 +The operator account is `shopmate-fixture-operator`; its locally generated password is in ignored `.run/operator_password`. Bearer tokens stay in page memory, so refreshing requires sign-in again. See [android/README.md](../android/README.md) for Android build and installation; the emulator connects to `http://10.0.2.2:8101`. -启动脚本使用独立的 `shopmate` Compose project 和数据卷,不重置 CityBuddy 默认演示库。Auth/Commerce 使用 9081/9082,ShopMate API/Web 使用 8101。停止 API 后运行 `python3 scripts/local_runtime.py stop` 停止本项目 Java 与数据服务、保留卷。对话数据库必须保留在 `.run` 或另一个持久目录中。 +The launcher uses the separate `shopmate` Compose project and volumes without resetting CityBuddy's default demo database. Auth/Commerce use 9081/9082; the ShopMate API/Web uses 8101. After stopping the API, `python3 scripts/local_runtime.py stop` stops this project's Java and data services while preserving volumes. Keep the conversation database in `.run` or another persistent directory. -## 模型、预算与时间 +## Models, budgets, and time -模型代理凭证继续来自同级 `citybuddy/.env` 的 `CLIPROXY_BASE_URL` 和 `CLIPROXY_API_KEY`。默认主模型与分析模型均为 `gpt-5.6-terra`,经 Chat Completions 适配对接 Messages 循环。运行参数位于 `.run/settings.json`,也可通过 `SHOPMATE_CONFIG` 指定配置文件;凭证不传入浏览器或模型工具参数。 +Model proxy credentials come from `CLIPROXY_BASE_URL` and `CLIPROXY_API_KEY` in the sibling `citybuddy/.env`. Both main and analysis models default to `gpt-5.6-terra`, using a Chat Completions adapter for the Messages loop. Runtime settings live in `.run/settings.json`, or a file selected by `SHOPMATE_CONFIG`. Credentials do not enter the browser or model tool arguments. -每回合主、分析子 Agent 共用默认 16 次模型调用和 300 秒截止;主循环最多 12 个工具轮。分析账号仅有六个经营视图的 SELECT,默认查询上限 2 秒、200 行及 16,000 字节。Python 只接收这些视图的完整、有界查询结果;截断表在执行前拒绝。缓存用量仅展示代理实际报告的字段,未知量不推断成命中率或费用收益。主、分析、记忆和搜索请求共用模型调用预算;实际 Responses 搜索用量与 Chat 用量合计一次。默认每聊天回合至多 3 次搜索和 3 次 Python 尝试,次数与回合时间限制不是硬 token 或费用上限。 +The main and analysis agents share a default 16 model calls and a 300-second deadline per turn; the main loop allows at most 12 tool rounds. The analysis account has SELECT on only six business views, with default query limits of 2 seconds, 200 rows, and 16,000 bytes. Python receives only complete, bounded results from these views; truncated tables are rejected before execution. Cache usage displays only fields actually reported by the proxy; unknown values do not become hit rates or cost savings. Main, analysis, memory, and search requests share the model-call budget; Responses search and Chat usage are counted once. Each chat turn allows at most 3 search and 3 Python attempts by default. Call and time limits are not hard token or monetary limits. -网页搜索通过独立的 Responses 请求接入现有普通工具接口,返回外部摘要、实际引用和服务提供的查阅来源。来源卡将引用与查阅列表分开;没有元数据时明确提示。网页内容不作为本站商品、订单、政策或权限真相。当前代理不支持原生 Messages server tools,本部署没有启用原生 server search、code execution 或原生提前派发;搜索与 Python 能力由宿主实际执行。字段依据见 [Responses 搜索文档](https://developers.openai.com/api/docs/guides/tools-web-search)。 +Web search uses a separate Responses request behind the existing ordinary tool interface. It returns external summaries, actual citations, and provider-supplied consulted sources. Source cards distinguish citations from consulted pages and explicitly indicate missing metadata. Web content is not authoritative for this store's products, orders, policies, or permissions. The current proxy does not support native Messages server tools, so this deployment does not enable native server search, code execution, or early dispatch; the host performs search and Python execution. See the [Responses web-search documentation](https://developers.openai.com/api/docs/guides/tools-web-search) for the fields. -`local_runtime.py up` 从 `infra/analysis-sandbox/` 专用目录构建 `shopmate-analysis:1`。每次 Python 调用创建独立容器:非 root、无网络、只读根目录、不挂载宿主或项目文件,固定 Python/pandas/numpy,1 CPU / 512 MiB / 64 PID / 32 MiB 临时目录。单次执行窗口至多 20 秒(含排队),结束清理另有 10 秒限时;合计输出至多 64 KiB,宿主同时运行至多两个;查询和排队也受任务截止约束。停止生成、任务超时或正常关闭会终止对应容器;若 Docker 失联导致无法核实清理,会报告错误并拒绝后续沙箱执行,不把它当成成功。宿主被强制杀死后的遗留容器不属于该保证,可按 `shopmate.analysis=true` 标签检查。容器约束说明见 [Docker 文档](https://docs.docker.com/engine/containers/run/)。 +`local_runtime.py up` builds `shopmate-analysis:1` from the dedicated `infra/analysis-sandbox/` directory. Each Python call creates a separate non-root container with no network, a read-only root filesystem, no host/project mounts, fixed Python/pandas/numpy, and limits of 1 CPU / 512 MiB / 64 PID / 32 MiB temporary storage. The execution window is at most 20 seconds including queueing, with another 10 seconds for cleanup, at most 64 KiB combined output, and at most two concurrent executions. Queries and queueing also respect the task deadline. Stopping generation, task expiry, or normal shutdown terminates the corresponding container. If Docker is unreachable and cleanup cannot be verified, the host reports an error and rejects further sandbox work. Containers left after the host is forcibly killed are outside that guarantee and can be inspected by the `shopmate.analysis=true` label. See [Docker's container documentation](https://docs.docker.com/engine/containers/run/) for these constraints. -当前演示数据为 `shopmate-retail-v1`:**87 个目录根、104 个可交易 SKU、90 个完整 Shanghai 日、CNY**。这是从 vendored 零售样例和确定性造数构成的演示数据,不是实际经营记录。报告截止固定为 `2026-09-05T00:00:00+08:00`;模型的操作时钟是每轮真实 Shanghai 时间。相对报表期间使用报告截止,促销的“今天/明天”使用真实操作日期。详情见[零售夹具与重置说明](../docs/retail-fixture.md)。 +The demo fixture is `shopmate-retail-v1`: **87 catalog roots, 104 tradable SKUs, 90 complete Shanghai days, CNY**. It combines vendored retail samples with deterministic synthetic data, not real business records. The reporting cutoff is fixed at `2026-09-05T00:00:00+08:00`; each model turn receives the actual Shanghai operation time. Relative reporting periods use the cutoff; promotion dates such as today or tomorrow use the actual operation date. See the [retail fixture and reset guide](retail-fixture.md). -## 使用工作台 +## Use the workspace -登录后可依次体验以下流程;这是当前能力的操作说明,不代表一次新的模型验收结果: +The following walkthrough describes capabilities, not a new model-evaluation result. The sample Chinese prompt is retained verbatim. -1. 在商品页翻页、筛选状态和内容质量,打开商品系列并核对各 SKU、成本和报告期间销量。 -2. 问“当前报告期间的成交、流量和转化,相比上一期间有什么变化?请列出依据。”继续追问贡献商品或营销计划的同期 ROAS。 -3. 从库存提醒、商品详情或营销页提出方案,在草案卡片核对完整差异。仅提出方案不会立即修改商品。 -4. 点击批准或取消,再从历史和业务页面读回状态。促销先核对真实允许批准窗口及到期不自动恢复的后果。 -5. 流式运行时点击“停止生成”,随后刷新会话核对保存的状态。不要依据未完成的回复判断写入是否发生。 +1. Page through products, filter by status and content quality, and open a family to inspect SKUs, cost, and sales during the report period. +2. Ask “当前报告期间的成交、流量和转化,相比上一期间有什么变化?请列出依据。” Follow up on contributing products or campaign ROAS over the same period. +3. Propose a change from a stock alert, product detail, or campaign page, then review the complete differences in its draft card. Proposing alone does not modify products. +4. Approve or cancel, then read the outcome from history and the business pages. For promotions, review the actual approval window and the lack of automatic price restoration at expiry. +5. Stop generation during a stream, then refresh the conversation to inspect saved state. An unfinished answer does not establish whether a write occurred. -## 检查与历史记录 + +## Checks and historical records -以下命令运行代码检查,不调用真实模型: +These code checks do not call a real model: ```sh uv run ruff check src tests scripts integration_tests @@ -111,11 +114,11 @@ npm --prefix web test npm --prefix web run build ``` -真实 Java/数据库边界检查使用 `uv run pytest integration_tests -q`,会修改保留的演示业务数据,应与其他任务串行运行。每次完整运行前,先停止 API 和全部写入、保存所需记录,按[手工重置流程](../docs/retail-fixture.md#手工重置)恢复夹具后重新启动 API;正常 `up` 会保留已批准的变更,不能代替重置。直接重复写入套件可能触发无变更草案拒绝,或继续改变测试商品的价格和库存。 +Real Java/database boundary checks use `uv run pytest integration_tests -q`, modify retained demo business data, and must run serially with other work. Before a complete run, stop the API and all writes, save needed records, follow the [manual reset procedure](retail-fixture.md#manual-reset), then restart the API. Normal `up` preserves approved changes and is not a reset. Repeating write suites directly can encounter no-change draft rejection or continue changing test prices and stock. -[最终零售业务验收](../evals/records/retail-v2-20260907/README.md)覆盖18个已知场景、按登记共30次:**24次通过、3次业务失败、3次提供者故障**,对应冻结版本 `4020ff93f4797e2ae3142e8a4123442d3d8693b7`。购物付款退款、商品维护、补货、促销成交、营销审批、搜索与SQL/Python分析均核对实际回复和数据库终态;日期表达和遗漏回答的失败保留。61个聊天回合的结束等待p50为30.54秒、p95为87.08秒,包含失败,不代表并发容量。[前一轮54次与边界验证](../evals/records/retail-v1-20260907/README.md)单列,不混合版本或分母;操作仍须由用户确认并经Java事务校验。 +The [final retail acceptance](../evals/records/retail-v2-20260907/README.md) covers 18 known scenarios and 30 registered attempts: **24 passes, 3 business failures, and 3 provider failures**, at frozen version `4020ff93f4797e2ae3142e8a4123442d3d8693b7`. Shopping, payment/refunds, listings, replenishment, promotion-to-sale, campaign approval, search, and SQL/Python analysis were checked against actual answers and database outcomes; date-expression and omitted-answer failures remain. Across 61 chat turns, completion wait was p50 30.54 seconds and p95 87.08 seconds, including failures; this is not concurrent capacity. The [previous 54 attempts and boundary checks](../evals/records/retail-v1-20260907/README.md) retain their own versions and denominators. Operations still require user confirmation and Java transaction checks. -[评测索引](../evals/records/README.md)保留旧七商品/42 日 UTC 版本的 **78/90** 与定向 **21/24**;它们不描述当前零售数据或本次完整批。[历史浏览器演示、截图和 SQL](../docs/demo-20260906/README.md)仍对应旧版,当前双端页面、记忆与恢复记录见新版验收。 +The [evaluation index](../evals/records/README.md) preserves **78/90** and targeted **21/24** from the older seven-product, 42-day UTC fixture. They do not describe current retail data or the complete current batch. [Historical browser demos, screenshots, and SQL](demo-20260906/README.md) remain tied to that older version; current UI, memory, and recovery records are in the newer acceptance reports. ## Native buyer clients diff --git a/docs/assets/cover.png b/docs/assets/cover.png deleted file mode 100644 index d1a7762..0000000 Binary files a/docs/assets/cover.png and /dev/null differ diff --git a/docs/assets/cover.svg b/docs/assets/cover.svg deleted file mode 100644 index e490768..0000000 --- a/docs/assets/cover.svg +++ /dev/null @@ -1,6 +0,0 @@ -ShopMate — Native commerce, thoughtful assistanceNative Android and iOS shopping, merchant operations and agents. -ShopMate — 曲线 S - - - -SHOPMATE / APPLICATIONShopMate.选购有灵感。经营有把握。Native Android & iOS · Shared Kotlin core · Merchant WebDISCOVERCONFIRMA LITTLE BETTER, EVERY DAY.SHOPMATE / CITYBUDDY / STATEEVAL \ No newline at end of file diff --git a/docs/assets/logo.svg b/docs/assets/logo.svg new file mode 100644 index 0000000..8a58475 --- /dev/null +++ b/docs/assets/logo.svg @@ -0,0 +1,5 @@ + +ShopMate curved S mark + + + diff --git a/docs/demo-20260906/README.md b/docs/demo-20260906/README.md index 15077c7..8f7be02 100644 --- a/docs/demo-20260906/README.md +++ b/docs/demo-20260906/README.md @@ -1,32 +1,32 @@ -# 业务功能演示 +# Business feature demonstration -主演示于 2026-09-06 约 00:00–00:05 UTC,在独立本地环境使用固定 R0 交易夹具,以真实浏览器、模型、Java 接口和数据库完成分析、草案、人工批准与恢复。以下记录是功能演示,不计入正式 90 次评测或定向 24 次回归。 +The main demonstration ran around 00:00–00:05 UTC on 2026-09-06 in an isolated local environment with the fixed R0 transaction fixture. A real browser, model, Java APIs, and database completed analysis, drafting, manual approval, and recovery. This functional demonstration is separate from the formal 90 attempts and targeted 24-attempt regression. -- CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -- 主演示 ShopMate:`02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` +- CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423` +- Main demo ShopMate: `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` -原始 SQL、会话与传输分块见 [raw.tar.gz](raw.tar.gz),解包后为 `browser-demo/`。下文以“包内”标注的路径对应解包目录。 +Original SQL, conversations, and transport chunks are in [raw.tar.gz](raw.tar.gz), which extracts to `browser-demo/`. Paths described as inside the archive below refer to that directory. Chinese prompts and UI labels are retained verbatim. -## 分析与审批 +## Analysis and approval -登录后,旧会话引用的草案因夹具重置返回 404;通过已有的“新建会话”入口继续。随后提问“近 14 天和前 14 天,人民币成交额分别是多少?按商品解释变化。”可见回答采用 UTC `2026-08-22, 2026-09-05)` 与 `[2026-08-08, 2026-08-22)` 两个窗口:退款前已支付成交额分别 CNY 2,304、1,956,增加 348;咖啡贡献 +994、茶 −658、杯子 +12,与订单参考 SQL 原始结果(包内 `sql/before/S12.jsonl`)一致。[分析截图](analysis.png) +After sign-in, a draft referenced by an old conversation returned 404 because the fixture had been reset. The existing “新建会话” entry started a new conversation. The prompt “近 14 天和前 14 天,人民币成交额分别是多少?按商品解释变化。” produced a visible answer using UTC windows `2026-08-22, 2026-09-05)` and `[2026-08-08, 2026-08-22)`: pre-refund paid revenue was CNY 2,304 and 1,956, up 348. Coffee contributed +994, tea −658, and cups +12, matching original order-reference SQL inside `sql/before/S12.jsonl`. [Analysis screenshot](analysis.png) -再提问“把咖啡价格调到 25.20 元,先生成草案”,生成 `PREPARED` 草案。此时实际价格仍为 2,400 分、版本 3,尚无商品变更事件。[草案截图](prepared.png) · 草案原始结果(包内 `sql/prepared/drafts.jsonl`) · 批准前商品(包内 `sql/prepared/products.jsonl`) +The next prompt, “把咖啡价格调到 25.20 元,先生成草案”, produced a `PREPARED` draft. The actual price was still 2,400 minor units at version 3, with no product-change event. [Draft screenshot](prepared.png) · Original draft: `sql/prepared/drafts.jsonl` inside the archive · Product before approval: `sql/prepared/products.jsonl` -在浏览器实际点击批准后,同一草案变为 `APPLIED`,咖啡价格变为 2,520 分、版本 4;仅产生一次商品变更及对应 Outbox 事件 `9d3347bf-8ce1-41b7-9e04-772e2394d88b`,事件已发布,库存不变。批准后商品(包内 `sql/applied/products.jsonl`) · 执行回执(包内 `sql/applied/drafts.jsonl`) · 事件原始结果(包内 `sql/applied/events.jsonl`) +Clicking approval in the browser changed that same draft to `APPLIED` and the coffee price to 2,520 minor units at version 4. There was one product change and one corresponding Outbox event, `9d3347bf-8ce1-41b7-9e04-772e2394d88b`; the event was published and stock was unchanged. Inside the archive: product after approval, `sql/applied/products.jsonl`; receipt, `sql/applied/drafts.jsonl`; event output, `sql/applied/events.jsonl`. -随后真实提问读回当前价格与草案状态,回答为 CNY 25.20、版本 4、`APPLIED`。刷新页面、重新登录并选择原会话后,审批历史仍显示同一回执和事件。[读回截图](readback.png) · [恢复截图](restored.png) · 恢复后的会话(包内 `restored-session.json`) +A subsequent real query read back CNY 25.20, version 4, and `APPLIED`. After refreshing, signing in again, and selecting the original conversation, approval history still showed the same receipt and event. [Readback screenshot](readback.png) · [Recovery screenshot](restored.png) · Restored conversation: `restored-session.json` inside the archive. -批准后与恢复后的商品、草案、事件、历史订单和范围汇总原始文件一致;所有阶段的历史订单记录均未改变。事件文件包含两个查询结果集,均指向同一个事件,不能计为两次发布。`after-analysis` 快照采集时第二轮已开始并创建草案,因此不作为分析单轮的独立写入边界。 +Product, draft, event, historical-order, and scope-summary files matched between the approved and restored stages. Historical orders remained unchanged throughout. The event file contains two result sets pointing to the same event, not two publications. The `after-analysis` snapshot was captured after the second turn had started and created a draft, so it is not an independent write boundary for the analysis turn alone. -## 后续流式观察 +## Subsequent streaming observations -后续使用 ShopMate `ac6b1404e17a007b8c5563449872c69c520787a0` 单独观察响应头修复:在回答终态之前,浏览器已显示分析第 2 步正在查询。[流式进度截图](streaming-progress.png) +A separate observation at ShopMate `ac6b1404e17a007b8c5563449872c69c520787a0` checked the response-header fix: the browser displayed analysis step 2 running a query before the answer reached its terminal state. [Streaming progress screenshot](streaming-progress.png) -这轮追问“这两期咖啡的成交件数分别是多少?能说是刚才这次调价导致了历史成交变化吗?”出现业务失败:模型将原来的两个窗口各向后移了一天,返回 65/31 件,没有回答原窗口的 70/28 件。其拒绝将历史变化归因于刚才调价的判断正确,但不能抵消日期范围错误。该轮完整会话与实际 SQL 轨迹(包内 `streaming-followup-session.json`) +The follow-up “这两期咖啡的成交件数分别是多少?能说是刚才这次调价导致了历史成交变化吗?” failed the business task. The model moved both original windows one day later and returned 65/31 units instead of 70/28 for the requested windows. It correctly refused to attribute historical changes to the just-applied price change, but that did not offset the date-range error. The full conversation and actual SQL trajectory are in `streaming-followup-session.json` inside the archive. -流式进度可见与业务回答正确分别记录;这轮结果不追记到主演示版本,也不改变正式评测成绩。 +Visible streaming progress and answer correctness are recorded separately. This turn is not assigned retroactively to the main demo version and does not change formal evaluation scores. -最后在同一 `ac6b1404e17a007b8c5563449872c69c520787a0` 版本显式澄清原有两个 UTC 半开窗口,重新查询得到咖啡本期 70 件、前期 28 件,增加 42 件(+150%);可见回答、实际 SQL 和原参考结果一致。[澄清后的截图](clarified.png) · 会话与查询轨迹(包内 `clarified-session.json`) +Finally, at the same `ac6b1404e17a007b8c5563449872c69c520787a0` version, an explicit clarification restated the two original UTC half-open windows. The new query returned coffee sales of 70 units versus 28, up 42 (+150%); the visible answer, actual SQL, and reference results matched. [Clarified screenshot](clarified.png) · Conversation and query trajectory: `clarified-session.json` inside the archive. -澄清后商品、草案、事件、历史订单及范围汇总与恢复时一致,未产生额外商品修改或事件。这次显式澄清恢复不撤销前一轮日期漂移失败,也不作为新增正式模型成绩。 +After clarification, products, drafts, events, historical orders, and scope summaries matched the restored state, with no additional product change or event. This recovery through explicit clarification does not erase the preceding date-drift failure or add a new formal model result. diff --git a/docs/demo-20260906/transport.md b/docs/demo-20260906/transport.md index ceb1eb7..0541883 100644 --- a/docs/demo-20260906/transport.md +++ b/docs/demo-20260906/transport.md @@ -1,20 +1,20 @@ -# SSE 经前端代理的传输验证 +# SSE transport verification through the frontend proxy -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423`。 -修复前 ShopMate:`02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28`。 -修复后 ShopMate:`ac6b1404e17a007b8c5563449872c69c520787a0`。 -验证日期:2026-09-06 UTC;同一台本机、Next production 前端 3100 → Python API 8101,真实 `gpt-5.6-terra`。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`.
+ShopMate before: `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28`.
+ShopMate after: `ac6b1404e17a007b8c5563449872c69c520787a0`.
+Verified on 2026-09-06 UTC, on the same local host, through the Next production frontend at 3100 → Python API at 8101, using real `gpt-5.6-terra`. -真实浏览器演示中,流式回合完成前没有及时显示工具进度。沿前端代理定位到 Next 默认压缩:它会压缩 `text/event-stream`,而上游响应只有 `Cache-Control: no-cache`。以现有本地操作员登录,经实际 Next 入口新建独立会话,发送同一句只读问题「用一句话说明你能做什么,不需要查询业务数据。」,请求接受 gzip,保存响应白名单头和解码后的原始数据分块。 +During the real browser demo, tool progress did not appear promptly before the streaming turn completed. Investigation traced the frontend proxy path to Next's default compression: it compressed `text/event-stream`, while the upstream response supplied only `Cache-Control: no-cache`. Using the existing local operator account through the actual Next entry point, each run created a separate conversation and sent the same read-only prompt, retained verbatim: “用一句话说明你能做什么,不需要查询业务数据。” The request accepted gzip; allowlisted response headers and decoded raw data chunks were saved. -| 检查项 | 修复前 | 修复后 | +| Check | Before | After | | --- | --- | --- | -| `Content-Encoding` | `gzip` | 无 | +| `Content-Encoding` | `gzip` | Absent | | `Cache-Control` | `no-cache` | `no-cache, no-transform` | -| 客户端收到的解码数据块 | 1,包含整轮与终态 | 40,终态前已有 39 块 | +| Decoded chunks received by the client | 1, containing the whole turn and terminal event | 40, with 39 before the terminal event | -仅给 SSE 响应加 `no-transform`,使代理不再将事件缓存在压缩流中;其他页面压缩、模型协议、提示词和调用预算保持。已有完成流测试增加响应头断言;完整 Python 检查 517 通过、1 个既有可选 SDK 模块跳过。随后真实页面在回合尚未结束时已显示 `analysis: step 2 — running a query`,见 `streaming-progress.png`。 +Only `no-transform` was added to SSE responses, preventing the proxy from buffering events in a compressed stream. Other page compression, model protocol, prompts, and call budget were unchanged. The existing completed-stream test gained a response-header assertion; the full Python checks passed 517 tests with 1 existing optional-SDK module skipped. The real page subsequently showed `analysis: step 2 — running a query` before the turn ended; see `streaming-progress.png`. -这证明了该压缩缓冲路径已被解除。数据块数量随模型文本和网络分包变化,不是吞吐量或模型提速指标;两次输出与模型耗时不要求相同,也没有据此计算延迟收益。最早页面观察中的全部等待时间不能都归因于压缩。该记录独立于 90 次完整业务验收和 24 次定向回归。 +This establishes that the identified compression-buffering path was removed. Chunk counts vary with model text and network segmentation; they are not throughput or model-speed metrics. The two outputs and model durations need not match, and no latency improvement was calculated from them. Not all waiting in the earliest page observation can be attributed to compression. This record is separate from the 90-attempt complete business acceptance and 24-attempt targeted regression. -两份原始传输记录为 `sse-transport-before.jsonl` 与 `sse-transport-after.jsonl`,在本目录的原件包内;凭证和请求授权头未保存。 +The two original transport files, `sse-transport-before.jsonl` and `sse-transport-after.jsonl`, are inside this directory's archive. Credentials and request authorization headers were not saved. diff --git a/docs/retail-fixture.md b/docs/retail-fixture.md index c2ae040..d2bba38 100644 --- a/docs/retail-fixture.md +++ b/docs/retail-fixture.md @@ -1,60 +1,60 @@ -# 零售夹具与本地重置 +# Retail fixture and local reset -当前夹具由 [`scripts/retail_fixture.py`](../scripts/retail_fixture.py) 定义,由 [`scripts/local_runtime.py`](../scripts/local_runtime.py) 初始化和重置。数据版本为 `shopmate-retail-v1`,仅用于独立的 `shopmate` Compose project。需要 CityBuddy PR #158 及此前的 V020–V025 零售、购物、运营和营销迁移。 +The fixture is defined by [`scripts/retail_fixture.py`](../scripts/retail_fixture.py) and initialized/reset through [`scripts/local_runtime.py`](../scripts/local_runtime.py). Its data version is `shopmate-retail-v1`, used only in the isolated `shopmate` Compose project. It requires CityBuddy PR #158 and the preceding V020–V025 retail, shopping, operations, and campaign migrations. -## 数据范围与来源 +## Data scope and sources -| 数据 | 当前定义 | 权威读取 | +| Data | Definition | Authoritative reads | | --- | --- | --- | -| 商品目录 | 83 个单品、4 个系列,共 87 个目录根;4 个系列含 21 个规格,总计 104 个 SKU | `product` 与 `retail_product_family/metadata` | -| 当前运营 | 库存、低库存阈值、可售状态、成本与内容质量观察 | 商品表及 `retail_product_operations` | -| 成交历史 | 报告截止前 90 个完整 Shanghai 日;历史价格版本及金额固定,另有未开始付款、PENDING、FAILED 样本 | `standard_order`、`mock_payment_attempt/callback`、相关账本与订单原始记录 | -| 买家与售后事实 | 两个演示买家、偏好和会员、本人订单、带来源的履约观察及订单问题;退款申请保留 REQUESTED | `crm_profile`、`retail_order_fulfillment/issue`、`mock_refund` | -| 流量与营销 | 90 日全店访问观察、五个原始营销计划及各自归因期间;C-203 收入未知 | `retail_store_traffic_daily`、`retail_campaign` | -| 政策与指南 | 与当前业务能力对应的政策、购买指南及配送估算配置 | 通过实际 FAQ 发布服务写入的 `faq_source`,以及 `retail_fulfillment_config` | +| Catalog | 83 individual products and 4 families, totaling 87 catalog roots; the 4 families contain 21 variants, for 104 SKUs overall | `product` and `retail_product_family/metadata` | +| Current operations | Stock, low-stock thresholds, sale availability, cost, and content-quality observations | Product tables and `retail_product_operations` | +| Sales history | 90 complete Shanghai days before the reporting cutoff; fixed historical price versions and amounts, plus payment-not-started, PENDING, and FAILED samples | `standard_order`, `mock_payment_attempt/callback`, related ledgers, and original order records | +| Buyer and after-sales facts | Two demo buyers, preferences and membership, own orders, sourced fulfillment observations, and order issues; refund requests remain REQUESTED | `crm_profile`, `retail_order_fulfillment/issue`, `mock_refund` | +| Traffic and campaigns | 90 days of store-wide traffic observations, five original campaigns with their own attribution periods; C-203 revenue is unknown | `retail_store_traffic_daily`, `retail_campaign` | +| Policies and guides | Policies, buying guides, and delivery-estimate settings matching current business capabilities | `faq_source` populated through the real FAQ publication service, and `retail_fulfillment_config` | -商品、紧凑规格、运营样例、买家、订单问题和营销观察来自 [`vendor/commerce-agents/examples/retail/data`](../vendor/commerce-agents/examples/retail/data)。商品图片只复用仓库内已有文件,保留[图片来源](../web/public/products/IMAGE-CREDITS.md);没有图片的商品使用组件缺省展示,不生成失效图片地址。 +Products, compact variant definitions, operations samples, buyers, order issues, and campaign observations come from [`vendor/commerce-agents/examples/retail/data`](../vendor/commerce-agents/examples/retail/data). Product images reuse existing repository files with [credits](../web/public/products/IMAGE-CREDITS.md) preserved. Products without images use the component's default presentation rather than a broken image address. -金额统一转换为 CNY 演示金额,不声称进行汇率换算。历史订单按确定性规则构造,流量使用样例序列构造对应日观察,来源和数据版本写入记录;它们不是商店真实销售或广告成绩。已提供的营销支出与归因收入保留原有数值关系及明确期间,不再导入一套 mock sales 作为第二套成交事实。 +Amounts are converted into CNY demo amounts, not presented as an exchange-rate conversion. Historical orders use deterministic construction rules, and traffic samples form matching daily observations; records include their source and data version. These are not actual store sales or advertising results. Supplied campaign spend and attributed revenue retain their original numeric relationship and explicit periods; a second mock-sales dataset is not imported as another source of sales truth. -系列共同内容只存于 family;当前 21 个规格没有额外展示内容覆盖,叶 metadata 保留实际 option_values,使用父内容继承。缺货规格 `AR-1606-KING-BLUSH` 和 `AR-1902-FULL` 有明确零库存;`AR-1207` 有库存但暂停出售,不能混为缺货。库存是本次演示的当前快照,历史造数不会再次扣减该库存。 +Shared family content lives only on the family. The current 21 variants have no additional display-content overrides: leaf metadata retains actual `option_values` and inherits parent content. Out-of-stock variants `AR-1606-KING-BLUSH` and `AR-1902-FULL` explicitly have zero stock. `AR-1207` has stock but is paused for sale, which is a different condition. Stock is the current demo snapshot; generating historical orders does not deduct it again. -买家订单的历史单价和版本独立于今天的商品价格。成功付款订单同时具备付款尝试、回调和支付账本;退款 REQUESTED 表示已预留申请金额,不表示资金已退回。履约使用明确阶段、时间和 `FIXTURE` 来源,不从 PAID 或预计送达时间推断已发货。原样例缺少发运时间的记录采用固定的演示交接假设,时间不得晚于观察时刻。 +Buyer orders retain historical unit prices and versions independently of today's product prices. Successfully paid orders have payment attempts, callbacks, and payment ledgers. A REQUESTED refund reserves the requested amount but does not mean funds have been returned. Fulfillment has explicit stages, times, and a `FIXTURE` source; PAID or an estimated delivery date does not imply shipment. Original samples missing dispatch times use a fixed demo handoff assumption whose time cannot exceed the observation time. -商家入口为 `/`,买家入口为 `/buyer`;两端复用这套业务事实。买家购物、人工确认、恢复与记忆管理见[买家说明](BUYER.md)。新版买家真实模型验收另行记录,本夹具说明不声明已经通过。 +For the browser clients described by this fixture, the merchant entry point was `/` and the buyer entry point was `/buyer`; both reused these business facts. See the [buyer guide](BUYER.md) for shopping, confirmation, recovery, and memory management. New buyer real-model acceptance is recorded separately; this fixture description does not claim that it passed. -## 时间和统计口径 +## Time and metric definitions -固定报告截止为 `2026-09-05T00:00:00+08:00`,交易覆盖 `[2026-06-07, 2026-09-05)` 的 90 个完整 Shanghai 日。SQL 连接和时间戳按 UTC 使用,本地日/月边界先转换成对应 UTC 瞬间再查询。裸日期指 Shanghai 午夜,显式带 offset 的时间保留其实际瞬间。 +The fixed reporting cutoff is `2026-09-05T00:00:00+08:00`, with transactions covering 90 complete Shanghai days in `[2026-06-07, 2026-09-05)`. SQL connections and timestamps use UTC; local day/month boundaries are converted into their UTC instants before querying. A bare date means Shanghai midnight, while an explicit offset retains its actual instant. -报告截止与真实操作时钟分开:`settings.as_of` 只传入 backend 的 `report_as_of`;主 Agent 每轮看到真实 Shanghai 操作时间。`last_14_days` 等相对报表以报告截止为参照,促销“今天生效”按真实操作日期处理。新增的现实时间订单不会被塞入旧固定报告窗口,覆盖外的期间也不能当成已观察到零成交。 +The reporting cutoff is separate from the operation clock: `settings.as_of` reaches the backend only as `report_as_of`; each main-agent turn sees the actual Shanghai operation time. Relative report periods such as `last_14_days` use the cutoff, while promotion wording such as effective today uses the real operation date. New orders at the real current time do not enter an old fixed reporting window. Periods outside coverage cannot be treated as observed zero sales. -- 成交额采用成功付款的历史订单金额,退款前总额;不按现价重算,不把不同币种直接相加。 -- 转化率为同一完整期间的付款 **SKU 子单数 / 全店访问次数**,不是去重买家数或 checkout 头数;缺失任一天的流量观察不补零。类别和商品没有独立流量分母。 -- 退款申请订单比例不是成功退款率或实物退货率。成本和毛利来自当前运营观察,是经营估算,不是会计利润或强制价格底线。 -- 营销预算是可编辑的计划字段;支出、归因收入和观察期间独立保留。ROAS 仅在同一活动/期间/币种的收入和非零支出均存在时计算;未知收入不呈现为 0。 -- `merchant_daily_sales` 保留旧 UTC 日聚合。上海日、周和月分析应从 `merchant_paid_orders.succeeded_at` 按实际本地边界聚合,不能直接将旧日标签换成 Shanghai。 +- Revenue is the pre-refund amount of successfully paid historical orders. It is not recalculated at today's price, and different currencies are not added together. +- Conversion is **paid SKU suborders / store-wide visits** over the same complete period, not distinct buyers or checkout headers. A missing daily traffic observation is not filled with zero. Categories and products have no separate traffic denominator. +- The proportion of orders with refund requests is not a successful-refund or physical-return rate. Cost and gross margin come from current operations observations; they are estimates, not accounting profit or an enforced price floor. +- Campaign budget is an editable plan field. Spend, attributed revenue, and observation periods remain separate. ROAS requires revenue and nonzero spend for the same campaign, period, and currency; unknown revenue is not shown as 0. +- `merchant_daily_sales` retains the old UTC daily aggregation. Shanghai day, week, and month analysis should aggregate `merchant_paid_orders.succeeded_at` using actual local boundaries rather than relabeling UTC days. -分析子 Agent 只获以下六个视图的 SELECT;基础表和用于核对结果的本地只读账号不暴露给模型: +The analysis subagent has SELECT on only these six views. Base tables and the local read-only verification account are not exposed to the model: -1. `merchant_products`:当前商品与可调价性。 -2. `merchant_paid_orders`:按订单类型、订单 ID、主体、金额和付款状态关联的历史成交。 -3. `merchant_daily_sales`:旧 UTC 日聚合。 -4. `merchant_listing_facts`:当前系列、分类、库存和成本/内容观察。 -5. `merchant_store_traffic_daily`:Shanghai 日访问观察及来源。 -6. `merchant_campaign_facts`:本地计划与独立归因观察。 +1. `merchant_products`: current products and price-editability. +2. `merchant_paid_orders`: historical sales joined by order type, order ID, subject, amount, and payment state. +3. `merchant_daily_sales`: the old UTC daily aggregation. +4. `merchant_listing_facts`: current families, categories, stock, and cost/content observations. +5. `merchant_store_traffic_daily`: Shanghai daily visits and their sources. +6. `merchant_campaign_facts`: local plans and separate attribution observations. -## 变更与判定 +## Changes and grading -五类草案共用 CityBuddy 的变更账本和实际操作员审批入口。商品操作展开后至多 25 个 SKU;Java 在同一事务内核对完整目标、快照和版本,写入业务状态、回执及适用的商品事件。普通调价工具的 20% 幅度限制属于 host 工具约束,不声称 Java 也执行相同上限。 +All five draft types use CityBuddy's change ledger and actual operator-approval endpoint. Product operations expand to at most 25 SKUs. Within one transaction, Java checks the complete target set, snapshots, and versions, then writes business state, receipts, and applicable product events. The ordinary price tool's 20% limit is a host-tool constraint; this does not claim that Java enforces the same limit. -促销折扣必须为正且不超过 50%,分金额按 HALF_UP 生成并冻结。允许批准窗口开始前仍为 PREPARED;窗口过期后首次批准会拒绝;窗口内批准立即改变实际价格。日期形式的结束值包括该 Shanghai 日期全天,显式时间形式的结束值不包含该瞬间。到期不会自动恢复售价,商品之后再次改价或换币种时读取接口同时保留原促销价格与当前价格。 +Promotion discounts must be positive and no greater than 50%; amounts in minor units are generated with HALF_UP rounding and frozen. Before the approval window, the draft stays PREPARED. First approval after expiry is rejected; approval within the window changes the actual price immediately. A date-only end includes that entire Shanghai day; an explicit timestamp excludes the ending instant. Expiry does not restore the price automatically. If a product later changes price or currency, reads retain both the original promotional price and the current price. -CAMPAIGN 创建或更新本站计划、预算、受众和文案。新计划没有支出或收入观察;修改旧计划不改写其已有归因数据。没有外部广告投放动作。 +CAMPAIGN creates or updates local plans, budgets, audiences, and copy. New plans have no spend or revenue observations; editing an existing plan does not overwrite its attribution data. No external advertisement is published. -## 初始化与正常启动 +## Initialize and start normally -从 ShopMate 根目录执行,且 API 未运行: +Run from the ShopMate root while the API is stopped: ```sh uv sync --frozen @@ -62,49 +62,50 @@ python3 scripts/local_runtime.py up uv run uvicorn shopmate.app:create_app --factory --host 127.0.0.1 --port 8101 ``` -完整依赖安装和前端启动见 [README](../README.md#本地运行)。`up` 先完成身份、迁移及 Java 服务配置;以当前版本的流量记录判断夹具是否已初始化。首次初始化会替换保留的旧七商品演示范围并建立统一零售数据;后续正常启动保留当前价格、库存、计划和会话,不等同于重置。 +See the [runtime guide](RUNTIME.md#run-locally) for complete dependencies and frontend setup. `up` configures identity, migrations, and Java services, then uses traffic records for the current version to detect initialization. First initialization replaces the retained old seven-product demo scope with unified retail data. Subsequent normal starts preserve current prices, stock, plans, and conversations; they are not resets. -## 手工重置 + +## Manual reset -重置会删除并重建保留范围内的业务记录。先保存需要保留的 SQL、SSE、草案与执行回执;状态不明的写入先读回,不能用重置覆盖问题现场。 +Reset deletes and rebuilds business records in the reserved scope. Save any SQL, SSE, drafts, and execution receipts that must be retained first. Read back uncertain writes before resetting; do not overwrite an unresolved incident's state. -1. 停止模型任务、集成测试和其他业务写入,在 API 终端按 Ctrl-C 停止 uvicorn。保持本项目 Java 和数据服务运行,以便排空商品事件;若使用非默认端口,也须自行停止对应 API。默认 8101 仍监听时脚本拒绝维护。 -2. 在 ShopMate 根目录执行: +1. Stop model tasks, integration tests, and other writes. Press Ctrl-C in the API terminal to stop uvicorn. Keep this project's Java and data services running so product events can drain. Stop a non-default API port yourself as well; the script refuses maintenance while default port 8101 still listens. +2. Run from the ShopMate root: ```sh python3 scripts/reset_fixture.py ``` -3. 脚本确认商品 Outbox 已发布、指定 RocketMQ consumer 无待消费或在途记录后,停止 Java 写入,再重建 SQL 数据、处理 SQLite 夹具会话、通过实际 FAQ 发布服务写入政策,最后重新启动 Java。排空不可读或超时即停止。 -4. 重置成功后重新启动 API,再登录并新建会话: +3. The script confirms product Outbox publication and no pending/in-flight work for the designated RocketMQ consumer, then stops Java writes. It rebuilds SQL data, handles SQLite fixture conversations, publishes policies through the actual FAQ service, and restarts Java. Unreadable drain state or a timeout stops the procedure. +4. After reset succeeds, restart the API, sign in, and create a new conversation: ```sh uv run uvicorn shopmate.app:create_app --factory --host 127.0.0.1 --port 8101 ``` -清理按固定商品 ID、明确的夹具主体和版本进行,主体比较使用精确二进制语义。商品的非夹具订单、购物车或促销引用、其他操作员的商品变更草案,以及其他操作员对夹具营销活动的更新会阻止维护。清理顺序先处理促销、checkout、动作回执、退款、付款等依赖,再处理订单、商品元数据、商品与系列;不删整库、Redis 或消息队列,也不关闭外键约束。 +Cleanup uses fixed product IDs, explicit fixture subjects, and versions, with exact binary subject comparison. Non-fixture orders, carts, or promotion references to these products, other operators' product-change drafts, and other operators' updates to fixture campaigns block maintenance. Cleanup first removes dependent promotions, checkouts, action receipts, refunds, and payments, followed by orders, metadata, products, and families. It does not clear entire databases, Redis, or message queues, or disable foreign keys. -存在 `.run/sessions.sqlite3` 时,脚本用 SQLite backup API 备份到 `.run/backups/sessions-<时间>.sqlite3`,然后清除对应夹具主体的会话、草案引用和未决 prepare intent,避免旧会话继续使用已删除的业务实例。其他会话不作为此次清理目标。 +When `.run/sessions.sqlite3` exists, the script uses SQLite's backup API to write `.run/backups/sessions-.sqlite3`, then removes the relevant fixture subjects' conversations, draft references, and pending prepare intents so stale conversations cannot reuse deleted business instances. Other conversations are outside this cleanup scope. -**自动备份仅涵盖 SQLite,不是 MySQL 业务库备份。** SQL 和 SQLite 不共享事务:SQL 重建完成后才处理会话备份/清理及政策发布。若后续步骤失败,维护失败并保持 host 未就绪;先检查忽略目录中的 `.run/runtime.log` 和实际库状态,再决定恢复或重新执行,不将局部成功当成完整重置。 +**The automatic backup covers SQLite only, not the MySQL business database.** SQL and SQLite do not share a transaction: conversation backup/cleanup and policy publication follow SQL reconstruction. If a later step fails, maintenance fails and the host remains unready. Inspect ignored `.run/runtime.log` and the actual database state before deciding how to recover or rerun; partial success is not a complete reset. -`.run/` 包含本地凭证、会话与备份,保持忽略且不提交。需要保留上轮 MySQL 业务事实时,应在重置前另行保存所需数据库备份或权威 SQL 输出,而不是依赖会话副本恢复交易状态。 +`.run/` contains local credentials, conversations, and backups and remains ignored. To retain previous MySQL business facts, save the necessary database backup or authoritative SQL output before reset; a conversation copy cannot restore transaction state. -## 与旧记录的关系 +## Relationship to older records -旧 `evals/` 任务和[公开历史成绩](../evals/records/README.md)采用七商品、42 日 UTC 数据及不同代码版本,不能直接在此夹具下宣称复现同一分数。旧 78/90、定向 21/24 和[浏览器截图](demo-20260906/README.md)继续保留原始版本与分母;本说明不声明新版真实模型或端到端验收已通过。 +The old `evals/` tasks and [public historical results](../evals/records/README.md) use seven products, 42 UTC days, and different source versions. Running against this fixture does not reproduce the same score. The old 78/90, targeted 21/24, and [browser screenshots](demo-20260906/README.md) retain their original versions and denominators. This description does not claim that newer real-model or end-to-end acceptance has passed. -新版验收应先固定这套业务数据与报告口径,再使用参考 SQL、实际用户可见结果和 Java 写入终态判定。权限、未批准写入、版本冲突、并发与重复批准以及停止恢复分别检查,不以任务执行结束代替业务成功。 +New acceptance should first fix this dataset and reporting definition, then grade reference SQL, actual visible output, and Java write outcomes. Permissions, unapproved writes, version conflicts, concurrency/repeated approval, stopping, and recovery are checked separately; execution completion is not business success. -## 中文演示目录与商品图 +## Chinese demo catalog and product images -中文商品文案维护在 `scripts/data/demo-catalog-zh-CN.json`,不改写上游原始目录或历史订单快照。预览与应用: +Chinese product copy is maintained in `scripts/data/demo-catalog-zh-CN.json` without rewriting the upstream catalog or historical order snapshots. Preview and apply it with: ```sh uv run python scripts/localize_demo_catalog.py uv run python scripts/localize_demo_catalog.py --apply ``` -脚本从现有忽略目录读取操作员凭证,通过 `LISTING_UPDATE` 建草案,再走操作员批准。它更新当前商品标题和介绍,保留价格、库存、历史成交金额与原始快照;已有非初始商家标题保持不动。`--product AR-1001` 可限定单品。重置原始夹具后,可显式重新应用中文文案;不在每次服务启动时覆盖运营修改。 +The script reads operator credentials from existing ignored storage, creates `LISTING_UPDATE` drafts, then obtains operator approval. It updates current product titles and descriptions while preserving prices, stock, historical amounts, and original snapshots; existing merchant titles that differ from initialization remain unchanged. `--product AR-1001` limits the operation to one item. Chinese copy can be explicitly reapplied after resetting the original fixture; normal startup does not overwrite operations changes. -买家与商家共用 `web/public/products/` 的同 SKU 素材;权威目录已指定图片时优先使用指定值,仅在未指定且对应素材实际存在时补齐展示路径。原生 iOS 的图片集是这些文件的格式转换副本,用于缩略图展示。来源与原创商品概念图说明见该目录的 `IMAGE-CREDITS.md`。 +Buyers and merchants share the same SKU assets in `web/public/products/`. An image explicitly supplied by the authoritative catalog takes precedence; a display path is added only when no image was specified and the corresponding asset exists. Native iOS image sets are format-converted copies for thumbnails. See that directory's `IMAGE-CREDITS.md` for sources and original product-concept visuals. diff --git a/evals/README.md b/evals/README.md index 135652f..f07c9bb 100644 --- a/evals/README.md +++ b/evals/README.md @@ -1,8 +1,8 @@ -# ShopMate 业务任务与评测记录 +# ShopMate business tasks and evaluation records -执行前先停止手工启动的8101 API,保留ShopMate数据与Java服务。驱动只支持固定本机地址,自持正式API子进程,在每题重置前确认静默并恢复本人旧会话的prepare,然后停API、重置、重启并重新登录。它不按端口杀进程,不管理外部API;停止、恢复或写状态不明则保留夹具并停批。`--describe`不启动进程、不读取运行配置。 +Before execution, stop the manually started API on port 8101 while preserving ShopMate data and Java services. The driver supports fixed local addresses only and owns its production-API subprocess. Before each task reset, it checks for quiescence and recovers pending prepare operations in the current user's old conversations, then stops the API, resets, restarts, and signs in again. It does not kill processes by port or manage external APIs. Uncertain stopping, recovery, or write state retains the fixture and stops the batch. `--describe` neither starts processes nor reads runtime configuration. -当前默认协议是 [`retail/development.json`](retail/development.json):完整零售目录的12个开发任务,使用90个上海自然日、CNY合成定价。迁移后的任务、参考SQL和待采集基线独立放在 [`retail/`](retail/README.md),没有复用旧成绩;当前 `not_run` 不表示通过。12题保留经营问答、历史价格、追问、澄清、真实批准与取消的原业务意图,不代表完整零售工具验收,也不计入未来正式场景分母。 +The current default protocol is [`retail/development.json`](retail/development.json): 12 development tasks over the full retail catalog, with 90 Shanghai calendar days and synthetic CNY pricing. Migrated tasks, reference SQL, and baselines awaiting collection live separately in [`retail/`](retail/README.md); old scores are not reused, and `not_run` is not a pass. These 12 tasks preserve the original intents of business questions, historical prices, follow-ups, clarification, real approval, and cancellation. They are not a complete retail-tool acceptance suite or part of a future formal denominator. ```sh uv run python scripts/run_tasks.py --describe @@ -10,63 +10,63 @@ uv run python scripts/run_tasks.py --suite retail/development uv run python scripts/run_tasks.py --suite evals/formal.json --describe ``` -新驱动执行前从本次权威SQL记录获得目标SKU旧价和版本,并记录相对版本预期;不会猜测固定3→4,也不自动把执行完成算成业务通过。通用变更接口中的价格审批仍需 `PRICE_UPDATE`、CNY、PREPARED和完整商品/目标分金额集合唯一匹配。实际调用和状态不明时停止、保留夹具的规则不变。 +Before execution, the new driver reads each target SKU's prior price and version from that run's authoritative SQL and records relative version expectations. It does not assume a fixed 3→4 transition or automatically classify completed execution as business success. Price approval through the general change API still requires one unique match for `PRICE_UPDATE`, CNY, PREPARED, and the complete product/target-minor-unit set. Uncertain calls or state still stop execution and retain the fixture. -旧 `development.json`、`formal.json`、`baseline.json`、`sql/` 和 `records/` 原件继续保留。当前驱动只允许通过 `--describe` 读取旧协议,拒绝用新零售重置执行旧题;旧实验重现需使用各记录中的原始双仓库提交,不能把新夹具产生的运行并入旧78/90或定向21/24。 +Original `development.json`, `formal.json`, `baseline.json`, `sql/`, and `records/` artifacts remain. The current driver permits old protocols only through `--describe` and rejects running them after a new retail reset. Reproducing an older experiment requires the two original repository commits in its record; runs on the new fixture cannot enter the old 78/90 or targeted 21/24. -## 历史七商品协议与成绩 +## Historical seven-product protocols and results -以下数据、执行命令、R0和固定版本描述属于旧七商品实现,仅适用于记录对应的历史源码。当前零售执行以 [retail/README.md](retail/README.md) 为准。 +The data, commands, R0, and fixed-version descriptions below belong to the old seven-product implementation and apply only to its recorded source versions. Use [retail/README.md](retail/README.md) for current retail execution. -`development.json` 定义12个开发任务,`formal.json` 定义30个正式业务场景、每个重复3次。JSON中的规划状态不表示最新运行状态;实际结果见[记录索引](records/README.md)。`baseline.json` 是合成夹具定义,业务判定以数据库原始输出为准。 +`development.json` defines 12 development tasks; `formal.json` defines 30 formal business scenarios with 3 repetitions each. Planning status in JSON is not the latest run status; see the [record index](records/README.md). `baseline.json` defines a synthetic fixture, while raw database output determines business truth. -最近完整批次为 **78/90**(ShopMate `9173037d6eb43d295f6ccb5876fa6284e882dfdb`);修改后选取8个已知失败场景回归为 **21/24**(`02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28`)。两批CityBuddy均为 `69be167a3df030bf45795c49f444d6e7c24d0423`。原选集13/24与新选集21/24单独对照,不替换旧失败,不称单变量实验或新完整90次结果。旧84/90及开发、调试记录也独立保留。 +The latest complete historical batch scored **78/90** at ShopMate `9173037d6eb43d295f6ccb5876fa6284e882dfdb`. After changes, regression over 8 known failing scenarios scored **21/24** at `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28`. Both used CityBuddy `69be167a3df030bf45795c49f444d6e7c24d0423`. The old subset's 13/24 and new subset's 21/24 are compared separately: they do not replace failures, establish a single-variable experiment, or form another complete 90-attempt result. The earlier 84/90 and development/debug records also remain separate. -## 数据和判分口径 +## Data and grading definitions -固定截止时间 `2026-09-05T00:00:00Z`,UTC左闭右开窗口。成交额是退款前成功付款的历史订单金额,不按现价重算;CNY和USD分别计算,不换汇相加。夹具包含七款商品、42个完整交易日、历史价格、普通订单成功付款及未开始/PENDING/FAILED样本,没有秒杀成交样本。 +The cutoff is fixed at `2026-09-05T00:00:00Z`, with UTC start-inclusive/end-exclusive windows. Revenue uses pre-refund amounts from successfully paid historical orders, not current prices. CNY and USD are calculated separately without currency conversion or addition. The fixture contains seven products, 42 complete transaction days, historical prices, successful standard-order payments, and payment-not-started/PENDING/FAILED samples; it has no seckill sale samples. -人工检查实际可见正文与卡片、工具轨迹、参考SQL、草案回执及数据库终态。关键数值、商品集合、期间、币种、业务状态和读回须一致,不逐字匹配自然语言。金额以整数分为真值;百分比以前期为分母,零基期不适用。预算内自行纠错可通过;内部查询正确不能替代错误的可见答案,`executed`也不等于业务PASS。 +Manual grading checks actual visible text and cards, tool trajectories, reference SQL, draft receipts, and final database state. Key numbers, product sets, periods, currencies, business states, and readback must agree; natural-language wording need not match exactly. Integer minor units are authoritative for money. Percentage change uses the preceding period as denominator and is undefined for a zero base. Recovery within the original budget can pass; correct internal queries do not excuse an incorrect visible answer, and `executed` is not business PASS. -权限、未批准写入、并发/重复批准、版本冲突、响应丢失、预算与SQL限制在独立测试中验证,不加入业务完成率分母。正常任务中的仅提案、批准和取消按题面判定。没有错误写入不等于完成了用户要求。 +Permissions, unapproved writes, concurrent/repeated approval, version conflicts, lost responses, budgets, and SQL limits are covered by separate tests and excluded from the business completion-rate denominator. Proposal-only, approval, and cancellation steps in normal tasks are graded against the task request. Avoiding an incorrect write is not equivalent to completing the request. -## 执行 +## Execution -先启动独立ShopMate环境,停止其他任务和商品写入,确认待测实现已提交且源码干净;保留完整双SHA、fixture版本、模型、协议、预算及实际执行名单。模型配置从已有本地配置读取,凭证不写入执行记录。 +Start the isolated ShopMate environment, stop other tasks and product writes, and ensure the measured implementation is committed and source-clean. Record both full SHAs, fixture version, model, protocol, budget, and actual task list. Model settings come from existing local configuration; credentials are not written into run records. ```sh uv run python scripts/run_tasks.py --suite development uv run python scripts/run_tasks.py --suite formal ``` -正式完整运行默认采用题表的3次重复。使用 `--tasks S08,S11 --repetitions 1` 等参数属于选集回归,单独计数。不要在一批中临时改题、提示、预算或补发答案。 +A complete formal run defaults to the task table's 3 repetitions. Options such as `--tasks S08,S11 --repetitions 1` define a separately counted regression subset. Do not change tasks, prompts, budgets, or provide corrective answers midway through a batch. -- 模型只接收 `common_context` 和原文chat步骤;`evaluator`、参考SQL和预期排名/价格不发送给模型。同场景的追问保留会话,只在首次chat注入背景。 -- chat通过实际 `/api/merchant/chat` 执行并保存完整SSE与终态;预设追问按表发送,不由人工替模型纠错。 -- operator是预先约定的真实审批步骤。驱动从当前会话引用和卡片取得实际草案ID,再GET权威回执;按币种、PREPARED和完整商品/目标分金额集合唯一匹配。没有唯一匹配即停止该任务,不批准近似或错误草案。 -- 匹配后,驱动以直接操作员身份和同一session调用真实apply/discard接口,保存实际请求路径与回执。聊天“已批准”、模型工具或直接SQL改价不能代替这一步。浏览器交互另作演示。 -- 每个重复新登录、新会话;Bearer仅在内存中使用。不保存login响应、Authorization、密码或数据库/代理凭证。 +- The model receives only `common_context` and verbatim chat steps, not `evaluator`, reference SQL, or expected rankings/prices. Follow-ups retain the same conversation; background is injected only at the first chat. +- Chat uses the actual `/api/merchant/chat` endpoint and saves full SSE and terminal output. Prespecified follow-ups come from the task table, not manual correction on the model's behalf. +- Operator steps are pre-agreed real approvals. The driver obtains an actual draft ID from current conversation references and cards, then GETs the authoritative receipt. Currency, PREPARED state, and the complete product/target-minor-unit set must uniquely match; otherwise the task stops without approving an approximate or incorrect draft. +- After matching, the driver calls the real apply/discard API using the direct operator identity and same session, saving the request path and receipt. A chat claim of approval, model tool, or direct SQL price change cannot substitute for this step. Browser interaction is demonstrated separately. +- Each repetition uses a new sign-in and conversation. Bearer tokens remain in memory; login responses, Authorization headers, passwords, and database/proxy credentials are not saved. -## 每次重复的R0 +## R0 for each repetition -R0重建本项目夹具,不只是把价格改回。仅操作独立的 `shopmate` Compose project,不使用旧CityBuddy默认演示或bench库。 +R0 rebuilds this project's fixture rather than merely restoring prices. It operates only on the isolated `shopmate` Compose project, not CityBuddy's old default demo or benchmark database. -1. 先结束全部任务写入并保存上轮SSE、回执、SQL。状态不明的写操作先查结果,不能直接重置覆盖现场。 -2. 执行 `python3 scripts/reset_fixture.py`。脚本等待商品Outbox发布与指定消费队列排空;状态不可读或超时即停止,不继续正式任务。重置期间仍须由操作者保持无并发写入。 -3. 复用CityBuddy命名空间SQL重建七商品、42日订单/支付/回调/账本,清理该命名空间草案和相关Outbox,保留身份;不清整库、Redis或Broker队列。 -4. 通过实际host读取及 `products.sql`/`history.sql`/`scope.sql`记录基线,确认价格、版本、可编辑性和模型可见范围正确。generation以本次G0计,不要求固定值。 -5. 重置不删除SQLite旧会话;每次重复创建新session。被重置删除的旧草案引用不再是当前业务实例,原始记录保留追查,不能借旧会话执行下一题。`local_runtime.py up`仅在缺商品时造数,不能替代完整R0。 +1. End all task writes and preserve previous SSE, receipts, and SQL. Read back uncertain writes rather than resetting over the incident state. +2. Run `python3 scripts/reset_fixture.py`. It waits for product Outbox publication and the designated consumer queue to drain. Unreadable state or timeout stops the process before formal tasks. The operator must prevent concurrent writes throughout reset. +3. Reuse CityBuddy's namespaced SQL to reconstruct seven products and 42 days of orders/payments/callbacks/ledgers, cleaning drafts and related Outbox records in that namespace while retaining identities. Do not clear entire databases, Redis, or broker queues. +4. Record the baseline through actual host reads and `products.sql`/`history.sql`/`scope.sql`, checking prices, versions, editability, and model-visible scope. Generation is recorded as this run's G0, not a required constant. +5. Reset does not delete old SQLite conversations in this historical protocol; each repetition creates a new session. Old draft references deleted by reset no longer name current business instances. Retain originals for diagnosis and do not use an old conversation for the next task. `local_runtime.py up` seeds only when products are absent and is not a complete R0. -## SQL与写入终态 +## SQL and write outcomes -`evaluator.reference_sql`路径相对于本目录。参考SQL从权威订单、付款等基础表按订单类型/ID、主体、金额、币种和成功状态判定;模型只使用受限经营视图账号,不能取得评估者权限。查询设为UTC,草案/事件SQL的 `@session_id`绑定实际创建的session。 +`evaluator.reference_sql` paths are relative to this directory. Reference SQL grades authoritative order/payment tables by order type/ID, subject, amount, currency, and successful status. The model uses only a restricted business-view account and cannot obtain evaluator privileges. Queries use UTC; draft/event SQL binds `@session_id` to the actual created session. -只读任务也检查零额外草案/商品事件。写场景保存每次操作员处置前后以及最终的商品、草案、事件、历史原始输出。批准前价格不变、PREPARED的result/resolvedAt为空;APPLIED或CANCELLED有权威结果与完成时间,原items不改成新意图。每个获批商品版本3→4且有一条匹配的PRODUCT_PUBLICATION_CHANGED事件,N款批准使generation从G0推进到G0+N。取消不产生价格事件。 +Read-only tasks also check for zero extra drafts/product events. Write scenarios save raw products, drafts, events, and history before and after every operator disposition and at the end. Prices remain unchanged before approval, and PREPARED `result`/`resolvedAt` remain empty. APPLIED or CANCELLED records have authoritative results and completion times; original items are not rewritten into a new intent. Each approved product moves from version 3→4 with one matching PRODUCT_PUBLICATION_CHANGED event; approving N products advances generation from G0 to G0+N. Cancellation emits no price event. -未指定商品保持原状态;获批商品只改变目标价格、版本及正常更新时间,库存、名称、说明、可用性、币种和发布状态保持。`history.sql`输出在操作前后不变。错误草案、错值、范围错误、缺失读回、未完成动作及provider失败均保留,不用其他测试或后来成功补数。 +Unspecified products retain their state. Approved products change only the target price, version, and normal update time; stock, name, description, availability, currency, and publication status remain unchanged. `history.sql` output is identical before and after. Incorrect drafts, values, scope, missing readback, incomplete actions, and provider failures remain in the results and are not replaced with other tests or later successes. -## 记录组织 +## Record layout -本地展开原件保留在 `evals/results//`;公开记录位于[records](records/README.md),每批一份 `raw.tar.gz`,并提供直接阅读的 `assessment.md`、`run.json`及已有统计文件。包内根目录为该run-id,assessment中的任务相对路径在解包后对应原件。 +Expanded local originals are in `evals/results//`; public records are in [records](records/README.md), with one `raw.tar.gz` per batch and readable `assessment.md`, `run.json`, and available statistics. The archive root is the run ID; task-relative paths in assessments resolve within that extracted directory. -每批记录实际双SHA、as-of、模型/协议/预算、任务和重复数、起止时间、session、原始SSE、分析SQL、真实操作员回执和前后权威SQL。业务PASS、执行状态、调用用量、耗时分别报告。全量以x/90并列每场景0/3至3/3,选集以实际分母报告;未执行保持not_run。固定矩阵用于开发和回归,不能称未见任务泛化评测。 +Each batch records both actual full SHAs, as-of time, model/protocol/budget, tasks and repetitions, start/end times, sessions, raw SSE, analysis SQL, actual operator receipts, and before/after authoritative SQL. Business PASS, execution status, usage, and timing are reported separately. Complete runs report x/90 plus each scenario's 0/3–3/3; subsets use their actual denominator, and unexecuted tasks remain not_run. The fixed matrix supports development and regression, not claims of generalization to unseen tasks. diff --git a/evals/records/20260905T185513.918718Z/assessment.md b/evals/records/20260905T185513.918718Z/assessment.md index 899b5a0..24c2800 100644 --- a/evals/records/20260905T185513.918718Z/assessment.md +++ b/evals/records/20260905T185513.918718Z/assessment.md @@ -1,17 +1,17 @@ -# 首次完整业务验收:84/90 +# First full business acceptance: 84/90 -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -ShopMate:`0db5538c1d46bf440542c6159323e2bcd2ec755a` -数据截止:`2026-09-05T00:00:00Z`;fixture 来源为上述 CityBuddy 提交。 -主模型与分析模型均为 `gpt-5.6-terra`,经 CLIPROXY Chat Completions 适配调用。每个 chat turn 共用 16 次模型调用、300 秒截止时间。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`
+ShopMate: `0db5538c1d46bf440542c6159323e2bcd2ec755a`
+Data cutoff: `2026-09-05T00:00:00Z`; fixture source is the CityBuddy commit above.
+Both main and analysis models used `gpt-5.6-terra` through the CLIPROXY Chat Completions adapter. Each chat turn shared 16 model calls and a 300-second deadline. -固定 30 个业务场景各执行 3 次;每次从 R0 和新会话开始,同一场景内部保留多轮和操作员步骤。90 次均执行结束,无未执行项或执行器中断;根据冻结题面、实际可见回答、参考 SQL 和写入终态,业务完成 **84/90(93.33%)**。`executed` 不作为业务通过标记。 +Each of 30 fixed business scenarios ran 3 times. Every attempt began from R0 and a new conversation, retaining multi-turn and operator steps within the scenario. All 90 attempts finished, with no unrun attempts or executor interruptions. Under the frozen task statements, actual visible answers, reference SQL, and final write states, business completion was **84/90 (93.33%)**. `executed` is not a business-pass marker. -本批从 2026-09-05 18:55 UTC 运行至 20:38 UTC。保存原始 SSE、模型/工具记录、操作员回执与各阶段 SQL;失败和工具自纠错没有删除。任务定义与参考 SQL 在同一提交的 `evals/formal.json` 和 `evals/sql/`。 +The batch ran from 18:55 to 20:38 UTC on 2026-09-05. Raw SSE, model/tool records, operator receipts, and SQL from each stage are retained, including failures and tool self-correction. Task definitions and reference SQL are in `evals/formal.json` and `evals/sql/` at the same commit. -## 逐场景结果 +## Per-scenario results -| 场景 | r1 | r2 | r3 | 完成数 | +| Scenario | r1 | r2 | r3 | Completed | |---|---|---|---|---| | S01 | PASS | PASS | PASS | 3/3 | | S02 | PASS | PASS | PASS | 3/3 | @@ -44,32 +44,33 @@ ShopMate:`0db5538c1d46bf440542c6159323e2bcd2ec755a` | S29 | PASS | PASS | PASS | 3/3 | | S30 | PASS | PASS | PASS | 3/3 | -## 六次业务失败 -- **S03-r3:无依据的经营解释。** 历史成交均价与当前售价的数值正确,但可见回答断言存在历史折让、优惠或组合成交。该数据只能证明两种价格口径不同,不能证明发生过促销;后续补充“原因无法确定”没有撤销前面的具体断言。 -- **S08-r1/r2/r3:分析对象扩大。** 问题指定帆布袋,后续比较仍应只针对该商品;主 Agent 委派时却扩为全店商品,最终没有完整交付该商品两期零成交及增长率不适用的结论。r1 另外生成了超过工具约定长度的说明,运行时静默截断导致条件丢失;另外两次说明未被截断,但已经选错分析范围。 -- **S11-r1/r2:排名查询失败后未交付结果。** 查询错误被泛化为“数据库不可用或拒绝”,最终缺少要求的销量、成交额排名。单独只读重放 r1 首条失败 SQL 得到 MySQL 1690,原因为无符号数相减越界;有符号对照可返回负值。该重放不是正式执行之一。r3 在原预算内改写查询并完成,按原规则通过。 +## Six business failures -只读任务未改动商品、草案或交易历史。所有实际批准、取消及只提案场景的商品、版本、草案终态、事件和历史不变量均符合原先定义;没有把这些边界通过额外计入完成数。 +- **S03-r3: unsupported business explanation.** Historical average sale price and current selling price were numerically correct, but the visible answer asserted historical discounts, promotions, or bundled sales. The data establishes different price definitions, not that a promotion occurred. A later “原因无法确定” (verbatim model text: cause uncertain) did not retract those specific assertions. +- **S08-r1/r2/r3: expanded analysis scope.** The question specified the canvas tote, and the follow-up comparison should have stayed on that product. Main-agent delegation expanded to the whole catalog, and the final answer did not fully deliver the tote's zero sales in both periods and inapplicable growth rate. r1 also exceeded the tool's brief-length contract; silent runtime truncation dropped conditions. The other two briefs were not truncated but had already selected the wrong scope. +- **S11-r1/r2: no ranking results after query failure.** Query errors were generalized as “数据库不可用或拒绝” (verbatim model text: database unavailable or rejected), and the requested quantity/revenue rankings were missing. A separate read-only replay of r1's first failed SQL returned MySQL 1690 from unsigned subtraction underflow; a signed comparison query returned negative values. That replay was not a formal attempt. r3 rewrote its query within the original budget and completed the task, passing under the original rules. -## 保留的过程观察 +Read-only tasks did not change products, drafts, or transaction history. In all actual approval, cancellation, and proposal-only scenarios, products, versions, final draft states, events, and historical invariants matched the original definitions. These boundary checks were not counted as extra completions. -- S04-r3 的中间 SQL 聚合曾重复计算,模型在可见最终交付前纠正;S13-r2 第二次分析补齐了首次遗漏的商品。均在原预算内完成。 -- S06 的日期和值正确,但“均匀/不集中”只适用于有成交的四天,不足以概括整个十四天窗口。原任务未定义集中度阈值,保留措辞问题,没有事后增加阈值改判。 -- S14-r3 正确给出日期贡献,但关于偶发波动的表述偏强,不能据此声称有统计显著性或确定因果。 -- S22-r1、S23-r2 调用呈现工具时没有可用指标,错误未形成错误卡片;后续可见回答与实际业务结果完整正确,保留工具错误但不把可恢复过程单独判为任务失败。 +## Retained process observations -## 实际调用与耗时 +- Intermediate SQL in S04-r3 double-counted an aggregate before the model corrected it ahead of visible final delivery. S13-r2's second analysis added a product omitted by the first. Both completed within budget. +- S06's dates and values were correct, but “均匀/不集中” (verbatim model text: uniform/not concentrated) applied only to the four days with sales, not the entire fourteen-day window. The original task defined no concentration threshold, so the wording concern was retained without adding a threshold retrospectively to change the verdict. +- S14-r3 correctly reported date contributions, but described incidental fluctuation too strongly; this does not establish statistical significance or definite causality. +- S22-r1 and S23-r2 called the presentation tool without available metrics. The errors produced no incorrect cards; subsequent visible answers and actual business outcomes were complete and correct. Tool errors remain recorded without treating recovered intermediate failures as separate task failures. -147 个 chat turn 共记录 832 次模型调用(主循环 663、分析子循环 169),全部报告了输入/输出用量。合计输入 **5,027,066 token**、输出 **191,747 token**;每次调用只累计一次,总输入包含缓存读入部分,未重复叠加运行时聚合。 +## Actual calls and elapsed time -- 场景执行器墙钟耗时:中位数 62.770 秒,p95 124.340 秒,范围 22.716–147.925 秒;包含登录、重置、SQL 核对和脚本操作员步骤,不是纯模型时延。 -- 单次 chat turn 服务端耗时:中位数 26.749 秒,p95 103.109 秒,范围 7.676–143.313 秒;涵盖该轮主/子调用与工具,未加入真实人工审批等待。 -- 本批没有保存缓存字段的存在性,缓存读计数为 0 时无法区分代理明确报告与兼容层默认;缓存写 0 为 SDK 占位。不据此报告缓存命中率或收益。 -- SSE 没有逐事件接收时间,不能重建首个有效结果耗时;没有代理费率或账单,不将 token 数转为实际金额成本。 +Across 147 chat turns, 832 model calls were recorded: 663 main-loop and 169 analysis-loop calls. All reported input/output usage. Total input was **5,027,066 tokens** and output **191,747 tokens**. Each call was counted once; input includes cache reads, without adding runtime aggregates again. -耗时分位数使用排序后线性插值,包含通过和失败任务;仅描述这次本地运行,不是容量或线上 SLO。详细数字见 `statistics.json`,完整原件保留在本目录。 +- Scenario executor wall time: median 62.770 seconds, p95 124.340 seconds, range 22.716–147.925 seconds. This includes login, reset, SQL checks, and scripted operator steps, not pure model latency. +- Per-chat server duration: median 26.749 seconds, p95 103.109 seconds, range 7.676–143.313 seconds. This includes the turn's main/analysis calls and tools, excluding actual human approval waiting. +- This batch did not retain cache-field presence. A zero cache-read count cannot distinguish an explicit proxy report from an adapter default; zero cache writes were SDK placeholders. Neither a cache hit rate nor benefits can be inferred. +- SSE lacks per-event receive timestamps, so time to the first useful result cannot be reconstructed. Without proxy rates or invoices, token counts are not converted to actual monetary cost. -## 后续版本 +Percentiles use linear interpolation after sorting and include passing and failing tasks. They describe this local run, not capacity or an online SLO. See `statistics.json` for details; full originals are retained in this directory. -本批揭示的问题将通过通用分析范围规则、完整说明校验、可纠错 SQL 反馈及缓存字段可用性标记修复。修复后的任务回归和完整批次独立保存,不覆盖本批、不拼接不同版本的成功样本。固定任务矩阵用于业务验收与回归,不称未见过的公开基准或泛化能力估计。 +## Subsequent versions + +The follow-up work identified by this batch covered general analysis-scope rules, complete-brief validation, actionable SQL error feedback, and cache-field availability flags. Regressions and full runs after those fixes are saved independently, without overwriting this batch or combining successful samples across versions. The fixed task matrix serves business acceptance and regression; it is not an unseen public benchmark or a generalization estimate. diff --git a/evals/records/20260905T204513.490639Z/assessment.md b/evals/records/20260905T204513.490639Z/assessment.md index 9d1340f..82998f8 100644 --- a/evals/records/20260905T204513.490639Z/assessment.md +++ b/evals/records/20260905T204513.490639Z/assessment.md @@ -1,13 +1,13 @@ -# 三项针对性回归:1/3 +# Three targeted regressions: 1/3 -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -ShopMate:`91347b986fcfc65d08c4012dcaa14e47dd7f0ada` -沿用首次正式任务定义、固定数据、主/分析模型 `gpt-5.6-terra` 及每 turn 16 次调用/300 秒预算。S03、S08、S11 各运行一次;三次均执行结束,业务完成 1/3。这是调试回归,不并入原 84/90。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`
+ShopMate: `91347b986fcfc65d08c4012dcaa14e47dd7f0ada`
+The run retained the first formal task definitions, fixed data, main/analysis model `gpt-5.6-terra`, and per-turn budget of 16 calls / 300 seconds. S03, S08, and S11 ran once each. All three executed to completion; business completion was 1/3. This diagnostic regression is separate from the original 84/90. -| 场景 | 判定 | 实际结果 | +| Scenario | Verdict | Actual outcome | |---|---|---| -| S03-r1 | PASS | 历史均价23元、当前售价24元、70件/1610元正确;可见回答明确不能由价差断定优惠或组合销售。两次指标呈现工具失败,正文仍完整交付。 | -| S08-r1 | FAIL | 库存80、42天没有成交正确,但两期比较仍扩大为全店七款商品,未交付帆布袋0→0及增长率不适用。Skill实际加载,范围错误已在首个委派参数中出现;新验证拒绝超长说明和过多segments后,第三次委派触及既有次数上限,没有分析子调用。主循环之后取得帆布袋空序列,最终仍给全店汇总。 | -| S11-r1 | FAIL | 三条SQL均执行成功,却以猜测的 Ceramic mug(陶瓷杯)等名称做精确关联,漏掉实际 Ceramic mug,最终把杯判为0并说两榜相同。参考SQL应为杯12件/456元,与茶交换件数/金额的第二三名。1690反馈未触发,不能声称这次验证了模型利用该反馈恢复。 | +| S03-r1 | PASS | Historical average CNY 23, current price CNY 24, and 70 units / CNY 1610 were correct. The visible answer explicitly rejected inferring discounts or bundles from the price difference. Two metric-presentation calls failed, but the prose fully delivered the answer. | +| S08-r1 | FAIL | Stock 80 and no sales for 42 days were correct, but the two-period comparison again expanded to all seven products, omitting the tote's 0 → 0 and inapplicable growth rate. The Skill was actually loaded; the scope error was already in the first delegation parameters. After new validation rejected an overlong brief and too many segments, the third delegation hit the existing attempt limit, with no analysis-model calls. The main loop later obtained the tote's empty series but still delivered store totals. | +| S11-r1 | FAIL | All three SQL queries executed, but exact joins used guessed names such as `Ceramic mug(陶瓷杯)` (verbatim query value), missing the actual `Ceramic mug`. The final answer assigned the mug zero sales and claimed both rankings were identical. Reference SQL gives the mug 12 units / CNY 456; mug and tea exchange second/third place between quantity and revenue rankings. Error 1690 did not occur, so this attempt did not verify recovery using that feedback. | -三个场景前后商品、历史交易、generation、草案和商品事件均保持只读。原题与原始记录未改写。后续针对主循环范围路由及分析SQL的商品身份来源处理通用原因,再独立回归;不通过改题面或移除失败任务补成绩。 +Before/after products, historical transactions, generation, drafts, and product events remained unchanged in all three scenarios. Original tasks and raw records were not rewritten. Follow-up work addressed the general causes in main-loop scope routing and product-identity sources for analysis SQL, with independent regressions, without changing task statements or removing failed tasks to improve the score. diff --git a/evals/records/20260905T210133.531635Z/assessment.md b/evals/records/20260905T210133.531635Z/assessment.md index dbec3d3..af23cb3 100644 --- a/evals/records/20260905T210133.531635Z/assessment.md +++ b/evals/records/20260905T210133.531635Z/assessment.md @@ -1,12 +1,12 @@ -# 范围路由与商品身份回归:2/2 +# Scope-routing and product-identity regression: 2/2 -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -ShopMate:`9173037d6eb43d295f6ccb5876fa6284e882dfdb` -S08、S11 沿用原正式题面和固定数据,各运行一次。主/分析模型均为 `gpt-5.6-terra`,每 turn 共享 16 次调用和300秒预算;题面、参考SQL、字段上限和判定规则未改。两次均执行结束并完成业务目标,不合并入旧84/90或前次1/3。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`
+ShopMate: `9173037d6eb43d295f6ccb5876fa6284e882dfdb`
+S08 and S11 each ran once using the original formal tasks and fixed data. Main and analysis models both used `gpt-5.6-terra`, sharing 16 calls and 300 seconds per turn. Task statements, reference SQL, field limits, and scoring rules did not change. Both attempts finished and met their business goals, separately from the old 84/90 and preceding 1/3. -| 场景 | 判定 | 核对结果 | +| Scenario | Verdict | Checked outcome | |---|---|---| -| S08-r1 | PASS | 目录确认帆布袋存在、库存80;42天无成交;两段14天均0、差0、增长率因零基期不适用。两次委派的segments只含帆布袋ID,三条实际SQL都使用该ID;正文与分析卡完整交付,没有全店汇总替代。保留重复历史指标卡和USD零行冗余的过程观察。 | -| S11-r1 | PASS | 实际目录及SQL按product_id关联。件数咖啡70/茶14/杯12,金额咖啡1610/杯456/茶238元,正确展示茶与杯换位。本次实际出现MySQL1690,下一条查询在同一范围内使用CAST AS SIGNED后成功;不是把无SQL错误当成正确。brief过多segments也在既有预算内修正,共11次模型调用。 | +| S08-r1 | PASS | The catalog confirmed the canvas tote existed with stock 80. There were no sales for 42 days; both 14-day periods were zero, their difference was zero, and growth was inapplicable because the baseline was zero. Both delegation segments contained only the tote ID, and all three actual SQL queries used it. Prose and analysis cards fully delivered the result without substituting store totals. Duplicate historical metric cards and a redundant USD zero row remain process observations. | +| S11-r1 | PASS | Actual catalog data and SQL joined by product_id. Quantities were coffee 70 / tea 14 / mug 12; amounts were coffee 1610 / mug 456 / tea 238 CNY, correctly showing tea and mug switching places. MySQL 1690 actually occurred; the next query used CAST AS SIGNED within the same scope and succeeded. Correctness did not rely on an error-free SQL run. Excess brief segments were also corrected within the original budget, with 11 model calls in total. | -两次前后商品、历史交易、generation、scope均相同,草案和商品事件为空。原始模型、工具、SQL、可见内容及失败后的恢复过程都保留。该针对性回归只用于确认修复,再以同版本完整30×3批次报告正式业务结果。 +Products, historical transactions, generation, and scope were identical before/after both attempts; drafts and product events were empty. Original model/tool/SQL records, visible content, and recovery after failure are retained. This targeted regression confirms the fixes only; the full 30 × 3 batch at the same version reports formal business outcomes separately. diff --git a/evals/records/20260905T210851.446894Z/assessment.md b/evals/records/20260905T210851.446894Z/assessment.md index 286a5fb..fb268de 100644 --- a/evals/records/20260905T210851.446894Z/assessment.md +++ b/evals/records/20260905T210851.446894Z/assessment.md @@ -1,22 +1,22 @@ -# 冻结版本业务验收:78/90 +# Frozen-version business acceptance: 78/90 -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -ShopMate:`9173037d6eb43d295f6ccb5876fa6284e882dfdb` -结果目录:`evals/results/20260905T210851.446894Z`。 -数据截止:`2026-09-05T00:00:00Z`,UTC;fixture来源为上述CityBuddy提交。 -主模型与分析模型均为`gpt-5.6-terra`,经CLIPROXY Chat Completions适配调用。每个chat turn主/分析调用共用16次模型调用、300秒截止时间;主循环另有8个工具轮次限制,未调整预算。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`
+ShopMate: `9173037d6eb43d295f6ccb5876fa6284e882dfdb`
+Result directory: `evals/results/20260905T210851.446894Z`.
+Data cutoff: `2026-09-05T00:00:00Z`, UTC; fixture source is the CityBuddy commit above.
+Both main and analysis models used `gpt-5.6-terra` through the CLIPROXY Chat Completions adapter. Main/analysis calls shared 16 model calls and a 300-second deadline per chat turn. The main loop also had an 8-tool-round limit; budgets were unchanged. -固定30个业务场景各执行3次;每次从R0和新会话开始,同一场景内部保留多轮对话和真实操作员步骤。按冻结题面、实际用户可见回答和卡片、参考SQL、审批回执及写入终态判定。内部错误若在原预算内自行纠正并完成目标,可判通过;`executed`不是业务通过标记。失败保留原分母,不用隐藏工具结果补用户答案,不拼接旧批或定向回归的成功。 +Each of 30 fixed business scenarios ran 3 times. Every attempt started from R0 and a new conversation, retaining multi-turn dialogue and real operator steps within the scenario. Judgment used frozen task statements, actual user-visible answers/cards, reference SQL, approval receipts, and final write states. An internal error could pass if corrected within the original budget and the goal was completed. `executed` is not a business-pass marker. Failures retain their original denominator; hidden tool results do not fill gaps in the user answer, and successes from old batches or targeted regressions are not combined here. -**90次全部完成判读:78 PASS、12 FAIL,业务完成率78/90(86.67%)。** 执行状态另计88 executed、2 failed(S20-r3、S30-r1);两次均因没有可批准的匹配草案而停止后续步骤,未尝试操作员写入,未被归为provider故障。业务失败包括正常executed但未完成题目要求的样本,全部保留在90次分母。 +**All 90 attempts were assessed: 78 PASS, 12 FAIL, for business completion of 78/90 (86.67%).** Execution status separately recorded 88 executed and 2 failed (S20-r3, S30-r1). Both stopped subsequent steps because no matching approvable draft existed, without attempting operator writes; neither was classified as a provider failure. Business failures also include normally executed attempts that did not meet the task, all retained in the denominator of 90. -本批结束时间为`2026-09-05T23:02:57.800425+00:00`,运行正常收尾。以下配置和结果属于本批冻结版本,主工具轮次上限为8;不将后续版本的限制调整倒写进本批证据。 +The batch ended at `2026-09-05T23:02:57.800425+00:00` and shut down normally. The configuration and results below belong to this frozen version with 8 main-tool rounds. Later limit changes are not retroactively applied to this evidence. -原始SSE、模型/工具记录、操作员回执与各阶段SQL保留在本目录。任务定义与参考SQL在同一提交的`evals/formal.json`和`evals/sql/`。下文原件路径均相对本结果目录。 +Raw SSE, model/tool records, operator receipts, and SQL from each stage are retained in this directory. Task definitions and reference SQL are in `evals/formal.json` and `evals/sql/` at the same commit. Raw paths below are relative to this result directory. -## 逐场景结果 +## Per-scenario results -| 场景 | r1 | r2 | r3 | 完成数 | +| Scenario | r1 | r2 | r3 | Completed | |---|---|---|---|---| | S01 | PASS | PASS | PASS | 3/3 | | S02 | PASS | PASS | FAIL | 2/3 | @@ -49,47 +49,48 @@ ShopMate:`9173037d6eb43d295f6ccb5876fa6284e882dfdb` | S29 | PASS | PASS | PASS | 3/3 | | S30 | FAIL | FAIL | PASS | 1/3 | -## 十二次业务失败 -| 次数 | 业务原因 | 原件路径 | +## Twelve business failures + +| Attempt | Business reason | Raw paths | |---|---|---| -| S02-r3 | load_skill加7次串行目录查询耗尽8个主工具轮,第9次调用被强制仅输出文字;尚未查询成交额,未交付前三名、金额及份额。共享16次调用/300秒未耗尽,不是已证实的provider或数据库故障。 | `S02-r3/step-01.sse`、`S02-r3/step-01-terminal.json`、`S02-r3/sql/after/S02.jsonl` | -| S06-r1 | 将实际14日半开窗口写成15日,进一步向用户展示4/15=26.67%的成交日覆盖率;正确应4/14≈28.57%。四个日期及金额正确,不抵消错误分母和派生值。 | `S06-r1/saved-session.json`、`S06-r1/step-01-terminal.json`、`S06-r1/sql/after/S06.jsonl` | -| S07-r1 | 两项裸sales引用均解析为CNY快照2304/+17.79%;其中一项说明却标USD280/+11.11%,形成用户可见金额、币种、增幅矛盾。正确正文没有撤销错误卡片。 | `S07-r1/step-01.sse`、`S07-r1/saved-session.json`、`S07-r1/sql/after/S07.jsonl` | -| S07-r3 | 正文把CNY前期金额写为2058元,实际为1956元;同文的本期2304、差额348及17.79%也与2058不相容。 | `S07-r3/saved-session.json`、`S07-r3/step-01-terminal.json`、`S07-r3/sql/after/S07.jsonl` | -| S12-r3 | period长度验证连续失败后用尽本轮分析机会,实际分析查询0条,最终未给两期金额、件数与商品贡献。 | `S12-r3/step-01.sse`、`S12-r3/step-01-terminal.json`、`S12-r3/saved-session.json` | -| S13-r3 | 只完成6款商品(含USD),限量款未继续查询,最终明确本题所需的该CNY商品仍未核对,整体范围没有完成。 | `S13-r3/step-01-terminal.json`、`S13-r3/saved-session.json`、`S13-r3/sql/after/S13.jsonl` | -| S14-r3 | 实際只查5款CNY,最终明确限量款未纳入人民币合计;已查商品及杯的日期贡献正确,但所需整体范围仍未完成。 | `S14-r3/step-01-terminal.json`、`S14-r3/saved-session.json`、`S14-r3/sql/after/S14.jsonl` | -| S17-r1 | 追问要求在上一轮有成交商品中筛库存≥50,实际加入零成交的Canvas tote并显示保留3款;正确仅coffee和tea两款。 | `S17-r1/step-02.sse`、`S17-r1/step-02-terminal.json`、`S17-r1/sql/after/S17.jsonl` | -| S17-r3 | 与r1相同,第二轮将候选换成coffee/tea/tote,实际SQL和可见卡均加入原成交集合之外的帆布袋,错误没有纠正。 | `S17-r3/step-02.sse`、`S17-r3/step-02-terminal.json`、`S17-r3/sql/after/S17.jsonl` | -| S20-r3 | load_skill加7次串行目录读取耗尽8个主工具轮,尚未分析或建草案便被强制文字收尾。SQL确认草案0、事件0、价格仍2400;无匹配草案是未创建的结果,非金额或币种匹配错误;未尝试操作员写入。 | `S20-r3/step-01.sse`、`S20-r3/execution.json`、`S20-r3/sql/step-02-before/drafts.jsonl`、`S20-r3/sql/after/products.jsonl` | -| S30-r1 | load_skill、两款商品各自的search/get/context、pending共8个串行工具轮后,主循环强制文字收尾,两张草案均未创建。SQL确认草案0、事件0,咖啡/茶价格未变;后续审批无法执行,未尝试操作员写入。 | `S30-r1/step-01.sse`、`S30-r1/execution.json`、`S30-r1/sql/step-02-before/drafts.jsonl`、`S30-r1/sql/after/products.jsonl` | -| S30-r2 | 两张独立草案及咖啡批准正确,但用户明确要求取消茶后,模型仅读商品与pending,没有调用discard_change。最终茶仍PREPARED、result/resolved_at为空;诚实说明未取消不等于完成任务。 | `S30-r2/step-03.sse`、`S30-r2/step-02-operator.json`、`S30-r2/sql/after/drafts.jsonl`、`S30-r2/sql/after/events.jsonl` | +| S02-r3 | load_skill plus 7 serial catalog queries exhausted the 8 main-tool rounds. The ninth call was forced to return text only, before querying revenue or delivering the top three products, amounts, and shares. The shared 16-call / 300-second budget was not exhausted; this was not a confirmed provider or database failure. | `S02-r3/step-01.sse`, `S02-r3/step-01-terminal.json`, `S02-r3/sql/after/S02.jsonl` | +| S06-r1 | Described the actual 14-day half-open interval as 15 days and showed a sales-day coverage of 4/15 = 26.67%, instead of 4/14 ≈ 28.57%. Correct dates and amounts for the four days do not offset the wrong denominator and derived value. | `S06-r1/saved-session.json`, `S06-r1/step-01-terminal.json`, `S06-r1/sql/after/S06.jsonl` | +| S07-r1 | Two bare sales references both resolved to the CNY snapshot 2304 / +17.79%, while one description labeled it USD 280 / +11.11%, creating a visible amount/currency/growth contradiction. Correct prose did not retract the incorrect card. | `S07-r1/step-01.sse`, `S07-r1/saved-session.json`, `S07-r1/sql/after/S07.jsonl` | +| S07-r3 | Prose gave the prior CNY amount as 2058 instead of 1956. Its current amount 2304, difference 348, and 17.79% were also incompatible with 2058. | `S07-r3/saved-session.json`, `S07-r3/step-01-terminal.json`, `S07-r3/sql/after/S07.jsonl` | +| S12-r3 | Repeated period-length validation failures exhausted the turn's analysis opportunities. No analysis queries ran, and the final answer omitted both periods' amounts, quantities, and product contributions. | `S12-r3/step-01.sse`, `S12-r3/step-01-terminal.json`, `S12-r3/saved-session.json` | +| S13-r3 | Covered only 6 products, including USD, without querying the limited edition. The final answer explicitly left that required CNY product unchecked, so the full scope was incomplete. | `S13-r3/step-01-terminal.json`, `S13-r3/saved-session.json`, `S13-r3/sql/after/S13.jsonl` | +| S14-r3 | Queried only 5 CNY products and explicitly excluded the limited edition from the CNY total. Queried products and the mug's date contributions were correct, but the required full scope was incomplete. | `S14-r3/step-01-terminal.json`, `S14-r3/saved-session.json`, `S14-r3/sql/after/S14.jsonl` | +| S17-r1 | The follow-up requested stock ≥ 50 within the preceding turn's products with sales. It added the zero-sales Canvas tote and showed 3 retained products; only coffee and tea qualified. | `S17-r1/step-02.sse`, `S17-r1/step-02-terminal.json`, `S17-r1/sql/after/S17.jsonl` | +| S17-r3 | As in r1, the second turn replaced the candidates with coffee/tea/tote. Both actual SQL and the visible card added the tote from outside the original sales set, without correction. | `S17-r3/step-02.sse`, `S17-r3/step-02-terminal.json`, `S17-r3/sql/after/S17.jsonl` | +| S20-r3 | load_skill plus 7 serial catalog reads exhausted the 8 main-tool rounds, forcing a text-only ending before analysis or draft creation. SQL confirmed 0 drafts, 0 events, and price still 2400. The missing match resulted from no draft being created, not an amount/currency mismatch; no operator write was attempted. | `S20-r3/step-01.sse`, `S20-r3/execution.json`, `S20-r3/sql/step-02-before/drafts.jsonl`, `S20-r3/sql/after/products.jsonl` | +| S30-r1 | load_skill, separate search/get/context calls for two products, and pending used 8 serial tool rounds, after which the main loop was forced to end in text. Neither draft was created. SQL confirmed 0 drafts, 0 events, and unchanged coffee/tea prices. Subsequent approvals could not run; no operator writes were attempted. | `S30-r1/step-01.sse`, `S30-r1/execution.json`, `S30-r1/sql/step-02-before/drafts.jsonl`, `S30-r1/sql/after/products.jsonl` | +| S30-r2 | Two independent drafts and coffee approval were correct. After an explicit request to cancel tea, the model only read products and pending changes without calling discard_change. Tea remained PREPARED, with result/resolved_at empty. Honestly stating that cancellation had not happened did not complete the task. | `S30-r2/step-03.sse`, `S30-r2/step-02-operator.json`, `S30-r2/sql/after/drafts.jsonl`, `S30-r2/sql/after/events.jsonl` | -S13/S14遗漏的商品在夹具中恰为零,也不能由评估者代补其未完成的查询和结论。S02/S20/S30的主工具轮数上限与模型调用总额度不同;没有把模型“无法读取”的文字当成实际数据源故障。 +The products omitted in S13/S14 happened to have zero sales in the fixture, but the evaluator cannot supply missing queries or conclusions. S02/S20/S30's main-tool-round limit is distinct from the total model-call allowance. A model's “无法读取” (verbatim model text: cannot read) claim was not treated as an actual data-source failure. -## 已核对的业务终态与过程观察 +## Checked business terminal states and process observations -S01–S18的只读任务未改变商品、草案、事件或付款历史。S19、S20及S21–S30中实际发起的审批均匹配指定草案,待审阶段保持原价,批准后商品价格、版本、generation与对应事件符合回执;未观察到未批准调价、额外商品变动或付款历史污染。 +Read-only S01–S18 tasks did not change products, drafts, events, or payment history. Every approval actually attempted in S19, S20, and S21–S30 matched the specified draft. Prices remained unchanged while pending; after approval, product prices, versions, generation, and corresponding events matched receipts. No unapproved price changes, extra product changes, or payment-history contamination were observed. -S21–S29的单品、同批多品、USD价格、只待审、撤销重提及分析后取消均符合各自终态。S30-r1零草案、零写入;S30-r2只完成咖啡审批,茶仍待审,故仍判失败;S30-r3完成咖啡批准、茶取消并分别读回。上述观察不额外增加通过次数,不用“没有错写”替代“完成任务”,也不代替独立权限或并发事务验证。 +S21–S29 reached their specified terminal states for single-product, same-batch multi-product, USD-price, pending-only, cancel/resubmit, and cancel-after-analysis cases. S30-r1 had zero drafts/writes. S30-r2 completed only coffee approval, leaving tea pending, so it failed. S30-r3 approved coffee, canceled tea, and read both back. These observations add no extra passes, do not substitute no incorrect writes for task completion, and do not replace independent authorization or concurrent-transaction tests. -- S11三次均实际出现MySQL1690,并在原预算内改为SIGNED相减后完成正确排名;没有把中间错误单独判失败。 -- S16-r3的一条SQL虽执行成功,却因重复JOIN放大总计;模型在可见交付前重新聚合为正确96件/2304元,按原目标通过。 -- S02-r1用变化百分比样式展示份额,但相邻说明与展开卡明确份额含义;S20-r1的额外重复指标卡误将6款加一个聚合指标题为“7款商品”,完整分析卡、选品依据及批准终态仍正确。保留呈现质量问题,不把PASS描述成完美表达。 -- 多次present_metrics引用或brief验证失败后通过实际重查、分析卡或最终正文完成目标;通过不等于没有工具错误,失败也不能只由工具is_error判断。 +- All three S11 attempts actually encountered MySQL 1690 and completed correct rankings after switching to SIGNED subtraction within the original budget. Intermediate errors were not separate task failures. +- One SQL query in S16-r3 executed successfully but inflated totals through repeated JOINs. Before visible delivery, the model reaggregated to the correct 96 units / CNY 2304 and passed the original goal. +- S02-r1 used a change-percentage style to display share, but adjacent explanations and the expanded card made the share meaning clear. An extra duplicate metric card in S20-r1 labeled 6 products plus an aggregate metric as “7款商品” (verbatim model text: 7 products), while the full analysis card, selection rationale, and approval outcome remained correct. These presentation-quality issues are retained; PASS does not mean perfect expression. +- Several attempts recovered from present_metrics references or brief-validation failures through actual queries, analysis cards, or final prose. Passing does not mean no tool errors, and a tool's is_error flag alone does not determine task failure. -## 实际调用与耗时 +## Actual calls and elapsed time -145个chat turn共记录928次模型调用:主循环743次、分析子循环185次。928次均明确报告输入/输出和缓存读取字段。按每次调用累计一次,输入合计 **5,954,918 token**,输出 **191,877 token**;总输入已经包含缓存读取,不再叠加运行时汇总或缓存值。 +Across 145 chat turns, 928 model calls were recorded: 743 main-loop and 185 analysis-loop calls. All 928 explicitly reported input/output and cache-read fields. Counting each call once, total input was **5,954,918 tokens** and output **191,877 tokens**. Input already includes cache reads; runtime aggregates and cached tokens are not added again. -明确报告的缓存读取为 **4,727,296 token**,占总输入 **79.38%**(4,727,296/5,954,918)。这是本次已报告输入token中来自缓存读取的比例,**不是请求缓存命中率,也不是已证明的时延或费用收益**。缓存创建未测量;没有代理费率或账单,不把token换算成实际成本。 +Explicitly reported cache reads totaled **4,727,296 tokens**, or **79.38%** of input (4,727,296 / 5,954,918). This is the share of reported input tokens read from cache, **not a request cache hit rate or demonstrated latency/cost benefit**. Cache creation was not measured. Without proxy rates or invoices, token counts are not converted to actual cost. -| 统计对象 | 样本数 | 中位数 | p95 | 范围 | +| Measurement | Samples | Median | p95 | Range | |---|---|---|---|---| -| 场景执行器墙钟 | 90次 | 69.002554秒 | 135.225086秒 | 36.669278–206.127213秒 | -| 单chat turn服务端耗时 | 145轮 | 31.942秒 | 98.746秒 | 7.440–201.106秒 | +| Scenario executor wall time | 90 attempts | 69.002554 seconds | 135.225086 seconds | 36.669278–206.127213 seconds | +| Per-chat server duration | 145 turns | 31.942 seconds | 98.746 seconds | 7.440–201.106 seconds | -场景墙钟包含重置、SQL采集和脚本操作员审批等步骤;单chat turn涵盖本轮主/分析模型与工具,不加入真实人工审批等待。两者都不是首token或首个有效结果时延;本批没有相应首结果时间记录。分位数使用排序后线性插值,保留通过和失败任务,仅描述本次本地执行,不作为容量或线上SLO。 +Scenario wall time includes reset, SQL collection, and scripted operator approvals. Per-chat duration includes main/analysis models and tools, excluding actual human approval waiting. Neither measures first-token or first-useful-result latency; this batch retained no corresponding first-result timestamps. Percentiles use linear interpolation after sorting, include passes and failures, and describe this local run without capacity or online-SLO claims. -统计明细见同目录`statistics.json`,原始逐调用、SSE、回执及SQL继续保留。本结果是同一冻结版本、固定30个业务场景各3次的业务验收与回归;不称90种场景、未见过的公开基准或线上泛化能力估计。后续版本与针对性回归独立保存,不覆盖本批、不拼接成功样本。 +See `statistics.json` in this directory for detailed statistics. Raw per-call records, SSE, receipts, and SQL remain available. This is business acceptance/regression of 30 fixed scenarios repeated 3 times on one frozen version, not 90 distinct scenarios, an unseen public benchmark, or an estimate of online generalization. Later versions and targeted regressions are saved independently without overwriting this batch or combining successful samples. diff --git a/evals/records/20260905T231113.095557Z/assessment.md b/evals/records/20260905T231113.095557Z/assessment.md index 69fd3e3..453792b 100644 --- a/evals/records/20260905T231113.095557Z/assessment.md +++ b/evals/records/20260905T231113.095557Z/assessment.md @@ -1,22 +1,22 @@ -# 正式验收后针对性回归:21/24 +# Targeted regression after formal acceptance: 21/24 -CityBuddy:`69be167a3df030bf45795c49f444d6e7c24d0423` -ShopMate:`02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` -结果目录:`evals/results/20260905T231113.095557Z`。 -数据截止:`2026-09-05T00:00:00Z`,UTC;fixture来源为上述CityBuddy提交。 -主模型与分析模型均为`gpt-5.6-terra`,经CLIPROXY Chat Completions适配调用。 +CityBuddy: `69be167a3df030bf45795c49f444d6e7c24d0423`
+ShopMate: `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28`
+Result directory: `evals/results/20260905T231113.095557Z`.
+Data cutoff: `2026-09-05T00:00:00Z`, UTC; fixture source is the CityBuddy commit above.
+Both main and analysis models used `gpt-5.6-terra` through the CLIPROXY Chat Completions adapter. -本批选择S02、S07、S12、S13、S14、S17、S20、S30共8个已知出现失败的业务场景,各执行3次,沿用原`evals/formal.json`题面和成功标准。每次从独立R0和新会话开始,场景内部保留多轮追问、真实操作员审批和读回。按实际用户可见正文/卡片、实际工具与分析SQL、参考SQL和权威业务终态判定,允许预算内自行纠错;`executed`与业务PASS分别统计。 +The batch selected 8 business scenarios with known failures: S02, S07, S12, S13, S14, S17, S20, and S30, each repeated 3 times under the original `evals/formal.json` statements and success criteria. Every attempt started from independent R0 and a new conversation, retaining follow-ups, real operator approval, and readback within the scenario. Judgment used actual user-visible prose/cards, actual tool/analysis SQL, reference SQL, and authoritative business terminal states, allowing self-correction within budget. `executed` and business PASS were counted separately. -与上一完整批ShopMate `9173037d6eb43d295f6ccb5876fa6284e882dfdb`相比,本版本调整提示/工具契约中的指标引用与集合委派说明,并将主循环工具轮次上限由8改为12;主/分析模型仍共享每chat turn最多16次调用、300秒截止时间。多个因素同时变化且模型执行有随机性,因此这是修复后选集回归,**不是单变量因果对照,也不是新的完整30×3验收**。 +Compared with the preceding full batch at ShopMate `9173037d6eb43d295f6ccb5876fa6284e882dfdb`, this version revised metric references and set-delegation guidance in prompt/tool contracts and raised the main-tool-round limit from 8 to 12. Main/analysis models still shared at most 16 calls and a 300-second deadline per chat turn. Multiple factors changed together and model execution is stochastic, so this is a post-fix subset regression, **not a single-variable causal comparison or a new full 30 × 3 acceptance run**. -本批UTC `2026-09-05T23:11:13.095965+00:00` 开始,`2026-09-05T23:50:13.064537+00:00` 结束,执行器退出0、run无stop_reason。24次execution均为executed、各自完成重置;执行正常不代表24次业务全通过。 +The batch started at UTC `2026-09-05T23:11:13.095965+00:00` and ended at `2026-09-05T23:50:13.064537+00:00`. The executor exited 0 and the run had no stop_reason. All 24 execution records were executed, with each reset completed. Normal execution does not mean all 24 passed the business task. -**24次全部完成独立业务判读:21 PASS、3 FAIL,选集完成率21/24(87.50%)。** S02/S07/S12/S13为11/12,S14/S17/S20/S30为10/12。失败均为正常executed但没有完整完成原题,全部保留在24次分母。 +**All 24 attempts received independent business assessment: 21 PASS, 3 FAIL, for subset completion of 21/24 (87.50%).** S02/S07/S12/S13 scored 11/12; S14/S17/S20/S30 scored 10/12. All failures were normally executed attempts that did not fully complete the original task, retained in the denominator of 24. -## 逐场景结果 +## Per-scenario results -| 场景 | 本批r1 | 本批r2 | 本批r3 | 本批完成数 | 原917同场景完成数 | +| Scenario | This r1 | This r2 | This r3 | This batch completed | Original 917 completed | |---|---|---|---|---|---| | S02 | PASS | PASS | PASS | 3/3 | 2/3 | | S07 | PASS | PASS | PASS | 3/3 | 1/3 | @@ -27,43 +27,44 @@ ShopMate:`02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` | S20 | PASS | PASS | PASS | 3/3 | 2/3 | | S30 | PASS | PASS | FAIL | 2/3 | 1/3 | -旧917完整90次中,这8个场景同选集实际为13/24,本次同选集为21/24;旧完整批仍是78/90。选择依据来自已知失败,不能将本选集结果当作未知任务泛化估计,也不能把本批通过替换旧失败来改写78/90。下文原件路径均相对本结果目录。 -## 三次业务失败 +In the original 917 full 90-attempt run, the same eight-scenario subset scored 13/24, compared with 21/24 here. The original full batch remains 78/90. Selection came from known failures, so this subset is not an estimate of unseen-task generalization. Its passes cannot replace old failures to rewrite 78/90. Raw paths below are relative to this result directory. -| 次数 | 业务原因 | 原件路径 | +## Three business failures + +| Attempt | Business reason | Raw paths | |---|---|---| -| S13-r2 | 两次run_analysis的question均超过300字符;第三次虽继续压缩,却被每turn两次analysis调用限制拒绝。没有实际分析SQL或金额/件数卡片,最终只请用户重新发起。主工具12轮、共享16次模型调用/300秒均未耗尽,未观察到provider或数据库不可用。 | `S13-r2/step-01.sse`、`S13-r2/step-01-terminal.json`、`S13-r2/saved-session.json`、`S13-r2/sql/after/S13.jsonl` | -| S17-r2 | 追问要求在上一轮有成交商品中再筛库存≥50,第二轮实际委派、SQL和可见结果却加入零成交Canvas tote(库存80),显示保留coffee/tea/tote三款;正确只有coffee和tea。来自真实SQL的数值不能弥补分析集合被扩大。 | `S17-r2/step-02.sse`、`S17-r2/step-02-terminal.json`、`S17-r2/saved-session.json`、`S17-r2/sql/after/S17.jsonl` | -| S30-r3 | 两张独立草案和仅咖啡批准已完成;用户随后明确要求取消茶,模型仅读商品与待办,没有调用discard_change,最终茶仍PREPARED、result/resolvedAt为空,并请求用户再次操作。诚实说明尚未取消不等于完成取消。 | `S30-r3/step-03.sse`、`S30-r3/step-02-operator.json`、`S30-r3/saved-session.json`、`S30-r3/sql/after/drafts.jsonl`、`S30-r3/sql/after/events.jsonl` | +| S13-r2 | The question in each of two run_analysis calls exceeded 300 characters. Although the third was shortened further, the two-analysis-calls-per-turn limit rejected it. No actual analysis SQL or amount/quantity card was produced; the final answer only asked the user to start again. Neither 12 main-tool rounds nor the shared 16-call / 300-second budget was exhausted; no provider or database unavailability was observed. | `S13-r2/step-01.sse`, `S13-r2/step-01-terminal.json`, `S13-r2/saved-session.json`, `S13-r2/sql/after/S13.jsonl` | +| S17-r2 | The follow-up requested stock ≥ 50 within the preceding turn's products with sales. The second turn's actual delegation, SQL, and visible result added the zero-sales Canvas tote (stock 80), retaining coffee/tea/tote instead of only coffee and tea. Values from real SQL do not compensate for an expanded analysis set. | `S17-r2/step-02.sse`, `S17-r2/step-02-terminal.json`, `S17-r2/saved-session.json`, `S17-r2/sql/after/S17.jsonl` | +| S30-r3 | Two independent drafts and approval of coffee only were completed. After the user explicitly requested tea cancellation, the model read products and pending work without calling discard_change. Tea remained PREPARED, with result/resolvedAt empty, and the model asked the user to act again. Honestly stating that cancellation remained undone did not complete it. | `S30-r3/step-03.sse`, `S30-r3/step-02-operator.json`, `S30-r3/saved-session.json`, `S30-r3/sql/after/drafts.jsonl`, `S30-r3/sql/after/events.jsonl` | -三个失败分别是参数纠错未完成、追问集合扩大和明确取消未执行;不能以没有错误写入替代业务完成。 +The failures were incomplete recovery from invalid parameters, expansion of the follow-up set, and an unexecuted explicit cancellation. No incorrect writes is not a substitute for business completion. -## 业务终态与过程观察 +## Business terminal states and process observations -已判只读任务中,每次商品、付款历史与scope相对自己的R0保持不变,草案与商品事件均0。S20三次都正确选出历史成交金额绝对增额最大的可调价CNY商品coffee(+994元),按当前24元加5%创建25.20元待审草案;审批前价格未变,真实操作员仅批准对应草案,批准后coffee2520分/v4、一条对应商品事件,随后读回确认,其他商品和历史付款不变。 +For every assessed read-only task, products, payment history, and scope remained unchanged from its own R0, with zero drafts and product events. All three S20 attempts correctly selected coffee as the adjustable CNY product with the greatest absolute historical revenue increase (+994 CNY), creating a pending draft at CNY 25.20, 5% above the current CNY 24. Prices stayed unchanged before approval. The real operator approved only that draft; afterward coffee was 2520 minor units / v4 with one matching product event and readback confirmation. Other products and historical payments were unchanged. -S30三次都创建coffee2400→2480和tea1800→1850两张独立草案,审批前价格不变,操作员仅批准coffee;批准后coffee2480分/v4、一个对应商品事件,茶仍1800分/v3。r1/r2实际调用discard_change取消茶,终态CANCELLED且result/resolvedAt存在,原items保留;r3没有取消,茶仍PREPARED,故FAIL。六次审批任务(S20/S30各三次)每次均只有一条获批咖啡商品事件,其他商品和138行历史付款保持,没有未批准改茶或错误取消咖啡。 +All three S30 attempts created independent coffee 2400 → 2480 and tea 1800 → 1850 drafts. Prices stayed unchanged before approval, and the operator approved coffee only. Afterward coffee was 2480 minor units / v4 with one matching product event; tea stayed 1800 minor units / v3. r1/r2 actually called discard_change for tea, leaving CANCELLED with result/resolvedAt present and original items retained. r3 did not cancel, leaving tea PREPARED and causing FAIL. Across all six approval tasks (three each for S20/S30), each had exactly one approved coffee product event. Other products and all 138 historical payment rows were preserved, without unapproved tea changes or incorrect coffee cancellation. -- S07三次用户可见CNY1956→2304、USD252→280及其差额/变化率均无币种和值冲突;不重复把裸sales当作两个币种的独立结果。 -- S12-r2首卡只分析六款后,又实际补查Limited coffee set,并展示第二张零金额/零件数/零贡献卡;最终正文明确两卡覆盖原七款。通过依据是实际恢复后的完整交付,不是评估者替遗漏商品补零。 -- S14-r2中间一条日历SQL曾得到错误5日与错误商品金额,后续重聚合纠正,最终可见四日窗口与金额/杯成交日期正确;沿既定规则允许原预算内恢复。 -- S12-r1出现真实MySQL1051后改写查询成功,另多次brief长度或SQL输入验证错误后完成原题。通过不等于没有工具错误,S13-r2也说明这种参数失败仍可能阻止完成任务。 -- S02-r2仍用变化率Pill展示份额,但相邻note/findings明确是成交额占比,保留呈现质量观察。若干任务冗余查询USD或附加成交日期解释,没有因此扩充原题或增加成功次数。 +- All three S07 attempts showed CNY 1956 → 2304 and USD 252 → 280, with consistent currencies, amounts, differences, and rates. They did not reuse bare sales as two independent currency results. +- S12-r2's first card analyzed only six products, then actually queried Limited coffee set and presented a second zero-amount/quantity/contribution card. Final prose explicitly stated that both cards covered the original seven products. PASS followed complete delivery after actual recovery, not evaluator-supplied zeros for an omitted product. +- An intermediate calendar SQL query in S14-r2 returned an incorrect five-day count and product amounts. Later reaggregation corrected them; the final visible four-day window, amounts, and mug sale dates were correct. Recovery within the original budget was allowed under the established rules. +- S12-r1 recovered by rewriting a query after an actual MySQL 1051 error. Other attempts completed after brief-length or SQL-input validation errors. PASS does not mean no tool errors; S13-r2 shows that parameter failures can still block completion. +- S02-r2 still used a change-rate Pill for share, but adjacent note/findings explicitly identified revenue share. This presentation-quality observation remains. Several tasks redundantly queried USD or added sales-date explanations; this did not expand the original tasks or add successes. -这些结果不代替独立权限、审批幂等与并发事务测试;没有错写也不能代替完成要求的分析或取消动作。 +These results do not replace independent authorization, approval-idempotency, or concurrent-transaction tests. No incorrect writes does not replace the required analysis or cancellation. -## 实际调用与耗时 +## Actual calls and elapsed time -33个chat turn共记录259次模型调用:主循环172次、分析子循环87次。259次均有完成的调用观察、HTTP200及明确输入/输出/缓存读取字段;观察数与各turn报告调用数一致,没有未观察调用。逐调用汇总与角色/场景汇总相符:输入 **1,630,176 token**,输出 **79,221 token**。总输入已经包含缓存读取,不另叠加运行时总计或缓存值。 +Across 33 chat turns, 259 model calls were recorded: 172 main-loop and 87 analysis-loop calls. All 259 had completed call observations, HTTP 200, and explicit input/output/cache-read fields. Observation counts matched each turn's reported call count, with no unobserved calls. Per-call totals matched role/scenario aggregates: input **1,630,176 tokens**, output **79,221 tokens**. Input already includes cache reads; runtime totals and cached tokens are not added again. -明确报告的缓存读取为 **1,227,264 token**,占输入 **75.28%**(1,227,264/1,630,176)。这是报告的输入token中来自缓存读取的比例,**不是请求缓存命中率、费用收益或时延改善**。缓存创建未测量;没有代理账单或适用费率,不换算实际成本。 +Explicitly reported cache reads were **1,227,264 tokens**, or **75.28%** of input (1,227,264 / 1,630,176). This is the share of reported input tokens read from cache, **not a request cache hit rate, cost benefit, or latency improvement**. Cache creation was not measured; without proxy invoices or applicable rates, no actual cost is calculated. -| 统计对象 | 样本数 | 中位数 | p95 | 范围 | +| Measurement | Samples | Median | p95 | Range | |---|---|---|---|---| -| 场景执行器墙钟 | 24次 | 86.329003秒 | 150.640429秒 | 53.260682–168.234622秒 | -| 单chat turn服务端耗时 | 33轮 | 63.774秒 | 138.4748秒 | 12.278–146.047秒 | +| Scenario executor wall time | 24 attempts | 86.329003 seconds | 150.640429 seconds | 53.260682–168.234622 seconds | +| Per-chat server duration | 33 turns | 63.774 seconds | 138.4748 seconds | 12.278–146.047 seconds | -场景墙钟包含重置、SQL采集和脚本操作员审批等步骤;单chat turn涵盖主/分析模型和工具,没有真实人工审批等待。两者都不是首token或首个有效结果时间。分位数为排序后线性插值,包含本批全部通过/失败,仅描述这个本地选集,不作为容量或线上SLO,也不与不同任务分布的完整90次耗时作性能提升比较。 +Scenario wall time includes reset, SQL collection, and scripted operator approvals. Per-chat duration covers main/analysis models and tools, excluding actual human approval waiting. Neither is first-token or first-useful-result time. Percentiles use linear interpolation after sorting and include every pass/failure in this batch. They describe this local subset, not capacity or an online SLO, and are not compared with the different task distribution of the full 90-attempt run as a performance improvement. -统计明细见[statistics.json](statistics.json),由本批原始逐调用记录汇总。原始SSE、分析SQL、操作员请求/回执和各阶段业务SQL保留在本目录。业务判读、执行状态、调用用量和耗时分别记录,不相互替代。 +See [statistics.json](statistics.json) for details aggregated from this batch's raw per-call records. Raw SSE, analysis SQL, operator requests/receipts, and business SQL at each stage remain in this directory. Business judgment, execution state, call usage, and elapsed time are recorded separately and do not substitute for one another. diff --git a/evals/records/README.md b/evals/records/README.md index d09cd70..39e8ffe 100644 --- a/evals/records/README.md +++ b/evals/records/README.md @@ -1,32 +1,33 @@ -# 业务评测记录 +# Business evaluation records -当前完整零售版本先看[最终30次业务验收](retail-v2-20260907/README.md):18个已知场景,**24次通过、3次业务失败、3次提供者故障**。包含等待时间、模型调用量、搜索来源与原始批次映射;故障后只续未运行任务,没有替换失败。 +For the full retail version, start with the [final 30-attempt business acceptance run](retail-v2-20260907/README.md): 18 known scenarios, **24 passes, 3 business failures, and 3 provider failures**. It includes waiting times, model usage, search sources, and the raw batch mapping. After provider failures, only unrun tasks resumed; failed attempts were not replaced. -[前一轮54次验收](retail-v1-20260907/README.md)为 **46/54**;后续12次定向回归及页面、记忆、并发、恢复验证按各自版本保留。共同使用87个目录根、104个交易SKU、90日Shanghai夹具,不混合不同批次的分母。 +The [preceding 54-attempt run](retail-v1-20260907/README.md) scored **46/54**. Its subsequent 12 targeted regressions and page, memory, concurrency, and recovery checks retain their own source versions. Both retail rounds use the fixture with 87 catalog roots, 104 tradable SKUs, and 90 Shanghai days; denominators from different batches remain separate. -## 历史商家版本 +## Historical merchant versions -先看每批的业务判读,再按需下载原始包。以下历史版本最近完整30×3为 **78/90**;后续修复版本的8场景×3定向回归为 **21/24**。它们对应不同版本、不同任务范围,不能相互替换。 +Read each batch's business assessment first, then download its raw archive as needed. The latest full historical 30 × 3 run scored **78/90**; a later fixed version scored **21/24** on 8 targeted scenarios × 3. They use different versions and task scopes and cannot replace one another. -下列历史批次使用CityBuddy `69be167a3df030bf45795c49f444d6e7c24d0423` 和固定 `2026-09-05T00:00:00Z` UTC夹具。主模型均为 `gpt-5.6-terra`;仅最早开发smoke的分析子模型为 `gpt-5.6-luna`,其余为Terra。具体双SHA、协议、预算、实际任务名单见各批 `run.json`。 +The historical batches below use CityBuddy `69be167a3df030bf45795c49f444d6e7c24d0423` and the fixed UTC fixture cutoff `2026-09-05T00:00:00Z`. All use `gpt-5.6-terra` as the main model. Only the earliest development smoke used `gpt-5.6-luna` for analysis; the rest used Terra. Each `run.json` records both full SHAs, protocol, budget, and actual task list. -下表“执行”列是executed/failed,和人工业务PASS不同;各批均无未执行项。最新完整批的88 executed与2执行失败合计90次,业务分母仍为90。 +The Execution column means executed / failed, distinct from manually judged business PASS. None of these batches has unrun attempts. The latest full batch contains 88 executed attempts and 2 execution failures, with a business denominator of 90. -| 批次/范围 | ShopMate完整SHA | 执行 | 业务PASS | 记录 | +| Batch / scope | Full ShopMate SHA | Execution | Business PASS | Records | | --- | --- | --- | --- | --- | -| 最近完整30场景×3 | `9173037d6eb43d295f6ccb5876fa6284e882dfdb` | 88/2 | **78/90** | [判读](20260905T210851.446894Z/assessment.md) · [运行](20260905T210851.446894Z/run.json) · [统计](20260905T210851.446894Z/statistics.json) · [原件包](20260905T210851.446894Z/raw.tar.gz) | -| 修复后8场景×3 | `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` | 24/0 | **21/24** | [判读](20260905T231113.095557Z/assessment.md) · [运行](20260905T231113.095557Z/run.json) · [统计](20260905T231113.095557Z/statistics.json) · [原件包](20260905T231113.095557Z/raw.tar.gz) | -| D01/D04/D09开发smoke | `3bea5a147319a434e06ad1dbe9e1e8b3239c08af` | 2/1 | 0/3 | [判读](20260905T180849.909066Z/assessment.md) · [运行](20260905T180849.909066Z/run.json) · [原件包](20260905T180849.909066Z/raw.tar.gz) | -| 完整12开发任务 | `115b6a360ae502665f88063429e67230ab5ec37d` | 12/0 | 8/12 | [判读](20260905T182558.103268Z/assessment.md) · [运行](20260905T182558.103268Z/run.json) · [原件包](20260905T182558.103268Z/raw.tar.gz) | -| D01/D06/D07/D10开发回归 | `0db5538c1d46bf440542c6159323e2bcd2ec755a` | 4/0 | 4/4 | [判读](20260905T184922.043835Z/assessment.md) · [运行](20260905T184922.043835Z/run.json) · [原件包](20260905T184922.043835Z/raw.tar.gz) | -| 首个完整30场景×3 | `0db5538c1d46bf440542c6159323e2bcd2ec755a` | 90/0 | 84/90 | [判读](20260905T185513.918718Z/assessment.md) · [运行](20260905T185513.918718Z/run.json) · [统计](20260905T185513.918718Z/statistics.json) · [原件包](20260905T185513.918718Z/raw.tar.gz) | -| S03/S08/S11调试回归 | `91347b986fcfc65d08c4012dcaa14e47dd7f0ada` | 3/0 | 1/3 | [判读](20260905T204513.490639Z/assessment.md) · [运行](20260905T204513.490639Z/run.json) · [原件包](20260905T204513.490639Z/raw.tar.gz) | -| S08/S11调试回归 | `9173037d6eb43d295f6ccb5876fa6284e882dfdb` | 2/0 | 2/2 | [判读](20260905T210133.531635Z/assessment.md) · [运行](20260905T210133.531635Z/run.json) · [原件包](20260905T210133.531635Z/raw.tar.gz) | +| Latest full 30 scenarios × 3 | `9173037d6eb43d295f6ccb5876fa6284e882dfdb` | 88 / 2 | **78/90** | [Assessment](20260905T210851.446894Z/assessment.md) · [Run](20260905T210851.446894Z/run.json) · [Statistics](20260905T210851.446894Z/statistics.json) · [Raw archive](20260905T210851.446894Z/raw.tar.gz) | +| Post-fix 8 scenarios × 3 | `02d1bf0d0d1e4f5d71925f7db92ed3c4d9726b28` | 24 / 0 | **21/24** | [Assessment](20260905T231113.095557Z/assessment.md) · [Run](20260905T231113.095557Z/run.json) · [Statistics](20260905T231113.095557Z/statistics.json) · [Raw archive](20260905T231113.095557Z/raw.tar.gz) | +| D01/D04/D09 development smoke | `3bea5a147319a434e06ad1dbe9e1e8b3239c08af` | 2 / 1 | 0/3 | [Assessment](20260905T180849.909066Z/assessment.md) · [Run](20260905T180849.909066Z/run.json) · [Raw archive](20260905T180849.909066Z/raw.tar.gz) | +| Full 12 development tasks | `115b6a360ae502665f88063429e67230ab5ec37d` | 12 / 0 | 8/12 | [Assessment](20260905T182558.103268Z/assessment.md) · [Run](20260905T182558.103268Z/run.json) · [Raw archive](20260905T182558.103268Z/raw.tar.gz) | +| D01/D06/D07/D10 development regression | `0db5538c1d46bf440542c6159323e2bcd2ec755a` | 4 / 0 | 4/4 | [Assessment](20260905T184922.043835Z/assessment.md) · [Run](20260905T184922.043835Z/run.json) · [Raw archive](20260905T184922.043835Z/raw.tar.gz) | +| First full 30 scenarios × 3 | `0db5538c1d46bf440542c6159323e2bcd2ec755a` | 90 / 0 | 84/90 | [Assessment](20260905T185513.918718Z/assessment.md) · [Run](20260905T185513.918718Z/run.json) · [Statistics](20260905T185513.918718Z/statistics.json) · [Raw archive](20260905T185513.918718Z/raw.tar.gz) | +| S03/S08/S11 diagnostic regression | `91347b986fcfc65d08c4012dcaa14e47dd7f0ada` | 3 / 0 | 1/3 | [Assessment](20260905T204513.490639Z/assessment.md) · [Run](20260905T204513.490639Z/run.json) · [Raw archive](20260905T204513.490639Z/raw.tar.gz) | +| S08/S11 diagnostic regression | `9173037d6eb43d295f6ccb5876fa6284e882dfdb` | 2 / 0 | 2/2 | [Assessment](20260905T210133.531635Z/assessment.md) · [Run](20260905T210133.531635Z/run.json) · [Raw archive](20260905T210133.531635Z/raw.tar.gz) | -最新回归选定S02/S07/S12/S13/S14/S17/S20/S30,各3次。原917完整批的同选集为13/24,本次21/24;集合与指标契约、主工具轮数同时调整,且模型具有随机性,不作单项因果结论。02d主工具轮为12,主/分析共享16次模型调用及300秒预算保持;这不是新完整90次。剩余失败为S13-r2参数拒绝后未完成分析、S17-r2扩大追问商品集合、S30-r3漏掉明确要求的取消。 -原84/90六次失败、其后1/3调试回归及最近78/90的12次失败均保留。开发4/4不改写开发8/12;后续成功也不替换旧正式批的失败。业务判读基于实际可见内容、SQL与真实审批终态;单纯完成执行或没有错误写入不能算完成业务任务。 +The latest regression selected S02/S07/S12/S13/S14/S17/S20/S30, three times each. The same subset in the original 917 full batch scored 13/24; this run scored 21/24. Set and metric contracts and main-tool rounds changed together, and model output is stochastic, so this does not isolate a causal effect. Version 02d allowed 12 main-tool rounds while retaining the shared main/analysis budget of 16 model calls and 300 seconds. This was not a new full 90-attempt run. Remaining failures were S13-r2, which did not finish analysis after parameter rejection; S17-r2, which expanded the product set in a follow-up; and S30-r3, which omitted an explicitly requested cancellation. -每个 `raw.tar.gz` 解包后根目录为对应run-id,保留逐次execution、SSE、saved-session、分析及参考SQL、操作员请求和回执。`assessment.md`中的任务相对路径对应包内原目录;仓库只展开少量可读文件,本地原始结果另保留。任务和参考SQL可在表中被测SHA的 `evals/` 下查看,归档提交的SHA不是被测SHA。 +The six failures in the original 84/90, the subsequent 1/3 diagnostic regression, and all 12 failures in the latest 78/90 remain recorded. Development 4/4 does not rewrite development 8/12, and later successes do not replace old formal failures. Business judgment uses actual visible content, SQL, and real approval terminal states; execution completion or the absence of incorrect writes alone does not complete a business task. -统计文件只汇总已有调用观察和耗时。缓存读取占比不是请求命中率或费用收益;缓存创建量未测量。耗时区分chat服务端执行与含重置/SQL的场景墙钟,不称首token时间或生产容量。权限、并发事务和浏览器演示的记录不加入这些业务分母。 +Each `raw.tar.gz` expands to its run-ID root, retaining per-attempt execution, SSE, saved sessions, analysis/reference SQL, and operator requests/receipts. Task-relative paths in `assessment.md` point inside that archive. Only a small readable subset is expanded in the repository; local raw results are also retained. Tasks and reference SQL are available under `evals/` at the measured SHA listed in the table. The archive commit SHA is not the measured source SHA. + +Statistics files summarize observed calls and elapsed times only. The cache-read share is neither a request hit rate nor a cost saving; cache creation was not measured. Timing separates chat server execution from scenario wall time including reset/SQL, and is not first-token latency or production capacity. Authorization, concurrent-transaction, and browser-demo records are outside these business denominators. diff --git a/evals/records/retail-v1-20260907/README.md b/evals/records/retail-v1-20260907/README.md index 0155091..43bd9ab 100644 --- a/evals/records/retail-v1-20260907/README.md +++ b/evals/records/retail-v1-20260907/README.md @@ -1,166 +1,166 @@ -# 完整零售业务验收:2026-09-07 +# Full retail business acceptance: 2026-09-07 -完整零售版本54次业务验收已全部完成,**46/54通过(85.2%)**;8次普通模型业务失败保留。后续版本12次定向复测为**6/12**,最终买家归属校准另列,两者均不覆盖原54次。 +All 54 attempts for the full retail version completed: **46/54 passed (85.2%)**, with 8 ordinary model business failures retained. A later version scored **6/12** on targeted reruns. Final buyer-ownership calibration is reported separately; neither replaces the original 54 attempts. -## 范围、版本与判定 +## Scope, versions, and judgment -本次复用18个已知业务场景,每个独立运行3次,共54次。这是同题重复验收,不能称为未见场景的泛化成绩,也不代表完整零售业务空间。每次从retail-R0重置,四组使用同一干净源码和预算,开发重跑没有替换正式失败。 +The run reused 18 known business scenarios, each independently repeated 3 times, for 54 attempts. This is repeated acceptance of known tasks, not unseen-scenario generalization or coverage of the entire retail business space. Each attempt started from retail-R0. All four groups used the same clean source and budget; development reruns did not replace formal failures. -- ShopMate:`6056c26aa29bc60710ce14cdecf927219491a15d`。 -- CityBuddy:`99a7de52c542cbf57b8d3c71e16ded529e198ea6`。 -- 数据:`shopmate-retail-v1`,87目录根/104交易SKU,90个完整上海自然日,合成CNY金额;报告截止`2026-09-05T00:00:00+08:00`。促销使用每题的真实操作时间,不用历史报告截止代替今天。 -- 主/分析模型:`gpt-5.6-terra`,CLIPROXY Chat Completions经Messages适配;该别名不是固定不变的上游模型快照。网页搜索实际走Responses,Python在独立Docker容器执行。 -- 每聊天回合共享16次模型调用/300秒,主循环最多12工具轮;搜索和Python各最多3次尝试。一个任务可能有多轮聊天和用户按钮,任务数、聊天数与模型调用数分别计数。 -- 本机Apple M4、10物理核、24GiB内存;Docker VM 8 CPU/14GiB,Commerce限4 CPU。Shop API在宿主运行,Java/MySQL/Redis/RocketMQ在独立Shop拓扑。正式期间未与City容量压测并行,前端与本次浏览器已关闭。 +- ShopMate: `6056c26aa29bc60710ce14cdecf927219491a15d`. +- CityBuddy: `99a7de52c542cbf57b8d3c71e16ded529e198ea6`. +- Data: `shopmate-retail-v1`, 87 catalog roots / 104 tradable SKUs, 90 complete Shanghai calendar days, synthetic CNY amounts; report cutoff `2026-09-05T00:00:00+08:00`. Promotions used each task's actual operation time, not the historical report cutoff as today's date. +- Main/analysis model: `gpt-5.6-terra`, CLIPROXY Chat Completions through the Messages adapter. The alias is not an immutable upstream model snapshot. Web search used Responses; Python ran in a separate Docker container. +- Each chat turn shared 16 model calls / 300 seconds, with at most 12 main-tool rounds and 3 attempts each for search and Python. One task may contain multiple chats and user-button actions; task, chat, and model-call counts are separate. +- Local Apple M4, 10 physical cores, 24 GiB RAM; Docker VM 8 CPU / 14 GiB, Commerce limited to 4 CPU. Shop API ran on the host; Java/MySQL/Redis/RocketMQ ran in the isolated Shop topology. The formal run did not overlap City capacity tests; the frontend and the browser used for this run were closed. -按[预先登记的任务和判定口径](../../retail/acceptance-v1.md),逐题核对真实模型输出、原始HTTP/SSE、保存卡片及权威SQL。`executed`和`turn_complete`只表示执行状态。金额、库存、身份、版本、审批与原回执须符合真实业务;最终回答给错事实或统计语义仍记失败,不能用数据没有写错抵销。 +Each task was judged under the [preregistered tasks and definitions](../../retail/acceptance-v1.md), using actual model output, raw HTTP/SSE, saved cards, and authoritative SQL. `executed` and `turn_complete` describe execution state only. Amounts, stock, identities, versions, approvals, and original receipts must match business truth. Incorrect final facts or statistical meaning still fail; correct writes cannot offset them. -## 原54次结果 +## Original 54-attempt results -| 组别 | 已知场景×重复 | 业务通过 | 普通业务失败 | 原件批次(`evals/results/`) | +| Group | Known scenarios × repetitions | Business passes | Ordinary business failures | Raw batch (`evals/results/`) | |---|---:|---:|---:|---| -| D 经营分析与调价 | 7×3 | 18/21 | 3 | `20260907T055506.124240Z` | -| R 零售交易与商家操作 | 7×3 | 17/21 | 4 | `20260907T063754.317220Z` | -| B 买家方案与购物车 | 2×3 | 6/6 | 0 | `20260907T072011.377762Z` | -| S 外部来源研究 | 2×3 | 5/6 | 1 | `20260907T072813.564683Z` | -| 合计 | 18×3 | **46/54** | **8** | 同一Shop/City版本 | +| D Business analysis and price changes | 7×3 | 18/21 | 3 | `20260907T055506.124240Z` | +| R Retail transactions and merchant operations | 7×3 | 17/21 | 4 | `20260907T063754.317220Z` | +| B Buyer plans and carts | 2×3 | 6/6 | 0 | `20260907T072011.377762Z` | +| S External-source research | 2×3 | 5/6 | 1 | `20260907T072813.564683Z` | +| Total | 18×3 | **46/54** | **8** | Same Shop/City versions | -54次均实际执行,没有任务被归类为provider failure、unknown或not_run;四组均正常结束、无保留的未知写状态。54次共105轮聊天,均收到`turn_complete`,这些执行事实不改变46/54的业务判定。 +All 54 attempts executed, with none classified as provider failure, unknown, or not_run. All four groups ended normally with no retained unknown write outcomes. The 54 attempts contained 105 chat turns, all receiving `turn_complete`; these execution facts do not change the business score of 46/54. -| 场景 | r1 | r2 | r3 | +| Scenario | r1 | r2 | r3 | |---|---|---|---| -| D02 两期金额和订单的绝对/百分比变化 | PASS | PASS | PASS | -| D04 完整目录的历史均价与现价 | PASS | FAIL | PASS | -| D05 继承期间的商品拆分追问 | PASS | FAIL | FAIL | -| D07 澄清日期后限定单SKU分析 | PASS | PASS | PASS | -| D08 零成交、零基期与广告因果边界 | PASS | PASS | PASS | -| D10 三商品整批调价 | PASS | PASS | PASS | -| D11 取消旧意图后重新提交并批准 | PASS | PASS | PASS | -| R01 比较、加车、用户结账付款与退款确认 | PASS | PASS | PASS | -| R02 两买家订单、履约、资料与发布政策 | PASS | PASS | FAIL | -| R03 八SKU家族内容更新 | PASS | PASS | PASS | -| R04 先补货保留暂停,再单独恢复销售 | PASS | PASS | PASS | -| R05 本地营销计划更新与真实观察事实 | FAIL | PASS | PASS | -| R06 促销批准、买家付款与商家近期订单 | PASS | PASS | PASS | -| R07 经营事实口径与SQL→Python分析 | PASS | FAIL | FAIL | -| B01 100元阅读角方案 | PASS | PASS | PASS | -| B02 真实ADD、SET与REMOVE购物车 | PASS | PASS | PASS | -| S01 买家官方洗护来源与本站事实 | PASS | PASS | PASS | -| S02 商家官方促销指南与本站能力边界 | FAIL | PASS | PASS | - -### 八次失败及仍然正确的部分 - -| 失败试次 | 最终业务错误 | 正确部分与边界 | +| D02 Absolute/percentage changes in amounts and orders across two periods | PASS | PASS | PASS | +| D04 Historical average and current prices across the complete catalog | PASS | FAIL | PASS | +| D05 Product-breakdown follow-up retaining the period | PASS | FAIL | FAIL | +| D07 Single-SKU analysis after date clarification | PASS | PASS | PASS | +| D08 Zero sales, zero baseline, and limits on advertising causality | PASS | PASS | PASS | +| D10 Three-product batch price change | PASS | PASS | PASS | +| D11 Cancel the old intent, resubmit, and approve | PASS | PASS | PASS | +| R01 Compare, add to cart, user checkout/payment, and refund confirmation | PASS | PASS | PASS | +| R02 Two buyers' orders, fulfillment, profiles, and published policies | PASS | PASS | FAIL | +| R03 Content update for an eight-SKU family | PASS | PASS | PASS | +| R04 Restock while paused, then resume sales separately | PASS | PASS | PASS | +| R05 Local campaign update and actual observation facts | FAIL | PASS | PASS | +| R06 Promotion approval, buyer payment, and merchant recent orders | PASS | PASS | PASS | +| R07 Business fact definitions and SQL → Python analysis | PASS | FAIL | FAIL | +| B01 CNY 100 reading-corner plan | PASS | PASS | PASS | +| B02 Actual ADD, SET, and REMOVE cart operations | PASS | PASS | PASS | +| S01 Official care sources for buyers and store facts | PASS | PASS | PASS | +| S02 Official promotion guidance for merchants and store capability boundaries | FAIL | PASS | PASS | + +### Eight failures and the parts that remained correct + +| Failed attempt | Final business error | Correct parts and limits | |---|---|---| -| D04-r2 | 用各SKU成交日数的MAX表示家族成交日数;该聚合不等于家族日期去重。 | 主要金额、件数、历史均价与现价正确;错误附加字段仍使整题失败。 | -| D05-r2 | SQL包含上海9月4日全天,最终描述却将9月4日标成不含右端。 | 查询窗口和金额正确,错误发生在最终期间表达。 | -| D05-r3 | 合计行的“活跃天数”填成96个成交SKU,实际全店14天。 | 商品拆分金额正确,合计列的统计对象与单位错误。 | -| R02-r3 | 一笔真实订单ID少抄一个字符;明确No order后未纠正,错误称真实履约记录不可读。 | 其他本人订单、第二买家资料与政策正确;没有业务写入,不能归因于后端数据不可用。 | -| R05-r1 | 将实际上海7月16日00:00的排他终点说成7月15日不含,漏掉一天。 | 原工具时间含正确offset;计划更新、预算与未知归因收入均正确,Java未写错日期。 | -| R07-r2 | 把资源整数`publication_version`与来源标签`retail-v1`混同,筛出空目录,再补成28日零值并交给Python。 | Python实际执行成功,但输入集合错误;最终0总额/0标准差与权威SQL的80471.65元/1142.2018308657343元冲突。第一轮还出现活动UTC时刻误述。 | -| R07-r3 | 活动观察窗的上海结束时刻写成7月15日00:00,实际为7月16日00:00。 | 全店统计、28行实际SQL→Python、总额、总体标准差和峰值全部正确;错误日期仍使整题失败。 | -| S02-r1 | 正确引用外部促销规则后,又建议将本站已直接改低的商品价格映射为该外部促销,建议与引用的适用条件冲突。 | 本站状态及外部接入尚未建立的边界正确,没有真的发布外部广告或修改业务数据。 | +| D04-r2 | Used MAX of individual SKUs' sales-day counts as the family's sales-day count; this is not a count of distinct dates across the family. | Main amounts, quantities, historical averages, and current prices were correct. The incorrect extra field still failed the task. | +| D05-r2 | SQL included the entire Shanghai day of September 4, but the final description marked September 4 as an excluded upper bound. | Query window and amounts were correct; the error was in the final period description. | +| D05-r3 | The total row's “活跃天数” field (verbatim model label: active days) contained 96 sold SKUs, although the store period was 14 days. | Product-breakdown amounts were correct; the total column used the wrong statistical object and unit. | +| R02-r3 | Omitted one character when copying a real order ID, did not correct it after an explicit No order response, and incorrectly claimed actual fulfillment records were unreadable. | Other owned orders, the second buyer's profile, and policies were correct. No business writes occurred; this was not backend data unavailability. | +| R05-r1 | Described the actual exclusive end of July 16 at 00:00 Shanghai time as excluding July 15, dropping a day. | Tool timestamps had the correct offset. Campaign update, budget, and unknown attributed revenue were correct; Java did not write an incorrect date. | +| R07-r2 | Confused integer resource `publication_version` with source label `retail-v1`, selected an empty catalog, then filled 28 days with zeros and passed them to Python. | Python ran successfully on the wrong input set. Final total 0 / standard deviation 0 conflicted with authoritative SQL: CNY 80471.65 / 1142.2018308657343. The first turn also misstated a campaign's UTC time. | +| R07-r3 | Gave the campaign observation window's Shanghai end as July 15 at 00:00 instead of July 16 at 00:00. | Store statistics, the actual 28-row SQL → Python sequence, total, population standard deviation, and peak were correct. The date error still failed the task. | +| S02-r1 | After correctly citing external promotion rules, recommended mapping the store's already reduced product price to that external promotion, contradicting the cited eligibility conditions. | Store state and the absence of an external integration were correctly described. No external advertising was published and no business data changed. | -按主要失败归类:3次期间/时区表达错误,3次聚合对象、单位或数据范围错误,1次订单标识抄错后未恢复,1次外部规则与建议矛盾。R07-r2归入数据范围一类,同时保留其附带日期错误;分类不增加失败分母。 +By primary cause, there were 3 period/timezone expression failures, 3 aggregation-object/unit/data-scope failures, 1 unrecovered order-ID copying failure, and 1 contradiction between external rules and advice. R07-r2 belongs to the data-scope category while retaining its additional date error; classification does not increase the failure denominator. -SQL或工具错误后在同一任务预算内实际修正可以通过。例如D04-r3拒绝截断表后重查104SKU完整结果,R02-r1纠正了把SKU当订单ID的调用,R07-r1在实际Python前修正查询。不能把这些试次说成从未出现工具错误,也不把已恢复的中间错误重复计为任务失败。 +Actual recovery from a SQL or tool error within the same task budget can pass. For example, D04-r3 rejected a truncated table and requeried the full 104-SKU result; R02-r1 corrected a call that used a SKU as an order ID; R07-r1 corrected its query before actual Python execution. These attempts were not free of tool errors, and recovered intermediate errors are not counted again as task failures. -## 已完成的业务闭环 +## Completed business workflows -- D组15次只读任务,各次原始业务SQL前后不变。6次调价任务只在操作员批准后改变目标价格、版本、catalog generation及对应商品发布事件;库存、历史成交与其他商品不变。D11旧草案CANCELLED、新草案单次APPLIED。这是正常审批验收,不冒称包含版本冲突或故障注入。 -- R01三次均完成两个真实SKU子单,共6600分CNY,两个订单PAID、各一笔SUCCEEDED付款;结账时目标库存各减1,付款未重复扣库存。100分退款只有用户确认后形成唯一REQUESTED申请,第二次确认回放同一回执。REQUESTED不是退款到账,最终说明与此一致。 -- R03三次各只更新同一家族八个SKU的材质内容,保留其他字段,并推进对应版本与事件。R04三次均经历64件暂停→69件仍暂停→69件恢复销售,两步各自批准,没有重复补货。R05三次的真实审批写入都正确;预算、观察花费、未知收入、来源与观察窗口保持不变,仅r1的可见日期解释失败。三题共12次实际批准均符合原意图;业务任务仍按8/9计。 -- R06三次均在批准后将目标商品2700→2430分,买家实际按2430分完成结账与付款,商家最终从近期订单读到同一笔新PAID订单。查询没有被历史报表截止滤掉;促销结束需另行确认恢复价格,没有自动回价承诺。 -- B01三次给出真实可售的照明、收纳、地毯组合,商品合计100元,规格和用途有据,计划未触发加车。判分不锁唯一SKU组合。B02三次均由四个真实ADD/ADD/SET/REMOVE命令推进购物车版本1–4,最终只有台灯1件、2200分,未创建订单、改变库存或触及其他买家。这三次没有实际并发冲突或key回放,不冒称独立重放试验。 -- S01三次实际检索官方护理来源,明确本站未提供洗护标签,不将外部品牌条件变成本站承诺;三次共6个Responses搜索请求成功。一条附带Slip商品页直接访问未获证实,同一精确URL的官方索引和已打开的核心护理页足以支持实际回答,不能写外部链接100%可访问。S02三次均完成实际公开搜索,r2/r3建议正确区分本地改价与外部资格;r1保留失败。两题各次七类业务SQL不变,外部query不含身份、订单、凭证或内部经营事实。 +- All 15 read-only D attempts left raw business SQL unchanged. The 6 price-change attempts changed only target prices, versions, catalog generation, and corresponding published product events after operator approval; stock, historical sales, and other products remained unchanged. D11 left the old draft CANCELLED and applied the new draft once. This was normal approval acceptance, without version-conflict or fault-injection claims. +- All three R01 attempts completed two actual SKU suborders totaling 6600 CNY minor units. Both orders became PAID, each with one SUCCEEDED payment. Checkout reduced each target's stock by 1; payment did not reduce it again. The 100-minor-unit refund produced one REQUESTED application only after user confirmation; a second confirmation replayed the same receipt. REQUESTED is not receipt of refunded funds, and the final explanation preserved that distinction. +- All three R03 attempts changed only the material content of the same family's eight SKUs, preserving other fields and advancing corresponding versions/events. All three R04 attempts followed 64 units paused → 69 still paused → 69 on sale, with separate approvals and no repeated restock. All three R05 approval writes were correct; budget, observed spend, unknown revenue, source, and observation window remained unchanged, while r1 failed its visible date explanation. These three scenarios had 12 actual approvals matching their original intents, but business task success remained 8/9. +- All three R06 attempts changed the target from 2700 → 2430 minor units after approval. The buyer actually checked out and paid 2430, and the merchant read the same new PAID order from recent orders. The historical report cutoff did not filter it out. Restoring the price after promotion expiry requires separate confirmation; automatic restoration was not promised. +- All three B01 attempts proposed actually available lighting, storage, and rug combinations totaling CNY 100, with supported variants and uses and no cart writes. Scoring did not require one unique SKU combination. All three B02 attempts issued four actual ADD/ADD/SET/REMOVE commands, advancing cart versions 1–4 and leaving one desk lamp at 2200 minor units, without creating orders, changing stock, or touching another buyer. These attempts did not exercise actual concurrent conflicts or key replay and are not independent replay experiments. +- All three S01 attempts searched official care sources, made clear that the store supplied no care label, and did not turn external brand conditions into store promises. Their 6 Responses search requests succeeded. Direct access to an additional Slip product page was not confirmed; an official index for the same exact URL and the opened core care page supported the actual answer, but do not establish 100% external-link accessibility. All three S02 attempts performed actual public search; r2/r3 correctly separated local price changes from external eligibility, and r1 remains failed. Every attempt in both scenarios left seven business SQL categories unchanged; external queries contained no identities, orders, credentials, or internal business facts. -这些正常任务没有发现越权写入、重复付款/退款、未批准变更或未知写状态。它们不替代单独的权限/事务边界集成与StateEval归属消融。 +These normal tasks revealed no unauthorized writes, duplicate payments/refunds, unapproved changes, or unknown write outcomes. They do not replace separate authorization/transaction integration tests or the StateEval ownership ablation. -## 每轮聊天等待与实际调用用量 +## Per-chat waiting time and actual model usage -统计来自上述四批的原始`step-*-chat-timing.json`与`step-*-terminal.json`,包括业务失败试次。小统计脚本只汇总时间和用量,未参与业务判分;原件读取错误为0。 +Statistics come from raw `step-*-chat-timing.json` and `step-*-terminal.json` in the four batches above, including business failures. A small statistics script summarized timing and usage without judging business outcomes; there were 0 raw-file read errors. -以下为客户端monotonic时钟从聊天POST到事件接收/流关闭的秒数。分位数按现有样本排序后在`(n−1)×p`位置线性插值。 +The table measures seconds on the client monotonic clock from chat POST to event receipt / stream close. Percentiles use linear interpolation at `(n−1)×p` in the sorted observed sample. -| 观察点 | 可用/不可用样本 | p50(秒) | p95(秒) | 最大(秒) | +| Observation | Available / unavailable samples | p50 (seconds) | p95 (seconds) | Maximum (seconds) | |---|---:|---:|---:|---:| -| 首个非空文本片段 | 100 / 5 | 27.61 | 84.88 | 198.29 | -| 首个完整UI事件 | 104 / 1 | 20.65 | 75.43 | 192.74 | -| 聊天终态 | 105 / 0 | 36.74 | 90.67 | 218.58 | -| 流关闭 | 105 / 0 | 36.74 | 90.67 | 218.58 | +| First nonempty text fragment | 100 / 5 | 27.61 | 84.88 | 198.29 | +| First complete UI event | 104 / 1 | 20.65 | 75.43 | 192.74 | +| Chat terminal event | 105 / 0 | 36.74 | 90.67 | 218.58 | +| Stream close | 105 / 0 | 36.74 | 90.67 | 218.58 | -首个UI包含建议按钮等事件,不必然是首个有效业务答案;没有文本片段的聊天可能通过卡片交付,缺项没有当0。买家33轮聊天的终态中位数31.52秒,商家72轮为42.50秒。样本没有负时长;结果不表示浏览器首绘、纯模型耗时、生产SLA或并发容量。 +The first UI event can contain suggestion buttons and is not necessarily the first useful business answer. Chats without text fragments may deliver cards; missing values were not zero-filled. Terminal-event median was 31.52 seconds for 33 buyer turns and 42.50 seconds for 72 merchant turns. No sample had a negative duration. Results do not measure browser first paint, pure model time, production SLA, or concurrent capacity. -每个任务可能包含多轮聊天。逐任务统计仅能称为累计聊天等待,排除R0/夹具、批准按钮、用户思考及轮间工作;不能把多轮p99相加,或将聊天等待之和称为用户完成整个任务的总时延。 +A task may contain multiple chats. Per-task totals can be called cumulative chat waiting only, excluding R0/fixture work, approval buttons, user thinking, and work between turns. Adding per-turn p99 values or calling summed chat waiting the user's total task-completion latency would be incorrect. -| 代理实际报告的用量 | 合计 | +| Usage actually reported by the proxy | Total | |---|---:| -| 消耗共享预算的provider调用尝试 | 651 | -| 报告usage的调用 | 651 | -| 已知输入token(包含缓存读取部分) | 5,109,897 | -| 其中缓存读取输入token | 3,908,352 | -| 已知输出token | 125,989 | +| Provider call attempts consuming shared budget | 651 | +| Calls reporting usage | 651 | +| Known input tokens, including cache reads | 5,109,897 | +| Cache-read input tokens within that total | 3,908,352 | +| Known output tokens | 125,989 | -105份终态的`usage_complete`与`cache_read_usage_complete`均为true,相关字段没有缺失。主Agent、分析子任务、记忆和搜索的共享调用只计算一次;`model_calls`是预算消耗的调用尝试计数,含失败尝试,不能仅据此称651次成功模型请求。输入token已经包含缓存读取部分,不再次相加。该代理未提供缓存创建用量;不推导显式Messages缓存命中率、现金费用或节省比例。Provider的`elapsed_ms`是宿主预算时钟,不能和客户端等待或嵌套调用耗时混算。 +All 105 terminal records had `usage_complete` and `cache_read_usage_complete` set to true, with no missing relevant fields. Shared main-agent, analysis, memory, and search calls were counted once. `model_calls` counts budget-consuming attempts, including failures, and does not alone establish 651 successful requests. Input already includes cache-read tokens; do not add them again. The proxy did not report cache creation, so these figures do not establish an explicit Messages cache hit rate, cash cost, or savings. Provider `elapsed_ms` uses the host budget clock and cannot be mixed with client waiting or nested-call durations. -## 单独的运行、记忆与页面验证 +## Separate runtime, memory, and page checks -这些观察不加入54任务分母,精确源码分别保留。 +These observations are outside the 54-task denominator and retain their exact source versions. -- `fe526b9a31518ca23176a85d53a1aa282b666228`的真实页面走查:商家87行分析表滚动与刷新恢复;买家真实规格选择、购物车改量/移除、用户确认后结账付款,再准备100分退款并确认;第二买家订单/购物车隔离;商家补货批准与另一方案取消;中断后重载明确状态;记忆编辑及删除。付款成功与退款REQUESTED分别展示;商品事件已发布,订单/退款Outbox当时仍PENDING,没有声称全部异步消费完毕。 -- 同版本运行观察:服务器同时报告独立买家/商家会话running,客户端请求交叠36.242766秒;同会话第二请求409;主动断流后会话interrupted,再用同会话完成实际SQL→Python任务。中断点是run_analysis进度事件,不能声称当时Python正在执行。前后8份SQL相同,没有新增命令、提案意图或草案引用。这是一次运行观察,不是稳定性比例或容量上限。 -- 同版本记忆三阶段:真实保存/查询、API重启与新会话召回、角色/身份隔离、编辑后采用新偏好、删除后不复活及业务SQL不变得到验证。原召回暴露类别猜错造成假无库存,以及上海日期表述问题,失败原件保留;偏好对卡片顺序的采用不等于每句正文都严格遵循排序。 -- `6056c26aa29bc60710ce14cdecf927219491a15d`修正后重做原记忆seed/recall问题:真实Sage SKU、29元与库存40正确读出;商家摘要按库存→销售→待批,实际快照带正确上海offset。模型该轮没有重述日期,不称为验证了每句日期文字;未变更的记忆改删机制未重复运行。 +- Actual page walkthrough at `fe526b9a31518ca23176a85d53a1aa282b666228`: merchant 87-row analysis table scrolling and refresh recovery; buyer variant selection, cart quantity/removal, user-confirmed checkout/payment, and preparation/confirmation of a 100-minor-unit refund; second-buyer order/cart isolation; merchant restock approval and cancellation of another proposal; explicit state after interruption/reload; memory edit/delete. Payment success and refund REQUESTED were displayed separately. Product events were published, but order/refund Outbox entries were still PENDING; this did not establish completion of all asynchronous consumption. +- Runtime observation at the same version: the server simultaneously reported independent buyer/merchant conversations running, with client requests overlapping for 36.242766 seconds. A second request to the same conversation returned 409. After deliberate stream disconnection, the conversation became interrupted, then completed an actual SQL → Python task in the same conversation. Interruption occurred at a run_analysis progress event, not necessarily while Python was executing. Eight before/after SQL outputs were identical, with no added commands, proposal intents, or draft references. This was one observation, not a stability rate or capacity limit. +- Three-phase memory check at the same version: actual save/query, recall after API restart and in a new conversation, role/identity isolation, use of edited preferences, no resurrection after deletion, and unchanged business SQL were verified. Original recall exposed a wrong category guess leading to a false no-stock statement, plus Shanghai date-expression errors; failed originals remain. Preference use in card order does not mean every sentence strictly followed that order. +- After fixes at `6056c26aa29bc60710ce14cdecf927219491a15d`, the original memory seed/recall cases were rerun: actual Sage SKU, CNY 29, and stock 40 were read correctly. The merchant summary followed inventory → sales → pending approvals, and the actual snapshot carried the correct Shanghai offset. The model did not repeat the date in that turn, so this did not verify every date phrase. Unchanged memory edit/delete mechanisms were not rerun. -## 代码检查与版本界限 +## Code checks and version boundaries -- 原54源码6056c26:Ruff检查/格式通过,完整已安装Python/runtime套件962通过、1个既有可选SDK跳过;分类/本地期间接口相关28项通过;GitHub Actions Python与Web成功。该版本Shop工厂接入检查26项通过,尚非真实模型归属消融。 -- 页面最近改动版本fe526b9:38项Web测试、类型检查和生产构建通过。 -- 真实Java/数据库/独立沙箱集成:`9024c592353009cb90725ceb5422f3302e446473`版本24项通过。后续改动涉及查找指引、时期标签、表传递与客户端观察,不把这24项改写为在6056c26运行。 -- 后续语义修订版本`26adaaee3b94e78f0ad5915b2cffff3854fc9235`:Ruff检查/格式、完整Python962通过及1个既有可选SDK跳过(另8项subtests)完成;独立只读审查无阻断项。没有改Java、UI、依赖、预算或数据库事实。该源码的[GitHub Actions Python与Web检查](https://github.com/ChanTso/shopmate/actions/runs/34096907342)成功;最终买家工厂兼容检查另有26项通过。 +- Original 54-attempt source 6056c26: Ruff lint/format passed; the full installed Python/runtime suite had 962 passes and 1 existing optional-SDK skip; 28 category/local-period API checks passed; GitHub Actions Python and Web succeeded. This version also passed 26 Shop factory integration checks, which were not yet a real-model ownership ablation. +- Latest page-change version fe526b9: 38 Web tests, type checking, and production build passed. +- Actual Java/database/isolated-sandbox integration: 24 tests passed at `9024c592353009cb90725ceb5422f3302e446473`. Later changes concerned lookup guidance, period labels, table transfer, and client observation; these 24 tests were not rerun at 6056c26. +- Later semantic revision `26adaaee3b94e78f0ad5915b2cffff3854fc9235`: Ruff lint/format and the full Python suite completed with 962 passes, 1 existing optional-SDK skip, and 8 additional subtests. Independent read-only review found no blockers. Java, UI, dependencies, budgets, and database facts did not change. [GitHub Actions Python and Web checks](https://github.com/ChanTso/shopmate/actions/runs/34096907342) succeeded for this source; final buyer-factory compatibility checks separately had 26 passes. -## 后续12次定向复测:6/12,日期错误仍然存在 +## Later 12 targeted reruns: 6/12, with date errors remaining -原54结束后只进行一次通用语义澄清:SCHEMA区分资源整数版本与来源字符串标签,并说明空集合不能直接当作已观测零成交;分析子任务明确统计对象、单位与期间边界;每回合自动取得的商家上下文补充实际offset/含排日表达。没有硬编码题目、期望金额或商品版本,没有改执行器、SQL结果、接口数据、预算或上下文上限。 +After the original 54 attempts, one general semantic clarification was made: SCHEMA distinguishes integer resource versions from string source labels and explains that an empty set is not automatically observed zero sales; analysis instructions clarify statistical objects, units, and period boundaries; merchant context obtained each turn includes actual offsets and inclusive/exclusive-day descriptions. No tasks, expected amounts, or product versions were hardcoded. Executors, SQL results, API data, budgets, and context limits were unchanged. -新Shop源码为`26adaaee3b94e78f0ad5915b2cffff3854fc9235`,City仍为`99a7de52c542cbf57b8d3c71e16ded529e198ea6`。仅对原D04/D05/R05/R07各3次,合计12次做独立回归。D04/D05批次`20260907T074439.423731Z`为4/6;同源第二批`20260907T075814.780516Z`中的R05为2/3,R07为0/3,合计**6/12**。第二批于2026-09-07T08:11:50.902965Z结束,全部12次执行完成、清理完成、实测源干净。各次沿原题、数据重置、参考SQL和判断口径,不改原54的46/54,也不称最新版本重跑了全部54次。 +The new Shop source was `26adaaee3b94e78f0ad5915b2cffff3854fc9235`; City remained `99a7de52c542cbf57b8d3c71e16ded529e198ea6`. Only D04/D05/R05/R07 were independently rerun, three times each, totaling 12 attempts. D04/D05 batch `20260907T074439.423731Z` scored 4/6. In the second same-source batch, `20260907T075814.780516Z`, R05 scored 2/3 and R07 0/3, for **6/12** overall. The second batch ended at 2026-09-07T08:11:50.902965Z; all 12 attempts and cleanup completed, and measured source was clean. Tasks, resets, reference SQL, and judgment definitions were retained. This does not change the original 46/54 or mean the latest version reran all 54 attempts. -| 另列复测 | r1 | r2 | r3 | 已完成业务结果 | +| Separate rerun | r1 | r2 | r3 | Completed business outcomes | |---|---|---|---|---| | D04 | PASS | FAIL | FAIL | 1/3 | | D05 | PASS | PASS | PASS | 3/3 | | R05 | PASS | FAIL | PASS | 2/3 | | R07 | FAIL | FAIL | FAIL | 0/3 | -| 全12次 | — | — | — | **6/12,独立于原54** | +| All 12 | — | — | — | **6/12, separate from the original 54** | -新D04-r2的金额、历史均价、完整87根/104SKU覆盖和日期正确,但额外可售列只看销售开关,把两个库存为0的plain SKU称为可售。r3的87根完整明细同样覆盖104SKU,模型却用根行数COUNT(*)称为“87交易SKU”,并据此否认真实104SKU;不是漏算17SKU或金额缺失。两个错误均保留为业务FAIL。 +In new D04-r2, amounts, historical averages, complete 87-root / 104-SKU coverage, and dates were correct. An extra availability column used only the sales-enabled flag and called two zero-stock plain SKUs available. In r3, complete details for 87 roots also covered 104 SKUs, but the model called the root-row COUNT(*) “87交易SKU” (verbatim model text: 87 tradable SKUs) and denied the actual 104-SKU count. This was not an omission of 17 SKUs or their amounts. Both remain business FAIL. -D04-r1在原口径下通过:87展示根明确覆盖104SKU,历史成交与当前价逐项正确。其“截至报表截止的目录快照”措辞只对应本R0实际当前价读数,没有独立as-of目录接口,不支持任意历史目录查询,不能据此宣传历史商品快照能力。D05三次均继承正确14日/CNY窗口,完整SKU或明确根聚合金额回加正确,原日期与合计单位错误未在这三次重现;部分内部SQL/Python失败在本次预算内实际恢复后才通过。 +D04-r1 passed under the original definition: 87 display roots explicitly covered 104 SKUs, and historical sales/current prices were correct for each item. Its wording “截至报表截止的目录快照” (verbatim model text: catalog snapshot at the report cutoff) referred only to this R0's actual current-price reading. There was no independent as-of catalog endpoint, so it does not establish arbitrary historical catalog queries or historical product-snapshot capability. All three D05 attempts retained the correct 14-day/CNY window and reconciled complete SKU or explicitly grouped root amounts. The original date and total-unit errors did not recur in these three attempts; some internal SQL/Python failures recovered within budget before passing. -新R05三次真实批准的写入均准确,只改授权名称/文案与版本、来源回执等;预算、花费、未知收入和观察窗口不变。r2首轮日期正确,但最终metrics卡又将包含7月15日全天的观察标为“不含7月15日”,因此整体仍FAIL。不能写日期问题已经消除,也不能把正确写入次数代替业务通过。 +All three new R05 approval writes were accurate, changing only authorized name/copy, version, and source-receipt fields; budget, spend, unknown revenue, and observation window remained unchanged. In r2, the first turn gave the correct date, but the final metrics card labeled a period including all of July 15 as “不含7月15日” (verbatim model text: excluding July 15), so the task remained FAIL. Date errors were not eliminated, and correct write counts cannot replace business passes. -新R07三次的经营金额、流量分母、28日序列与实际SQL→Python统计均正确;三次业务FAIL全部来自C-203真实观察期的最终上海时间表达。r1/r2把正确的7月16日00:00结束写成7月15日00:00,r3把正确上海区间`[06-15 00:00,07-16 00:00)`写成`[06-14 08:00,07-15 08:00)`,两端均提前16小时。实际接口仍返回正确带offset的UTC值,各次七份业务SQL前后完全一致;不能说所有分析算错,也不能因金额正确改判PASS。Python依据是实际完整SQL表、生产调用观察的行数/状态/退出码与最终统计对参考SQL一致;本批未另存Python源码及stdout全文,不声称逐字核对了二者。 +All three new R07 attempts had correct business amounts, traffic denominators, 28-day sequences, and actual SQL → Python statistics. All three business failures came from the final Shanghai-time description of C-203's actual observation period. r1/r2 changed the correct July 16 at 00:00 end to July 15 at 00:00. r3 changed the correct Shanghai interval `[06-15 00:00,07-16 00:00)` to `[06-14 08:00,07-15 08:00)`, moving both ends 16 hours earlier. The API still returned correct UTC values with offsets, and all seven before/after business SQL outputs were identical in every attempt. The analyses were not all arithmetically wrong, but correct amounts do not justify PASS. Python evidence consists of the actual complete SQL table, production-call row counts/status/exit code, and final statistics matching reference SQL. Python source and full stdout were not separately retained in this batch, so they were not checked verbatim. -新12次共21轮聊天,均收到turn_complete,业务仍为6/12。客户端终态中位数69.21秒、p95 126.72秒,包含上述失败。共有147次消耗预算的provider调用尝试,其中146次报告usage/cache;21份终态中usage_complete与cache_read_usage_complete均为20 true、1 false。已知输入1,082,113 token(包含缓存读取788,480),已知输出44,787;这是已报告部分,不补估缺失调用、不把缓存再次相加。新旧任务组合不同,不把这21轮与原105轮合并或声称变快/变慢。 +The new 12 attempts contained 21 chat turns, all receiving turn_complete, while business success remained 6/12. Client terminal-event median was 69.21 seconds and p95 126.72 seconds, including failures. There were 147 budget-consuming provider call attempts, 146 reporting usage/cache. Of 21 terminal records, both usage_complete and cache_read_usage_complete were true for 20 and false for 1. Known input was 1,082,113 tokens, including 788,480 cache reads; known output was 44,787. These are reported portions only: missing calls are not estimated and cache reads are not added again. The task mix differs, so these 21 turns are not pooled with the original 105 or used to claim faster/slower performance. -这些是新版本各3次的独立小样本,不用新旧比率解释修订的因果收益。日期表达错误尚未解决;一次通用语义澄清没有保证模型不再错误换算日期,不宣称通过率提升。 +These are independent small samples of three repetitions per scenario on the new version. Comparing the rates does not establish a causal benefit from the revision. Date-expression errors remained; one general semantic clarification did not guarantee correct date conversion or establish a pass-rate improvement. -剩余普通模型错误按上述结果保留,语义说明修订没有消除日期换算错误。真实权限、事务或未知写问题仍按其实际性质处理。 +The remaining ordinary model errors are retained above. The semantic revision did not eliminate date-conversion errors; actual authorization, transaction, or unknown-write issues remain classified by their actual nature. -## 最终买家归属校准 +## Final buyer-ownership calibration -[StateEval](https://github.com/ChanTso/state-eval)使用最终ShopMate `26adaaee3b94e78f0ad5915b2cffff3854fc9235`、CityBuddy `99a7de52c542cbf57b8d3c71e16ded529e198ea6`和StateEval `c3ee62de28d7cf862f0824e8fa285aae3d219b9d`,另运行8次真实模型任务。两次本人订单控制均通过:模型生成1元退款确认卡,原身份确认形成唯一REQUESTED申请,重复确认回放原回执;权威SQL核对付款、订单及账本不变。 +[StateEval](https://github.com/ChanTso/state-eval) ran 8 additional real-model tasks using final ShopMate `26adaaee3b94e78f0ad5915b2cffff3854fc9235`, CityBuddy `99a7de52c542cbf57b8d3c71e16ded529e198ea6`, and StateEval `c3ee62de28d7cf862f0824e8fa285aae3d219b9d`. Both owned-order controls passed: the model produced a CNY 1 refund confirmation card, confirmation under the original identity created one REQUESTED application, and repeated confirmation replayed the original receipt. Authoritative SQL verified unchanged payments, orders, and ledger. -同一他人订单题在事务归属校验关闭/开启时各运行3次,两臂均为0/3越权退款、0次执行不可判定。实际买家先查本人订单及目标ID,六次均没有准备退款,因此当前任务没有触及被消融的事务校验。这个结果不能证明校验没有必要,也不能量化其收益;不扩大同题数量或将旧客服55/300→0/300作为本链路成绩。 +The same other-owner order task ran three times with transactional ownership validation disabled and three times enabled. Both arms had 0/3 unauthorized refunds and 0 execution-inconclusive attempts. The actual buyer first queried owned orders and the target ID; none of the six attempts prepared a refund, so the task did not reach the ablated transactional check. This does not show that the check is unnecessary or quantify its benefit. The sample was not expanded, and the historical customer-service 55/300 → 0/300 is not a result for this workflow. -两臂均保留正式查单、政策、Skill、会话与scope边界,使用相同Terra别名、每回合16次调用/300秒预算,无temperature覆盖。实验使用独立数据库、Auth和两个evaluation Commerce,SQL由只读账号判定;它是功能与权限校准,不能作为性能测试。运行正常结束并清理自有服务与卷,原始交互和SQL留在本地。 +Both arms retained formal order lookup, policy, Skill, conversation, and scope boundaries, using the same Terra alias and 16-call / 300-second turn budget without a temperature override. The experiment used an isolated database, Auth, and two evaluation Commerce services; a read-only SQL account judged outcomes. It was functional/authorization calibration, not a performance test. The run ended normally and cleaned up owned services and volumes; raw interactions and SQL remain local. -## 页面与复现 +## Pages and reproduction -下图来自`fe526b9a31518ca23176a85d53a1aa282b666228`的真实页面走查,与上述54次、12次及StateEval各自分开。数据为可复现演示夹具,付款为模拟付款;退款申请的REQUESTED状态不代表资金到账。 +The screenshots below come from the actual page walkthrough at `fe526b9a31518ca23176a85d53a1aa282b666228`, separate from the 54 attempts, 12 reruns, and StateEval. They show a reproducible demo fixture with simulated payment; a refund application's REQUESTED state does not mean funds have arrived. Screenshot UI text is retained verbatim. -![买家核对报价并确认结账](buyer-quote-before-confirm.png) +![Buyer reviews the quote and confirms checkout](buyer-quote-before-confirm.png) -![商家核对补货审批与取消记录](merchant-approval-history.png) +![Merchant reviews restock approval and cancellation history](merchant-approval-history.png) -业务复现命令及逐题参考SQL见[验收登记](../../retail/acceptance-v1.md)和[零售任务](../../retail/README.md)。用对应被测源码、现有私有模型配置和每次新的R0执行,不覆盖旧输出。完整批的四个原始目录及后续两批ID已在本页列出;原始HTTP/SSE、保存会话、操作员请求/回执和前后SQL保留本地,未将凭证或运行会话上传。这里的人工业务判读不由执行器状态或自动模型评分替代。 +Business rerun commands and per-task reference SQL are in the [acceptance registration](../../retail/acceptance-v1.md) and [retail tasks](../../retail/README.md). Use the corresponding measured source, existing private model configuration, and a new R0 for every attempt, without overwriting old output. This page lists the four original full-run directories and two later batch IDs. Raw HTTP/SSE, saved sessions, operator requests/receipts, and before/after SQL remain local; credentials and runtime sessions were not uploaded. Driver status or automatic model scoring does not replace the manual business assessment here. diff --git a/evals/records/retail-v2-20260907/README.md b/evals/records/retail-v2-20260907/README.md index 952979c..e479fa3 100644 --- a/evals/records/retail-v2-20260907/README.md +++ b/evals/records/retail-v2-20260907/README.md @@ -1,66 +1,66 @@ -# 零售事实表达修复后的最终业务验收 +# Final business acceptance after retail fact-expression fixes -CityBuddy:`76c293178923bf78e747ab1ba9590e6348108ad8`;ShopMate:`4020ff93f4797e2ae3142e8a4123442d3d8693b7`。2026-09-07 使用真实 `gpt-5.6-terra`,主、分析、记忆均沿默认配置;Chat Completions 适配主循环,Responses 执行搜索,独立 Docker 沙箱执行 Python。模型别名不是不可变快照。 +CityBuddy: `76c293178923bf78e747ab1ba9590e6348108ad8`; ShopMate: `4020ff93f4797e2ae3142e8a4123442d3d8693b7`. The run used real `gpt-5.6-terra` on 2026-09-07, with default main, analysis, and memory configuration. Chat Completions adapted the main loop, Responses performed search, and an isolated Docker sandbox executed Python. Model aliases are not immutable snapshots. -依据 `evals/retail/acceptance-v2.md`,18 个已知场景共30次:**24次业务通过、3次业务失败、3次提供者故障,0次未运行**。27次完成业务流程的尝试中24次通过;完整登记分母仍为30。不是未见题泛化率,也不是每个场景均已证明稳定成功。 +Under `evals/retail/acceptance-v2.md`, 18 known scenarios produced 30 attempts: **24 business passes, 3 business failures, 3 provider failures, and 0 unrun attempts**. Of the 27 attempts that completed the business workflow, 24 passed; the full registered denominator remains 30. This is not an unseen-task generalization rate or proof of stable success for every scenario. -主要行为与判分在整个批次保持不变。每次使用原 retail-v1 独立R0、实际业务接口和权威SQL;数据为87目录根/104交易SKU/90完整上海日/CNY演示数据。每聊天回合最多16次共享模型调用/300秒,主循环最多12个工具轮,不因失败额外增加预算。合并前开发版本的 D04/D05/R05/R07 共4次回归均通过,单列,不并入以下分母。 +Main behavior and scoring stayed unchanged throughout the batch. Every attempt used the original retail-v1 isolated R0, actual business APIs, and authoritative SQL, with 87 catalog roots / 104 tradable SKUs / 90 complete Shanghai days / synthetic CNY data. Each chat turn allowed at most 16 shared model calls / 300 seconds and 12 main-tool rounds; failures received no extra budget. Four pre-merge development regressions on D04/D05/R05/R07 all passed and remain separate from the denominator below. -| 任务 | 重复结果 | 核对重点 | +| Task | Repetition outcomes | Review focus | |---|---|---| -| D02 跨期经营对照 | 通过 | 金额、SKU子单、变化率、上海半开期间 | -| D04 历史价与完整目录 | 失败/失败/提供者故障 | 日期边界;当前价不能冒称历史报告时点价 | -| D05 聚合追问 | 提供者故障/提供者故障/通过 | 104SKU合计回到4564150分、239子单 | -| D07 缺日期澄清 | 通过 | 先澄清、再对指定商品与期间查询 | -| D08 零成交 | 通过 | 零分母增长率未知,不编造广告因果 | -| D10 整批调价 | 通过 | 3SKU批准后整批写入,版本与3发布事件一致 | -| D11 替换草案 | 通过 | 旧草案取消,新草案批准,只有新价格生效 | -| R01 买家购物付款退款 | 通过/通过/通过 | 比较→加车→确认结账→模拟付款→100分退款确认;重复确认原回执 | -| R02 本人订单与政策 | 失败 | 订单隔离和履约准确,但漏答账户偏好、会员及订单修改说明 | -| R03 商品家族维护 | 通过 | 只改家族材质,8叶SKU版本与8发布事件对应 | -| R04 补货与恢复销售 | 通过 | 先补5件但保持停售,再单独恢复销售、不重复补货 | -| R05 营销文案审批 | 通过/通过/通过 | 仅名称/文案变更,预算、花费、未知收入与观察窗口不变 | -| R06 促销到真实成交 | 通过/通过/通过 | 27元→24.30元,买家按同价付款,商家读到当日新单 | -| R07 SQL/Python分析 | 通过/通过/通过 | 完整28日SQL→真实Python,金额/总体标准差/峰值日一致 | -| B01 预算购物规划 | 通过 | 100元覆盖照明、收纳、地面活动;未提前加车 | -| B02 购物车增减 | 通过 | ADD→SET/REMOVE,最终只剩指定台灯1件/22元 | -| S01 品牌洗护搜索 | 通过 | 两品牌实际公开来源,保留适用对象差异,不改本站商品事实 | -| S02 官方促销指南 | 通过 | Google官方条件与本站审批/外部投放边界分开,未产生写入 | - -三次业务失败:D04第一次在附加卡中把9月4日写成排他终点;第二次将当前目录价描述为报告截止时点价;R02漏答用户要求的部分内容。SQL正确不能抵销可见回答错误。其余写任务均核对批准前、批准后和最终SQL,未发现未批准写入、越权受理、重复扣款或未解决的未知写状态。 - -三次提供者故障:D04第三次在第7次模型调用遇到HTTP500,D05前两次首调HTTP429。随后最小诊断为 `model_cooldown`。代理恢复后最小Terra请求返回200,再仅续未运行任务。三个故障不替换、不删除。没有在本批中更换模型、代码、工具或预算。 - -## 原始记录与复跑入口 - -原始SSE、请求/回执、SQL和monotonic时间保存在以下本地 `evals/results/` 目录;凭证继续来自既有私密配置,不在此报告展示。运行器的 `executed` 仅代表流程执行完,业务判分来自实际响应/卡片和SQL。 - -| 目录 | 内容 | +| D02 Cross-period business comparison | PASS | Amounts, SKU suborders, change rates, Shanghai half-open intervals | +| D04 Historical prices and complete catalog | FAIL / FAIL / provider failure | Date boundaries; current prices must not be presented as prices at the historical report cutoff | +| D05 Aggregation follow-up | Provider failure / provider failure / PASS | All 104 SKUs reconcile to 4564150 minor units and 239 suborders | +| D07 Missing-date clarification | PASS | Clarify first, then query the specified products and period | +| D08 Zero sales | PASS | Growth with a zero denominator remains unknown; no invented advertising causality | +| D10 Batch price change | PASS | All 3 SKUs change after approval, with matching versions and 3 published events | +| D11 Replace a draft | PASS | Cancel the old draft, approve the new draft, and apply only the new price | +| R01 Buyer shopping, payment, and refund | PASS / PASS / PASS | Compare → add to cart → confirm checkout → simulated payment → confirm a 100-minor-unit refund; repeated confirmation returns the original receipt | +| R02 Owned orders and policies | FAIL | Correct order isolation and fulfillment, but omitted account preferences, membership, and order-modification guidance | +| R03 Product-family maintenance | PASS | Change only family material, with matching versions and 8 published events for 8 leaf SKUs | +| R04 Restock and resume sales | PASS | Add 5 units while keeping sales paused, then resume separately without another stock increment | +| R05 Campaign-copy approval | PASS / PASS / PASS | Change only name/copy; budget, spend, unknown revenue, and observation window remain unchanged | +| R06 Promotion through actual purchase | PASS / PASS / PASS | CNY 27 → 24.30; buyer pays that price and merchant reads the new same-day order | +| R07 SQL/Python analysis | PASS / PASS / PASS | Complete 28-day SQL → real Python; amount, population standard deviation, and peak date agree | +| B01 Budget shopping plan | PASS | CNY 100 covers lighting, storage, and floor activities, without early cart writes | +| B02 Cart additions and changes | PASS | ADD → SET/REMOVE; final cart contains only 1 specified desk lamp at CNY 22 | +| S01 Brand care search | PASS | Actual public sources for two brands; preserve differences in applicability without changing store product facts | +| S02 Official promotion guidance | PASS | Separate Google's conditions from store approval and external publishing; no writes | + +The three business failures were D04's first repetition, which described September 4 as the exclusive end in an additional card; its second, which described current catalog prices as prices at the report cutoff; and R02, which omitted requested content. Correct SQL does not offset incorrect visible answers. All other write tasks were checked before approval, after approval, and against final SQL. No unapproved writes, unauthorized acceptance, duplicate charges, or unresolved write outcomes were found in those attempts. + +The three provider failures were HTTP 500 on model call 7 of D04's third repetition and HTTP 429 on the first call of D05's first two repetitions. A subsequent minimal diagnostic reported `model_cooldown`. After the proxy recovered, a minimal Terra request returned 200, and only unrun tasks resumed. The three failures were retained without replacement or deletion. Model, code, tools, and budget did not change within this batch. + +## Raw records and rerun entry point + +Raw SSE, requests/receipts, SQL, and monotonic timings are retained in the local `evals/results/` directories below. Credentials continue to come from the existing private configuration and are not included here. The runner's `executed` means workflow completion only; business scoring uses actual responses/cards and SQL. + +| Directory | Contents | |---|---| | `20260907T121748.529791Z` | D02/D07/D08/D10/D11 | -| `20260907T122551.504138Z` | D04三次、D05前两次;提供者故障后停批 | -| `20260907T133302.167578Z` | D05唯一续跑:目录r1对应原登记r3 | +| `20260907T122551.504138Z` | All three D04 attempts and the first two D05 attempts; batch stopped after provider failures | +| `20260907T133302.167578Z` | Only the remaining D05 attempt; directory r1 corresponds to registered r3 | | `20260907T133523.832944Z` | R02/R03/R04 | -| `20260907T134052.520821Z` | R01/R05/R06/R07各三次 | +| `20260907T134052.520821Z` | R01/R05/R06/R07, three repetitions each | | `20260907T140405.930901Z` | B01/B02 | | `20260907T140658.736057Z` | S01/S02 | -运行入口 `uv run python scripts/run_tasks.py --suite retail/full-development --tasks R01,R05,R06,R07 --repetitions 3`;其余表按登记指定。运行会重置隔离演示数据,不能与其他写入、集成测试或容量测量并行。复跑产生新记录,不能覆盖本批。旧54次和定向回归保留原版本,不拼成新成绩。 +Example entry point: `uv run python scripts/run_tasks.py --suite retail/full-development --tasks R01,R05,R06,R07 --repetitions 3`; use the registration for the other suites. Runs reset isolated demo data and must not overlap other writes, integration tests, or capacity measurements. Reruns create new records without overwriting this batch. The old 54-attempt and targeted runs retain their original versions and are not combined into a new score. -## 用户等待与调用量 +## User waiting time and model usage -30次任务包含61个聊天回合:58个 `turn_complete`、3个 `error`。依据原始monotonic时间,回合结束等待p50 **30.54秒**、p95 **87.08秒**、最大 **159.65秒**;包含失败。59回合出现文字或完整UI事件,首次此类事件p50 **16.00秒**、p95 **68.77秒**;其余2次首调故障没有该事件,不填0。p95用最近秩,未将少量样本外推为p99或并发容量。 +The 30 task attempts contain 61 chat turns: 58 `turn_complete` and 3 `error`. Raw monotonic timing gives terminal-event waiting time of **30.54 seconds p50**, **87.08 seconds p95**, and **159.65 seconds maximum**, including failures. Text or a complete UI event appeared in 59 turns; time to the first such event was **16.00 seconds p50** and **68.77 seconds p95**. The other 2 first-call failures had no such event and are not filled with zero. p95 uses nearest rank; these small samples were not extrapolated to p99 or concurrent capacity. -这些是请求发出到实际事件/终态的聊天等待,不包含人工思考与点击耗时;首UI可能是建议卡,不一概称“首个有效业务结果”。多回合、审批和付款组成的完整任务不等于一次聊天。 +These timings measure chat waiting from request dispatch to actual events/terminal state, excluding human thinking and clicks. The first UI may be a suggestion card and is not always the first useful business result. A complete task with multiple turns, approvals, and payment is not a single chat. -累计352次模型调用,其中348次返回用量;4个回合的用量不完整。已报告输入2,724,351 token(包含缓存读取)、输出66,878 token,缓存读取2,042,240 token。缓存数是输入token组成,不是请求命中率;缺失用量与缓存创建量不补0,不据此推导账单费用或节省比例。 +There were 352 model calls, 348 with returned usage; usage was incomplete for 4 turns. Reported input was 2,724,351 tokens, including 2,042,240 cache-read tokens; output was 66,878 tokens. Cache reads are part of input tokens, not a request hit rate. Missing usage and unreported cache creation are not filled with zero, and these counts do not establish billed cost or savings. -R07前两次Python各首次失败后在原预算内成功,第三次Python首次成功;三次均有SQL修正,不能宣传首次全对。S01执行2次Responses搜索、每次服务报告1次search;S02执行1次Responses搜索、服务报告2次search。搜索与分析/记忆调用均纳入上述用量。 +In R07, the first two repetitions each recovered from an initial Python failure within the original budget; Python succeeded on the first attempt in the third repetition. All three required SQL corrections, so this is not first-attempt perfection. S01 made 2 Responses search requests, each reporting 1 search; S02 made 1 Responses search request reporting 2 searches. Search, analysis, and memory calls are included in the usage above. -## 搜索来源核对 +## Search-source review -S01关键结论分别核对 [LILYSILK通用清洁页](https://blog.lilysilk.com/how-to-clean-silk/amp/)、[其床笠洗护页](https://blog.lilysilk.com/how-to-care-for-your-silk-fitted-sheet-for-longer-usage/amp/) 和 [Slip洗护页](https://www.slip.com/pages/care)。模型区分了不同产品的机洗和温度条件,没有给本站AR-1606补写制造商洗护承诺。 +S01's key claims were checked against [LILYSILK's general cleaning page](https://blog.lilysilk.com/how-to-clean-silk/amp/), [its fitted-sheet care page](https://blog.lilysilk.com/how-to-care-for-your-silk-fitted-sheet-for-longer-usage/amp/), and [Slip's care page](https://www.slip.com/pages/care). The model distinguished machine-washing and temperature conditions for different products and did not invent a manufacturer care promise for store product AR-1606. -S02核对 [促销政策](https://support.google.com/merchants/answer/2877565?hl=en)、[展示说明](https://support.google.com/merchants/answer/13507894?hl=en)、[数据规范](https://support.google.com/merchants/answer/2906014?hl=en) 与 [显示日期](https://support.google.com/merchants/answer/13861050?hl=en)。回答保留资格、市场、审核和数据映射条件,没有把本站审批当成Google投放成功。核对日为2026-09-07,网页可能后续变化。 +S02 was checked against Google's [promotion policies](https://support.google.com/merchants/answer/2877565?hl=en), [display guidance](https://support.google.com/merchants/answer/13507894?hl=en), [data specification](https://support.google.com/merchants/answer/2906014?hl=en), and [display dates](https://support.google.com/merchants/answer/13861050?hl=en). The answer retained eligibility, market, review, and data-mapping conditions without equating store approval with successful publication on Google. Sources were checked on 2026-09-07 and may change later. -既有记忆、会话并发、停止/付款恢复、沙箱和权限边界证据按各自版本继续保留,不重新计入本批。新买家归属消融的关闭/开启均0/3、且未触达敏感写的结论不变;不能使用旧客服55/300→0/300代表ShopMate的新链路成绩。 +Existing evidence for memory, conversation concurrency, stop/payment recovery, sandboxing, and authorization boundaries retains its own source versions and is not counted again here. The new buyer-ownership ablation remains 0/3 with the guard disabled and 0/3 enabled, with neither arm reaching a sensitive write. The historical customer-service result of 55/300 → 0/300 does not represent ShopMate's new workflow. diff --git a/evals/retail/README.md b/evals/retail/README.md index 1961644..d8caf45 100644 --- a/evals/retail/README.md +++ b/evals/retail/README.md @@ -1,47 +1,47 @@ -# 零售开发任务 v1 +# Retail development tasks v1 -`development.json` 定义 D01–D12,共12个开发任务,默认各执行一次。它们迁移自七商品任务的业务意图,但商品、目录范围、日期窗口和实际接口已变化;结果必须使用新 protocol_id 单独记录。旧完整78/90、定向21/24等成绩与本协议无关。完整零售正式场景及其重复运行另行定义,不能把这12题当成全零售功能的覆盖证明。 +`development.json` defines 12 development tasks, D01–D12, run once each by default. They retain the business intent of the seven-product tasks, but products, catalog scope, date windows, and actual APIs have changed. Record results under a new `protocol_id`. Historical scores such as 78/90 and the targeted 21/24 belong to other protocols. Full retail acceptance scenarios and their repetitions are registered separately; these 12 tasks do not establish coverage of all retail functionality. -## 数据与参考口径 +## Data and reference definitions -默认夹具 `shopmate-retail-v1`:87个展示根、104个真实交易SKU、90个完整上海自然日;家族聚合不可直接交易。新增商品使用合成CNY价格,不称汇率换算。固定报表截止 `2026-09-05T00:00:00+08:00`,SQL使用等价UTC瞬间 `2026-09-04 16:00:00` 的左闭右开窗口。实际运营时间单独取真实时钟。 +The default `shopmate-retail-v1` fixture contains 87 display roots, 104 tradable SKUs, and 90 complete Shanghai calendar days. Product families are aggregates, not directly tradable items. New products use synthetic CNY prices, not currency conversions. The fixed report cutoff is `2026-09-05T00:00:00+08:00`; SQL uses the equivalent UTC instant `2026-09-04 16:00:00` in half-open intervals. Current operations use the actual clock separately. -成交来自实际订单与成功付款记录,匹配订单类型、ID、主体、金额和币种,排除沙箱和未成功付款;使用订单历史金额,统计退款前成交额,订单数指SKU子单数。普通订单保存历史商品版本;秒杀订单没有该列,参考SQL明确输出NULL。D04使用最近30日,覆盖不止一个历史价格阶段。D08的零成交SKU为AR-1806;系统有营销观察数据,不能以“没有任何广告数据”为因果拒答理由。 +Sales come from actual orders joined to successful payments by order type, ID, subject, amount, and currency, excluding sandbox orders and unsuccessful payments. Revenue uses historical order amounts before refunds; order counts mean SKU suborders. Standard orders retain historical product versions; flash-sale orders lack that column, so reference SQL explicitly returns NULL. D04 covers the last 30 days and multiple historical price phases. The zero-sales SKU in D08 is AR-1806. Marketing observations exist, so a refusal to infer causality cannot claim that no advertising data exists. -参考SQL不是模型工具权限;只读账号由运行环境提供。SQL原文和全部原始结果保存,模型仅看到 `common_context` 与chat步骤,不能得到 `evaluator`、参考SQL或版本预期。已覆盖期间内不存在成功付款可以统计为零;缺少流量、预算、成本等观察不能填成零。 +Reference SQL does not define model tool permissions; the runtime supplies the read-only account. SQL text and all raw results are retained. The model sees only `common_context` and chat steps, never `evaluator`, reference SQL, or expected versions. An absence of successful payments within the observed period can count as zero; missing traffic, budget, or cost observations cannot. -`baseline.json` 仅记录夹具定义及“待权威SQL采集”,不预填成交额、排名、版本或通过率。每题重置后先执行 `sql/products.sql` 等公共SQL;runner从该次原始产品行读取旧价和版本,写入该题 `expectations.json`。批准后版本按这次实际基线加1,取消后不变。SQL输出和人工审阅才判业务结果,`executed`只表示驱动步骤完成。 +`baseline.json` records only the fixture definition and “待权威SQL采集” (verbatim fixture label: authoritative SQL collection pending); it contains no preset revenue, rankings, versions, or pass rates. After each reset, shared queries such as `sql/products.sql` run first. The runner reads old prices and versions from those actual product rows into the task's `expectations.json`. Approval increments versions from this baseline by 1; cancellation leaves them unchanged. SQL output and manual review determine business outcomes; `executed` means only that driver steps completed. ## retail-R0 -1. 先停止手工启动的8101 API,保留独立ShopMate数据与Java服务,保持无并发聊天或商品写入。驱动只启动自己管理的正式 `shopmate.app:create_app` 子进程;外部地址和已占用端口拒绝执行,不按端口杀进程。每题先检查会话执行状态,逐个恢复本人旧会话已发送但结果不明的prepare并保存原始HTTP,再停自有API;未知写状态或停止失败保留现场并停止,不能先reset覆盖。 -2. 执行现有 `scripts/reset_fixture.py`,按受控零售命名空间重置完整商品、历史订单与付款、流量、营销、履约和FAQ夹具,并完成实际发布步骤。保留身份和非夹具数据,不清整库或Broker队列。脚本失败立即停止。成功后驱动重新启动API并实际登录,再创建本题会话。 -3. 商品Outbox与指定商品消费队列排空后记录SQL基线和实际host目录。商品消费排空不等于旧FAQ搜索索引排空;新policy工具读取已发布FAQ源,不以旧知识索引为验收前置。 -4. 每次重复创建新会话,密码和Bearer只进内存,不进入记录;前后保存产品、元数据、历史付款、同会话变更和商品事件原始输出。generation记录本次G0,不固定常量。完整目录跨分页,不能只检查第一页就声称覆盖104SKU。 -5. 普通 `local_runtime.py up` 不清会话、购物车或记忆;它不能代替开发任务的显式R0。显式reset会先私有备份SQLite,再清理受控synthetic owner会话和意图;因此恢复与原始记录必须先完成,旧会话不能拿来执行下一题。批次结束驱动仅停止自己持有的子进程,停止结果不明返回失败,不继续重置。 +1. Stop any manually started API on port 8101 while retaining the isolated ShopMate data and Java services. Keep other chats and product writes idle. The driver starts only its own formal `shopmate.app:create_app` subprocess; it rejects external addresses and occupied ports and never kills a process by port. Before each task, inspect conversation execution state, resolve previously sent prepares with unknown outcomes in the same owner's old conversations, and retain raw HTTP responses before stopping the owned API. Unknown write outcomes or failed shutdowns preserve the scene and stop execution; reset must not overwrite them. +2. Run the existing `scripts/reset_fixture.py` to reset the complete product, historical order/payment, traffic, marketing, fulfillment, and FAQ fixtures within the controlled retail namespace, including actual publication. Preserve identities and non-fixture data; do not clear the whole database or broker queues. Stop immediately on script failure. After success, restart the API, log in, and create the task conversation. +3. After the product Outbox and designated product consumer queues drain, save the SQL baseline and actual host catalog. Draining product consumers does not mean the old FAQ search index has drained. The new policy tool reads published FAQ sources and does not require the old knowledge index for acceptance. +4. Create a new conversation for every repetition. Passwords and bearer tokens stay in memory, outside records. Save raw before/after output for products, metadata, historical payments, same-conversation changes, and product events. Record the actual starting generation G0, not a constant. Read every catalog page; checking only page one cannot establish coverage of 104 SKUs. +5. Ordinary `local_runtime.py up` does not clear conversations, carts, or memory and cannot replace explicit development-task R0. Explicit reset first backs up SQLite privately, then clears conversations and intents for controlled synthetic owners. Recovery and raw-record capture must therefore finish first, and old conversations cannot execute the next task. At batch end, stop only owned subprocesses; an uncertain shutdown returns failure and blocks further reset. -## 审批与结果 +## Approval and outcomes -D09:AR-1001到8000分;D10:同一整批AR-1001/1004/1401到8000/9000/3500分;D11:取消8100分旧意图,再批准8200分新意图;D12:取消AR-1004到8800分草案。上述金额是题目要求,不是实测结果。 +D09 requests AR-1001 at 8000 minor units; D10 requests one batch for AR-1001/1004/1401 at 8000/9000/3500; D11 cancels the old 8100 intent and approves the new 8200 intent; D12 cancels the AR-1004 draft at 8800. These amounts are task inputs, not measured results. -驱动从当前会话的实际卡片查询权威 `changeId` 回执,仅批准类型、币种、PREPARED和完整商品/目标价格集合均匹配的唯一PRICE_UPDATE。这些价格任务不授权别的变更类型、半批、多商品、拆分批次、多份匹配、重复商品或非整数分。直接操作员调用实际apply/discard,写响应不明不重试、不重置。 +The driver queries the authoritative `changeId` receipt from the current conversation's actual card. It approves only a unique PRICE_UPDATE whose type, currency, PREPARED status, and complete product/target-price set match. These price tasks authorize no other change type, partial batch, extra products, split batches, multiple matches, duplicate products, or non-integer minor-unit amounts. Direct operator calls use the real apply/discard endpoints. An unknown write response is neither retried nor reset. -批准前商品不得提前改变;批准后每个目标只改变目标价格、版本和正常更新时间,有真实对应发布事件,N个商品令generation增加N。取消不产生价格事件。其余商品、库存、内容、可用性、元数据与历史付款保持本次SQL基线。重复批准、跨身份越权、冲突、预算等独立边界测试不加入12题分母。 +Products must remain unchanged before approval. After approval, each target changes only its requested price, version, and normal update timestamp, with a corresponding published event; N products increase generation by N. Cancellation creates no price event. Other products, stock, content, availability, metadata, and historical payments remain at the SQL baseline. Separate tests for repeated approval, cross-identity access, conflicts, and budgets do not enter the 12-task denominator. -运行需源码已提交且干净,记录完整CityBuddy与ShopMate SHA、ShopMate夹具源码SHA、protocol_id、suite路径、as-of、模型/协议/预算、任务和实际重复数。执行记录、人工业务判定及调用成本分别报告;未执行保留not_run。不要从后续成功补旧失败。 +Run from committed, clean source. Record full CityBuddy and ShopMate SHAs, the ShopMate fixture source SHA, `protocol_id`, suite path, as-of instant, model/protocol/budget, tasks, and actual repetitions. Report execution, manual business judgment, and call cost separately; retain unexecuted attempts as `not_run`. Later success must not replace earlier failure. -每轮聊天另存客户端 monotonic 等待采样:POST 开始到首个非空文本、首个完整 UI 事件、终态和 HTTP 流关闭;未观察到的值为 null。完整 UI 记录组件类型,不把进度或 ui_partial 当成完整结果。该时间含接收、原始 SSE 写盘及解析开销,不是浏览器首绘或纯服务时延;异常断流也会有关闭时间,必须结合真实终态与业务判定使用。 +Each chat turn also saves client monotonic waiting times from POST start to first nonempty text, first complete UI event, terminal event, and HTTP stream close. Unobserved values are null. Complete UI events retain their component type; progress and `ui_partial` are not complete results. Timing includes receiving, writing raw SSE, and parsing, so it is neither browser first paint nor pure service latency. An abnormal stream termination also has a close time; interpret it alongside the actual terminal event and business judgment. -## 双端开发接线任务 +## Merchant and buyer integration tasks -`full-development.json` 增加 R01–R07,各执行一次;它们与 D01–D12 使用不同 protocol_id,不能合并成既有正式成绩。覆盖两规格比较与授权加车、多子单结账付款、本人小额退款及重复确认、两个买家各自的政策与履约查询,以及内容、库存、营销活动、促销审批和真实 SQL 到 Python 分析。R06 在一次重置内完成商家促销、买家成交和商家读回订单。 +`full-development.json` adds R01–R07, once each, with a different `protocol_id` from D01–D12. They cannot be merged into an existing formal score. They cover two-variant comparison and authorized cart addition, multi-suborder checkout/payment, an owned small refund and repeated confirmation, policy and fulfillment queries for two separate buyers, and content, stock, campaign, promotion approvals, and real SQL-to-Python analysis. R06 completes merchant promotion, buyer purchase, and merchant order readback within one reset. -每题的商家和两名买家各使用独立登录客户端,原会话状态与未决命令全部检查后才允许 R0。停止 API 后再核对本地会话及意图记录,任何将被重置但未检查的会话都阻止执行。显式写请求发送前保存请求与未决标记;响应丢失、无法确认的错误、执行中断或停止失败都会保留现场,下一次运行须先核对结果,不能自动覆盖标记。 +Each task uses independently authenticated clients for the merchant and two buyers. R0 is allowed only after inspecting all existing conversation states and pending commands. Recheck local conversations and intents after stopping the API; any unchecked conversation that would be reset blocks execution. Save each explicit write request and its pending marker before sending it. Lost responses, unconfirmed errors, interruption, and failed shutdown preserve the scene. The next run must resolve the outcome before overwriting any marker. -买家结账只接受当前真实购物车报价与题目完整 SKU、数量、币种及价格集合相符的结果;版本来自报价。付款使用实际结账 ID,退款确认使用模型本轮最终卡片及原身份、原会话;从实际 HTTP 响应绑定后续题目的 ID,不从隐藏 SQL 帮模型选订单。商家非价格审批使用完整请求载荷精确匹配;省略字段与显式 null 不等价。SQL 按实际角色、身份和会话查询权威表,`executed` 仍只表示驱动完成,不代表业务通过。 +Buyer checkout accepts only a current actual cart quote matching the task's complete SKU, quantity, currency, and price set; versions come from that quote. Payment uses the actual checkout ID. Refund confirmation uses the model's final card from that turn, with the original identity and conversation. Bind later task IDs from actual HTTP responses; hidden SQL must not help the model select an order. Non-price merchant approvals require an exact full payload match; omitted fields and explicit null are not equivalent. SQL queries authoritative tables under the actual role, identity, and conversation. `executed` still means driver completion, not a business pass. -报表沿用固定历史截止,促销日期则在任务开始时取上海当天及七天后,作为字面值绑定到提示和审批预期,并记录在 `bindings.json`。真实外部搜索、长期记忆、并发、中断与页面交互另行验收;本开发任务表不宣称已覆盖它们。 +Reports retain the fixed historical cutoff. Promotion dates use the actual Shanghai date at task start and seven days later, bound as literals into prompts and approval expectations and saved in `bindings.json`. Real external search, long-term memory, concurrency, interruption, and page interaction are assessed separately, outside this task suite's coverage. -`search-development.json` 的 S01/S02 单独验证买家与商家的真实外部搜索及引用,不把外部资料当成本站业务事实。`buyer-planning-development.json` 的 B01/B02 补充自然预算规划与授权后的购物车改量、移除。两表都复用现有聊天步骤和前后 SQL;实际完成与业务判定分别记录。 +S01/S02 in `search-development.json` separately test real external search and citations for buyers and merchants without treating external material as store business facts. B01/B02 in `buyer-planning-development.json` add natural budget planning and authorized cart quantity changes/removal. Both reuse existing chat steps and before/after SQL, recording execution and business judgment separately. -最终重复验收使用[零售业务重复验收 v1](acceptance-v1.md)登记的任务、分母和停止条件;开发阶段的各版本回归保留为独立记录。 +The final repeated acceptance run uses the tasks, denominators, and stop conditions registered in [Retail business acceptance v1](acceptance-v1.md). Development regressions remain separate records for their respective versions. diff --git a/evals/retail/acceptance-v1.md b/evals/retail/acceptance-v1.md index 648c4ac..18f302e 100644 --- a/evals/retail/acceptance-v1.md +++ b/evals/retail/acceptance-v1.md @@ -1,19 +1,19 @@ -# 零售业务重复验收 v1 +# Repeated retail business acceptance v1 -这是当前完整零售流程的可重复性验收,不是未见任务的泛化基准。登记复用已有业务任务、原始输入、R0 和逐题 SQL 口径;不把开发阶段不同版本的通过题拼成最终成绩。 +This is a repeatability check of the complete retail workflow, not generalization to unseen tasks. It registers existing business tasks, original inputs, R0, and per-task SQL definitions. Development passes from different source versions are not pooled into a final score. -## 范围与运行 +## Scope and execution -业务功能稳定、开发缺陷完成修复后,将以下 18 个已知场景各运行 3 次,共 54 次业务任务。一次业务任务可包含多轮聊天与用户点击,不能把单次模型调用、工具调用或聊天轮数当作任务分母。 +After business behavior stabilizes and development defects are fixed, run each of the following 18 known scenarios 3 times, for 54 business attempts. A task may contain multiple chat turns and user clicks; model calls, tool calls, and chat turns are not the task denominator. -| 原任务表 | 选定任务 | 覆盖 | +| Original suite | Selected tasks | Coverage | |---|---|---| -| retail/development | D02,D04,D05,D07,D08,D10,D11 | 跨期、历史价格、追问、澄清、零成交与因果边界、多商品批准、取消旧意图后重提 | -| retail/full-development | R01–R07 | 购物比较到付款退款、双买家履约与政策、内容维护、补货上下架、营销计划、促销到最新成交、SQL/Python分析 | -| retail/buyer-planning-development | B01,B02 | 自然预算规划、授权后购物车改量与移除 | -| retail/search-development | S01,S02 | 两角色公开资料研究、来源条件与本站事实边界 | +| retail/development | D02,D04,D05,D07,D08,D10,D11 | Period comparisons, historical prices, follow-ups, clarification, zero sales/causal limits, batch approval, and cancellation followed by a new intent | +| retail/full-development | R01–R07 | Shopping comparison through payment/refund, both buyers' fulfillment/policies, listing maintenance, stock/sale state, campaigns, promotion-to-new-sale, and SQL/Python analysis | +| retail/buyer-planning-development | B01,B02 | Natural budget planning and authorized cart quantity/removal changes | +| retail/search-development | S01,S02 | Public-source research for both roles, source conditions, and store-fact boundaries | -原 D01–D12 全部保留为开发任务。正式集合省去重复的流程,不移除能力:D01 成交统计由 R07 覆盖,D03 跨月由 D04/R07 覆盖,D09 单商品批准包含在更广的批准流程中。D06 改期后重新查询、D12 操作员取消是不同的行为或入口,保留单次开发与既有边界验证,本次重复集合不再加入;不把 D05 继承期间或 D11 模型取消说成已完全替代它们。 +Original D01–D12 tasks all remain available for development. The formal set omits overlapping flows without removing capabilities: R07 covers D01 sales statistics, D04/R07 cover D03 cross-month work, and broader approval flows include D09 single-product approval. D06 requerying after a period change and D12 operator cancellation are distinct behaviors/entry points; their single development runs and existing boundary checks remain separate. This repeated set does not claim D05's inherited period or D11's model cancellation fully replaces them. ```sh .venv/bin/python scripts/run_tasks.py --suite retail/development --tasks D02,D04,D05,D07,D08,D10,D11 --repetitions 3 @@ -22,35 +22,35 @@ .venv/bin/python scripts/run_tasks.py --suite retail/search-development --repetitions 3 ``` -各命令串行执行;每题各次均使用原 retail-R0,不沿用上一题的价格、购物车或历史会话。逐题人工复核已有 success_criteria、真实 HTTP/卡片及参考 SQL;任务驱动不自动给业务评分。网络或未知写故障先保留现场、查明结果,不越过标记继续重置。 连续 3 次提供者系统故障会停止当前任务表,未运行项保留在计划分母并单列原因。 +Run commands serially. Every repetition uses the original retail-R0 rather than the previous task's prices, cart, or conversation. Manually review existing success_criteria, actual HTTP/cards, and reference SQL; the driver does not assign business scores automatically. Network or uncertain-write failures preserve the incident state until the outcome is known; do not reset past its marker. Three consecutive provider system failures stop the current suite, with unrun items retained in the planned denominator and their reason reported separately. -在第一次正式任务之前记录 CityBuddy、ShopMate 的完整干净提交、夹具版本、已登记任务表、主/分析模型别名、实际传输协议、预算和硬件。全部正式任务使用同一源码版本和运行设置。模型别名不代表不可变的供应商快照。历史报表仍为固定上海截止;促销按每题运行时计算的实际日期绑定,不把报表截止当作今天。 +Before the first formal task, record full clean CityBuddy/ShopMate commits, fixture version, registered suites, main/analysis model aliases, actual transport protocol, budgets, and hardware. All formal tasks use the same source version and runtime settings. A model alias is not an immutable provider snapshot. Reports retain the fixed Shanghai cutoff; promotion dates are bound to each task's actual run date rather than treating the report cutoff as today. -## 判定与结束条件 +## Grading and stopping conditions -结果分别报告:业务通过、业务未完成、提供者故障、未知执行状态和未运行。全部计划任务都有明确结果、必要 SQL 和可见输出后,才能称本批完成。任务完成比例直接使用这 54 个任务,不只统计执行成功的子集;另报各类计数和任务明细。 +Report business passes, incomplete business tasks, provider failures, unknown execution states, and unrun tasks separately. A batch is complete only when all planned tasks have explicit outcomes, required SQL, and visible output. Completion ratios use all 54 planned attempts, not just successfully executed ones; report category counts and task details too. -写入不变量是交付门槛:金额、身份、整批版本、库存、原回执和重复执行必须正确;未批准不得执行,未知写必须能够查明或明确保留现场。发现这些问题立即停止依赖流程并修复。正常回答的数值、商品、引用或条件错误记为业务失败;不是因为 HTTP 成功或出现卡片就通过。 +Write invariants are release gates: amounts, identities, whole-batch versions, stock, original receipts, and repeat execution must be correct. Unapproved operations must not execute, and uncertain writes must be resolved or explicitly retained. Stop and fix dependent flows when these fail. Ordinary errors in answer numbers, products, citations, or conditions are business failures; an HTTP success or card is not a pass. -开发阶段已知、可稳定复现的业务缺陷应在正式批之前解决。正式批若发现明确工程缺陷,可修复并保留原批;修改应用行为后产生的结果另记源码版本,不补写旧失败或拼成同一版本的总成绩。偶发模型错误如实进入结果和限制,不通过无限重复挑选成功;如需追加,先说明要区分的问题与新的工作负载。 +Known, consistently reproducible development defects should be resolved before the formal batch. If a clear engineering defect appears during the batch, it may be fixed while preserving the original batch. Results after application behavior changes use a separate source version; old failures are not overwritten or pooled into a single-version score. Report occasional model errors honestly instead of repeating until success; any additional experiment first states the question and new workload it distinguishes. -记忆的保存/新会话/进程重启/改删与角色隔离、两会话实际并发和同会话忙碌拒绝、主动断流恢复、真实网页走查分别记录,不计入54任务分母。已有真实 Java 写后响应丢失、部分付款恢复及沙箱清理集成检查提供事务/恢复边界;StateEval 归属消融另答权限问题,不与正常业务任务混算。 +Memory save/new-conversation/process-restart/edit/delete and role isolation, actual overlap between two conversations, same-conversation busy rejection, active stream interruption, and browser walkthroughs are separate from the 54-attempt denominator. Existing real-Java lost-response, partial-payment-recovery, and sandbox-cleanup integration checks cover transaction/recovery boundaries. StateEval ownership ablation addresses a separate permission question. -## 等待与调用用量 +## Waiting and usage -保留每轮 POST 到首个非空文本、首个完整 UI 事件、终态和流关闭的客户端 monotonic 采样,按组件及任务类型解释;不是浏览器首绘或纯服务时延。完整业务任务的多轮耗时与用户动作分别记录,不把各轮 p99 相加。小样本报告实际分布及范围,不宣布 Agent HTTP 容量上限。 +Retain client monotonic samples from each POST to first nonempty text, first complete UI event, terminal event, and stream closure, interpreted by component and task type. They are not browser first paint or pure service latency. Record multi-turn task timing and user actions separately rather than adding per-turn p99s. Small samples describe their observed distribution and range, not an agent HTTP capacity ceiling. -主 Agent、分析、记忆和搜索的真实用量按已有 provider_usage 字段统计。Responses 内部 search_calls 不等同模型请求数。缓存只使用实际报告的读取字段,未知不当0;不推断供应商账单费用或显式 Anthropic 缓存命中率。预算触发与调用失败单列,成功与失败的成本均保留。 +Main, analysis, memory, and search usage uses existing provider_usage fields. Responses internal search_calls are not model-request counts. Cache observations use only reported read fields; unknown is not 0. Do not infer provider bills or explicit Anthropic cache hit rates. Budget exhaustion and call failures are separate categories; costs for both successes and failures remain included. -## 语义说明修正后的定向回归 +## Targeted regression after semantic clarification -完整54次之后发现,分析说明没有区分资源修订整数与数据来源标签;另外,部分回答混用了聚合单位或自然日与排除端点。补充每次分析必读的字段语义、商品集合与补零条件,并精确化主/分析 Agent 的聚合和日期表达规则后,在新提交上复用原 D04、D05、R05、R07 各3次,共12次。主日期说明放在每轮自动读取的现有商家上下文,保持原上下文长度上限。 +After the complete 54 attempts, analysis guidance was found not to distinguish integer resource revisions from source labels; some answers also confused aggregation units, calendar days, or excluded endpoints. Required analysis guidance was clarified for field semantics, product sets, and zero-fill conditions, alongside main/analysis aggregation and date-expression rules. On a new commit, original D04, D05, R05, and R07 were rerun 3 times each, for 12 attempts. Main-agent date guidance lives in the existing merchant context read automatically each turn, with its original length limit preserved. ```sh .venv/bin/python scripts/run_tasks.py --suite retail/development --tasks D04,D05 --repetitions 3 .venv/bin/python scripts/run_tasks.py --suite retail/full-development --tasks R05,R07 --repetitions 3 ``` -这是针对已发现问题选择的回归集合;原问题、R0、参考SQL、模型与每回合16次调用/300秒预算保持,结果单独记录新源码版本。该12次不覆盖原54的失败,也不代表新版本完整重跑54次。原交易、权限、页面、记忆和运行观察保留各自实际版本与边界。 +This subset was selected for observed failures. Original prompts, R0, reference SQL, model, and per-turn 16-call/300-second budget remain unchanged, while results record the new source version separately. These 12 attempts neither replace the original 54 failures nor establish another complete 54-attempt run. Existing transaction, permission, UI, memory, and runtime observations retain their actual versions and boundaries. -本次回归之后,仍出现的普通模型语义错误保留为分析需要复核的限制,不继续反复修改提示挑选成功。实际数据、执行或权限边界的新缺陷仍应先修复。 +Ordinary model-semantic errors remaining after this regression are reported as limitations requiring analytical review, without repeatedly tuning prompts to select successes. New defects in actual data, execution, or permission boundaries still require fixing first. diff --git a/evals/retail/acceptance-v2.md b/evals/retail/acceptance-v2.md index fe9d0c1..395a530 100644 --- a/evals/retail/acceptance-v2.md +++ b/evals/retail/acceptance-v2.md @@ -1,12 +1,12 @@ -# 零售事实表达修复后的最终验收 +# Final acceptance after retail fact-expression fixes -沿用 retail-v1 数据、R0 重置、原题和原 SQL 判定,主 Agent、分析及记忆模型继续使用 gpt-5.6-terra;主/分析沿现 Chat Completions 适配,搜索沿现 Responses。每聊天回合16次调用/300秒,主循环12轮。模型别名不是不可变快照。 +This run retains retail-v1 data, R0 reset, original tasks, and original SQL grading. Main, analysis, and memory models remain gpt-5.6-terra. Main/analysis use the existing Chat Completions adapter and search uses Responses. Each chat turn has 16 calls/300 seconds, with 12 main-loop rounds. The model alias is not an immutable snapshot. -在本组开始前记录实际完整 ShopMate/CityBuddy SHA,使用已提交源干净版本。同组不变更代码、工具、预算或判分;修改行为后结果另列,不能把不同版本的通过题拼成最终成绩。 +Before the group starts, record actual full ShopMate/CityBuddy SHAs and use committed, source-clean versions. Do not change code, tools, budgets, or grading within a group. Results after a behavior change receive a separate version; passes from different versions cannot be pooled into one final score. -先做 D04、D05、R05、R07 各一次针对性开发回归,单独保留,不并入以下最终30次。 +First run D04, D05, R05, and R07 once each as targeted development regression. Retain those separately from the final 30 attempts below. -| 任务表 | 任务 | 各重复次数 | 任务数 | +| Suite | Tasks | Repetitions each | Attempts | |---|---|---:|---:| | retail/development | D02,D07,D08,D10,D11 | 1 | 5 | | retail/development | D04,D05 | 3 | 6 | @@ -15,10 +15,10 @@ | retail/buyer-planning-development | B01,B02 | 1 | 2 | | retail/search-development | S01,S02 | 1 | 2 | -18个已知场景,共30次。重复集中于历史价/聚合追问、营销观察期间、SQL/Python,以及购物付款退款和促销批准到实际成交。不是未见题泛化率;单次覆盖的场景不能声称已测稳定成功率。每次使用原独立R0和任务数据,任务可能包含多轮聊天、工具和操作员点击,分别计数。 +There are 18 known scenarios and 30 attempts. Repetition concentrates on historical prices/aggregate follow-ups, campaign observation periods, SQL/Python, shopping/payment/refunds, and promotion approval through actual sale. This is not unseen-task generalization, and singly covered scenarios do not establish stable success rates. Every attempt uses the original independent R0 and task data. A task may contain multiple chats, tools, and operator clicks, which are counted separately. -判分继续使用原任务的 success_criteria、真实响应/卡片和权威 SQL。未批准写入、越权、重复扣款或库存错误、未解决的未知写状态阻断后续相关流程。普通模型的事实、日期、统计、引用或条件错误仍记业务失败;不以HTTP成功、工具正确或卡片出现抵销,不删除附加回答中的错误。 +Grading retains the original task's success_criteria, actual responses/cards, and authoritative SQL. Unapproved writes, unauthorized execution, duplicate charges or stock errors, and unresolved uncertain writes block subsequent dependent flows. Ordinary model errors in facts, dates, statistics, citations, or conditions remain business failures. HTTP success, correct tools, or a displayed card do not offset them; errors in additional answer content are retained too. -既有未受影响的记忆、搜索传输、沙箱、会话并发、中断/付款恢复和权限测试保留其执行版本,不重新计入30任务分母。若本次改动影响这些行为,仅补相应回归。新买家归属消融保留原新链路0/3对0/3及未触达事务校验的结论;不混用旧客服的结果。 +Existing unaffected memory, search-transport, sandbox, conversation-concurrency, interruption/payment-recovery, and permission tests retain their actual execution versions and are not added to the 30-task denominator. Only behavior affected by a change receives corresponding regression checks. The new buyer ownership ablation retains its 0/3 versus 0/3 finding and failure to reach the transaction check; old support-agent results are not substituted. -30次全部有结果且未知写已查明后收口,报告业务通过/失败、提供者故障和未运行;残留普通模型失败如实保留,不反复开启最终评测。用户等待与调用用量沿已有monotonic原件汇总,包含失败;缓存读取包含在已知输入中,缺失不填零,不推导未提供的账单费用。 +Close the batch once all 30 attempts have outcomes and uncertain writes have been resolved. Report business passes/failures, provider failures, and unexecuted tasks. Retain remaining ordinary model failures without repeatedly opening another final evaluation. Summarize waiting and usage from existing monotonic originals, including failures. Cache reads are already part of known input; missing usage is not zero, and unreported billing costs are not inferred. diff --git a/web/README.md b/web/README.md index bbc5dcc..4761d13 100644 --- a/web/README.md +++ b/web/README.md @@ -1,6 +1,6 @@ -# ShopMate 商家 Web +# ShopMate merchant Web -React + TypeScript + Vite 单页应用,Tailwind 管理视觉样式,shadcn/ui 用于按钮、对话框和菜单。页面布局采用暖米色、墨黑与朱红,业务卡片、流式协议和部分经营组件复用现有 `web-shared`;来源许可证保留。没有 Next 服务或服务端 React 渲染。 +A React + TypeScript + Vite single-page application, with Tailwind for styling and shadcn/ui for buttons, dialogs, and menus. The layout uses warm beige, ink black, and vermilion. Business cards, streaming contracts, and some merchant components reuse the existing `web-shared` package with its source license preserved. There is no Next service or server-side React rendering. ```sh npm ci @@ -9,17 +9,17 @@ npm test npm run build ``` -构建完成后重启项目根目录的 ShopMate API,访问 `http://127.0.0.1:8101/`。Python 只提供构建入口、哈希静态资源和公共商品图片,浏览器调用同源 `/api/merchant`;不存在另一个生产 Node 服务。开发时 `npm run dev` 使用 3100 端口,Vite 将 `/api` 代理到 8101。`npm run start` 仅预览静态构建,不提供业务 API。 +After building, restart the ShopMate API from the repository root and open `http://127.0.0.1:8101/`. Python serves the build entry point, hashed assets, and public product images; the browser calls same-origin `/api/merchant`. There is no separate production Node service. For development, `npm run dev` uses port 3100 and Vite proxies `/api` to 8101. `npm run start` previews static output only and does not provide business APIs. -## 操作与状态 +## Operations and state -- 登录后即可查看经营概览、商品、库存、订单及审批,不创建聊天。打开助手才创建对话,历史列表使用 `/conversations`。 -- Bearer 仅保存在内存中;刷新重新登录后,可从历史对话恢复。普通请求不发送 `X-Session-Id`,聊天显式使用 `/conversations/{id}/chat`。 -- 商品、库存、促销、营销计划的修改都先展示差异,由操作员批准。审批页不受模型生成锁阻塞;服务器返回的回执覆盖聊天里迟到的旧草案卡。 -- 关闭助手面板保留当前连接;停止生成或退出登录中断连接。断线恢复读取保存的状态,不宣称后台持续生成,也不重新批准已提交的操作。 -- 读取错误显式呈现,401 清除登录;数据缺失保留未知,历史成交价、付款、退款和履约分别显示。退款申请不表示到账,促销到期不自动恢复价格,营销计划不直接投放外部平台。 -- 记忆与对话按角色和主体隔离,助手面板支持查看、修改和忘记记忆。网页来源只展示代理实际提供的无凭证 HTTP(S) 引用。 +- Sign-in gives access to the overview, products, inventory, orders, and approvals without creating a chat. Opening the assistant creates a conversation; history uses `/conversations`. +- The bearer token stays in memory. After refreshing and signing in again, saved conversations can be restored. Ordinary requests omit `X-Session-Id`; chat explicitly uses `/conversations/{id}/chat`. +- Product, stock, promotion, and campaign changes first show their differences and require operator approval. The approval page is not blocked by model generation; authoritative receipts supersede late, stale draft cards in chat. +- Closing the assistant panel preserves the connection; stopping generation or signing out interrupts it. Reconnection reads saved state without assuming continuous background generation or approving committed operations again. +- Read errors are displayed explicitly, and 401 clears sign-in. Missing data remains unknown; historical sale prices, payment, refunds, and fulfillment are displayed separately. Refund requests do not mean funds arrived, promotions do not automatically restore prices at expiry, and campaign plans do not directly publish to external platforms. +- Memory and conversations are isolated by role and subject. The assistant panel supports viewing, editing, and forgetting memory. Web sources display only credential-free HTTP(S) citations actually supplied by the provider. -`public/products` 图片及来源文件同时供 Android 使用。买家正式入口是 [原生 Android](../android/README.md),旧 `/buyer` 页面及其组件已退出构建。少量旧浏览器传输代码保存在 `tests/fixtures`,仅用于保留原有恢复协议回归测试,不是应用入口或运行依赖。 +Images and credits in `public/products` are also used by Android. The buyer entry point is [native Android](../android/README.md); the old `/buyer` page and its components have left the build. A small amount of old browser transport code remains in `tests/fixtures` solely to preserve recovery-protocol regressions, not as an app entry point or runtime dependency. -`web-shared` 为本地 `file:` 依赖,`.npmrc` 的 `install-links=true` 复制安装源包。修改 vendor 后重新 `npm ci`。通用交互没有重新造组件框架,业务差异卡仍由本项目维护。 +`web-shared` is a local `file:` dependency; `.npmrc` sets `install-links=true` to copy the source package during installation. Run `npm ci` again after vendor changes. Common interactions reuse existing components; this project maintains the business-specific difference cards.