Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .devin/wiki.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"repo_notes": [
{
"content": "Use current source and README.md, docs/EXPERIMENTS.md and each result record as authority. StateEval is an independent evaluation harness, not the transaction service or a model runtime. Acting through agent/business endpoints and judging through a separate read-only SQL account are different paths; /api/eval/state and audit endpoints are diagnostic, not the oracle. Core contains no CityBuddy imports or SQL; adapters own business knowledge. Preserve hard-gate order: final business state, forbidden side effects, permission violations. Treat tool trajectories as diagnostics. Keep the historical support-agent campaign and the current ShopMate buyer calibration separate, each with its original code versions, model configuration, task set and denominator. Historical 55/300 versus 0/300 does not describe current ShopMate; current foreign-order pairs were 0/3 versus 0/3 and did not reach a sensitive write, so they do not demonstrate incremental ownership-check effectiveness. Two own-order positive controls passed; repeated confirmation is not another model trial. A REQUESTED refund is an accepted request, not funds arrival. Never imply model aliases are immutable snapshots, ablations affect production, or local pilot results prove general safety."
}
],
"pages": [
{
"title": "Overview and evaluation design",
"purpose": "Explain the authorization question, independent acting/judging paths, gate order and relationship to external CityBuddy and ShopMate."
},
{
"title": "Local setup and reproducible runs",
"purpose": "Document the current Makefile, pinned runtime, sibling checkouts, positive controls, isolated fixtures, source-clean measured revisions and safe cleanup from active guides/scripts."
},
{
"title": "Core engine and target adapters",
"purpose": "Describe core task/turn/assertion/gate types, adapter seam, trial execution, CityBuddy and ShopMate adapters, real buyer factories and independent SQL grading."
},
{
"title": "Campaigns and result interpretation",
"purpose": "Explain ownership/session experiments, equal-budget controls and denominators. Keep historical formal results, current buyer calibration and positive controls separate; include non-activated and inconclusive outcomes."
},
{
"title": "Artifacts, SQL evidence and failure handling",
"purpose": "Describe raw transcripts, independent before/after SQL, outcome files, retained-fixture flags and operational failures without treating tool logs or diagnostic APIs as business truth."
},
{
"title": "Testing, methodology and contribution",
"purpose": "Cover boundary checks, unit/integration tests, Actions pinned dependencies, contribution rules and prior-art references. Explain limits and reproduction without fabricating coverage or results."
}
]
}
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
![StateEval · Evaluate agent actions through independent business-state checks](docs/assets/cover.png)

# StateEval

**English** · [简体中文](README.zh-CN.md) · [Contributing](CONTRIBUTING.md)

[![check](https://github.com/ChanTso/state-eval/actions/workflows/check.yml/badge.svg?branch=main)](https://github.com/ChanTso/state-eval/actions/workflows/check.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ChanTso/state-eval)

**Let the agent act. Use independent SQL to check what it actually changed.**

Expand Down
3 changes: 1 addition & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
![StateEval · 从独立业务终态判断 Agent 行为](docs/assets/cover.png)

# StateEval

[English](README.md) · **简体中文** · [贡献指南](CONTRIBUTING.md)

[![check](https://github.com/ChanTso/state-eval/actions/workflows/check.yml/badge.svg?branch=main)](https://github.com/ChanTso/state-eval/actions/workflows/check.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ChanTso/state-eval)

**让 Agent 执行业务,用独立 SQL 检查它最终做了什么。**

Expand Down
Binary file removed docs/assets/cover.png
Binary file not shown.
1 change: 0 additions & 1 deletion docs/assets/cover.svg

This file was deleted.