Repository navigation
Commit be8561d
feat(qa): AI 可执行的平台功能标准测试清单 — 15 域 178 项 + 能力覆盖 ratchet + CI 门禁 (#6557)
* feat(qa): 平台功能标准测试清单 — AI 可执行的常备台账 (docs/qa/platform-checklist)
Release verification lived in one-off shapes — a checkbox issue per release
(#3358) and a hand table (docs/plans/release-15.1-test-plan.md) — that could
not accumulate: items were not reusable across releases, ticks had no revision
to pin against, and every fixture gap a sweep found (#3408/#3409/#3415) was
rediscovered from prose. This lands the standing, machine-readable successor:
- areas/*.json — 9 areas, 32 items seeded from the #3358 evidence runs. Each
item: clause-grained acceptance (clause + oracle + verify + evidence),
declared fixtures with knownGaps, negative sides (both sides of every gate),
false-positive traps, immutable <area>.<slug> ids, revision + history for
append/change governance, `automated` refs for the 🤖 lane.
- README.md — item anatomy; append/change/retire lifecycle (append-only, never
delete); release sweep = filter over the ledger (since:vN ∪ P0); relationship
to dogfood-verification, @objectstack/verify, showcase coverage.ts, e2e-live,
and why spec's inert TestScenarioSchema is deliberately not reused.
- RUNNER.md — AI execution protocol: verdict vocabulary with first-class
`partial`/`blocked`, oracle hierarchy (server truth > screenshot > DOM),
fail requires reproduction ×2 + automation self-check + filed issue, trap
vocabulary distilled from the #3358 retractions, durable run records under
runs/ pinned to item revisions.
- scripts/check-platform-checklist.mjs (+ lint.yml gate, house style):
structural invariants only — unique immutable ids, revision == last history
entry, every acceptance clause names its oracle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 补齐清单缺失能力域 — ai 与 integration-system (11 域 46 项)
Closes the two coverage holes the seed import left: nothing covered the AI
metadata kinds (agent/tool/skill, MCP surfaces) or the integration/system
services (declarative connectors, webhooks, jobs, email templates).
- areas/ai.json — agent/tool/skill metadata round-trip (variants matrix),
MCP HTTP transport both-sides (enabled 501/off + /mcp/skill public),
stdio fail-closed + RLS/FLS parity (from #3358 §9), run_action
ai.exposed gate + audit (15.1 §A9), validate_expression. Showcase ships
no AI seeds (ADR-0063) — fixture requirements declared explicitly.
- areas/integration-system.json — declarative connector lifecycle from the
15.1 §B rows (#2994/#3062 boot materialization, #3049 degraded husk +
atomic recovery, #3059 stdio default-deny allowlist, #3024 spec-path
escape rejection, #2985 descriptor-only boot audit, objectui#2563
designer picker), webhook live-fire + retired-trigger build gate, job
scheduled run, email-template variable rendering.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 深化 dashboards 与 i18n 域到深测契约 (rev 2)
First two areas rewritten to the deep-test contract: concrete
draft-door/API steps, clause-grained acceptance citing the exact spec
sources (dashboard.zod.ts LEGACY key list as a variants matrix,
chart.zod.ts wrong-layer guidance, translation.zod.ts metadataForms
group), rejected-draft-not-persisted checks, per-recipient locale
controls, and round-trip/symmetry clauses. Existing items bumped to
revision 2 with history entries; new items added per area.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 深化 identity-auth/automation/access-security/records-forms 域(检查点)
Progressive deepening checkpoint — validator green at 11 areas / 74 items.
identity-auth complete (auth-method matrix from AuthPluginConfigSchema +
PUBLIC_AUTH_FEATURES, delegated-admin invitation gates ADR-0105 D8,
admin lifecycle ops with audit attribution); automation and
access-security rewritten to the deep contract (ExecutionStepLogSchema
region tags, runRlsProofs delegation with skip-audit clause, readonlyWhen
locks, owner-anchor forge/transfer/disown probes); records-forms
in progress by its agent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 深化 platform-core 与 approvals 域(检查点,80 项)
platform-core: /ready probe, authored-nav-vs-served diff, new
seed-integrity item (authored row baselines, verbatim value spot-diff,
upsert idempotence — the #3408/#3415 silent-seed class as first-class
FAILs), metadata registry serving + draft→publish round-trip with deny
gates. approvals: per-group signoff now pins the exact seed wiring
(seed-approval-demo.ts distinct holders, run resume down the approve
edge, actor derived from session #3800), quorum keeps its fixture
blocker plus a runnable clamp-contrast clause.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 平台测试清单缺口全排查 — 补齐至 170 项,修正 3 处错误豁免
Five-angle capability-coverage sweep (console UI surfaces · spec enums ·
routes/settings · built-in apps page-by-page · docs claims) closed a class
of systematic omissions and grew the checklist 84 → 170 items across 15
areas, authored by 8 area-partitioned writer agents grounded in real source.
Headline gaps closed:
- The three built-in console apps (Setup/Studio/Account) had ZERO nav
coverage — nav-surfaces-render only swept the showcase app. Added
builtin-apps-nav-render (P0) + settings-hub-roundtrip, session-list-revoke,
api-key-ui-lifecycle, org/team management, oauth/consent, audit-log-browser,
notification-center, package-lifecycle, app-management, and more.
- Three coverage.json waivers were FACTUALLY STALE and are corrected to real
items: api (showcase authors 2 live declarative endpoints), datasource
(read-only SQLite federation fixture), mapping (showcase_inquiry_feed).
Ratchet now 25 kinds mapped / 3 waived (book/doc display-only, hook indirect).
- sharing-rules-widen (the marquee gap): criteria rules → materialized
sys_record_share, explain attribution, §A13 seed-not-clobber.
- Enumerable-surface matrices previously missing: validation rule types (6),
action locations (7, ACTION_LOCATIONS), gantt/kanban/calendar drag
persistence, bulk select-all-matching, adhoc filter/sort builder, grid
personalization, lookup create-new, concurrent-edit OCC-409, discussion
@mentions, and route-ledger-live-parity extended to all ledgers (#3361 class).
Every item stays honest: 8 blocked items name the missing fixture rather than
faking coverage; product defects found while grounding (saved-report schedule
cross-owner delete, AppManagement client-only stubs, undo without ifMatch) are
encoded as expected-fail probes and catalogued in FOLLOW-UPS.md for maintainer
decision, alongside docs-drift findings (retired capabilities still advertised).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): 变体新鲜度门 + hook/页面/字段下钻 + sweep 自动化配方(177 项)
Automation to keep the checklist current with minimal human effort, plus a
deep-drill round on the surfaces a maintainer asked about (objects, field
types, hooks, custom pages, cascading fields).
Automation:
- scripts/check-platform-checklist.mjs gains a VARIANTS-FRESHNESS ratchet:
a matrix item pins the spec enum it was authored against via `enumSource`
{file, export, expect}; the validator extracts the enum's current member
count from spec source and fails CI when it drifts, with a precise fix
instruction. Closes the gap the kind-level coverage ratchet left — the
latter catches a NEW metadata kind, enumSource catches a NEW value inside
an existing kind's enum. Pinned: field types (49), chart types (20),
action locations (7), webhook triggers (5), flow node types (20).
- SWEEP.md — the five-angle gap-hunt as a committed runbook, so "跑一轮
coverage sweep" is an AI-executable refresh (the human-effort half); the
deterministic gates are the zero-human half.
Deep-drill items (the 4th stale waiver corrected → coverage 26 mapped / 2
waived, only book/doc display-only remain):
- records-forms.object-hook-lifecycle (un-waives `hook` — showcase ships 4
observable hooks): 6 lifecycle events, CEL previous-vs-record transition
semantics (#4770/#4784), async+retry, onError abort-rollback vs log, priority.
- studio-authoring custom-page items: declarative block composition + data
binding, page variables + action interpolation, html-parsed vs react-executed
source tiers with the OS_PAGE_REACT capability gate.
- records-forms field drills: per-type write-path constraint enforcement (22
variants, exact ADR-0114 codes), multilevel cascade clear-semantics, encrypted
field fail-closed. Two server-side integrity gaps found → FOLLOW-UPS D7/D8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): coverage-sweep 内部 skill — 一句话手动触发清单缺口重扫
Manual trigger for the five-angle checklist gap hunt: the maintainer says
"跑一轮 coverage sweep" (or asks "X 有测试吗" needing verification) and the
AI session loads the orchestration contract — worktree-first, five read-only
hunters in parallel, per-area writer partitioning, central coverage.json
reconciliation with waiver re-audit, validator-green landing, FOLLOW-UPS for
defects. Canonical method stays in docs/qa/platform-checklist/SWEEP.md; the
skill is the trigger + contract, not a second copy. metadata.internal: true
per template-consistency (sibling to dogfood-verification/pm-dispatch).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* docs(qa): FOLLOW-UPS 头部措辞随提交状态更新
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* docs(qa): 首次真实执行记录 — platform-core.console-login (pass) + 证据
Demonstrates the runs/ format end-to-end: an opus runner drove the item in
headless Chromium against an isolated showcase boot. Verdict pass across all
6 clauses + the wrong-password negative, each clause pinned to a server-truth
oracle (sign-in/get-session status, on-the-wire 401 deny side) and a captured
screenshot. runs/evidence/2026-08-08-console-login/ carries the 8 screenshots
+ 2 network traces.
The record also shows the protocol earning its keep: the runner's automation
self-check (RUNNER rule 2) caught a false "dead shell" P0 — a cookie-only clear
left the console authed via its localStorage bearer token; only a full
credential clear produced the real redirect-to-login. And it surfaced a genuine
run-time precondition (the vendored /_console dist builds separately from the
showcase workspace closure), now recorded in the env block rather than faked
around.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* docs(qa): 测试结果不进仓库 — runs/ 改为格式契约 + git-ignore 结果
Per maintainer decision: a run record is OUTPUT about one build, not source.
Removed the committed console-login run record + evidence PNGs, git-ignore
runs/* (keeping only README + .gitignore), and rewrote the RUNNER/README
guidance so results live in the executing environment (CI artifact / tracking
issue / external QA store), never in git. The checklist under areas/ stays the
durable version-controlled contract; a run is a transient assertion about a
build and belongs with that build's other artifacts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* feat(qa): checklist-run 执行 skill + 选择器解析脚本
The execution counterpart to coverage-sweep (which authors items). Lets the
maintainer test a specified feature point or feature FILE on demand:
- scripts/checklist-select.mjs — deterministic selector resolver. Turns an item
id / area: / capability: / priority: / surface: / since:vN / **file:<path>**
into the concrete runnable item set. `file:` maps a framework source file to
the items whose `source[]` cites it ("test whatever covers this file"), with
narrowest-useful precedence (exact/basename before dir-level). Blocked items
excluded by default. Carries a --self-test (12 cases) now wired into the
check:platform-checklist gate, so the resolver stays healthy under the same CI.
- .claude/skills/checklist-run/SKILL.md — the runner skill (internal, marked
internal:true per template-consistency.test.ts). Trigger + selection +
isolation/parallelism contract; defers the judging protocol to RUNNER.md and
the boot know-how to dogfood-verification. Plans the run by surface (build the
vendored console only for browser/mixed items), isolates per item (own port +
file DB), dispatches opus runner subagents few-at-a-time for browser items,
and keeps results OUT of the repo per the runs/ policy.
README documents the two-skill split (coverage-sweep authors, checklist-run
executes) and the selector grammar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* chore(qa): 平台测试清单校验改为定期手动,从 CI 移除(维护者决定)
By maintainer decision the platform test checklist is a QA ledger, not a
per-PR code gate. Removed the `check:platform-checklist` step from lint.yml so
no unrelated PR is ever blocked by checklist drift; the script stays for manual
/ periodic use (`pnpm check:platform-checklist`), run before a release or after
a large platform surface lands. README + the lint.yml note updated to say so.
Trade-off understood and accepted: a NEW metadata kind landing on main (like
`capability` did) will no longer fail a PR — it surfaces at the next manual run
instead. The coverage ratchet still exists; it just isn't a blocking gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
* docs(qa): README 记录「运行节奏」并全面校准去-CI 后的措辞
Added an 'Operating cadence — when to run this (it is NOT in CI)' section to
the checklist README: run `pnpm check:platform-checklist` before a release,
after a large platform surface lands, when touching the checklist, or beside a
coverage-sweep/checklist-run — with the trade-off (a new kind/enum that lands
between runs is caught at the next manual run, not on merge) spelled out and
the one-line re-add if latency ever matters more than PR independence.
Corrected every now-stale "fails CI" reference across README, coverage.json
$comment, and 4 area-file history entries to say the manual check flags/catches
the drift instead of blocking a PR. Validator green (178 items, 27/2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD9f6FYyMraUWYeJf53V43
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 64d764e commit be8561d
28 files changed
Lines changed: 17400 additions & 0 deletions
File tree
- .claude/skills
- checklist-run
- coverage-sweep
- .github/workflows
- docs/qa/platform-checklist
- areas
- runs
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
196 | 204 | | |
197 | 205 | | |
198 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
0 commit comments