From e12d7eec64403f7aeaf8da6373657efef1473aa5 Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 11 Jul 2026 16:55:19 -0700 Subject: [PATCH 1/3] pstack: add create-verification-skill and maintain-verification-skill Generalizes the control-glass approach (feature map, doctor, proof standards, harness-first) for any language or platform. The generator interviews the repo, writes a project-local verify skill + seeded feature map, and must prove its own output by running it once. The maintainer is the upkeep loop: source wave per feature, one live pass, at most one PR. setup-pstack gains an optional final step offering the generator. Validated by 4 cloud agents generating against real repos (go TUI, node CLI, HTTP service, full-stack web app) - all four proof runs passed, and their friction reports drove 6 revisions. --- pstack/.cursor-plugin/plugin.json | 2 +- pstack/README.md | 2 + .../skills/create-verification-skill/SKILL.md | 44 +++++++++++++++++++ .../maintain-verification-skill/SKILL.md | 37 ++++++++++++++++ pstack/skills/setup-pstack/SKILL.md | 4 ++ 5 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 pstack/skills/create-verification-skill/SKILL.md create mode 100644 pstack/skills/maintain-verification-skill/SKILL.md diff --git a/pstack/.cursor-plugin/plugin.json b/pstack/.cursor-plugin/plugin.json index 6606378..ec2d640 100644 --- a/pstack/.cursor-plugin/plugin.json +++ b/pstack/.cursor-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "pstack", "displayName": "pstack", - "version": "0.10.4", + "version": "0.11.0", "description": "if you want to go fast, go deep first. pstack helps you write less, but higher quality code. rigorous agent workflows you can parallelize with confidence.", "author": { "name": "Lauren Tan" diff --git a/pstack/README.md b/pstack/README.md index 434c837..0509394 100644 --- a/pstack/README.md +++ b/pstack/README.md @@ -116,6 +116,8 @@ the full rules and playbooks live in [`skills/poteto-mode/SKILL.md`](./skills/po | [`/typescript-best-practices`](./skills/typescript-best-practices/SKILL.md) | you're reading or editing typescript. grounds the type-system-discipline principle in syntax. | | [`/figure-it-out`](./skills/figure-it-out/SKILL.md) | no bundled playbook fits. designs a rigorous, auditable playbook for the task. | | [`/show-me-your-work`](./skills/show-me-your-work/SKILL.md) | you want a reviewable decision trail. logs decisions to a tsv you can commit. | +| [`/create-verification-skill`](./skills/create-verification-skill/SKILL.md) | your project has no scripted way to prove app behavior. generates a project-local verify skill with a feature map, for any language or platform. | +| [`/maintain-verification-skill`](./skills/maintain-verification-skill/SKILL.md) | your verify skill's feature map has drifted from the app. source wave + one live pass, at most one PR of proven corrections. | | [`/unslop`](./skills/unslop/SKILL.md) | you're cleaning up writing. removes AI tells. | diff --git a/pstack/skills/create-verification-skill/SKILL.md b/pstack/skills/create-verification-skill/SKILL.md new file mode 100644 index 0000000..59434ac --- /dev/null +++ b/pstack/skills/create-verification-skill/SKILL.md @@ -0,0 +1,44 @@ +--- +name: create-verification-skill +description: "Generate a project-local verification skill that drives your app the way a user does — any language, framework, or platform. Use for /create-verification-skill, \"make a control skill for this repo\", or when a project has no scripted way to prove UI/CLI/service behavior." +disable-model-invocation: true +--- + +# Create a verification skill + +Every serious project needs a scripted way to drive the real app and prove behavior: launch it, exercise a feature the way a user would, and capture evidence. This skill generates that as a project-local skill (`.cursor/skills/verify-/`) tailored to the repo. You write the generator's output for the next agent, not for a human: it will be read cold, mid-task, by an agent that has never seen the app. + +## 1. Interview the repo, not the user + +Answer these from the codebase and only ask the user what you cannot observe: + +- **Surface:** what does a user actually touch? A web UI, a CLI/TUI, a desktop app, an API, a mobile app, a library? A repo can have several; pick the primary one and note the rest. +- **Run:** how does the app start locally? Prefer the repo's own documented dev command (package scripts, Makefile, README quickstart). Note ports, env vars, seed data, auth. +- **Drive:** how can an agent interact with it programmatically? Existing harnesses first — Playwright/Cypress specs, expect scripts, PTY helpers, curl-able endpoints, a debug port. Only then pick a generic recipe: browser/CDP for web and Electron, a tmux/PTY harness for CLI/TUI, plain HTTP for services. +- **Observe:** what evidence can be captured? Screenshots, terminal transcripts, response bodies, logs, exit codes, DB state. +- **Isolate:** can two instances run side by side (ports, data dirs, profiles)? If not, say so in the generated skill: refusing to double-drive a shared instance beats corrupting the user's session. + +If the checkout doesn't build or start as-is, fix that first (or report it precisely) before generating; a skill written against a broken base teaches wrong steps. When an irrelevant missing asset blocks startup (a static dir the API never serves, a sample config), the generated skill may create it, clearly marked as verification scaffolding, and remove it in cleanup. + +## 2. Generate the skill + +Write `.cursor/skills/verify-/SKILL.md` with these sections, each grounded in what the interview actually found (no placeholders left): + +- **Launch:** the exact command that starts the app for verification, and how to tell it's ready (a log line, a port answering, a prompt). Include teardown. For a short-lived CLI or TUI there is no server to keep alive: launch means build the binary (or install deps) once, then start each drive in its own isolated PTY or tmux session. +- **Doctor:** one read-only check that answers "is this instance worth driving?" — process up, right version/build, port owned by us, auth valid. An agent runs this first whenever anything looks off. +- **Drive:** the harness recipe with real selectors/commands from this repo, not examples. Prefer stable handles (ARIA labels, data attributes, prompt strings, route paths) over coordinates and tab order. +- **Evidence:** what to capture for a proof and where it goes. State the proof standards: exercise the real user path, not internal setters or test-only endpoints; capture the action and the resulting state, not just the final screen; verify side effects (files written, rows inserted, messages sent) alongside what's visible; mocks only where a production boundary already isolates the external system. When the safe path is a dry-run or test mode, verify what it actually skips by observing (files, network, git refs) rather than trusting its name: some dry-runs still touch the network or open a browser. +- **Cleanup:** how to tear down instances the run created. Never kill by process name; kill what you started. Cleanup removes instances and scratch state, never the evidence: proof artifacts survive the teardown, in a location the skill names. +- **Helpers:** any script the skill ships is executable and its invocation is shown in the skill body. A helper the reader has to reverse-engineer is not a helper. + +## 3. Seed the feature map + +Create `.cursor/skills/verify-/features/README.md` plus one file per user-facing feature you can identify (aim for the top 3-5 to start, from routes, commands, menus, or docs). Each file answers, from the user's point of view: what the feature is, how to reach it, how to drive it with the harness, and what observable end state proves it works. The map is the repo's maintained verification source; a proof that drives one convenient entry point is incomplete when the map lists others. + +## 4. Prove the generated skill before handing it over + +Run its own instructions end to end once: launch, doctor, drive ONE mapped feature (one is enough; the map exists so later runs can cover the rest), capture evidence, clean up. Fix what fails. A generated skill that was never executed is a draft, not a deliverable. + +## 5. Offer the maintenance loop + +Point the user at `/maintain-verification-skill` for keeping the map honest as the app changes. Suggest a cadence only if they ask. diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md new file mode 100644 index 0000000..140f930 --- /dev/null +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -0,0 +1,37 @@ +--- +name: maintain-verification-skill +description: "Periodic pass that keeps a project's verification skill and feature map honest: parallel source readers per feature, one live session driving every feature, at most one PR of proven corrections. Use for /maintain-verification-skill or \"audit the verify skill\"." +disable-model-invocation: true +--- + +# Maintain a verification skill + +A feature map rots the moment the app changes. This skill is the upkeep loop for a skill generated by `/create-verification-skill` (or any project-local verification skill with a feature map). The unit of rigor is the feature, not every sentence: cover every feature file from source and exercise every feature live, without terminalising every bullet. + +## Outcomes + +Pick one, and say which: + +- **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. +- **changed** — one PR ships proven doc, harness, or map corrections. +- **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. + +## Edit scope + +Only edit the verification skill's own directory (its SKILL.md, features/, and any harness scripts it owns). Never edit product code during a run: a behavior the map describes that the app no longer does is either doc drift (fix the map) or a product regression (report it, don't paper over it in docs). + +## Pass + +1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory. + +2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe. + +3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. + +4. **Live pass.** Required even when source looks clean. One coordinator-owned app instance: launch per the verification skill, run its doctor first. Drive the recipes serially against that instance; reuse state, but exercise every feature at least once. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. + +5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it. App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. + +6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. + +Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. diff --git a/pstack/skills/setup-pstack/SKILL.md b/pstack/skills/setup-pstack/SKILL.md index 7774223..08d9c28 100644 --- a/pstack/skills/setup-pstack/SKILL.md +++ b/pstack/skills/setup-pstack/SKILL.md @@ -56,3 +56,7 @@ interrogate reviewers: claude-opus-4-8-thinking-xhigh, gpt-5.5-high-fast, grok-4 ### 6. Confirm Tell the user the rule was written and that it applies to new sessions. Re-running this skill updates it. + +### 7. Offer a verification skill (optional) + +Check whether the project has a way to drive the real app for proof (a `verify-*` skill, or an existing harness). If not, offer once: "want a project-local verification skill, so agents can drive the app the way a user does and prove changes work? I can generate one with /create-verification-skill." On yes, read and follow `create-verification-skill`. On no, move on without pushing. From cbb153917105622910f4d84be945ba6e22d04c61 Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 11 Jul 2026 17:04:02 -0700 Subject: [PATCH 2/3] Address bugbot: frontmatter spec, teardown, launch-model deference, target discovery --- pstack/skills/create-verification-skill/SKILL.md | 4 ++-- pstack/skills/maintain-verification-skill/SKILL.md | 6 ++++-- pstack/skills/setup-pstack/SKILL.md | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/pstack/skills/create-verification-skill/SKILL.md b/pstack/skills/create-verification-skill/SKILL.md index 59434ac..db06674 100644 --- a/pstack/skills/create-verification-skill/SKILL.md +++ b/pstack/skills/create-verification-skill/SKILL.md @@ -22,7 +22,7 @@ If the checkout doesn't build or start as-is, fix that first (or report it preci ## 2. Generate the skill -Write `.cursor/skills/verify-/SKILL.md` with these sections, each grounded in what the interview actually found (no placeholders left): +Write `.cursor/skills/verify-/SKILL.md` with YAML frontmatter (`name: verify-` and a `description` that names the app, the surface, and when to reach for it — without frontmatter the skill never registers) and these sections, each grounded in what the interview actually found (no placeholders left): - **Launch:** the exact command that starts the app for verification, and how to tell it's ready (a log line, a port answering, a prompt). Include teardown. For a short-lived CLI or TUI there is no server to keep alive: launch means build the binary (or install deps) once, then start each drive in its own isolated PTY or tmux session. - **Doctor:** one read-only check that answers "is this instance worth driving?" — process up, right version/build, port owned by us, auth valid. An agent runs this first whenever anything looks off. @@ -37,7 +37,7 @@ Create `.cursor/skills/verify-/features/README.md` plus one file per user-f ## 4. Prove the generated skill before handing it over -Run its own instructions end to end once: launch, doctor, drive ONE mapped feature (one is enough; the map exists so later runs can cover the rest), capture evidence, clean up. Fix what fails. A generated skill that was never executed is a draft, not a deliverable. +Run its own instructions end to end once: launch, doctor, drive ONE mapped feature (one is enough; the map exists so later runs can cover the rest), capture evidence, clean up. After cleanup, confirm the evidence still exists at the named location — a cleanup that eats the proof fails this step. Fix what fails, and run the generated cleanup after every failed iteration too, so broken attempts don't strand processes and ports. A generated skill that was never executed is a draft, not a deliverable. ## 5. Offer the maintenance loop diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index 140f930..aa12a26 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -22,15 +22,17 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a ## Pass +0. **Locate the target.** Find the verification skill to maintain: the project-local skill whose body has launch/drive sections and a feature map (usually `.cursor/skills/verify-*/`). Several candidates → ask which one; none → stop and point at `/create-verification-skill` instead of inventing a target. + 1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory. 2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe. 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. -4. **Live pass.** Required even when source looks clean. One coordinator-owned app instance: launch per the verification skill, run its doctor first. Drive the recipes serially against that instance; reuse state, but exercise every feature at least once. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. +4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Run its doctor first and don't drive until doctor passes; a failing doctor is `blocked`, not a license to drive an unhealthy instance. Exercise every feature at least once. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. When the pass ends — clean, changed, or blocked — run the verification skill's cleanup so no instances, browsers, or databases outlive the run (evidence stays, per the skill). -5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it. App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. +5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. 6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. diff --git a/pstack/skills/setup-pstack/SKILL.md b/pstack/skills/setup-pstack/SKILL.md index 08d9c28..9a77f29 100644 --- a/pstack/skills/setup-pstack/SKILL.md +++ b/pstack/skills/setup-pstack/SKILL.md @@ -59,4 +59,4 @@ Tell the user the rule was written and that it applies to new sessions. Re-runni ### 7. Offer a verification skill (optional) -Check whether the project has a way to drive the real app for proof (a `verify-*` skill, or an existing harness). If not, offer once: "want a project-local verification skill, so agents can drive the app the way a user does and prove changes work? I can generate one with /create-verification-skill." On yes, read and follow `create-verification-skill`. On no, move on without pushing. +Check whether the project has a way to drive the real app for proof (a `verify-*` skill, or an existing harness). If not, offer once: "want a project-local verification skill, so agents can drive the app the way a user does and prove changes work? I can generate one with /create-verification-skill." On yes, invoke `/create-verification-skill` (resolves wherever pstack is installed — workspace, user, or plugin). On no, move on without pushing. From 45f66012b3806f6618cf5895275b91f070cec21f Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 11 Jul 2026 17:10:25 -0700 Subject: [PATCH 3/3] Rewrite live pass: per-session health checks, doctor-drift retry, per-failure cleanup, teardown after re-proof --- pstack/skills/maintain-verification-skill/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index aa12a26..61c7560 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -30,7 +30,7 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. -4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Run its doctor first and don't drive until doctor passes; a failing doctor is `blocked`, not a license to drive an unhealthy instance. Exercise every feature at least once. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. When the pass ends — clean, changed, or blocked — run the verification skill's cleanup so no instances, browsers, or databases outlive the run (evidence stays, per the skill). +4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR.