From 809c189ed4911d5fa4e00d0e3e0190fbed7e4025 Mon Sep 17 00:00:00 2001 From: Jennifer Power Date: Fri, 28 Aug 2026 08:39:16 -0400 Subject: [PATCH 1/3] feat: stage-aware schema and jobs-to-be-done model Reframe the PRD model around jobs-to-be-done: job executors replace personas, desired-outcomes replace KPIs, journeys replace workflows. Add stage-aware constraints (ACs required only at Ready, FR `satisfies` required at Ready) and an in-file FR/journey coverage floor enforced by `cue vet`. Add valid/invalid conformance testdata. Assisted-by: Claude Code --- README.md | 89 ++++++++---------- examples/parent-prd.yaml | 24 ++--- examples/phase-prd-draft.yaml | 22 +++++ examples/phase-prd.yaml | 23 ++--- module/AGENTS.md | 18 +++- prd.cue | 94 +++++++++++-------- testdata/invalid/bad-outcome-id.yaml | 6 ++ .../invalid/executor-missing-core-job.yaml | 4 + testdata/invalid/journey-uses-persona.yaml | 9 ++ testdata/invalid/orphan-fr.yaml | 10 ++ testdata/invalid/parent-has-kpis.yaml | 6 ++ testdata/invalid/phase-has-outcomes.yaml | 11 +++ .../invalid/ready-fr-missing-satisfies.yaml | 10 ++ testdata/valid/parent-valid.yaml | 19 ++++ testdata/valid/phase-ready-valid.yaml | 23 +++++ 15 files changed, 254 insertions(+), 114 deletions(-) create mode 100644 examples/phase-prd-draft.yaml create mode 100644 testdata/invalid/bad-outcome-id.yaml create mode 100644 testdata/invalid/executor-missing-core-job.yaml create mode 100644 testdata/invalid/journey-uses-persona.yaml create mode 100644 testdata/invalid/orphan-fr.yaml create mode 100644 testdata/invalid/parent-has-kpis.yaml create mode 100644 testdata/invalid/phase-has-outcomes.yaml create mode 100644 testdata/invalid/ready-fr-missing-satisfies.yaml create mode 100644 testdata/valid/parent-valid.yaml create mode 100644 testdata/valid/phase-ready-valid.yaml diff --git a/README.md b/README.md index 58c1d11..a7e2828 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,19 @@ # Intake Kit -A structured product requirements intake toolkit. CUE-validated PRD authoring paired with a multi-agent AI review -council that gates PRD advancement from Draft to Review. +A structured product requirements intake toolkit: CUE-validated PRD authoring paired with a multi-agent AI review +council that gates a PRD from Draft to Ready. Installs as a [Lola](https://github.com/LobsterTrap/lola) module and works with Claude Code, Cursor, Gemini CLI, and OpenCode. ## What's in the box -- **CUE schema** (`prd.cue`) — validates PRD YAML documents. Enforces structure for stakeholders, functional - requirements, acceptance criteria, workflows, and lifecycle state. -- **prd-review skill** (`module/skills/prd-review/`) — a Lola skill that checks CUE schema conformance, then dispatches - 5 specialist review agents (`module/agents/prd-*.md`) against a PRD family and emits a BLOCKED / NEEDS REVISION / - APPROVED verdict. -- **Example PRDs** (`examples/`) — parent and phase templates ready to copy and fill in. +- **CUE schema** (`prd.cue`) — validates PRD YAML: stakeholders, functional requirements, journeys, acceptance + criteria, and lifecycle state. +- **prd-review skill** (`module/skills/prd-review/`) — checks schema conformance, then dispatches review agents against + a PRD family and emits a BLOCKED / NEEDS REVISION / APPROVED verdict. A default Guard-only pass gates nothing; + `--full` runs the 5-agent council and is the only pass that gates `Draft → Ready`. +- **Example PRDs** (`examples/`) — parent and phase templates to copy and fill in. ## Install @@ -21,73 +21,60 @@ OpenCode. lola install github.com/unbound-force/intake-kit ``` -## PRD Document Model +## Usage -PRDs are structured YAML. A **parent PRD** defines the initiative (title, description, personas, NFRs, scope). **Phase -PRDs** reference the parent and carry delivery-specific fields (FRs with ACs, workflows, state). - -``` -parent-prd.yaml # what + why -├── phase1-prd.yaml # phase 1 FRs, workflow, state -└── phase2-prd.yaml # phase 2 FRs, workflow, state -``` - -See `examples/` for the full field set. - -## Validating PRDs - -Requires [CUE](https://cuelang.org/docs/introduction/installation/). +Validate a PRD against the schema (requires [CUE](https://cuelang.org/docs/introduction/installation/)): ```bash cue vet prd.cue -d '#PRDDocument' your-prd.yaml ``` -## Running the Review Council - -The `prd-review` skill runs in any AI coding assistant that supports Lola skills. +Run the review council (in any AI assistant that supports Lola skills): ``` -/prd-review prds/my-feature.yaml prds/my-feature-phase1.yaml +/prd-review prds/my-feature.yaml prds/my-feature-phase1.yaml # Guard-only, gates nothing +/prd-review --full prds/my-feature.yaml prds/my-feature-phase1.yaml # full council, gates Draft → Ready ``` -**Phase 0 is schema conformance.** Before any content review, the PRD family is checked against the CUE schema -(`#PRDDocument`), resolved from the CUE Central Registry first, falling back to a local file if you pass -`--schema `. A structural violation (bad ID format, invalid enum, disallowed field) blocks immediately with the -raw `cue vet` error — no point reviewing behavior in a file that doesn't even parse against the schema. +## PRD document model -**Phases 2 – 4 dispatch 5 specialist agents** to review content and quality, by default in parallel (pass `--serial` to run them -sequentially in one context instead — slower, but roughly 1/5th the token cost, since the PRD text is read once -instead of once per agent): +PRDs are structured YAML. A **parent PRD** defines the initiative (title, description, job executors, NFRs, scope) and +carries `desired-outcomes` — solution-agnostic, measurable needs. **Phase PRDs** reference the parent and carry +delivery fields (FRs, journeys, state; ACs required only at Ready). An FR `satisfies` a desired outcome (required at +Ready); an NFR may optionally. -| Agent | Reviews | -|---|---| -| Guard | Intent fidelity, scope discipline, persona/ID/workflow integrity, FR-to-value traceability | -| Adversary | Security gaps (auth boundaries, trust, credential scope) + ambiguity/completeness | -| Tester | Behavioral language, testability, AC quality | -| Operator | Implicit deployment/environment/connectivity assumptions | -| Curator | Evidence capture, audit trail, retention, provenance, open-question hygiene | +``` +parent-prd.yaml # what + why +├── phase1-prd.yaml # phase 1 FRs, journeys, state +└── phase2-prd.yaml # phase 2 FRs, journeys, state +``` -Each agent returns a JSON verdict; a Verify phase confirms every finding's evidence is a literal quote from the PRD -before it's allowed into the report. The council then produces a consolidated finding report with severity levels -(BLOCKER, WARNING, INFO) and a verdict that gates PRD state advancement from Draft to Review. +`cue vet` validates each file independently (structure + in-file FR/journey coverage). Cross-document checks — a +journey executor missing from the parent, a `satisfies` reference naming no real outcome, and similar — need reasoning +across files and are owned by the review council's Guard agent, not the schema. -See `module/agents/prd-*.md` for each agent's full review criteria, and `module/skills/prd-review/SKILL.md` for the -full phase-by-phase process. +See `examples/` for the full field set. -## Project Layout +## Project layout ``` module/ ← installable Lola module - agents/ ← standalone review agent definitions (prd-guard, prd-adversary, ...) - skills/prd-review/ ← orchestration skill (schema check + agent dispatch + verdict) + agents/ ← review agent definitions + skills/prd-review/ ← schema check + agent dispatch + verdict commands/ ← command entry points prd.cue ← CUE schema (published to registry) cue.mod/ ← CUE module definition examples/ ← example PRD YAML files ``` -See [AGENTS.md](AGENTS.md) for the full developer guide. +The review council roster, dispatch modes, and verdict phases live in +`module/skills/prd-review/SKILL.md`. See [AGENTS.md](AGENTS.md) for the +developer guide. ## License -Apache-2.0 — see [LICENSE](LICENSE). +Apache-2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE). + +The lola-eval harness scaffolding under `.taskfiles/` is adapted from the +[Review Council](https://github.com/lolables/lola-mod-review-council) project (Apache-2.0); see [NOTICE](NOTICE) for +attribution. diff --git a/examples/parent-prd.yaml b/examples/parent-prd.yaml index 4632a89..c85e951 100644 --- a/examples/parent-prd.yaml +++ b/examples/parent-prd.yaml @@ -19,26 +19,28 @@ stakeholders: - role: "Stakeholder Representative" handle: "@handle" -personas: - - "Example Persona" +job-executors: + - id: example-executor + label: "Example Executor" + core-job: "accomplish the core job this initiative serves, in its context" + +desired-outcomes: + - id: DO-MF-001 + statement: "minimize the time it takes to so " + executor-id: example-executor scope: - in_scope: + in-scope: - "Thing we are building" - out_of_scope: + out-of-scope: - "Thing we are not building" -nonfunctional_requirements: +nonfunctional-requirements: - id: NFR-MF-001 title: "Example NFR" description: >- Describe the non-functional requirement here. -kpis: - - metric: "Example metric this initiative moves" - target: "Qualitative direction is acceptable in Draft; add a number once validated with stakeholders." - baseline: "Current value or state, if known" - -open_questions: +open-questions: - question: >- An open question that needs resolution before or during delivery. diff --git a/examples/phase-prd-draft.yaml b/examples/phase-prd-draft.yaml new file mode 100644 index 0000000..684759d --- /dev/null +++ b/examples/phase-prd-draft.yaml @@ -0,0 +1,22 @@ +header: + schema-version: "0.1.0" + version: "0.1.0" + last-updated: "2026-01-01" + parent: my-feature +slug: my-feature-phase1-draft +phase: "Phase 1 — Early" +state: + status: Draft +open-questions: + - question: "Does the downstream consumer require a specific wire format?" +journeys: + - label: "Example Journey" + executor: example-executor + steps: + - label: "Step One" + description: "What happens in this step." + implements: [FR-MFP-001] +functional-requirements: + - id: FR-MFP-001 + title: "Example Functional Requirement" + satisfies: [] diff --git a/examples/phase-prd.yaml b/examples/phase-prd.yaml index bcb774c..bae8f34 100644 --- a/examples/phase-prd.yaml +++ b/examples/phase-prd.yaml @@ -14,26 +14,27 @@ stakeholders: handle: "@handle" state: - status: Draft + status: Ready dependencies: - description: "What this phase depends on (another team, system, or decision)" blocking: true context: "Why it's blocking and what unblocks it" -workflow: - label: "Example Workflow" - steps: - - label: "Step One" - description: "What happens in this step." - implements: - - FR-MFP-001 +journeys: + - label: "Example Journey" + executor: example-executor + steps: + - label: "Step One" + description: "What happens in this step." + implements: + - FR-MFP-001 -functional_requirements: +functional-requirements: - id: FR-MFP-001 title: "Example Functional Requirement" - persona: "Example Persona" - acceptance_criteria: + satisfies: [DO-MF-001] + acceptance-criteria: - id: AC-MFP-001-01 description: "First acceptance criterion for FR-MFP-001." - id: AC-MFP-001-02 diff --git a/module/AGENTS.md b/module/AGENTS.md index b4302e4..41848f2 100644 --- a/module/AGENTS.md +++ b/module/AGENTS.md @@ -4,7 +4,19 @@ Skills in this module support structured product requirements intake. ## Skills -- **prd-review** — Multi-agent review council for PRDs. Invoked via `/prd-review` before advancing a PRD from Draft to Review. Checks CUE schema conformance, then runs 5 specialist review agents (Guard, Adversary, Tester, Operator, Curator) — in parallel by default, or `--serial` for lower token cost — and emits a BLOCKED / NEEDS REVISION / APPROVED verdict. +- **prd-review** — Two-tier multi-agent review council. Invoked via `/prd-review`. Runs `cue vet` (schema conformance + FR/journey coverage floor), then dispatches for a `Draft` PRD by invocation: a fresh PRD gets a `cue vet` + Guard review; `--full` on a matured PRD runs all 5 agents (Guard, Adversary, Tester, Operator, Curator) and gates `Draft → Ready`, mutating `state.status` only after the author confirms. A `Ready` or later PRD triggers no gating dispatch; `Ready → Approved` is a manual human sign-off outside the toolkit. Parallel by default, `--serial` for lower token cost. + +## Lifecycle + +`Draft → [full council] → Ready → Approved(human)`. +The PRD clusters FRs under journeys (ACs optional; structural coverage +enforced by `cue vet`); the fresh PRD gets a `cue vet` + Guard review; +the author matures it (outside the toolkit); then the full 5-agent council +(`/prd-review --full`) gates `Draft → Ready` — on pass it asks the author +and mutates `state.status` to `Ready` only on confirmation. `Ready → +Approved` is a MANUAL human sign-off entirely outside the toolkit: no +agent, skill, or command dispatches for it, gates it, or writes +`status: Approved`. `Superseded` retires a PRD. ## Review Agents @@ -17,8 +29,8 @@ schema. PRDs are structured YAML validated against a CUE schema. The document model uses parent-child relationships: -- **Parent PRDs** define the initiative: title, description, personas, NFRs, KPIs, stakeholders, scope. -- **Phase PRDs** define delivery phases: FRs with ACs, workflows, dependencies, state tracking. +- **Parent PRDs** define the initiative: title, description, job executors, NFRs, desired outcomes, stakeholders, scope. +- **Phase PRDs** define delivery phases: FRs (ACs required only at Ready), journeys, dependencies, state tracking. ## Key Constraints diff --git a/prd.cue b/prd.cue index d07f67d..d9c8895 100644 --- a/prd.cue +++ b/prd.cue @@ -18,15 +18,28 @@ package prds parent?: string & =~"^[a-z][a-z0-9-]*$" } -#WorkflowStep: { +#JourneyStep: { label: string & !="" description: string & !="" implements: [...string] & [_, ...] } -#Workflow: { - label: string & !="" - steps: [...#WorkflowStep] & [_, ...] +#Journey: { + label: string & !="" + executor: string & !="" + steps: [...#JourneyStep] & [_, ...] +} + +#JobExecutor: { + id: string & =~"^[a-z][a-z0-9-]*$" + label: string & !="" + "core-job": string & !="" +} + +#DesiredOutcome: { + id: string & =~"^DO-[A-Z]+-\\d{3}$" + statement: string & !="" + "executor-id": string & =~"^[a-z][a-z0-9-]*$" } #AcceptanceCriteria: { @@ -35,16 +48,17 @@ package prds } #FunctionalRequirement: { - id: string & =~"^FR-[A-Z]+-\\d{3}$" - title: string & !="" - persona: string & !="" - acceptance_criteria: [...#AcceptanceCriteria] & [_, ...] + id: string & =~"^FR-[A-Z]+-\\d{3}$" + title: string & !="" + satisfies: [...string] + "acceptance-criteria": [...#AcceptanceCriteria] } #NonFunctionalRequirement: { id: string & =~"^NFR-[A-Z]+-\\d{3}$" title: string & !="" description: string & !="" + satisfies?: [...string] } #OpenQuestion: { @@ -52,12 +66,6 @@ package prds context?: string } -#KPI: { - metric: string & !="" - target: string & !="" // qualitative acceptable in Draft - baseline?: string -} - #Dependency: { description: string & !="" blocking: bool | *false @@ -65,7 +73,7 @@ package prds } #State: { - status: "Draft" | "Review" | "Approved" | "Superseded" + status: "Draft" | "Ready" | "Approved" | "Superseded" remarks?: string } @@ -73,39 +81,49 @@ package prds header: #PRDHeader slug?: string & =~"^[a-z][a-z0-9-]*$" - // Root-level fields stakeholders?: [...#Stakeholder] & [_, ...] - title?: string & !="" + title?: string & !="" description?: string & !="" features?: [...string] & [_, ...] - personas?: [...string] & [_, ...] scope?: { - in_scope: [...string] - out_of_scope: [...string] + "in-scope": [...string] + "out-of-scope": [...string] } - nonfunctional_requirements?: [...#NonFunctionalRequirement] & [_, ...] - kpis?: [...#KPI] & [_, ...] - - // Phase-level fields - phase?: string & !="" - state?: #State - workflow?: #Workflow - functional_requirements?: [...#FunctionalRequirement] & [_, ...] + "nonfunctional-requirements"?: [...#NonFunctionalRequirement] & [_, ...] + + phase?: string & !="" + state?: #State + journeys?: [...#Journey] & [_, ...] + "functional-requirements"?: [...#FunctionalRequirement] & [_, ...] dependencies?: [...#Dependency] & [_, ...] - open_questions?: [...#OpenQuestion] + "open-questions"?: [...#OpenQuestion] - // A document is either a phase (has `phase`) or a parent (does not). - // Each shape has its own required fields — `header` alone is not a - // valid document either way. - // Discriminate parent vs phase document shape: `!= _|_` means the - // field is set (not bottom/undefined). if phase != _|_ { - workflow: #Workflow - functional_requirements: [...#FunctionalRequirement] & [_, ...] + journeys: [...#Journey] & [_, ...] + _fr="functional-requirements": [...#FunctionalRequirement] & [_, ...] + + // CUE-native FR/journey coverage floor: ORPHAN_FR (every FR is + // implemented by at least one journey step) and UNKNOWN_FR (every + // `implements` id names a real FR in this phase). A failure names + // the offending id, e.g. `_ck_orphan_fr.1: undefined field: "FR-MFP-002"`. + _frSet: {for f in _fr {(f.id): true}} + _implementedFR: {for j in journeys for s in j.steps for id in s.implements {(id): true}} + _ck_orphan_fr: [for f in _fr {_implementedFR[f.id] & true}] + _ck_unknown_fr: [for j in journeys for s in j.steps for id in s.implements {_frSet[id] & true}] + + // Maturity gate: acceptance criteria are optional at early stage (Draft) + // and required once a phase reaches Ready or Approved. + if state != _|_ { + if state.status == "Ready" || state.status == "Approved" { + "functional-requirements": [...{"acceptance-criteria": [_, ...]}] + "functional-requirements": [...{satisfies: [_, ...]}] + } + } } if phase == _|_ { - title: string & !="" - personas: [...string] & [_, ...] + title: string & !="" + "job-executors": [...#JobExecutor] & [_, ...] + "desired-outcomes"?: [...#DesiredOutcome] } } diff --git a/testdata/invalid/bad-outcome-id.yaml b/testdata/invalid/bad-outcome-id.yaml new file mode 100644 index 0000000..4800bf5 --- /dev/null +++ b/testdata/invalid/bad-outcome-id.yaml @@ -0,0 +1,6 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01"} +title: "X" +job-executors: + - {id: a, label: "A", core-job: "do a thing in a context"} +desired-outcomes: + - {id: OUTCOME-1, statement: "minimize time to x", executor-id: a} diff --git a/testdata/invalid/executor-missing-core-job.yaml b/testdata/invalid/executor-missing-core-job.yaml new file mode 100644 index 0000000..f247583 --- /dev/null +++ b/testdata/invalid/executor-missing-core-job.yaml @@ -0,0 +1,4 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01"} +title: "X" +job-executors: + - {id: a, label: "A"} diff --git a/testdata/invalid/journey-uses-persona.yaml b/testdata/invalid/journey-uses-persona.yaml new file mode 100644 index 0000000..3560751 --- /dev/null +++ b/testdata/invalid/journey-uses-persona.yaml @@ -0,0 +1,9 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01", parent: my-feature} +phase: "P1" +journeys: + - label: "J" + persona: "Old Persona" + steps: + - {label: "S", description: "d", implements: [FR-X-001]} +functional-requirements: + - {id: FR-X-001, title: "T", satisfies: []} diff --git a/testdata/invalid/orphan-fr.yaml b/testdata/invalid/orphan-fr.yaml new file mode 100644 index 0000000..5cf7237 --- /dev/null +++ b/testdata/invalid/orphan-fr.yaml @@ -0,0 +1,10 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01", parent: my-feature} +phase: "P1" +journeys: + - label: "J" + executor: a + steps: + - {label: "S", description: "d", implements: [FR-X-001]} +functional-requirements: + - {id: FR-X-001, title: "T", satisfies: []} + - {id: FR-X-002, title: "Orphan", satisfies: []} diff --git a/testdata/invalid/parent-has-kpis.yaml b/testdata/invalid/parent-has-kpis.yaml new file mode 100644 index 0000000..37c2eee --- /dev/null +++ b/testdata/invalid/parent-has-kpis.yaml @@ -0,0 +1,6 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01"} +title: "X" +job-executors: + - {id: a, label: "A", core-job: "do a thing in a context"} +kpis: + - {metric: "m", target: "t"} diff --git a/testdata/invalid/phase-has-outcomes.yaml b/testdata/invalid/phase-has-outcomes.yaml new file mode 100644 index 0000000..e6e0664 --- /dev/null +++ b/testdata/invalid/phase-has-outcomes.yaml @@ -0,0 +1,11 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01", parent: my-feature} +phase: "P1" +journeys: + - label: "J" + executor: a + steps: + - {label: "S", description: "d", implements: [FR-X-001]} +functional-requirements: + - {id: FR-X-001, title: "T", satisfies: []} +desired-outcomes: + - {id: DO-X-001, statement: "minimize time to x", executor-id: a} diff --git a/testdata/invalid/ready-fr-missing-satisfies.yaml b/testdata/invalid/ready-fr-missing-satisfies.yaml new file mode 100644 index 0000000..f9f1a4c --- /dev/null +++ b/testdata/invalid/ready-fr-missing-satisfies.yaml @@ -0,0 +1,10 @@ +header: {schema-version: "0.1.0", version: "0.1.0", last-updated: "2026-01-01", parent: my-feature} +phase: "P1" +state: {status: Ready} +journeys: + - label: "J" + executor: a + steps: + - {label: "S", description: "d", implements: [FR-X-001]} +functional-requirements: + - {id: FR-X-001, title: "T", satisfies: [], acceptance-criteria: [{id: AC-X-001-01, description: "d"}]} diff --git a/testdata/valid/parent-valid.yaml b/testdata/valid/parent-valid.yaml new file mode 100644 index 0000000..3819b98 --- /dev/null +++ b/testdata/valid/parent-valid.yaml @@ -0,0 +1,19 @@ +header: + schema-version: "0.1.0" + version: "0.1.0" + last-updated: "2026-01-01" +slug: my-feature +title: "My Feature" +description: "An initiative." +job-executors: + - id: auditor + label: "Compliance Auditor" + core-job: "verify a control set is met before an external audit" +nonfunctional-requirements: + - id: NFR-MF-001 + title: "Example NFR" + description: "A constraint." +desired-outcomes: + - id: DO-MF-001 + statement: "minimize the time it takes to detect a misconfigured control before an audit" + executor-id: auditor diff --git a/testdata/valid/phase-ready-valid.yaml b/testdata/valid/phase-ready-valid.yaml new file mode 100644 index 0000000..f4bca33 --- /dev/null +++ b/testdata/valid/phase-ready-valid.yaml @@ -0,0 +1,23 @@ +header: + schema-version: "0.1.0" + version: "0.1.0" + last-updated: "2026-01-01" + parent: my-feature +slug: my-feature-phase1 +phase: "Phase 1" +state: + status: Ready +journeys: + - label: "Detect journey" + executor: auditor + steps: + - label: "Scan" + description: "Scan controls." + implements: [FR-MFP-001] +functional-requirements: + - id: FR-MFP-001 + title: "Control scan" + satisfies: [DO-MF-001] + acceptance-criteria: + - id: AC-MFP-001-01 + description: "Scan completes and reports drift." From 6d0b2a550a4c518b4c325cda88077f07fa58aa7d Mon Sep 17 00:00:00 2001 From: Jennifer Power Date: Fri, 28 Aug 2026 08:39:19 -0400 Subject: [PATCH 2/3] feat: two-tier stage-aware review council Split dispatch by PRD state and invocation: a fresh Draft gets a `cue vet` + Guard review; `--full` runs the 5-agent council and is the only pass that gates Draft to Ready, mutating state.status only on author confirmation. Ready and later trigger no gating dispatch. Assisted-by: Claude Code --- module/agents/prd-curator.md | 8 +- module/agents/prd-guard.md | 117 ++++++++++----- module/agents/prd-tester.md | 31 +++- module/commands/prd-review.md | 6 +- module/skills/prd-review/SKILL.md | 135 +++++++++++++----- .../references/reviewer-protocol.md | 11 +- 6 files changed, 213 insertions(+), 95 deletions(-) diff --git a/module/agents/prd-curator.md b/module/agents/prd-curator.md index e9665b7..5f5db27 100644 --- a/module/agents/prd-curator.md +++ b/module/agents/prd-curator.md @@ -9,11 +9,11 @@ description: Reviews PRD evidence capture, audit trail, retention, provenance, a Curator reviews documentation completeness for a system that will eventually need to prove what happened: evidence capture, audit trail, retention, and provenance, plus whether a PRD leaves loose ends behind as -it moves toward Review or Approved. Any PRD for a system that handles +it moves toward Ready or Approved. Any PRD for a system that handles compliance evidence, state mutation, or compliance reporting must state requirements for what gets captured, how long it is kept, and how its origin is established — absence of these requirements is a defect, not a -gap to address later in design. Curator also owns `open_questions` and +gap to address later in design. Curator also owns `open-questions` and `dependencies` hygiene: an unresolved question or dependency with no actionable context, or one that quietly blocks a stated requirement, is left dangling rather than closed out. @@ -28,7 +28,7 @@ the dispatching `prd-review` skill (see its `$SKILL_DIR` convention). 1. **Read & Map** — read every provided PRD file. Build a map of every state-mutating operation, every evidence-artifact mention, every - compliance-framework reference, and every entry in `open_questions` + compliance-framework reference, and every entry in `open-questions` and `dependencies`. No findings yet. Steps 2–3 (Evaluate, Self-Check) per Phased Process Skeleton in @@ -53,7 +53,7 @@ Steps 2–3 (Evaluate, Self-Check) per Phased Process Skeleton in requirements as manual submission? **Open-question hygiene** -- Does every entry in `open_questions` carry enough `context` to be +- Does every entry in `open-questions` carry enough `context` to be actionable by a future reader who wasn't in the room when it was written? - Does any unresolved `open_question` block a requirement elsewhere in diff --git a/module/agents/prd-guard.md b/module/agents/prd-guard.md index 63abbb2..cef5c29 100644 --- a/module/agents/prd-guard.md +++ b/module/agents/prd-guard.md @@ -1,5 +1,5 @@ --- -description: Reviews PRD intent fidelity, scope discipline, persona/ID/workflow integrity, and FR-to-value-proposition traceability. Dispatched by the prd-review skill. +description: Reviews PRD intent fidelity, scope discipline, executor/outcome/ID/journey integrity, and FR-to-value-proposition traceability. Dispatched by the prd-review skill. --- # Agent: PRD Guard @@ -8,10 +8,11 @@ description: Reviews PRD intent fidelity, scope discipline, persona/ID/workflow Guard reviews whether a PRD family holds together as one coherent document and whether it is honest about where it came from. That splits -into two halves of the same question: internally, personas, IDs, and -workflow references must be consistent across the parent and every phase -file with no drift, gaps, or collisions; externally, every functional -requirement must trace back to a stated user need, every `features` list +into two halves of the same question: internally, job executors, +desired outcomes, IDs, and journey references must be consistent across +the parent and every phase file with no drift, gaps, or collisions; +externally, every functional requirement must trace back to a stated +user need, every `features` list in the family — the parent's and any phase's own — must point at real originating issues, and a reader who has never seen those issues must still understand — from the PRD alone — what @@ -28,11 +29,13 @@ the dispatching `prd-review` skill (see its `$SKILL_DIR` convention). ## Phased Process 1. **Read & Map** — read every provided PRD file. Build a field map: the - parent's `personas` list, every FR/AC/NFR ID across all phases, every - workflow step and its `implements` list, every `features` list in the + parent's `job-executors` list, the parent's `desired-outcomes` list, + every FR/AC/NFR ID across all phases (and each FR/NFR's `satisfies` + list), every journey (with its `executor`) and every step's + `implements` list, every `features` list in the family (`features` may appear on the parent, on any phase file, or both — check each file, not just the parent), the family's - `stakeholders`, the parent's `kpis` if present, and `state.status` per + `stakeholders`, and `state.status` per phase. No findings yet. Steps 2–3 (Evaluate, Self-Check) per Phased Process Skeleton in @@ -42,8 +45,13 @@ drafting any finding tied to it. ## Feature Traceability Grounding +`features` is optional in the schema (`features?:`). A file that declares +no `features` list is valid — never flag an absent or omitted `features` +list as a finding at any severity. This procedure applies only to +`features` entries that are actually present. + `features` is not parent-exclusive — a phase file may declare its own -`features` list alongside `phase`/`functional_requirements`. Run this +`features` list alongside `phase`/`functional-requirements`. Run this procedure against every `features` entry in every file across the family, not just the parent's. @@ -61,11 +69,39 @@ For each `features` entry: ## Review Criteria -**Persona consistency** -- Does every `persona` string in every phase's functional requirements - exactly match a string in the parent PRD's `personas` list? -- Is any persona referenced by shorthand ("ProdSec", "Dev") instead of - the full string defined in the parent? +**Executor consistency** +- Read the parent PRD's `job-executors` list and every journey's + `executor` id across all phase files. This check spans documents, so + no single-file tool performs it — it is yours. +- `UNKNOWN_EXECUTOR`: does every journey `executor` id exactly match an + `id` in the parent's `job-executors`? A journey naming an executor the + parent never declared is a BLOCKER. +- `DANGLING_EXECUTOR`: is every executor in the parent's `job-executors` + used by at least one journey somewhere in the family? A declared + executor no journey serves is a WARNING. +- Is any executor referenced by shorthand ("ProdSec", "Dev") instead of + the full `id` defined in the parent? (INFO / WARNING per severity.) + +**Outcome consistency** +- Read the parent PRD's `desired-outcomes` list (each with `id`, + `statement`, `executor-id`) and every FR/NFR `satisfies` list across + all phase files. This check spans documents, so no single-file tool + performs it — it is yours. +- `UNKNOWN_OUTCOME`: does every FR or NFR `satisfies` id exactly match an + `id` in the parent's `desired-outcomes`? A requirement claiming to + satisfy an outcome the parent never declared is a BLOCKER. +- `ORPHAN_OUTCOME`: is every entry in the parent's `desired-outcomes` + satisfied by at least one FR or NFR somewhere in the family? A + declared outcome no requirement satisfies is a WARNING. +- `UNKNOWN_EXECUTOR_REF`: does every `desired-outcomes` entry's + `executor-id` exactly match an `id` in the parent's `job-executors`? + An outcome referencing an executor the parent never declared is a + BLOCKER. +- Cross-doc maturity gate: once any phase's `state.status` reaches + `Ready` or `Approved`, does the parent have at least one non-empty + `desired-outcomes` entry? `cue vet` validates each file in isolation + and cannot see this parent/phase relationship — it is yours to check. + Missing outcomes at that maturity is a BLOCKER. **ID continuity** - Are FR IDs sequential within each phase, with no gaps? @@ -75,18 +111,21 @@ For each `features` entry: - Do FR or AC IDs collide across phases? Does phase 2 continue numbering from where phase 1 left off, rather than restarting? -**Workflow integrity** -- Does every FR in a phase appear in at least one workflow step's - `implements` list? -- Does every FR ID named in an `implements` list actually exist in that - phase's `functional_requirements`? -- Does every workflow step have a non-empty `label` and `description`? +**Journey integrity** +- Hard FR coverage — every FR implemented by some journey step + (`ORPHAN_FR`), every `implements` id existing (`UNKNOWN_FR`) — is + enforced mechanically by `cue vet` against the schema. Do NOT + re-report those; assume `cue vet` ran and passed before you were + dispatched. +- Instead, flag the semantic gaps `cue vet` cannot see: a journey step + whose `description` does not actually exercise the FRs it + `implements`, a journey whose executor's core job the steps never + serve, or a step with a placeholder/empty `label` or `description`. **Cross-references and state** - Do open questions and context fields reference other PRDs by phase name or title, rather than by FR/NFR ID? -- Is the parent's `features` list present and non-empty? -- Is `state.status` present and one of `Draft`, `Review`, `Approved`, +- Is `state.status` present and one of `Draft`, `Ready`, `Approved`, `Superseded` for every phase? **Stakeholder completeness** @@ -102,9 +141,6 @@ For each `features` entry: requiring the reader to open the originating issues? - Is a value proposition stated — what gets better, for which user, and by how much (qualitative is acceptable in Draft)? -- Where the parent states `kpis`, does each have a `target` that's - actually measurable — even qualitatively in Draft — rather than an - unfalsifiable restatement of the metric name itself? - Does every FR map to a stated user need, or does any FR exist purely for internal engineering convenience? - Do the FRs collectively cover the scope implied by `description`, with @@ -116,24 +152,26 @@ For each `features` entry: | Finding type | Severity | |---|---| -| Persona string in a phase FR does not match the parent `personas` list | BLOCKER | -| FR ID named in a workflow step's `implements` does not exist in that phase | BLOCKER | +| Journey `executor` id not in the parent `job-executors` list (UNKNOWN_EXECUTOR) | BLOCKER | +| FR or NFR `satisfies` id not in the parent `desired-outcomes` list (UNKNOWN_OUTCOME) | BLOCKER | +| `desired-outcomes` entry's `executor-id` names no parent executor (UNKNOWN_EXECUTOR_REF) | BLOCKER | +| Parent has no non-empty `desired-outcomes` once any phase reaches Ready/Approved | BLOCKER | | Duplicate FR or AC ID within or across phases | BLOCKER | -| Parent `features` list absent or empty | BLOCKER | | No `stakeholders` entry across the family has `approver: true` | WARNING | | Duplicate `(role, handle)` stakeholder pair within a file | WARNING | | `description` does not explain the user problem | BLOCKER | -| FR not referenced in any workflow step | WARNING | +| Journey step `description` does not exercise the FRs it `implements` | WARNING | +| Parent executor declared but used by no journey (DANGLING_EXECUTOR) | WARNING | +| Parent outcome declared but satisfied by no FR or NFR (ORPHAN_OUTCOME) | WARNING | | ID gap within a phase | WARNING | | FR IDs restart from 001 in a non-first phase (should continue) | WARNING | | Cross-reference uses an FR/NFR ID from another PRD instead of a name | WARNING | | `state.status` absent | WARNING | | FR with no traceable user value | WARNING | | Value proposition absent or too vague to evaluate | WARNING | -| `kpis` entry present but `target` unfalsifiable/restates the metric | WARNING | | Large scope gap between `description` and the FRs | WARNING | | Possible duplication with an already-delivered PRD | WARNING | -| Minor persona string whitespace or case difference | INFO | +| Minor executor id whitespace or case difference | INFO | | `description` requires reading the originating issues to understand | INFO | | `features` entry unverifiable — `gh` unavailable or fetch failed | INFO | | Fetched issue contradicts or doesn't support the FR claiming to satisfy it | WARNING | @@ -147,14 +185,17 @@ owns only its row; every other domain belongs to another agent. If you catch yourself doing any of these, stop: -- About to flag a persona mismatch without having opened the parent - PRD's `personas` list in this pass — go read it first. +- About to flag an executor mismatch without having opened the parent + PRD's `job-executors` list in this pass — go read it first. +- About to flag an outcome mismatch without having opened the parent + PRD's `desired-outcomes` list in this pass — go read it first. - About to flag a "duplicate work" finding based on a feature's title alone, without reading the FRs of the PRD it allegedly duplicates. - About to treat a `gh issue view` failure as evidence the FR is unsupported — a failed fetch proves nothing; it only means unverified. -- About to flag missing workflow coverage for an FR you haven't - confirmed actually exists in that phase's `functional_requirements`. +- About to re-report an ORPHAN_FR / UNKNOWN_FR that `cue vet` already + owns — that's the schema's job; flag only the semantic gap it can't + see. - About to invent a value-proposition gap because the wording feels thin, without checking whether the qualitative bar for Draft state is met. - About to run Feature Traceability Grounding against only the parent's @@ -164,12 +205,14 @@ If you catch yourself doing any of these, stop: | Excuse | Reality | |---|---| -| "The persona strings look close enough, probably fine." | Exact string match is the whole check — "close" is a BLOCKER, not a pass. | +| "The executor ids look close enough, probably fine." | Exact id match is the whole check — "close" is a BLOCKER, not a pass. | +| "This FR's `satisfies` id is close to an outcome id, probably a typo, not worth flagging." | Exact id match is the whole check — a mismatched `satisfies` id is UNKNOWN_OUTCOME, a BLOCKER. | | "`gh` isn't set up here, I'll just skip the features check." | Skipping produces silence; emit the INFO-unverified finding instead — it's the required outcome, not an escape hatch. | | "This FR clearly relates to the description, I don't need to trace it." | "Clearly relates" is not traceable — cite the specific need the FR maps to or flag the gap. | -| "The workflow probably covers this FR somewhere." | Grep the `implements` lists you built in Read & Map — don't guess coverage you haven't confirmed. | +| "I should double-check FR coverage myself." | Hard coverage is `cue vet`'s job (schema-enforced) — re-reporting it is noise; spend your pass on the executor/outcome checks and semantic gaps. | | "Numbering restarted but it's only phase 2, not a big deal." | It's a WARNING by the table above regardless of how minor it feels — calibrate from the table, not from vibes. | | "I already checked the parent's `features`, that covers this PRD." | Phase files can declare their own `features` too — an unchecked phase-level entry is a silent gap, not coverage. | +| "This PRD has no `features` list, that's a gap I should flag." | `features` is optional (`features?:`) — an absent list is valid and is never a finding. Only present entries are graded. | ## Output diff --git a/module/agents/prd-tester.md b/module/agents/prd-tester.md index 67a58ad..2a41902 100644 --- a/module/agents/prd-tester.md +++ b/module/agents/prd-tester.md @@ -74,10 +74,10 @@ finding (see Severity Calibration). **For each FR:** - Does it cover exactly one concern? - Does it describe behavior, not implementation? -- Does its `persona` field read as a real, well-formed string (Guard - separately checks that string against the parent's `personas` list — - Tester's concern is whether the string itself is meaningful, not - whether it matches the parent)? +- Does each journey's `executor` id read as a real, well-formed job + executor (Guard separately matches that id against the parent's + `job-executors` list — you only assess whether the executor makes the + journey's ACs judgeable from a real user's standpoint)? - Do its ACs describe user-observable behavior, not implementation detail? - Do its ACs avoid naming protocols, HTTP status codes, message schemas, @@ -87,6 +87,17 @@ finding (see Severity Calibration). - Does any AC restate NFR behavior instead of establishing where/when the FR applies? +**For each `desired-outcomes` entry's `statement`:** +- `#DesiredOutcome.statement` is free text — CUE cannot enforce its + shape, so this check is yours alone. +- Does it read as a well-formed outcome statement: a direction + (minimize/increase/reduce), a metric/unit, an object of control, and a + context clarifier? +- Does it restate a feature or solution instead of stating a + solution-agnostic need? +- Does it lack a measurable direction, making it impossible to tell + whether the outcome improved or regressed? + ## Severity Calibration | Finding type | Severity | @@ -102,6 +113,7 @@ finding (see Severity Calibration). | Mixed concerns in one requirement | WARNING | | Table-stakes AC | WARNING | | NFR behavior restated in an AC | WARNING | +| `desired-outcomes` statement restates a feature or lacks a measurable direction | BLOCKER | | Minor phrasing imprecision | INFO | ## Out of Scope @@ -112,9 +124,9 @@ See Domain Ownership in `reviewer-protocol.md` for the full map. If you catch yourself doing any of these, stop: -- About to flag a persona string as "wrong" because it doesn't match the - parent PRD — that comparison is Guard's job; you only judge whether the - string itself is well-formed. +- About to flag an executor id as "wrong" because it doesn't match the + parent PRD's `job-executors` list — that comparison is Guard's job; you + only judge whether the executor makes the journey's ACs judgeable. - About to flag a named external service purely because it creates a deployment assumption — that's Operator's domain unless the issue is specifically that it's a named technology in Draft. @@ -135,6 +147,10 @@ If you catch yourself doing any of these, stop: - About to let an AC pass because it names a status code but "everyone knows what that means" — table-stakes and implementation-detail ACs are findings regardless of how conventional they are. +- About to accept a `desired-outcomes` statement because it sounds + aspirational — check it actually has a direction, a metric/unit, an + object of control, and a context clarifier; missing any of these is a + BLOCKER, not a style nit. ## Rationalization Table @@ -147,6 +163,7 @@ If you catch yourself doing any of these, stop: | "The technology name is only in a comment-like aside, not the main requirement text." | Location doesn't matter — what matters is whether it's an implementation choice (BLOCKER) or a compatibility constraint (INFO); classify it, don't wave it through on placement. | | "We need this data model for compatibility, so naming it is fine — no need to check further." | State that it's a required compatibility constraint explicitly if the PRD doesn't already say so; an unlabeled name is a WARNING (ambiguous), not an automatic pass. | | "It's just in the description paragraph, not a real requirement field." | The description is exactly where an implementation choice most often hides — check it like any other field. | +| "The outcome statement names a real user benefit, that's good enough." | A benefit without a direction, metric/unit, object of control, and context clarifier can't be judged pass/fail later — flag it, don't credit the intent. | ## Output diff --git a/module/commands/prd-review.md b/module/commands/prd-review.md index 402e1c0..b501434 100644 --- a/module/commands/prd-review.md +++ b/module/commands/prd-review.md @@ -1,11 +1,11 @@ --- -description: Run the PRD Review Council (5 specialist agents) against a PRD family. +description: Run the PRD Review Council against a PRD family — Guard review pass by default, or the full five-agent council with --full. --- Run the PRD Review Council on the specified PRD files. -Usage: /prd-review [--serial] [--schema ] [phase1.yaml phase2.yaml ...] +Usage: /prd-review [--full] [--serial] [--schema ] [phase1.yaml phase2.yaml ...] -Checks CUE schema conformance (registry first, falling back to `--schema ` if the published schema can't be resolved), then dispatches 5 specialist agents (Guard, Adversary, Tester, Operator, Curator) against the PRD family and emits a BLOCKED / NEEDS REVISION / APPROVED verdict. Default `parallel` mode dispatches all 5 as independent subagents (faster, ~5x PRD ingestion cost). `--serial` runs all 5 personas sequentially in one context, reusing the already-read PRD text (slower, ~1x ingestion cost). +Checks CUE schema conformance and FR/journey coverage in one `cue vet` pass (the structural floor), then selects review depth for a `Draft` PRD by invocation: by default a fresh PRD gets a `cue vet` + Guard review (gates nothing); with `--full` a matured PRD gets the full five agents (Guard, Adversary, Tester, Operator, Curator) that gate `Draft → Ready` — on APPROVED the skill asks the author to confirm before mutating `state.status` to `Ready`, and emits a BLOCKED / NEEDS REVISION / APPROVED verdict. A `Ready` or later PRD triggers no gating dispatch; `Ready → Approved` is a manual human sign-off outside this command. Default `parallel` mode dispatches agents as independent subagents; `--serial` runs them sequentially in one context for lower token cost. $ARGUMENTS diff --git a/module/skills/prd-review/SKILL.md b/module/skills/prd-review/SKILL.md index 6665e64..5c4d688 100644 --- a/module/skills/prd-review/SKILL.md +++ b/module/skills/prd-review/SKILL.md @@ -1,6 +1,6 @@ --- name: prd-review -description: Multi-lens AI review council for PRDs. Run before advancing a PRD from Draft to Review state. +description: Multi-lens AI review council for PRDs. On a Draft PRD, runs a cue vet + Guard review by default; with --full, runs the full five-agent council that gates Draft → Ready. --- # Helper paths @@ -11,13 +11,15 @@ description: Multi-lens AI review council for PRDs. Run before advancing a PRD f # PRD Review Council Multi-lens AI review council for structured PRDs. Checks CUE schema -conformance, then dispatches 5 specialist review agents against a PRD -family (parent + phase files). Produces a consolidated finding report and -a verdict that gates state advancement. +conformance and FR/journey coverage in one `cue vet` pass, then +dispatches review agents against a PRD family (parent + phase files) — +Guard only by default, all 5 specialists with `--full`. Produces a +consolidated finding report and, on `--full`, a verdict that gates +`Draft → Ready`. ## When to Use -- PRD is transitioning from `Draft` → `Review` state +- PRD is at `Draft`: a fresh PRD, or a matured PRD ready for the full `--full` council that gates `Draft → Ready` - User invokes `/prd-review` with PRD file paths - User asks for "PRD review" or "run the review council on this PRD" @@ -44,19 +46,25 @@ unreachable and you want a local fallback: If no paths are given, ask for them before proceeding. -**Review mode is a cost/speed tradeoff, not a capability fallback:** +**Review mode is a cost/speed tradeoff, not a capability fallback.** It +applies to whichever agents Phase 2 selects — Guard alone by default, all +5 with `--full`: -- **`parallel`** (default) — all 5 agents dispatch as independent +- **`parallel`** (default) — the dispatched agents run as independent subagents, each ingesting its own copy of the full PRD family plus `reviewer-protocol.md`. For a PRD family of N tokens this costs roughly - 5N tokens of ingestion (plus five ~1K-token persona files), against - ~1N for a single-context read. Faster wall-clock, higher token cost — - the cost grows with PRD family size, not just with a fixed checklist. -- **`serial`** — the orchestrator adopts each of the 5 personas in this + kN tokens of ingestion for k dispatched agents (plus one ~1K-token + persona file per agent — one at Guard-only default, five with `--full`), + against ~1N for a single-context read. Faster wall-clock, higher token + cost — the cost grows with PRD family size and agent count, not just + with a fixed checklist. +- **`serial`** — the orchestrator adopts each dispatched persona in this same context, one at a time, reusing the PRD text already read in Phase 1 (see below) instead of re-ingesting it per agent. Cost stays - near ~1N regardless of how many agents run. Slower wall-clock (5 - sequential passes instead of 5 concurrent ones), lower token cost. + near ~1N regardless of how many agents run. Slower wall-clock at + `--full` (5 sequential passes instead of 5 concurrent ones; a no-op + distinction at Guard-only default, where there's only one agent either + way), lower token cost. - If the user does not specify a mode and the host cannot do named-agent dispatch at all, `serial` is the only option regardless of request — this is a hard capability fallback, separate from the cost/speed choice @@ -101,36 +109,64 @@ catch (they read for behavior and quality, not structural validity). cue vet -d '#PRDDocument' ``` 4. If every file passes (exit 0), proceed to Phase 1. -5. If any file fails, stop — do not dispatch the 5 agents. Emit the same - report format as Phase 4 below, with `Verdict: BLOCKED`, +5. If any file fails, stop — do not dispatch any review agents. Emit the + same report format as Phase 5 below, with `Verdict: BLOCKED`, `Schema conformance: BLOCKED`, a single BLOCKERs entry containing the raw `cue vet` error output verbatim (it already names the exact field, - constraint, and file/line), and empty WARNINGs/INFO sections — the 5 - agents never ran. A structurally invalid PRD isn't reviewable for - behavioral quality until the schema violation itself is fixed. + constraint, and file/line), and empty WARNINGs/INFO sections — no + agents ran. A structurally invalid PRD isn't reviewable for behavioral + quality until the schema violation itself is fixed. ### Phase 1: Preparation Read all provided PRD files and hold the raw file text verbatim in -context for each one — not just extracted fields. Phase 3 needs to +context for each one — not just extracted fields. Phase 4 needs to byte-match `evidence` quotes against this raw text later; extracting only -a field summary here would force a second full read in Phase 3. +a field summary here would force a second full read in Phase 4. Also extract, for convenience during Dispatch and Synthesis: -- **Parent**: `title`, `slug`, `description`, `personas`, `nonfunctional_requirements`, `kpis`, `features`, `stakeholders`, `open_questions` (each with its own `context`, if present) -- **Each phase**: `phase`, `state`, `workflow`, `functional_requirements` (with ACs), `dependencies`, and `features` if this phase file declares its own (see the Feature Traceability note in Phase 2's Guard row — `features` isn't parent-exclusive) +- **Parent**: `title`, `slug`, `description`, `job-executors`, `nonfunctional-requirements`, `desired-outcomes`, `features`, `stakeholders`, `open-questions` (each with its own `context`, if present) +- **Each phase**: `phase`, `state`, `journeys`, `functional-requirements` (with ACs), `dependencies`, and `features` if this phase file declares its own (see the Feature Traceability note in Phase 3's Guard row — `features` isn't parent-exclusive) -Note the PRD `state.status`. If already `Approved` or `Superseded`, warn the user and ask whether to continue. +Note the PRD `state.status` for use in Phase 2's stage selection. -### Phase 2: Dispatch +### Phase 2: Stage selection + +Read `state.status` on each phase file (from the text held in Phase 1) +and select review depth by the LEAST-mature phase in the family and +whether `--full` was passed: + +| Least-mature `state.status` | Invocation | Agents dispatched | Gate | +|---|---|---|---| +| `Draft` | default | **Guard only** + the Phase 0 `cue vet` floor | none — sanity pass, stays `Draft` | +| `Draft` | `--full` | **All 5** (Guard, Adversary, Tester, Operator, Curator) + floor | **`Draft → Ready`** | +| `Ready` / `Approved` / `Superseded` | any | none by default | none — warn the author; `--full` may re-run all 5 for feedback but gates nothing | + +Why Guard-only at a fresh Draft: acceptance criteria are untrusted and +optional at Draft, so the Tester (AC quality), Adversary (security +detail), Operator (deployment), and Curator (evidence/retention) lenses +have nothing stable to review yet — running them produces noise and burns +~4x the tokens for findings the author will invalidate at the next edit. +Guard's coherence/value/traceability/executor lens is the one that IS +meaningful on a PRD, and `cue vet` already caught every structural +defect. The full council earns its cost only once the PO has matured the +PRD and runs `--full` to gate `Draft → Ready`. + +Both PRD tiers are `state.status: Draft`; the tier is chosen by the +`--full` flag, not the status value. There is no `Ready → Approved` gate in this skill — that transition is a manual human sign-off outside it. +This skill never dispatches for that transition, never gates it, and NEVER writes `state.status: Approved`. + +### Phase 3: Dispatch + +Dispatch only the agents selected in Phase 2 — Guard alone by default, all five with `--full`. Hand each agent the PRD context, file paths, and `$SKILL_DIR/references/reviewer-protocol.md`: | Agent | File | Reviews | |---|---|---| -| Guard | `module/agents/prd-guard.md` | Intent fidelity, scope discipline, persona/ID/workflow integrity, FR-to-value traceability | +| Guard | `module/agents/prd-guard.md` | Intent fidelity, scope discipline, executor/ID/journey integrity, FR-to-value traceability | | Adversary | `module/agents/prd-adversary.md` | Security gaps + ambiguity/completeness | | Tester | `module/agents/prd-tester.md` | Behavioral language, testability, AC quality | | Operator | `module/agents/prd-operator.md` | Deployment/environment/connectivity assumptions | @@ -141,24 +177,26 @@ Hand each agent the PRD context, file paths, and every `features` entry across the whole PRD family, not just the parent's. -**`parallel` mode** (default, host-capable): dispatch all 5 as independent -subagents concurrently. Each receives the PRD file paths and +**`parallel` mode** (default, host-capable): dispatch the agents selected +in Phase 2 — Guard alone by default, all five with `--full` — as +independent subagents concurrently. Each receives the PRD file paths and `reviewer-protocol.md` and reads the PRD family itself — it does not inherit the orchestrator's Phase 1 context. **`serial` mode** (requested, or forced by host capability): stay in this -context. For each of the 5 agents in turn, adopt its persona from its +context. For each dispatched agent in turn, adopt its persona from its full file, apply its Phased Process against the PRD text already held from Phase 1 — do not re-read the PRD files — collect its JSON output, -then move to the next persona. This is what makes `serial` cheaper: the -PRD ingestion cost is paid once in Phase 1, not once per persona. +then move to the next persona (a single pass at Guard-only default; five +passes with `--full`). This is what makes `serial` cheaper: the PRD +ingestion cost is paid once in Phase 1, not once per persona. Each agent returns its own fenced JSON block per `reviewer-protocol.md` — no prose, no summaries, no praise. -### Phase 3: Verify +### Phase 4: Verify -For every finding across all 5 agents' JSON output, confirm `evidence` is +For every finding across the dispatched agents' JSON output, confirm `evidence` is a literal substring of the PRD file `location` is scoped to (the parent or phase file containing the cited ID or section), checked against the raw text held from Phase 1. Drop any finding whose `evidence` cannot be @@ -169,11 +207,11 @@ This is an LLM re-check pass over the raw PRD text already held from Phase 1 — no re-reading files, no scripts, no schema-as-code, keeping the module script-free per `AGENTS.md`. -### Phase 4: Synthesis & Verdict +### Phase 5: Synthesis & Verdict Collect the findings that survived Verify. Group by severity: -- **BLOCKER** — must be resolved before PRD advances to Review +- **BLOCKER** — must be resolved before PRD advances to Ready - **WARNING** — author must respond with rationale to skip, or resolve - **INFO** — advisory, no response required @@ -208,16 +246,35 @@ Schema conformance: PASSED | SKIPPED () | BLOCKED ``` -If `APPROVED`: state the PRD may advance to `Review` state. -If `NEEDS REVISION`: list each WARNING and state the author must resolve or respond before merge. -If `BLOCKED`: list each BLOCKER and state nothing advances until they are resolved. +The verdict's meaning depends on the tier selected in Phase 2: +- **Default Draft (cue vet + Guard) review** — never gates a state + transition. `APPROVED` means the PRD is coherent enough to mature; + keep sharpening it, then run `/prd-review --full` for the gating + council. `NEEDS REVISION`/`BLOCKED` keep it at `Draft`. +- **`--full` Draft council** — the only gating pass. + - `APPROVED`: the matured PRD is eligible to advance `Draft → Ready`. + ASK the PO to confirm ("Advance from Draft to Ready? [y/N]"). + ONLY after an explicit in-session `yes`, edit the PRD file's + `state.status: Draft` to `Ready`, show the diff, and report it. If + the PO declines or does not answer, leave it at `Draft`. Never + advance without that confirmation. + - `NEEDS REVISION`/`BLOCKED`: keep it at `Draft`; do not mutate. + +`Ready → Approved` is a manual human sign-off OUTSIDE this toolkit. This +skill NEVER writes `state.status: Approved` and never gates that +transition — not even on explicit request. If asked, tell the author +they set `Approved` by hand. ## Rules -- Never edit PRD files. +- Never edit PRD files, with one exception: mutating `state.status: Draft` + to `Ready` in Phase 5, and only after explicit in-session PO + confirmation on an `APPROVED` `--full` verdict. Never write + `state.status: Approved` — that transition is a manual human sign-off + outside this skill. - Report findings only — do not propose rewrites unless the user explicitly asks after the report. - If a PRD file cannot be read, report `BLOCKED` with reason. -- If Phase 0 finds a schema violation, report `BLOCKED` with the raw `cue vet` output and stop — do not dispatch the 5 agents against a structurally invalid file. +- If Phase 0 finds a schema violation, report `BLOCKED` with the raw `cue vet` output and stop — do not dispatch any review agents against a structurally invalid file. - If Phase 0 is skipped (no schema found, or `cue` unavailable), say so plainly in the report — never imply schema conformance was checked when it wasn't. - Walk one finding at a time if user asks for interactive mode — otherwise emit the full report. - Findings from different agents may overlap in location. Do not deduplicate across agents — each agent owns its own scope. diff --git a/module/skills/prd-review/references/reviewer-protocol.md b/module/skills/prd-review/references/reviewer-protocol.md index a41dce4..2f3f7cf 100644 --- a/module/skills/prd-review/references/reviewer-protocol.md +++ b/module/skills/prd-review/references/reviewer-protocol.md @@ -18,7 +18,7 @@ row, don't flag it — the owning agent will. | Domain | Owner | |---|---| -| Persona/ID/workflow consistency, feature traceability, scope/value coherence | Guard | +| Executor/outcome/ID/journey consistency, feature traceability, scope/value coherence | Guard | | Auth boundaries, credential scope, trust model, ambiguity hiding a security decision | Adversary | | Behavioral language, testability, AC quality, technology-naming-as-implementation-choice | Tester | | Deployment, environment, and connectivity assumptions | Operator | @@ -42,8 +42,9 @@ same for all 5 agents and aren't restated per agent: ## Evidence Discipline - Read every provided PRD file in full before drafting any finding. -- Ground every cited identifier — FR ID, NFR ID, AC ID, persona string, - workflow step — in text you have actually read in the cited file. +- Ground every cited identifier — FR ID, NFR ID, AC ID, executor id, + desired-outcome id, journey step — in text you have actually read in + the cited file. - Never assert a field is absent without having read the whole section of the file where it would appear. - Never fabricate a quote. `evidence` is a byte-for-byte substring of the @@ -82,12 +83,12 @@ no prose before or after it. ```json { "agent": "prd-guard", - "sections_read": ["parent.personas", "phase1.functional_requirements"], + "sections_read": ["parent.job-executors", "phase1.functional-requirements"], "verdict": "APPROVE | REQUEST_CHANGES", "findings": [ { "severity": "BLOCKER|WARNING|INFO", - "location": "FR-003-02", + "location": "FR-MFP-002", "title": "Short headline naming the defect", "evidence": "", "issue": "What is wrong, one sentence", From 5304da30cb1e061ccc4776fed868c3856c36c3e7 Mon Sep 17 00:00:00 2001 From: Jennifer Power Date: Fri, 28 Aug 2026 08:39:19 -0400 Subject: [PATCH 3/3] test: add lola-eval harness and CUE conformance CI Add the lola-eval behavioral harness (config, provisioner, taskfiles) with the case-001 schema baseline, plus a CUE conformance test and CI workflow. Harness scaffolding adapted from lola-mod-review-council. Assisted-by: Claude Code --- .github/workflows/ci.yml | 33 ++++ .github/workflows/publish-cue.yml | 5 - .gitignore | 11 ++ .lola-eval/baseline.json | 7 + .lola-eval/config.yaml | 39 +++++ .lola-eval/provision.sh | 129 ++++++++++++++++ .../tests/case-001-unknown-executor/prompt.md | 11 ++ .../tests/case-001-unknown-executor/rubric.md | 83 ++++++++++ .../starter/parent.yaml | 28 ++++ .../starter/phase1.yaml | 28 ++++ .../case-001-unknown-executor/starter/prd.cue | 129 ++++++++++++++++ .../tests/case-001-unknown-executor/task.yaml | 21 +++ .taskfiles/lola-eval.yml | 146 ++++++++++++++++++ .taskfiles/scripts/ensure-lola-eval.sh | 116 ++++++++++++++ NOTICE | 28 ++++ Taskfile.yml | 34 ++++ 16 files changed, 843 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 .lola-eval/baseline.json create mode 100644 .lola-eval/config.yaml create mode 100644 .lola-eval/provision.sh create mode 100644 .lola-eval/tests/case-001-unknown-executor/prompt.md create mode 100644 .lola-eval/tests/case-001-unknown-executor/rubric.md create mode 100644 .lola-eval/tests/case-001-unknown-executor/starter/parent.yaml create mode 100644 .lola-eval/tests/case-001-unknown-executor/starter/phase1.yaml create mode 100644 .lola-eval/tests/case-001-unknown-executor/starter/prd.cue create mode 100644 .lola-eval/tests/case-001-unknown-executor/task.yaml create mode 100644 .taskfiles/lola-eval.yml create mode 100755 .taskfiles/scripts/ensure-lola-eval.sh create mode 100644 NOTICE create mode 100644 Taskfile.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..2d02dd2 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,33 @@ +name: CI +on: + push: + branches: + - "main" + pull_request: + branches: + - "main" + +permissions: {} + +jobs: + test: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Setup CUE + uses: cue-lang/setup-cue@a93fa358375740cd8b0078f76355512b9208acb1 # v1.0.1 + with: + version: "v0.17.0" + - name: Install Task + uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0 + with: + version: 3.x + - name: Run vet + run: task vet-examples + - name: Run test + run: task test \ No newline at end of file diff --git a/.github/workflows/publish-cue.yml b/.github/workflows/publish-cue.yml index f4556e9..b1245d9 100644 --- a/.github/workflows/publish-cue.yml +++ b/.github/workflows/publish-cue.yml @@ -20,11 +20,6 @@ jobs: uses: cue-lang/setup-cue@a93fa358375740cd8b0078f76355512b9208acb1 # v1.0.1 with: version: "v0.17.0" - - name: Validate examples - run: | - for f in examples/*.yaml; do - cue vet prd.cue "$f" -d '#PRDDocument' - done - name: Login to CUE Central Registry uses: cue-labs/registry-login-action@66d40052b0206031343e17173425fa10508968d0 # v1.0.3 - name: Publish module diff --git a/.gitignore b/.gitignore index 8d29bba..04158ef 100644 --- a/.gitignore +++ b/.gitignore @@ -11,5 +11,16 @@ node_modules/ .claude/ .superpowers/ +# lola-eval harness: project-local venv + generated run artifacts +.venv/ +.lola-eval/out/ +.lola-eval/runs.db +.lola-eval/results/ +.lola-eval/*.xml +# Provisioned into starters by provision.sh; regenerated, never committed +.lola-eval/tests/*/starter-clean/ +.lola-eval/tests/*/starter/.gitconfig +.lola-eval/tests/*/starter/.opencode/ + # Working artifacts (never committed) docs/superpowers/ diff --git a/.lola-eval/baseline.json b/.lola-eval/baseline.json new file mode 100644 index 0000000..0689d5f --- /dev/null +++ b/.lola-eval/baseline.json @@ -0,0 +1,7 @@ +{ + "_schema_version": 2, + "claude-code/claude-sonnet-4-6/case-001-unknown-executor/project/none": { + "composite": 0.925, + "rubric_pass_threshold": 0.6 + } +} diff --git a/.lola-eval/config.yaml b/.lola-eval/config.yaml new file mode 100644 index 0000000..bd55dbf --- /dev/null +++ b/.lola-eval/config.yaml @@ -0,0 +1,39 @@ +# .lola-eval/config.yaml — Mode 1 (in-repo) spike scaffold. +# +# The project provisions its own pack (see provision.sh) and the harness +# measures the agent's behaviour against that environment. Trimmed to a +# single test cell for the initial spike; widen the matrix once the +# harness has proven itself here. + +# Cases live in `.lola-eval/tests/`, not the harness default `test_sets/`. +tests_dir: tests + +# Pinned to a concrete model id so composite scores stay reproducible. +targets: + - cli: claude-code + models: + - claude-sonnet-4-6 + +# Bare-model "none" pass doubles run cost; off until we want a lift study. +calculate_baseline: false + +threshold: + mode: absolute + tolerance: 0.15 + timeout_is_failure: true + +concurrency: 4 + +timeouts: + runner_seconds: 14400 # 4h; don't let the harness kill slow cells + agent_seconds: 1800 # 30m per agent (matches task.yaml timeout_seconds) + +judges: + - {cli: claude-code, model: claude-sonnet-4-6} # single judge keeps drift comparable + +aggregation: mean + +ci: + junit_xml: true + github_summary: true + html_report: true diff --git a/.lola-eval/provision.sh b/.lola-eval/provision.sh new file mode 100644 index 0000000..0b79d94 --- /dev/null +++ b/.lola-eval/provision.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# provision.sh — copy the current module/ into each starter repo so the +# agent under test runs /prd-review with THIS commit's version of the +# intake-kit module. +# +# Usage: provision.sh [module_dir] +# module_dir defaults to ../module (relative to this script). +# +# For each starter/ under .lola-eval/tests/: +# 1. Remove any prior .lola/ and CLI integration dirs inside the starter +# 2. Copy module/ into starter/.lola/modules/intake-kit/module/ +# 3. Write starter/.lola/modules/intake-kit/.lola/source.yml +# 4. Copy top-level AGENTS.md, README.md, LICENSE into the module root +# 5. Copy CLI integration files directly (.claude/, .opencode/) instead +# of calling `lola install` (too slow for batch use) +# 6. Write a clean .gitconfig for eval isolation +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MODULE_DIR="${1:-$(cd "$SCRIPT_DIR/../module" && pwd)}" +TESTS_DIR="$SCRIPT_DIR/tests" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +if [[ ! -d "$MODULE_DIR" ]]; then + echo "provision.sh: module dir not found: $MODULE_DIR" >&2 + exit 1 +fi + +if [[ ! -d "$TESTS_DIR" ]]; then + echo "provision.sh: tests dir not found: $TESTS_DIR" >&2 + exit 1 +fi + +provisioned=0 + +for starter in "$TESTS_DIR"/*/starter; do + [[ -d "$starter" ]] || continue + case_name="$(basename "$(dirname "$starter")")" + + # Clean prior provisioning (module files + CLI integration dirs) + rm -rf "$starter/.lola" "$starter/.claude" "$starter/.opencode" + + # Create lola module structure + mod_dest="$starter/.lola/modules/intake-kit" + mkdir -p "$mod_dest/module" + mkdir -p "$mod_dest/.lola" + + # Copy the module contents (exclude .git to avoid embedded repo warnings) + cp -a "$MODULE_DIR/." "$mod_dest/module/" + find "$mod_dest" -name .git -type d -exec rm -rf {} + 2>/dev/null || true + + # Copy top-level files that lola expects at the module root + for f in AGENTS.md README.md LICENSE CHANGELOG.md; do + if [[ -f "$PROJECT_ROOT/$f" ]]; then + cp "$PROJECT_ROOT/$f" "$mod_dest/$f" + fi + done + + # Write lola source metadata + cat > "$mod_dest/.lola/source.yml" <<'YAML' +source: local://provision +type: local +YAML + + # Install CLI integration files directly by copying from module/. This + # replaces `lola install` (15-30s per invocation). The output is + # deterministic: agents/, commands/, skills/, references/ are straight + # copies of the module source files. + for target_dir in .claude .opencode; do + mkdir -p "$starter/$target_dir/agents" + + if [[ -d "$MODULE_DIR/agents" ]]; then + cp "$MODULE_DIR/agents/"*.md "$starter/$target_dir/agents/" 2>/dev/null || true + fi + + if [[ -d "$MODULE_DIR/commands" ]]; then + mkdir -p "$starter/$target_dir/commands" + cp "$MODULE_DIR/commands/"*.md "$starter/$target_dir/commands/" 2>/dev/null || true + fi + + # Skills copy is recursive — carries the nested references/ under each skill. + if [[ -d "$MODULE_DIR/skills" ]]; then + cp -a "$MODULE_DIR/skills/." "$starter/$target_dir/skills/" + fi + + if [[ -d "$MODULE_DIR/references" ]]; then + cp -a "$MODULE_DIR/references/." "$starter/$target_dir/references/" + fi + done + + # Write clean git config for eval isolation + cat > "$starter/.gitconfig" <<'GIT' +[user] + name = lola-eval + email = eval@localhost +[commit] + gpgsign = false +[init] + defaultBranch = main +GIT + + # Create starter-clean/ — same source, no module artifacts. Used by + # pack_id=none baseline runs for genuine bare-model comparison. + clean="$TESTS_DIR/$case_name/starter-clean" + rm -rf "$clean" + cp -a "$starter" "$clean" + rm -rf "$clean/.lola" "$clean/.claude" "$clean/.opencode" + for f in AGENTS.md CLAUDE.md; do + if [[ -f "$clean/$f" ]]; then + sed -i '//,//d' "$clean/$f" + sed -i '//,//d' "$clean/$f" + sed -i '//d; //d' "$clean/$f" + sed -i '/^## Lola Skills$/,/^/d' "$clean/$f" 2>/dev/null || true + if [[ ! -s "$clean/$f" ]] || ! grep -q '[^[:space:]]' "$clean/$f" 2>/dev/null; then + rm -f "$clean/$f" + fi + fi + done + + provisioned=$((provisioned + 1)) + echo "provision.sh: provisioned $case_name (+ starter-clean)" +done + +if [[ $provisioned -eq 0 ]]; then + echo "provision.sh: no starter dirs found under $TESTS_DIR" >&2 + exit 1 +fi + +echo "provision.sh: $provisioned case(s) provisioned" diff --git a/.lola-eval/tests/case-001-unknown-executor/prompt.md b/.lola-eval/tests/case-001-unknown-executor/prompt.md new file mode 100644 index 0000000..927a296 --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/prompt.md @@ -0,0 +1,11 @@ +Review this PRD family using the intake-kit prd-review skill. + +The initiative has a parent PRD (`parent.yaml`) and one phase PRD +(`phase1.yaml`). The `prd-review` skill and its agents are pre-installed +in this project's `.lola/` directory. Run the review with the local +schema as fallback: + + /prd-review --schema prd.cue parent.yaml phase1.yaml + +Report all findings from the review, including their severity and the +specific location (file and id) each finding refers to. diff --git a/.lola-eval/tests/case-001-unknown-executor/rubric.md b/.lola-eval/tests/case-001-unknown-executor/rubric.md new file mode 100644 index 0000000..6d2afd1 --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/rubric.md @@ -0,0 +1,83 @@ +--- +rubric_version: "1" +pass_threshold: 0.6 +weights: + detection: 0.5 + evidence: 0.25 + false_positives: 0.15 + no_flapping: 0.10 +--- + +# Rubric: case-001-unknown-executor + +The agent was asked to run `/prd-review` on a two-file PRD family +(`parent.yaml` + `phase1.yaml`) that is schema-valid but contains one +seeded cross-file defect: + +- The phase journey "Create a short link" names + `executor: ghost-admin`, but the parent's `job-executors` list + declares only `end-user`. This is the **UNKNOWN_EXECUTOR** condition — + a journey referencing an executor id that no parent job-executor + defines. It is a BLOCKER and is only catchable by the Guard agent's + cross-file check, since each file passes `cue vet` in isolation. + +Score each component in [0.0, 1.0]. + +## detection (weight 0.5) + +Did the review identify the unknown executor `ghost-admin` (the journey +referencing an executor absent from the parent's job-executors list)? + +- 1.0 — the review explicitly flags `ghost-admin` as an executor not + declared in the parent (UNKNOWN_EXECUTOR or an equivalent description), + at BLOCKER severity. +- 0.5 — the review notes an executor/journey problem but is vague about + which executor is undefined or does not treat it as blocking. +- 0.0 — the defect is not reported. + +## evidence (weight 0.25) + +Does the finding cite the specific location — the phase file's journey and +the executor id `ghost-admin`, and ideally that the parent declares only +`end-user`? + +- 1.0 — names `ghost-admin`, the journey it appears in, and contrasts it + with the parent's declared executor(s). +- 0.5 — names the executor OR the journey but not both, or is approximate. +- 0.0 — no specific location cited. + +## false_positives (weight 0.15) + +Did the review avoid inventing findings that do not exist? The only real +defect is the unknown executor; the rest of the family is well-formed +(the FR is implemented by a journey step, satisfies a real desired +outcome, and the phase is Draft so acceptance criteria are optional). + +- 1.0 — zero fabricated findings. +- 0.5 — one fabricated finding. +- 0.0 — two or more fabricated findings. + +## no_flapping (weight 0.10) + +Did the agent load its instruction files (SKILL.md, phase files, agent +definitions) cleanly? + +- 1.0 — clean load, no searching or retrying. +- 0.5 — minor searching behavior. +- 0.0 — extensive searching, multiple retries, or errors. + +## output + +Return strict JSON: + +``` +{ + "components": { + "detection": "", + "evidence": "", + "false_positives": "", + "no_flapping": "" + }, + "explanation": "" +} +``` diff --git a/.lola-eval/tests/case-001-unknown-executor/starter/parent.yaml b/.lola-eval/tests/case-001-unknown-executor/starter/parent.yaml new file mode 100644 index 0000000..2292cca --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/starter/parent.yaml @@ -0,0 +1,28 @@ +header: + schema-version: "0.1.0" + version: "0.1.0" + last-updated: "2026-01-01" + +slug: link-shortener +title: "Link Shortener" +description: >- + Help users turn a long URL into a short, shareable link. + +stakeholders: + - role: "Product Owner" + handle: "@owner" + - role: "Requestor" + handle: "@requestor" + approver: true + - role: "Stakeholder Representative" + handle: "@rep" + +job-executors: + - id: end-user + label: "End User" + core-job: "turn a long URL into a short link I can share and reuse" + +desired-outcomes: + - id: DO-LS-001 + statement: "minimize the time it takes to create a short link so sharing isn't interrupted" + executor-id: end-user diff --git a/.lola-eval/tests/case-001-unknown-executor/starter/phase1.yaml b/.lola-eval/tests/case-001-unknown-executor/starter/phase1.yaml new file mode 100644 index 0000000..d140fef --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/starter/phase1.yaml @@ -0,0 +1,28 @@ +header: + schema-version: "0.1.0" + version: "0.1.0" + last-updated: "2026-01-01" + parent: link-shortener + +slug: link-shortener-phase1 +phase: "Phase 1 — Create Links" + +state: + status: Draft + +journeys: + - label: "Create a short link" + # Seeded defect: `ghost-admin` is not declared in the parent's + # job-executors list (only `end-user` is). This is well-formed per the + # CUE schema (matches the executor id regex) and passes `cue vet`, so + # only the Guard agent's cross-file check catches it: UNKNOWN_EXECUTOR. + executor: ghost-admin + steps: + - label: "Shorten a URL" + description: "The user pastes a long URL and receives a short link." + implements: [FR-LS-001] + +functional-requirements: + - id: FR-LS-001 + title: "Shorten a URL" + satisfies: [DO-LS-001] diff --git a/.lola-eval/tests/case-001-unknown-executor/starter/prd.cue b/.lola-eval/tests/case-001-unknown-executor/starter/prd.cue new file mode 100644 index 0000000..d9c8895 --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/starter/prd.cue @@ -0,0 +1,129 @@ +package prds + +#SchemaVersion: =~"^[0-9]+\\.[0-9]+\\.[0-9]+$" +#Date: =~"^[0-9]{4}-[0-9]{2}-[0-9]{2}$" + +#StakeholderRole: "Product Owner" | "Requestor" | "Stakeholder Representative" | "Technical Owner" | "Alternate Technical Owner" + +#Stakeholder: { + role: #StakeholderRole + handle: string & !="" + approver: bool | *false +} + +#PRDHeader: { + "schema-version": #SchemaVersion + version: #SchemaVersion + "last-updated": #Date + parent?: string & =~"^[a-z][a-z0-9-]*$" +} + +#JourneyStep: { + label: string & !="" + description: string & !="" + implements: [...string] & [_, ...] +} + +#Journey: { + label: string & !="" + executor: string & !="" + steps: [...#JourneyStep] & [_, ...] +} + +#JobExecutor: { + id: string & =~"^[a-z][a-z0-9-]*$" + label: string & !="" + "core-job": string & !="" +} + +#DesiredOutcome: { + id: string & =~"^DO-[A-Z]+-\\d{3}$" + statement: string & !="" + "executor-id": string & =~"^[a-z][a-z0-9-]*$" +} + +#AcceptanceCriteria: { + id: string & =~"^AC-[A-Z]+-\\d{3}-\\d{2}$" + description: string & !="" +} + +#FunctionalRequirement: { + id: string & =~"^FR-[A-Z]+-\\d{3}$" + title: string & !="" + satisfies: [...string] + "acceptance-criteria": [...#AcceptanceCriteria] +} + +#NonFunctionalRequirement: { + id: string & =~"^NFR-[A-Z]+-\\d{3}$" + title: string & !="" + description: string & !="" + satisfies?: [...string] +} + +#OpenQuestion: { + question: string & !="" + context?: string +} + +#Dependency: { + description: string & !="" + blocking: bool | *false + context?: string +} + +#State: { + status: "Draft" | "Ready" | "Approved" | "Superseded" + remarks?: string +} + +#PRDDocument: { + header: #PRDHeader + slug?: string & =~"^[a-z][a-z0-9-]*$" + + stakeholders?: [...#Stakeholder] & [_, ...] + title?: string & !="" + description?: string & !="" + features?: [...string] & [_, ...] + scope?: { + "in-scope": [...string] + "out-of-scope": [...string] + } + "nonfunctional-requirements"?: [...#NonFunctionalRequirement] & [_, ...] + + phase?: string & !="" + state?: #State + journeys?: [...#Journey] & [_, ...] + "functional-requirements"?: [...#FunctionalRequirement] & [_, ...] + dependencies?: [...#Dependency] & [_, ...] + + "open-questions"?: [...#OpenQuestion] + + if phase != _|_ { + journeys: [...#Journey] & [_, ...] + _fr="functional-requirements": [...#FunctionalRequirement] & [_, ...] + + // CUE-native FR/journey coverage floor: ORPHAN_FR (every FR is + // implemented by at least one journey step) and UNKNOWN_FR (every + // `implements` id names a real FR in this phase). A failure names + // the offending id, e.g. `_ck_orphan_fr.1: undefined field: "FR-MFP-002"`. + _frSet: {for f in _fr {(f.id): true}} + _implementedFR: {for j in journeys for s in j.steps for id in s.implements {(id): true}} + _ck_orphan_fr: [for f in _fr {_implementedFR[f.id] & true}] + _ck_unknown_fr: [for j in journeys for s in j.steps for id in s.implements {_frSet[id] & true}] + + // Maturity gate: acceptance criteria are optional at early stage (Draft) + // and required once a phase reaches Ready or Approved. + if state != _|_ { + if state.status == "Ready" || state.status == "Approved" { + "functional-requirements": [...{"acceptance-criteria": [_, ...]}] + "functional-requirements": [...{satisfies: [_, ...]}] + } + } + } + if phase == _|_ { + title: string & !="" + "job-executors": [...#JobExecutor] & [_, ...] + "desired-outcomes"?: [...#DesiredOutcome] + } +} diff --git a/.lola-eval/tests/case-001-unknown-executor/task.yaml b/.lola-eval/tests/case-001-unknown-executor/task.yaml new file mode 100644 index 0000000..4527317 --- /dev/null +++ b/.lola-eval/tests/case-001-unknown-executor/task.yaml @@ -0,0 +1,21 @@ +task_version: "2" +description: | + A two-file PRD family (parent + one Draft phase) with a single seeded + cross-file defect: the phase's journey names executor `ghost-admin`, + which is not in the parent's job-executors list (only `end-user` is). + The family is schema-valid and passes `cue vet`, so the defect is + invisible to the structural floor — only the Guard agent's cross-file + referential check should catch it (UNKNOWN_EXECUTOR, BLOCKER). +timeout_seconds: 1800 +followup_messages: + - | + Emit a structured summary of the review you just performed. + Use this exact format: + + VERDICT: + FINDINGS_COUNT: + FINDINGS: + - : + - : + FALSE_POSITIVES: + SCOPE_USED: diff --git a/.taskfiles/lola-eval.yml b/.taskfiles/lola-eval.yml new file mode 100644 index 0000000..0a30509 --- /dev/null +++ b/.taskfiles/lola-eval.yml @@ -0,0 +1,146 @@ +# .taskfiles/lola-eval.yml — eval automation for intake-kit. +# +# Adapted from the Review Council project (Apache-2.0): +# https://github.com/lolables/lola-mod-review-council +# See the repository NOTICE file for attribution. +# +# Mirrors the lola-mod-review-council methodology: the harness is +# bootstrapped into a project-local .venv via uv on first use, so no +# system RPM or sudo is required. A lola-eval already on PATH wins. +version: "3" + +vars: + VENV_DIR: "{{.ROOT_DIR}}/.venv" + # Floating on main: upstream ships no tags, so tracking the branch is how + # fixes arrive. `task lola-eval:update` is the only thing that advances an + # already-bootstrapped checkout. Edit to pin (…git@) for reproducibility. + LOLA_EVAL_SPEC: "lola-eval @ git+https://github.com/complytime-labs/lola-eval.git@main" + # A host lola-eval (RPM, pipx) wins; otherwise the bootstrap makes the + # .venv path real. Task resolves this while parsing, before any dep runs, + # so it must name a fixed path rather than report "not installed". + LOLA_EVAL: + sh: command -v lola-eval 2>/dev/null || echo "{{.VENV_DIR}}/bin/lola-eval" + BOOTSTRAP: "{{.ROOT_DIR}}/.taskfiles/scripts/ensure-lola-eval.sh" + PROVISION: "{{.ROOT_DIR}}/.lola-eval/provision.sh" + MODULE_DIR: "{{.ROOT_DIR}}/module" + RESULTS_DIR: "{{.ROOT_DIR}}/.lola-eval" + +# XDG_CACHE_HOME must be set so the eval provider and the sandbox reset +# resolve the same cache root. Git vars isolate the eval's git operations +# from the caller's global/system config. +env: + XDG_CACHE_HOME: + sh: echo "${XDG_CACHE_HOME:-$HOME/.cache}" + XDG_STATE_HOME: + sh: echo "${XDG_STATE_HOME:-$HOME/.local/state}" + GIT_CONFIG_GLOBAL: /dev/null + GIT_CONFIG_SYSTEM: /dev/null + GIT_AUTHOR_NAME: Lola Eval + GIT_AUTHOR_EMAIL: lola-eval@invalid + GIT_COMMITTER_NAME: Lola Eval + GIT_COMMITTER_EMAIL: lola-eval@invalid + +tasks: + default: + silent: true + cmd: task --list 2>/dev/null + + _ensure-lola-eval: + internal: true + desc: Install the harness into .venv when it is missing or does not match the spec + cmds: + - bash "{{.BOOTSTRAP}}" "{{.LOLA_EVAL}}" "{{.VENV_DIR}}" "{{.LOLA_EVAL_SPEC}}" + + _ensure-init: + internal: true + desc: Auto-run lola-eval init when config.yaml is missing + deps: [_ensure-lola-eval] + status: + - test -f "{{.RESULTS_DIR}}/config.yaml" + cmds: + - "{{.LOLA_EVAL}} init" + + _preflight: + internal: true + desc: Guard — verify eval runtime deps (promptfoo, bubblewrap) are on PATH + silent: true + preconditions: + - sh: command -v promptfoo >/dev/null 2>&1 + msg: | + promptfoo not found on PATH. Install the pinned version: + npm install -g promptfoo@0.121.19 + lola-eval shells to `npx --no-install promptfoo` and will NOT + download it — every cell fails `no_run_produced` until present. + - sh: command -v bwrap >/dev/null 2>&1 + msg: | + bubblewrap (bwrap) not found — eval agents would run UNSANDBOXED + and read host config/skills, skewing results. Install: + sudo dnf install -y bubblewrap + + doctor: + desc: Verify lola-eval installation and config (bootstraps harness; no API cost) + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} doctor" + + update: + desc: Reinstall the harness from the current tip of LOLA_EVAL_SPEC + cmds: + - bash "{{.BOOTSTRAP}}" --force "{{.LOLA_EVAL}}" "{{.VENV_DIR}}" "{{.LOLA_EVAL_SPEC}}" + + provision: + desc: Copy current module/ into all starter repos + cmds: + - bash "{{.PROVISION}}" "{{.MODULE_DIR}}" + sources: + - "{{.MODULE_DIR}}/**/*" + + test: + desc: Provision starters and run full eval suite + deps: [provision, _ensure-init, _preflight] + cmds: + - "{{.LOLA_EVAL}} test" + + test-case: + desc: Run a single eval case (set CASE=case-001-unknown-executor) + deps: [provision, _ensure-init, _preflight] + requires: + vars: [CASE] + cmds: + - "{{.LOLA_EVAL}} test --case {{.CASE}}" + + baseline-update: + desc: Promote last-run results to baseline.json + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} baseline update" + + baseline-diff: + desc: Compare current baseline against last run + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} baseline diff" + + report: + desc: Generate HTML report from latest run + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} report" + + clean: + desc: Remove ephemeral eval artifacts + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} clean --cache" + - | + for starter in {{.ROOT_DIR}}/.lola-eval/tests/*/starter; do + rm -rf "$starter/.lola" "$starter/.claude" "$starter/.opencode" "$starter/.gitconfig" + done + - echo "Cleaned eval artifacts and provisioned starters" + + clean-all: + desc: Remove all eval artifacts including runs.db + deps: [_ensure-init] + cmds: + - "{{.LOLA_EVAL}} clean --cache --state" + - task: clean diff --git a/.taskfiles/scripts/ensure-lola-eval.sh b/.taskfiles/scripts/ensure-lola-eval.sh new file mode 100755 index 0000000..6676487 --- /dev/null +++ b/.taskfiles/scripts/ensure-lola-eval.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# Reused verbatim from the Review Council project (Apache-2.0): +# https://github.com/lolables/lola-mod-review-council +# See the repository NOTICE file for attribution. +# +# Bring the lola-eval harness into a managed virtualenv, or confirm the one +# already there is the one the Taskfile asks for. +# +# Usage: ensure-lola-eval.sh [--force] +# +# binary the lola-eval path the Taskfile resolved. When it is not the one +# inside , the host already has lola-eval installed and +# this script keeps its hands off. +# venv-dir the virtualenv this script owns outright. It is cleared and +# rebuilt, never patched, so nothing else may live in it. +# spec a PEP 508 direct reference, e.g. +# 'lola-eval @ git+https://host/lola-eval.git@main' +# --force rebuild even when the venv already matches . With a +# floating ref the spec string never changes, so this is the only +# way to pick up new upstream commits. +# +# Why a script rather than a `status:` block on the task: Task resolves a +# global `sh:` var when it parses the file, before any dep can run, so the +# Taskfile cannot both discover lola-eval and install it in one invocation. +# The var therefore names a fixed path and this script makes that path real. +# Keeping the create/install/skip/fail decision here also puts it somewhere +# that shellcheck lints and module/tests/test-ensure-lola-eval.sh can drive. +set -euo pipefail + +force=0 +if [[ "${1:-}" == "--force" ]]; then + force=1 + shift +fi + +if [[ $# -ne 3 ]]; then + echo "Usage: ensure-lola-eval.sh [--force] " >&2 + exit 2 +fi + +binary="$1" +venv="$2" +spec="$3" +managed="$venv/bin/lola-eval" +# Inside the venv on purpose: the stamp describes that specific tree, so +# clearing the venv must take the stamp with it. Kept out of git by the +# existing .venv/ entry in .gitignore. +stamp="$venv/.lola-eval-spec" + +# A lola-eval the host installed itself — an RPM, pipx, a system package — wins +# over anything this script would build, and the Taskfile has already applied +# that precedence by the time we get here. Building a venv anyway would review +# with a different harness than the operator chose, so the only thing left to +# check is that the path actually works. +if [[ "$binary" != "$managed" ]]; then + if [[ -x "$binary" ]]; then + exit 0 + fi + echo "ERROR: lola-eval resolved to '$binary', which is not executable." >&2 + echo " Remove that entry from PATH to let this project manage its own" >&2 + echo " virtualenv, or repair the installation it points at." >&2 + exit 1 +fi + +# The hot path: this runs ahead of every eval task, so an up-to-date venv must +# cost two stat calls and a read, with no subprocess and nothing on the network. +if [[ "$force" -eq 0 && -x "$managed" && -f "$stamp" ]]; then + installed="$(cat "$stamp")" + if [[ "$installed" == "$spec" ]]; then + exit 0 + fi +fi + +if ! command -v uv >/dev/null 2>&1; then + cat >&2 <<-'MISSING_UV' + ERROR: uv not found on PATH — cannot install the lola-eval harness. + + Install it: + macOS: brew bundle (from the repo root; uv is in the Brewfile) + any: curl -LsSf https://astral.sh/uv/install.sh | sh + + Then re-run this task. To use a lola-eval you installed yourself instead, + put it on PATH and this project will defer to it. + MISSING_UV + exit 1 +fi + +echo "Installing the lola-eval harness into $venv" +echo " spec: $spec" + +# Dropped before the first thing that can fail, so that a half-built venv is +# never left carrying a stamp that says it is current. The next run then +# rebuilds instead of trusting the wreckage. +rm -f "$stamp" + +# --clear rebuilds from scratch: a respec is a version change, and layering a +# second install over the first leaves whichever files the new one does not +# overwrite. --no-project stops uv walking up to any pyproject.toml above the +# repo. The floor is lola-eval's own Requires-Python; naming a single version +# instead would reject the perfectly good interpreter most hosts already have. +uv venv --clear --no-project --python '>=3.11' "$venv" +uv pip install --quiet --python "$venv/bin/python" "$spec" + +# A resolution that succeeds without producing the console script is possible — +# a renamed entry point, a wheel built without one — and reporting success here +# would hand the caller a path that does not exist, surfacing later as a bare +# "no such file" from inside whichever eval task ran next. +if [[ ! -x "$managed" ]]; then + echo "ERROR: the install reported success but produced no executable at" >&2 + echo " $managed" >&2 + exit 1 +fi + +printf '%s\n' "$spec" >"$stamp" +"$managed" --version diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..dba7282 --- /dev/null +++ b/NOTICE @@ -0,0 +1,28 @@ +Intake Kit +Copyright 2026 Unbound Force Contributors + +https://github.com/unbound-force/intake-kit + +This product includes software developed as part of the Review Council +project (https://github.com/lolables/lola-mod-review-council), specifically +the lola-eval harness scaffolding: + + .taskfiles/scripts/ensure-lola-eval.sh (used verbatim) + .taskfiles/lola-eval.yml (adapted for this module) + +Those files are used under the Apache License, Version 2.0. Their upstream +attribution is retained per Apache License 2.0, Sections 4(c) and 4(d): + + Review Council + Copyright 2026 Unbound Force Contributors + Copyright 2026 Lolables Contributors + + The Review Council in turn derives from the Unbound Force project + (https://github.com/unbound-force/unbound-force). + +Changes from the Review Council harness: + + - Reused .taskfiles/scripts/ensure-lola-eval.sh verbatim. + - Adapted .taskfiles/lola-eval.yml to the intake-kit module layout: + provisions module/ into per-case starters, defines the discovery and + prd-review eval cases, and pins a single Claude Code target cell. diff --git a/Taskfile.yml b/Taskfile.yml new file mode 100644 index 0000000..4ce7502 --- /dev/null +++ b/Taskfile.yml @@ -0,0 +1,34 @@ +version: '3' + +includes: + lola-eval: .taskfiles/lola-eval.yml + +tasks: + test: + desc: Validate schema fixtures — testdata/valid/* must pass cue vet, testdata/invalid/* must fail it + cmds: + - | + set -u + fail=0 + for f in testdata/valid/*.yaml; do + if cue vet prd.cue "$f" -d '#PRDDocument' >/dev/null 2>&1; then + echo "PASS (valid) $(basename "$f")" + else + echo "FAIL (valid) $(basename "$f") — expected to pass but cue vet errored"; fail=1 + fi + done + for f in testdata/invalid/*.yaml; do + if cue vet prd.cue "$f" -d '#PRDDocument' >/dev/null 2>&1; then + echo "FAIL (invalid) $(basename "$f") — expected cue vet to error but it passed"; fail=1 + else + echo "PASS (invalid) $(basename "$f")" + fi + done + exit $fail + + vet-examples: + desc: Validate example PRDs against the schema + cmds: + - cue vet prd.cue examples/parent-prd.yaml -d '#PRDDocument' + - cue vet prd.cue examples/phase-prd.yaml -d '#PRDDocument' + - cue vet prd.cue examples/phase-prd-draft.yaml -d '#PRDDocument'