Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/NEXT_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,11 @@ Decision:
cron and operator-approved dogfood items. Each recommendation carries
`blocked_by_external_event` so agents can distinguish local work from
cron/operator wait states.
That local scorecard review is now applied for the four non-external
candidates: local-first privacy boundary, setup/doctor/MCP smoke, loop memory
and continuation, and release stability are 9.5/10 in the scorecard. Product
planning, Codex/Claude operator dogfood, web operations, scheduled
`ui-patrol`, and native-dialog approved dogfood remain pending.
- PR #478 moved that quality evidence gate onto the installed product CLI. Main
CI run `28753458359` passed Node 22 and Node 24 after merge, so `prompt-coach
quality-evidence --require-complete` is now a current default-branch release
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@
| Axis | Current level after latest evidence | 9.5 bar | Evidence that must exist |
| --- | --- | --- | --- |
| Product planning and positioning | 9.2/10 | 9.5 bar: every active first-screen surface, plugin surface, README path, and backlog slice says PromptLane is prompt improvement first, loop-aware continuation second, with no product-facing Loopdeck drift. | Packaging guard, README/plugin metadata, repo metadata, docs/PROMPTLANE.md, docs/NEXT_BACKLOG.md, goal audit, expected-impact evidence. |
| Local-first privacy boundary | 9.1/10 | 9.5 bar: every hook, MCP, CLI, server, web, export, loop, and dogfood path proves no prompt body, raw path, provider credential, transcript body, compact summary, or external provider call leaks outside the allowed storage layer. | Focused privacy tests, raw-free fixtures, dogfood:first-coach-loop, dogfood:loop-memory-approval, smoke:mcp-coach-loop, browser E2E, release smoke. |
| Local-first privacy boundary | 9.5/10 | 9.5 bar: every hook, MCP, CLI, server, web, export, loop, and dogfood path proves no prompt body, raw path, provider credential, transcript body, compact summary, or external provider call leaks outside the allowed storage layer. | Focused privacy tests, raw-free fixtures, dogfood:first-coach-loop, dogfood:loop-memory-approval, smoke:mcp-coach-loop, browser E2E, release smoke. |
| Codex and Claude Code integration | 9.0/10 | 9.5 bar: setup, doctor, hook capture, MCP registration, plugin install guidance, slash commands, statusline, and recovery copy are all verified for both tools in isolated smoke and at least one real operator dogfood pass. | smoke:agent-setup, smoke:hooks, dogfood:first-coach-loop, docs/DOGFOOD_CODEX_CLAUDE_2026-07-05.md, AGENT-HARNESS. |
| Setup, doctor, and MCP smoke | 9.0/10 | 9.5 bar: setup and doctor smoke proves capture readiness; MCP smoke proves score/improve/clarify/record loop; failure states produce raw-free recovery actions instead of generic errors. | smoke:agent-setup, smoke:mcp-coach-loop, storage_unavailable tests, package checks. |
| Loop memory and continuation | 9.1/10 | 9.5 bar: collect, brief, outcome, memory candidate, memory approval, instruction patch proposal, and apply gate are proven through CLI and MCP with evidence-first rules and no automatic instruction writes. | Loop unit tests, storage evidence guards, dogfood:first-coach-loop, dogfood:loop-memory-approval, prompt-linked outcome evidence. |
| Setup, doctor, and MCP smoke | 9.5/10 | 9.5 bar: setup and doctor smoke proves capture readiness; MCP smoke proves score/improve/clarify/record loop; failure states produce raw-free recovery actions instead of generic errors. | smoke:agent-setup, smoke:mcp-coach-loop, storage_unavailable tests, package checks. |
| Loop memory and continuation | 9.5/10 | 9.5 bar: collect, brief, outcome, memory candidate, memory approval, instruction patch proposal, and apply gate are proven through CLI and MCP with evidence-first rules and no automatic instruction writes. | Loop unit tests, storage evidence guards, dogfood:first-coach-loop, dogfood:loop-memory-approval, prompt-linked outcome evidence. |
| Web UI and operational evidence | 8.6/10 | 9.5 bar: archive, detail, coach, saved draft reuse, settings, loops, exports, projects, and mobile layout have screenshots or browser assertions, plus scheduled `ui-patrol` artifact evidence. | corepack pnpm ui-patrol, workflow_dispatch run `28717406758`, scheduled `ui-patrol`, browser E2E, screenshot artifacts, in-app Browser audit. |
| Release stability | 9.0/10 | 9.5 bar: Node 22 and 24 CI, pack dry-run, release smoke, first coach loop dogfood, package contents, dependency audit, and release checklist all agree on shipped files and commands. | GitHub Actions, corepack pnpm pack:dry-run, smoke:release, dogfood:first-coach-loop, dogfood:loop-memory-approval, docs/RELEASE_CHECKLIST.md. |
| Release stability | 9.5/10 | 9.5 bar: Node 22 and 24 CI, pack dry-run, release smoke, first coach loop dogfood, package contents, dependency audit, and release checklist all agree on shipped files and commands. | GitHub Actions, corepack pnpm pack:dry-run, smoke:release, dogfood:first-coach-loop, dogfood:loop-memory-approval, docs/RELEASE_CHECKLIST.md. |

## Evidence Progress Ledger

Expand Down Expand Up @@ -203,6 +203,11 @@
This strengthens the local proof for privacy, Codex/Claude integration,
setup/MCP smoke, loop memory, and release stability while keeping scheduled
`ui-patrol` and native-dialog dogfood as separate blockers.
- Local scorecard review promoted the four non-external candidate axes to
9.5/10: local-first privacy boundary, setup/doctor/MCP smoke, loop memory and
continuation, and release stability. This removes those scorecard-axis
blockers while leaving product planning, Codex/Claude operator dogfood, web
operations, scheduled `ui-patrol`, and native-dialog approved dogfood pending.
- PR #478 proved that installed CLI path on the default branch; main CI run
`28753458359` passed Node 22 and Node 24 after merge, so future agents can use
the product CLI itself to decide whether 9.5 is still blocked before claiming
Expand Down
69 changes: 42 additions & 27 deletions src/cli/commands/quality-evidence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@ describe("quality-evidence CLI command", () => {
const parsed = JSON.parse(json) as {
check: string;
status: string;
scorecard_axes: Array<{ id: string; status: string }>;
scorecard_axes: Array<{
id: string;
current_level: string;
status: string;
}>;
axis_evidence_coverage: Array<{
id: string;
status: string;
Expand All @@ -32,8 +36,37 @@ describe("quality-evidence CLI command", () => {
expect(parsed.check).toBe("promptlane_95_quality");
expect(parsed.status).toBe("pending");
expect(parsed.scorecard_axes).toHaveLength(7);
expect(parsed.scorecard_axes).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "local_first_privacy_boundary",
current_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "setup_doctor_and_mcp_smoke",
current_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "loop_memory_and_continuation",
current_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "release_stability",
current_level: "9.5/10",
status: "meets_target",
}),
]),
);
expect(parsed.axis_evidence_coverage).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "local_first_privacy_boundary",
status: "complete",
remaining_evidence: [],
}),
expect.objectContaining({
id: "web_ui_and_operational_evidence",
status: "blocked_external",
Expand All @@ -54,18 +87,7 @@ describe("quality-evidence CLI command", () => {
}),
]),
);
expect(parsed.scorecard_review_candidates).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "local_first_privacy_boundary",
required_review: "scorecard_level_below_9_5",
}),
expect.objectContaining({
id: "release_stability",
required_review: "scorecard_level_below_9_5",
}),
]),
);
expect(parsed.scorecard_review_candidates).toEqual([]);
expect(parsed.scorecard_review_candidates).not.toEqual(
expect.arrayContaining([
expect.objectContaining({
Expand All @@ -86,10 +108,10 @@ describe("quality-evidence CLI command", () => {
]),
);
expect(parsed.recommended_next_slices[0]).toMatchObject({
id: "scorecard_review_candidates",
priority: 10,
blocked_by_external_event: false,
command: "prompt-coach quality-evidence --json",
id: "scheduled_ui_patrol_cron_review",
priority: 90,
blocked_by_external_event: true,
command: "corepack pnpm evidence:ui-patrol",
});
expect(parsed.recommended_next_slices).not.toEqual(
expect.arrayContaining([
Expand Down Expand Up @@ -119,8 +141,9 @@ describe("quality-evidence CLI command", () => {
expect(text).toContain("PromptLane 9.5 quality evidence");
expect(text).toContain("Status: pending");
expect(text).toContain("Scorecard axes: 7");
expect(text).toContain("Blockers: 9");
expect(text).toContain("Blockers: 5");
expect(text).toContain("Axis evidence coverage");
expect(text).toContain("local_first_privacy_boundary: complete");
expect(text).toContain(
"web_ui_and_operational_evidence: blocked_external",
);
Expand All @@ -131,18 +154,10 @@ describe("quality-evidence CLI command", () => {
);
expect(text).toContain("remaining=native_dialog_approved_dogfood");
expect(text).toContain("Scorecard review candidates");
expect(text).toContain(
"local_first_privacy_boundary: review=scorecard_level_below_9_5",
);
expect(text).toContain(
"release_stability: review=scorecard_level_below_9_5",
);
expect(text).toContain("- none");
expect(text).toContain("scheduled_ui_patrol");
expect(text).toContain("native_dialog_approved_dogfood");
expect(text).toContain("Recommended next slices");
expect(text).toContain("scorecard_review_candidates");
expect(text).toContain("prompt-coach quality-evidence --json");
expect(text).toContain("external event: no");
expect(text).toContain("scheduled_ui_patrol_cron_review");
expect(text).toContain("corepack pnpm evidence:ui-patrol");
expect(text).toContain("external event: yes");
Expand Down
10 changes: 10 additions & 0 deletions src/packaging/plugin-files.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -973,6 +973,10 @@ describe("plugin packaging files", () => {
expect(content).toContain("scorecard_level_below_9_5");
expect(content).toContain("recommended_next_slices");
expect(content).toContain("blocked_by_external_event");
expect(content).toContain("local-first privacy boundary");
expect(content).toContain("setup/doctor/MCP smoke");
expect(content).toContain("loop memory");
expect(content).toContain("release stability");
expect(content).toContain("web_user_flow_current_main_evidence");
expect(content).toContain("corepack pnpm dogfood:web-user-flow");
expect(content).toContain("browser e2e passed");
Expand All @@ -987,6 +991,12 @@ describe("plugin packaging files", () => {
expect(content).toContain("native_dialog_approved_dogfood");
expect(content).toContain("scheduled_ui_patrol");
}
expect(plan).toContain("| Local-first privacy boundary | 9.5/10 |");
expect(plan).toContain("| Setup, doctor, and MCP smoke | 9.5/10 |");
expect(plan).toContain("| Loop memory and continuation | 9.5/10 |");
expect(plan).toContain("| Release stability | 9.5/10 |");
expect(plan).toContain("| Codex and Claude Code integration | 9.0/10 |");
expect(plan).toContain("| Web UI and operational evidence | 8.6/10 |");
for (const content of [localEvidence, backlog, plan]) {
expect(content).toContain("corepack pnpm smoke:hooks");
expect(content).toContain("hook binary smoke passed");
Expand Down
105 changes: 61 additions & 44 deletions src/packaging/quality-evidence-script.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,68 @@ describe("quality 9.5 evidence script", () => {
target_level: "9.5/10",
status: "below_target",
}),
expect.objectContaining({
id: "local_first_privacy_boundary",
current_level: "9.5/10",
target_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "setup_doctor_and_mcp_smoke",
current_level: "9.5/10",
target_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "loop_memory_and_continuation",
current_level: "9.5/10",
target_level: "9.5/10",
status: "meets_target",
}),
expect.objectContaining({
id: "web_ui_and_operational_evidence",
current_level: "8.6/10",
status: "below_target",
}),
expect.objectContaining({
id: "release_stability",
current_level: "9.5/10",
target_level: "9.5/10",
status: "meets_target",
}),
]),
);
expect(parsed.blockers).toHaveLength(5);
expect(parsed.blockers).not.toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "scorecard_axis:local_first_privacy_boundary",
}),
expect.objectContaining({
id: "scorecard_axis:setup_doctor_and_mcp_smoke",
}),
expect.objectContaining({
id: "scorecard_axis:loop_memory_and_continuation",
}),
expect.objectContaining({
id: "scorecard_axis:release_stability",
}),
]),
);
expect(parsed.blockers).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "scorecard_axis:product_planning_and_positioning",
status: "below_target",
}),
expect.objectContaining({
id: "scorecard_axis:codex_and_claude_code_integration",
status: "below_target",
}),
expect.objectContaining({
id: "scorecard_axis:web_ui_and_operational_evidence",
status: "below_target",
}),
expect.objectContaining({
id: "scheduled_ui_patrol",
status: "pending_no_schedule_run",
Expand All @@ -114,14 +167,12 @@ describe("quality 9.5 evidence script", () => {
expect.arrayContaining([
expect.objectContaining({
id: "local_first_privacy_boundary",
status: "partial",
status: "complete",
satisfied_evidence: expect.arrayContaining([
"privacy_raw_free_regression_sweep",
"local_95_evidence_sweep",
]),
remaining_evidence: expect.arrayContaining([
"scorecard_level_below_9_5",
]),
remaining_evidence: [],
}),
expect.objectContaining({
id: "web_ui_and_operational_evidence",
Expand Down Expand Up @@ -149,37 +200,7 @@ describe("quality 9.5 evidence script", () => {
]),
);
expect(parsed.scorecard_review_candidates).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "local_first_privacy_boundary",
required_review: "scorecard_level_below_9_5",
satisfied_evidence: expect.arrayContaining([
"privacy_raw_free_regression_sweep",
"local_95_evidence_sweep",
]),
}),
expect.objectContaining({
id: "setup_doctor_and_mcp_smoke",
required_review: "scorecard_level_below_9_5",
satisfied_evidence: expect.arrayContaining([
"local_95_evidence_sweep",
]),
}),
expect.objectContaining({
id: "loop_memory_and_continuation",
required_review: "scorecard_level_below_9_5",
satisfied_evidence: expect.arrayContaining([
"local_95_evidence_sweep",
]),
}),
expect.objectContaining({
id: "release_stability",
required_review: "scorecard_level_below_9_5",
satisfied_evidence: expect.arrayContaining([
"local_95_evidence_sweep",
]),
}),
]),
[],
);
expect(parsed.scorecard_review_candidates).not.toEqual(
expect.arrayContaining([
Expand All @@ -195,11 +216,11 @@ describe("quality 9.5 evidence script", () => {
"Do not claim 9.5 completion while blockers remain pending.",
);
expect(parsed.recommended_next_slices[0]).toMatchObject({
id: "scorecard_review_candidates",
axis: "scorecard_review",
priority: 10,
blocked_by_external_event: false,
command: "prompt-coach quality-evidence --json",
id: "scheduled_ui_patrol_cron_review",
axis: "web_ui_and_operational_evidence",
priority: 90,
blocked_by_external_event: true,
command: "corepack pnpm evidence:ui-patrol",
});
expect(parsed.recommended_next_slices).not.toEqual(
expect.arrayContaining([
Expand All @@ -224,10 +245,6 @@ describe("quality 9.5 evidence script", () => {
);
expect(parsed.recommended_next_slices).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "scorecard_review_candidates",
blocked_by_external_event: false,
}),
expect.objectContaining({
id: "scheduled_ui_patrol_cron_review",
blocked_by_external_event: true,
Expand Down
23 changes: 23 additions & 0 deletions tasks/todo.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# 작업 계획

## 2026-07-06 PromptLane Local Scorecard Axis Promotion

- [x] CHECK: `scorecard_review_candidates` identified four axes whose local
evidence was present and whose only remaining gap was
`scorecard_level_below_9_5`.
- [x] RED: quality evidence script and CLI tests required local-first privacy,
setup/doctor/MCP smoke, loop memory/continuation, and release stability to
report `9.5/10` and `meets_target`; tests failed while the scorecard still
reported them below target.
- [x] GREEN: the 9.5 scorecard now promotes those four non-external axes while
leaving product planning, Codex/Claude operator dogfood, web operations,
scheduled `ui-patrol`, and native-dialog approved dogfood pending.
- [x] EFFECT: `quality-evidence` now has fewer scorecard blockers and no local
scorecard-review recommendation ahead of the explicit external blockers.

### 판단 기준

- Do not promote web operations until scheduled `ui-patrol` evidence exists.
- Do not promote Codex/Claude integration until operator-approved native-dialog
dogfood evidence exists.
- Do not claim full 9.5 completion while product planning or external blockers
remain pending.

## 2026-07-06 PromptLane Scorecard Review Recommendation

- [x] CHECK: `scorecard_review_candidates` existed, but
Expand Down