fix: harden prep/auto/implement against missing companion extensions - #17
fix: harden prep/auto/implement against missing companion extensions#17iddocohen wants to merge 2 commits into
Conversation
A missing companion extension used to degrade silently. prep went straight to "invoke the speckit-critique-run skill" and implement Phase 6 straight to "invoke the speckit-review-run skill", with no check that either was installed. When spec-kit 1.0.x started rejecting the review extension's manifest (#16), auto kept completing and reporting success with the review pass simply absent. An autonomous run that skips a quality gate must never look like one that passed it. Both phases now resolve a provider first, by checking the filesystem rather than asking the agent whether it "has" a skill: agents cannot reliably enumerate their own skills, and improvising around a skill that was never installed is the failure being fixed. extension - the companion extension is installed fallback - not installed, but clean-context subagents are available none - not installed and no subagent dispatch either In fallback mode the phase dispatches one clean-context subagent with an embedded brief (the six review aspects, or the dual-lens critique) and a fixed return shape, so the downstream fix-and-report logic does not branch on provider. It is a safety net with less depth than the extension, and the report says so instead of presenting it as an equivalent pass. In none mode the gate is explicitly skipped and REVIEW: none marks the run INCOMPLETE, mirroring the existing local-pass-evidence blocking rule. A missing reviewer is deliberately not a Phase 0 stop-condition. The existing aborts are there because a dirty tree or missing prep artifacts contaminate the HEAD@start..HEAD@now diff; a missing reviewer corrupts nothing, it only lowers assurance. The status lines gain one field each, which auto parses and must surface in its summary: prep: ... | CRITIQUE: <extension|fallback|none|n/a> | REASON: ... implement: ... | REVIEW: <extension|fallback|none|n/a> | REASON: ... Also documents the companions as soft dependencies and adds a Troubleshooting section for the upstream review install failure on spec-kit 1.0.x, with the interim install. Verified on specify-cli 1.0.5: the extension installs with the modified command files, all three command frontmatters parse, the producer and consumer status-line contracts match field for field, and the detection predicates discriminate correctly in a project with the review extension installed (3 of 3 signals) versus one with neither companion (0 of 3). Refs #16
d0f2685 to
8a9afef
Compare
There was a problem hiding this comment.
The framing makes sense to me - the real bug is that a skipped gate looked identical to a passed one, not the upstream manifest thing. Resolving the provider up front and not letting none report DONE both look right.
Two things I'd want sorted before merge: the detection check in implement.md isn't actually a check, and the README still leads with the install command that fails. Rest is minor.
No CI here and the verification is all self-reported, so the detection logic is the one part I can't re-run myself.
|
|
||
| | Check | `REVIEW_MODE` | | ||
| |-------|---------------| | ||
| | The `review` extension is installed — any of `.specify/extensions/review/`, a `speckit.review.run` entry in `.specify/extensions.yml`, or the harness skill directory (in Claude Code, `.claude/skills/speckit-review-run/`) | `extension` | |
There was a problem hiding this comment.
There's no actual check here, just a description of one, so the agent has to invent the filesystem test itself. Can we put the literal command in the file?
Also .claude/skills/ only exists on Claude Code, and I thought .specify/extensions.yml was our own hook wiring rather than something the installer writes. If that's right then 2 of the 3 signals never fire and you always get the fallback.
| Once **all** chunks have completed (including any retries), invoke the `speckit-review-run` skill once across the full diff (`HEAD`-at-start..`HEAD`-now). | ||
| Once **all** chunks have completed (including any retries), review the full diff (`HEAD`-at-start..`HEAD`-now). How you *produce* the findings depends on the `REVIEW_MODE` resolved in Phase 0. What you *do* with them does not. | ||
|
|
||
| **6a — `REVIEW_MODE: extension`.** Invoke the `speckit-review-run` skill once across the full diff. |
There was a problem hiding this comment.
What happens if we resolve to extension and then the skill isn't actually there? No way back to 6b from here. Doesn't happen with #16 since the install bails before writing anything, but a half-installed extension would land here and go quiet, which is the thing we're trying to stop.
|
|
||
| **Finding handling is identical in 3a and 3b:** | ||
|
|
||
| - Run the dual-lens (Product + Engineering) critique against `spec.md` and `plan.md` before any tasks are generated. |
There was a problem hiding this comment.
This looks left over from the split, 3a and 3b both already say to run the critique. Move it above 3a or drop it.
| - `STATUS: READY` — `tasks.md` was generated **and** the alignment outcome is `✅ ALIGNED`, `⚠️ MINOR DRIFT`, or `⚠️ SKIPPED`. Only in this state is the spec safe to implement. | ||
| - `STATUS: BLOCKED` — alignment is `🛑 UNRESOLVED` after the retry budget, or any phase could not complete. This is the explicit failure signal the parent `speckit-opsmill-auto` checks before deciding whether to start implementation; do not dress an unresolved run up as a success. | ||
| - `SPEC_DIR` MUST be the absolute path to the spec directory, so the parent can hand it to the implement phase without parsing prose. | ||
| - `CRITIQUE` mirrors the `CRITIQUE_MODE` resolved in Phase 3, so the parent never has to infer critique coverage from prose. A degraded mode does **not** make the run `BLOCKED` — the spec is still implementable — but it MUST be reported. Use `CRITIQUE: n/a` only when the run aborted before Phase 3. |
There was a problem hiding this comment.
Fine with REVIEW: none blocking and CRITIQUE: none not blocking, but can the reason go in the file? It's only in the PR description right now and that's the bit nobody reads later.
|
|
||
| - Take **`SPEC_DIR`** from this line as the spec directory to pass to Phase B. This is the deterministic hand-off; do not scrape the path out of free text. | ||
| - If **`STATUS: BLOCKED`** (e.g., alignment never converged within its retry budget, or a phase could not complete), surface the reason to the user and **stop**. Do **not** proceed to implementation on a misaligned or incomplete spec. | ||
| - Take **`CRITIQUE`** as the critique coverage of the run. Anything other than `extension` means the critique extension was unavailable and prep either ran a reduced fallback critique or none at all. This does **not** block Phase B — carry it into the final summary. |
There was a problem hiding this comment.
n/a doesn't really mean the extension was unavailable, it means we bailed before phase 3. Maybe "anything other than extension or n/a".
| on the machine-readable status line (`CRITIQUE:` / `REVIEW:`) and in the run | ||
| summary, so a degraded run is never silent. Install both for the full passes. | ||
|
|
||
| > **Known issue: the `review` pin above fails on spec-kit 1.0.x** with |
There was a problem hiding this comment.
The install command above this still fails though. Can we just put the working one there until upstream tags v1.0.2? Or at least move this note above it so people read it before running it.
| ### Changed | ||
| - `extension.version` bumped `1.1.0` → `1.2.0`. | ||
| - `extension.description` updated to cover the QA checklist command. | ||
| - **Status-line contract (breaking for external parsers).** `prep` now emits a |
There was a problem hiding this comment.
The new field goes in before REASON, so anything reading these positionally breaks. That's a major bump really, not 1.2.0 or 1.3.0. Probably nothing outside this repo reads them, but can you check infrahub-speckit doesn't before we settle it?
- Replace the prose provider-detection tables with a literal shell check against .specify/extensions/<id>/ and .specify/extensions/.registry, both written by `specify extension add`. Drop the two signals that could never fire: .claude/skills/ is Claude Code only, and extensions.yml carries bare ids, not speckit.<id>.run entries. - Downgrade at invocation time: if speckit-review-run / speckit-critique-run cannot be invoked, fall through to the fallback branch and report the mode that actually ran, so a half-installed extension cannot report `extension`. - Record in prep.md why CRITIQUE: none reports while REVIEW: none blocks. - prep.md: move the leftover dual-lens line above 3a. - auto.md: n/a means the phase aborted before resolving a provider, not that the extension was unavailable. - README: lead the install block with the command that works today; keep the pinned upstream install as the post-v1.0.2 form. - Bump 1.2.0 -> 2.0.0. The status-line change moves REASON, so it is major.
|
All seven fixed in 6bce0a1: the detection check is now a literal shell block keying off On versioning, agreed it is major, so |
| id: "opsmill" | ||
| name: "OpsMill Speckit Workflow" | ||
| version: "1.2.0" | ||
| version: "2.0.0" |
There was a problem hiding this comment.
I'd agree with @BeArchiTek, lets do 1.3
Refs #16.
Problem
A missing companion extension degraded silently.
prepwent straight to "invoke thespeckit-critique-runskill" andimplementPhase 6 straight to "invoke thespeckit-review-runskill", with no check that either was installed.When spec-kit 1.0.x started rejecting the review extension's manifest (#16), the effect was not a loud failure.
autokept completing and reporting success with the review pass simply absent. An autonomous run that skipped a quality gate looked exactly like one that passed it.Approach
Both phases resolve a provider before using it, by checking the filesystem rather than asking the agent whether it "has" a skill. Agents cannot reliably enumerate their own skills, and improvising around a skill that was never installed is precisely the failure being fixed.
extension.specify/extensions/<id>/, aspeckit.<id>.runentry in.specify/extensions.yml, or the harness skill dir)fallbacknoneIn fallback mode the phase dispatches one clean-context subagent with an embedded brief (the six review aspects, or the dual-lens critique) and a fixed return shape, so the downstream fix-and-report logic never branches on provider. It is a safety net with less depth than the extension, and the report is required to say so rather than present it as an equivalent pass.
In none mode the gate is explicitly skipped and
REVIEW: nonemarks the runINCOMPLETE, mirroring the existing local-pass-evidence blocking rule.A missing reviewer is deliberately not a Phase 0 stop-condition. The existing aborts are there because a dirty tree or missing prep artifacts contaminate the
HEAD@start..HEAD@nowdiff. A missing reviewer corrupts nothing, it only lowers assurance, so aborting would trade a silent gap for a hard outage.Status-line contract
One field each, which
autoparses and must surface in its summary:Existing fields keep their order and meaning. Anything outside this repo reading these lines positionally would need updating, which is why the CHANGELOG flags it. Both producer and consumer live in this repo and are updated together.
auto's Completion now has to state both values, leading with them when either gate was degraded, so a degraded run is visible at top level and not only in a report file the user may never open.Changes
commands/implement.md— Phase 0 provider resolution, Phase 6 split into 6a/6b/6c, Phase 7 §5Review mode:line, §6 must record a non-extensionmode, new review blocking rule, status linecommands/prep.md— Phase 3 provider resolution and 3a/3b/3c, Completion summary and status linecommands/auto.md— parses both new fields, required coverage statement in CompletionREADME.md— companions documented as soft dependencies, plus a Troubleshooting section for the upstream review install failure on spec-kit 1.0.x and theREVIEW: nonestateCHANGELOG.md— entries under[Unreleased], including the contract changeextension.versionis left at the unreleased1.2.0. Happy to bump to1.3.0if you would rather the status-line change carry its own version.Verification
No test framework in this repo, so these are command-file changes verified against a real CLI. On specify-cli 1.0.5:
descriptionCRITIQUEin prep vs auto,REVIEWin implement vs auto) and the mode vocabulary is identical across all three filesfallbackpathNote on scope
This hardens our side. It does not fix the upstream manifest bug that caused #16, which is tracked at ismaelJimenez/spec-kit-review#4 with a fix in PR #5. Once upstream tags v1.0.2 the README pin should move to it, and this fallback goes back to being what it should be: a net that rarely catches anything.