Repository navigation
Commit 0bbe400
feat(spec,core,cli)!: a scenario's requires is checked before it runs — unmet params or services skip it with a reason; requires.plugins retires into requires.services (#20511)
Fixes #20289
Clause-②: yes (narrowing)
**BREAKING** — `scenarios[].requires.plugins` is removed (a
`retiredKey()` tombstone whose refusal names `requires.services`).
Shipped as `minor` under the launch-window convention; the ADR-0087
disposition is `registered qa-scenario-requires-plugins-retired`,
carried by the adr-0087 marker in
`.changeset/20289-requires-services-skip.md` (the file
`check:adr-0087-registration` reads).
Executes ruling **B** on the card (`5863822176`, batch 232, item 1,
maintainer 「同意」): the fifth and last key of the `qa-runner` family. The
four sibling keys landed in PR #20341; this PR completes the card.
## What changes
- **`@objectstack/spec`** (`qa/testing.zod.ts`)
- `requires.params` — judged against the environment of the process
running `os test`: each variable must be set and non-empty (an
unconfigured CI secret arrives as an empty string, so empty counts as
unset).
- `requires.services` — NEW, `z.array(CoreServiceName)`: each key must
be declared by the target's discovery document as `enabled` with `status
=== 'available'` (ADR-0076 D12). A misspelled key is refused when the
suite loads.
- `requires.plugins` — `retiredKey()` tombstone. The prescription names
`requires.services` and carries the plugin → service mapping, derived
from `CORE_SERVICE_PROVIDER` (the provider table both discovery builders
read) so it cannot name a package that does not fill the slot. This
answers the ruling's confidence gap for out-of-repo authors.
- ADR-0087: `RETIRED_KEYS_BY_MAJOR[18]` gains
`qa/TestScenario:requires.plugins`; D3 semantic entry
`qa-scenario-requires-plugins-retired`; no D2 conversion (a QA suite is
a loose JSON file `os test` loads, never a stack collection member or a
stored row — the `rest-api-config-dead-keys-retired` precedent).
- Liveness: `qa.scenarios.requires` `dead` → `live` (evidence
`runner.ts#judgeRequirements`, `test.ts#summaryLine`; producer
`http-adapter.ts#readTargetServices` + the discovery builders, and
`test.ts#run` for the env). `state-counts.md`: `qa` 8/1 → 9/0. README
row updated.
- **`@objectstack/core`** (`qa/`)
- `TestRunner` judges `requires` before the first step, setup included.
Every unmet entry is reported (params first, then services); a service
reason lists what the target declares available.
- `TestResult.status: 'passed' | 'failed' | 'skipped'` and, on a skip,
`skipped: { reason, unmet[], availableServices? }`. `passed` stays,
`false` on a skip.
- `TestExecutionAdapter.readTargetServices?()` — optional.
`HttpTestAdapter` answers it from the SAME memoised discovery probe its
record actions use: the document is now kept instead of read for
`routes.data` and dropped. A suite that requires no service issues no
extra request (an `api_call`-only suite still probes nothing). An
adapter without the method skips a service requirement rather than
running it.
- `new TestRunner(adapter, { env })` — optional, defaults to this
process's environment.
- **`@objectstack/cli`** (`os test`)
- Prints each skipped scenario as `⏭️ Scenario: NAME [ID] (skipped)`
with a `Skipped:` reason line.
- Counts skips apart: `SUCCESS: 3 scenarios passed. 1 skipped (not run,
not counted as passed).` With nothing skipped the summary lines are
byte-identical to before.
- A run in which every selected scenario was skipped prints `No scenario
ran: …` instead of `SUCCESS`, exits 0, and exits 1 under
`--fail-on-empty` (description and flag help updated).
## Measured before building (the dispatch's mechanism assumptions)
1. **The adapter's discovery fetch — HOLDS, with one precision.**
`HttpTestAdapter` issues `GET {apiBase}/discovery` at most once per
adapter (memoised promise), and `os test` builds one adapter per run. It
was LAZY: fired only by the first record action, and it kept
`routes.data` only. Both producers (`metadata-protocol` `getDiscovery`,
`runtime` `getDiscoveryInfo`) emit `services[name].{enabled,status}`
keyed by `CoreServiceName` members only. So no fetch is added: the
`requires.services` judgement is a second trigger of the same memoised
probe, and a run still issues at most one discovery request (pinned: 4
CLI runs against a stub → 4 discovery hits).
2. **`requires` read nowhere on `main`, ledger row `dead` — HOLDS**
(`git grep` over `packages/core/src/qa` and `packages/cli/src`: zero
`scenario.requires` reads; `liveness/qa.json` row `dead`).
3. **`--fail-on-empty` today** fails exactly two things: a glob that
loaded no suite, and a `--tags` selection that selected zero scenarios.
DESELECTED scenarios never reach the runner and appear only in the
`--tags … selected N of M; K deselected` line; a SKIPPED scenario was
selected, reached the runner and was refused by its own precondition, so
it prints its own line and reason and is counted on the summary. The
all-skipped case joins the same posture as the other two.
4. **Retirement route:** `requires` is a non-strict `z.object()` ⇒
`retiredKey()` tombstone (a bare deletion would strip it silently).
ADR-0087 for a key nested at `scenarios[].requires.plugins`: one
`RETIRED_KEYS_BY_MAJOR[18]` entry spelled
`qa/TestScenario:requires.plugins` (nested key of an inline block, no
`authorable-surface/` line of its own — the
`api/RestApiConfig:documentation.enabled` precedent) plus one D3
semantic entry; no D2 conversion. The liveness walk classifies
`requires` as one block, so the tombstone needs no row.
## Pins (the ruling's five, plus the kit)
| Ruling pin | Where |
|:--|:--|
| an unmet `services` entry skips and lists the declared services |
`core/src/qa/runner.test.ts` (structured `unmet` + `availableServices`),
`cli/test/qa-requires-skip-run.test.ts` (printed reason) |
| a met one runs | same two files, CONTROL cases |
| an unmet `params` entry skips naming the variable | same two files;
the CLI run sets the variable in the child and the scenario runs |
| an all-skipped run is not a pass, and exits 1 under `--fail-on-empty`
| `cli/test/qa-requires-skip-run.test.ts` (no `SUCCESS`, exit 0; exit 1
strict) |
| a `plugins` key is refused at parse with the prescription |
`spec/src/qa/testing.test.ts` (code, path, prescription, `tsc` `never`),
`cli/test/qa-suite-schema-load.test.ts` (refused at `os test` load) |
Plus: `spec/src/qa/requires-plugins-retirement.test.ts` (ADR-0087
registration + the tree-scoped absence pin over
`packages`/`examples`/`skills`/`content`/`scripts`, inside the radius
`@objectstack/spec` already declares; registered in
`vitest.repo-tests.json`), `core/src/qa/http-adapter.test.ts` (one probe
answers both questions; failure readings),
`cli/test/qa-requires-summary.test.ts` (summary line shapes).
## Verification
All code runs at head `75d17a11` (the final commit; the only
later-than-code commit regenerated `testing.mdx`, and the targeted pass
ran on that same tree before committing it). Suites and builds ran under
`os-verify-lock`; lock seconds are shared-box figures.
- `@objectstack/spec` `test` (local): `Test Files 572 passed (572)` ·
`Tests 16797 passed | 1 todo (16798)`.
- `@objectstack/spec` `test:repo`: `Test Files 39 passed (39)` · `Tests
694 passed (694)` (includes the new tree-scoped pin).
- `@objectstack/core` `test`: `Test Files 59 passed (59)` · `Tests 1555
passed (1555)`; `src/qa` alone `69 passed`.
- `@objectstack/cli`, the `os test` files — unit tier
(`qa-suite-schema-load`, `qa-requires-summary`, `qa-tags-selection`,
`vitest-tiers-partition`): `46 passed`; integration tier
(`qa-requires-skip-run`, `qa-names-and-tags-run`, spawning
`bin/run-dev.js` against a `node:http` stub): `15 passed`. The rest of
the cli suite is declared to CI (`pnpm test` runs both tiers);
`qa-empty-glob-exit-code.e2e.test.ts` was named and matched no vitest
project in this package — NOT MEASURED here, reason: not a member of
either project.
- `typecheck` exit 0 for `@objectstack/spec`, `@objectstack/core`,
`@objectstack/cli` (each including `check:test-typecheck` OK). Reverse
check of the type channel: the `@ts-expect-error` on an authored
`requires.plugins` in `spec/src/qa/testing.test.ts` is consumed (a stale
`.d.ts` would leave it unused and red), and core/cli tests typecheck
`requires.services`, a key only the rebuilt `.d.ts` carries.
- `check:generated`: first run `✗ 1 of 15 artifact(s) stale`
(`check:docs`); `--fix` regenerated
`content/docs/references/qa/testing.mdx`; re-run `✓ All 15 generated
artifacts are up to date`.
- `check:liveness`: `qa 9 classified (live 9)`; `✓
packages/spec/liveness/state-counts.md is current`.
- **Ablation** (`scripts/ablation-replace.mjs`, committed tree):
`runner.ts` anchor `if (unmet.length === 0) return undefined;` flipped
to a greater-or-equal-zero comparison (never skip). Anchor 1 → 0, blob
`77e3f4c5a76c` → `5df21094f9fd`. `runner.test.ts`: `7 failed | 23 passed
(30)` — exactly the seven skip-asserting pins; the met-service CONTROL,
the no-probe pin, the `failed`-status pin and the 20 pre-existing tests
stayed green. Expected direction: red; observed: red. Restore: blob
`77e3f4c5a76c` == HEAD, `git diff HEAD` empty. No `dist` leg: the
subject is imported relatively from `src`.
- **Gates**: `dispatch-gates --commands --repo
objectstack-ai/objectstack` at `75d17a11` derived 122 families; all 122
ran with exit codes recorded to disk; `--ran`: `122 derived, 122 run, 0
NOT-MEASURED, 0 UNRUN`. 120 exit 0. Four first answered exit 3
(PREREQUISITE NOT MET: `check:skill-examples`,
`check:dual-build-cjs-loads`, `check:i18n-coverage`,
`check:type-check-debt`) and exit 0 after their named prerequisites were
built. Two exit 1, neither this diff's:
- `check-empty-changeset --base origin/main` — the DELIBERATE CORRECTION
of the pending note (section below). Expected red.
- `check:platform-checklist` — `areas/identity-auth.json: ABSENT SYMBOL
packages/plugins/plugin-auth/src/auth-plugin.ts#twoFactor`. The same
single problem at the base `fc0db22b` and at current `main` `2b24b8b8`
(control runs in a detached worktree); this diff touches neither file.
- **Lint, a measured narrowing:** the 25 changed paths, asked of
`eslint.config.mjs` itself: 16 linted by the config, 9 ignored
(`.md`/`.mdx`/`.json`). `eslint --no-inline-config --format json` over
the 16: 16 results, 0 errors, 0 warnings. `parserOptions.project` /
`projectService` are null for all 16 (no type-aware linting), so no
untouched file's verdict can move. The repo-wide `pnpm lint` is CI's.
- Mergeability: a driver-free `merge-tree` of this head onto `main`
`2b24b8b8` is clean (the one shared file, `migrations/registry.ts`, is
disjoint prose on main's side). `main` is not merged in; CI validates
the merge ref.
- Not measured: a booted showcase. The skip path was driven end to end
against a stub that answers discovery the way both producers do.
## Texts this PR would otherwise falsify, corrected in the same change
- `content/docs/deployment/cli.mdx` §`os test`: said `requires` is not
checked; now documents both keys, the skip line and the exit posture.
- `docs/qa/platform-checklist/areas/cli.json`
(`cli.qa-suite-execution`): knownGap and the `qa.json` source note;
revision 5 → 6.
- `packages/spec/scripts/liveness/check-liveness.mts` header comment:
named `scenario.requires` as the one key still unread.
- **A pending release note — a deliberate correction, confirmation
requested here.** `.changeset/20289-os-test-names-tags.md` (PR #20341's,
not yet released) ends with a `@objectstack/spec` bullet saying
`TestScenario.requires` "is still checked by nothing … a scenario that
declares a plugin the target lacks still runs". This PR makes that false
in the same release, so the bullet now says the ledger moved the four
keys to `live` and that `requires` is checked in this release under its
own note. Nothing else in that note changed. This is the DELIBERATE
CORRECTION class `check-empty-changeset.mjs` names: `Check Changeset` is
expected to stay RED on that row (it is not a required context),
`skip-changeset` is not applied, and the correction needs a person's
confirmation on this PR.
## Acceptance notes
- **`requires.services` is closed over `CoreServiceName`.** The ruling
says "array of service keys"; both discovery producers key `services` by
`CoreServiceName` members only, so the closed enum loses nothing and
turns a misspelling into a parse refusal instead of a runtime skip — the
contract-tightening direction. The ruling's "a misspelling skips loudly"
still holds for a valid key the target does not declare. If the seat
reads the ruling as an open vocabulary, it is a one-line change to
`z.array(z.string())`.
- **`TestResult.passed` is kept** (`false` on a skip) beside the new
`status`. A consumer that counts `!passed` as a failure reads a skip as
a failure — loud, never a silent pass; the changeset tells it to read
`status`.
- **Out of scope, unchanged:** a suite whose `scenarios` is `[]` (no
`--tags`, nothing skipped) still prints `SUCCESS: All 0 scenarios
passed.` and exits 0 under `--fail-on-empty` — the all-skipped posture
does not widen to it.
- `@objectstack/core` and `@objectstack/cli` are `minor` (additive
public fields and an optional interface method; new output and exit
posture); `@objectstack/spec` is `minor` with the BREAKING banner. All
three share the changesets fixed group.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 9e9bb46 commit 0bbe400
25 files changed
Lines changed: 1536 additions & 70 deletions
File tree
- .changeset
- content/docs
- deployment
- references/qa
- docs/qa/platform-checklist/areas
- packages
- cli
- src/commands
- test
- core/src/qa
- spec
- liveness
- scripts/liveness
- src
- migrations
- entries
- retired-keys
- semantic
- qa
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1715 | 1715 | | |
1716 | 1716 | | |
1717 | 1717 | | |
1718 | | - | |
1719 | | - | |
1720 | | - | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
| 1749 | + | |
1721 | 1750 | | |
1722 | 1751 | | |
1723 | 1752 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
119 | | - | |
| 119 | + | |
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| |||
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | | - | |
162 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
163 | 164 | | |
164 | 165 | | |
165 | 166 | | |
| |||
224 | 225 | | |
225 | 226 | | |
226 | 227 | | |
227 | | - | |
| 228 | + | |
228 | 229 | | |
229 | 230 | | |
230 | 231 | | |
| |||
0 commit comments