Skip to content

Commit 0bbe400

Browse files
feat(spec,core,cli)!: a scenario's requires is checked before it runs — unmet params or services skip it with a reason; requires.plugins retires into requires.services (#20511)
Fixes #20289 Clause-②: yes (narrowing) **BREAKING** — `scenarios[].requires.plugins` is removed (a `retiredKey()` tombstone whose refusal names `requires.services`). Shipped as `minor` under the launch-window convention; the ADR-0087 disposition is `registered qa-scenario-requires-plugins-retired`, carried by the adr-0087 marker in `.changeset/20289-requires-services-skip.md` (the file `check:adr-0087-registration` reads). Executes ruling **B** on the card (`5863822176`, batch 232, item 1, maintainer 「同意」): the fifth and last key of the `qa-runner` family. The four sibling keys landed in PR #20341; this PR completes the card. ## What changes - **`@objectstack/spec`** (`qa/testing.zod.ts`) - `requires.params` — judged against the environment of the process running `os test`: each variable must be set and non-empty (an unconfigured CI secret arrives as an empty string, so empty counts as unset). - `requires.services` — NEW, `z.array(CoreServiceName)`: each key must be declared by the target's discovery document as `enabled` with `status === 'available'` (ADR-0076 D12). A misspelled key is refused when the suite loads. - `requires.plugins` — `retiredKey()` tombstone. The prescription names `requires.services` and carries the plugin → service mapping, derived from `CORE_SERVICE_PROVIDER` (the provider table both discovery builders read) so it cannot name a package that does not fill the slot. This answers the ruling's confidence gap for out-of-repo authors. - ADR-0087: `RETIRED_KEYS_BY_MAJOR[18]` gains `qa/TestScenario:requires.plugins`; D3 semantic entry `qa-scenario-requires-plugins-retired`; no D2 conversion (a QA suite is a loose JSON file `os test` loads, never a stack collection member or a stored row — the `rest-api-config-dead-keys-retired` precedent). - Liveness: `qa.scenarios.requires` `dead` → `live` (evidence `runner.ts#judgeRequirements`, `test.ts#summaryLine`; producer `http-adapter.ts#readTargetServices` + the discovery builders, and `test.ts#run` for the env). `state-counts.md`: `qa` 8/1 → 9/0. README row updated. - **`@objectstack/core`** (`qa/`) - `TestRunner` judges `requires` before the first step, setup included. Every unmet entry is reported (params first, then services); a service reason lists what the target declares available. - `TestResult.status: 'passed' | 'failed' | 'skipped'` and, on a skip, `skipped: { reason, unmet[], availableServices? }`. `passed` stays, `false` on a skip. - `TestExecutionAdapter.readTargetServices?()` — optional. `HttpTestAdapter` answers it from the SAME memoised discovery probe its record actions use: the document is now kept instead of read for `routes.data` and dropped. A suite that requires no service issues no extra request (an `api_call`-only suite still probes nothing). An adapter without the method skips a service requirement rather than running it. - `new TestRunner(adapter, { env })` — optional, defaults to this process's environment. - **`@objectstack/cli`** (`os test`) - Prints each skipped scenario as `⏭️ Scenario: NAME [ID] (skipped)` with a `Skipped:` reason line. - Counts skips apart: `SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` With nothing skipped the summary lines are byte-identical to before. - A run in which every selected scenario was skipped prints `No scenario ran: …` instead of `SUCCESS`, exits 0, and exits 1 under `--fail-on-empty` (description and flag help updated). ## Measured before building (the dispatch's mechanism assumptions) 1. **The adapter's discovery fetch — HOLDS, with one precision.** `HttpTestAdapter` issues `GET {apiBase}/discovery` at most once per adapter (memoised promise), and `os test` builds one adapter per run. It was LAZY: fired only by the first record action, and it kept `routes.data` only. Both producers (`metadata-protocol` `getDiscovery`, `runtime` `getDiscoveryInfo`) emit `services[name].{enabled,status}` keyed by `CoreServiceName` members only. So no fetch is added: the `requires.services` judgement is a second trigger of the same memoised probe, and a run still issues at most one discovery request (pinned: 4 CLI runs against a stub → 4 discovery hits). 2. **`requires` read nowhere on `main`, ledger row `dead` — HOLDS** (`git grep` over `packages/core/src/qa` and `packages/cli/src`: zero `scenario.requires` reads; `liveness/qa.json` row `dead`). 3. **`--fail-on-empty` today** fails exactly two things: a glob that loaded no suite, and a `--tags` selection that selected zero scenarios. DESELECTED scenarios never reach the runner and appear only in the `--tags … selected N of M; K deselected` line; a SKIPPED scenario was selected, reached the runner and was refused by its own precondition, so it prints its own line and reason and is counted on the summary. The all-skipped case joins the same posture as the other two. 4. **Retirement route:** `requires` is a non-strict `z.object()` ⇒ `retiredKey()` tombstone (a bare deletion would strip it silently). ADR-0087 for a key nested at `scenarios[].requires.plugins`: one `RETIRED_KEYS_BY_MAJOR[18]` entry spelled `qa/TestScenario:requires.plugins` (nested key of an inline block, no `authorable-surface/` line of its own — the `api/RestApiConfig:documentation.enabled` precedent) plus one D3 semantic entry; no D2 conversion. The liveness walk classifies `requires` as one block, so the tombstone needs no row. ## Pins (the ruling's five, plus the kit) | Ruling pin | Where | |:--|:--| | an unmet `services` entry skips and lists the declared services | `core/src/qa/runner.test.ts` (structured `unmet` + `availableServices`), `cli/test/qa-requires-skip-run.test.ts` (printed reason) | | a met one runs | same two files, CONTROL cases | | an unmet `params` entry skips naming the variable | same two files; the CLI run sets the variable in the child and the scenario runs | | an all-skipped run is not a pass, and exits 1 under `--fail-on-empty` | `cli/test/qa-requires-skip-run.test.ts` (no `SUCCESS`, exit 0; exit 1 strict) | | a `plugins` key is refused at parse with the prescription | `spec/src/qa/testing.test.ts` (code, path, prescription, `tsc` `never`), `cli/test/qa-suite-schema-load.test.ts` (refused at `os test` load) | Plus: `spec/src/qa/requires-plugins-retirement.test.ts` (ADR-0087 registration + the tree-scoped absence pin over `packages`/`examples`/`skills`/`content`/`scripts`, inside the radius `@objectstack/spec` already declares; registered in `vitest.repo-tests.json`), `core/src/qa/http-adapter.test.ts` (one probe answers both questions; failure readings), `cli/test/qa-requires-summary.test.ts` (summary line shapes). ## Verification All code runs at head `75d17a11` (the final commit; the only later-than-code commit regenerated `testing.mdx`, and the targeted pass ran on that same tree before committing it). Suites and builds ran under `os-verify-lock`; lock seconds are shared-box figures. - `@objectstack/spec` `test` (local): `Test Files 572 passed (572)` · `Tests 16797 passed | 1 todo (16798)`. - `@objectstack/spec` `test:repo`: `Test Files 39 passed (39)` · `Tests 694 passed (694)` (includes the new tree-scoped pin). - `@objectstack/core` `test`: `Test Files 59 passed (59)` · `Tests 1555 passed (1555)`; `src/qa` alone `69 passed`. - `@objectstack/cli`, the `os test` files — unit tier (`qa-suite-schema-load`, `qa-requires-summary`, `qa-tags-selection`, `vitest-tiers-partition`): `46 passed`; integration tier (`qa-requires-skip-run`, `qa-names-and-tags-run`, spawning `bin/run-dev.js` against a `node:http` stub): `15 passed`. The rest of the cli suite is declared to CI (`pnpm test` runs both tiers); `qa-empty-glob-exit-code.e2e.test.ts` was named and matched no vitest project in this package — NOT MEASURED here, reason: not a member of either project. - `typecheck` exit 0 for `@objectstack/spec`, `@objectstack/core`, `@objectstack/cli` (each including `check:test-typecheck` OK). Reverse check of the type channel: the `@ts-expect-error` on an authored `requires.plugins` in `spec/src/qa/testing.test.ts` is consumed (a stale `.d.ts` would leave it unused and red), and core/cli tests typecheck `requires.services`, a key only the rebuilt `.d.ts` carries. - `check:generated`: first run `✗ 1 of 15 artifact(s) stale` (`check:docs`); `--fix` regenerated `content/docs/references/qa/testing.mdx`; re-run `✓ All 15 generated artifacts are up to date`. - `check:liveness`: `qa 9 classified (live 9)`; `✓ packages/spec/liveness/state-counts.md is current`. - **Ablation** (`scripts/ablation-replace.mjs`, committed tree): `runner.ts` anchor `if (unmet.length === 0) return undefined;` flipped to a greater-or-equal-zero comparison (never skip). Anchor 1 → 0, blob `77e3f4c5a76c` → `5df21094f9fd`. `runner.test.ts`: `7 failed | 23 passed (30)` — exactly the seven skip-asserting pins; the met-service CONTROL, the no-probe pin, the `failed`-status pin and the 20 pre-existing tests stayed green. Expected direction: red; observed: red. Restore: blob `77e3f4c5a76c` == HEAD, `git diff HEAD` empty. No `dist` leg: the subject is imported relatively from `src`. - **Gates**: `dispatch-gates --commands --repo objectstack-ai/objectstack` at `75d17a11` derived 122 families; all 122 ran with exit codes recorded to disk; `--ran`: `122 derived, 122 run, 0 NOT-MEASURED, 0 UNRUN`. 120 exit 0. Four first answered exit 3 (PREREQUISITE NOT MET: `check:skill-examples`, `check:dual-build-cjs-loads`, `check:i18n-coverage`, `check:type-check-debt`) and exit 0 after their named prerequisites were built. Two exit 1, neither this diff's: - `check-empty-changeset --base origin/main` — the DELIBERATE CORRECTION of the pending note (section below). Expected red. - `check:platform-checklist` — `areas/identity-auth.json: ABSENT SYMBOL packages/plugins/plugin-auth/src/auth-plugin.ts#twoFactor`. The same single problem at the base `fc0db22b` and at current `main` `2b24b8b8` (control runs in a detached worktree); this diff touches neither file. - **Lint, a measured narrowing:** the 25 changed paths, asked of `eslint.config.mjs` itself: 16 linted by the config, 9 ignored (`.md`/`.mdx`/`.json`). `eslint --no-inline-config --format json` over the 16: 16 results, 0 errors, 0 warnings. `parserOptions.project` / `projectService` are null for all 16 (no type-aware linting), so no untouched file's verdict can move. The repo-wide `pnpm lint` is CI's. - Mergeability: a driver-free `merge-tree` of this head onto `main` `2b24b8b8` is clean (the one shared file, `migrations/registry.ts`, is disjoint prose on main's side). `main` is not merged in; CI validates the merge ref. - Not measured: a booted showcase. The skip path was driven end to end against a stub that answers discovery the way both producers do. ## Texts this PR would otherwise falsify, corrected in the same change - `content/docs/deployment/cli.mdx` §`os test`: said `requires` is not checked; now documents both keys, the skip line and the exit posture. - `docs/qa/platform-checklist/areas/cli.json` (`cli.qa-suite-execution`): knownGap and the `qa.json` source note; revision 5 → 6. - `packages/spec/scripts/liveness/check-liveness.mts` header comment: named `scenario.requires` as the one key still unread. - **A pending release note — a deliberate correction, confirmation requested here.** `.changeset/20289-os-test-names-tags.md` (PR #20341's, not yet released) ends with a `@objectstack/spec` bullet saying `TestScenario.requires` "is still checked by nothing … a scenario that declares a plugin the target lacks still runs". This PR makes that false in the same release, so the bullet now says the ledger moved the four keys to `live` and that `requires` is checked in this release under its own note. Nothing else in that note changed. This is the DELIBERATE CORRECTION class `check-empty-changeset.mjs` names: `Check Changeset` is expected to stay RED on that row (it is not a required context), `skip-changeset` is not applied, and the correction needs a person's confirmation on this PR. ## Acceptance notes - **`requires.services` is closed over `CoreServiceName`.** The ruling says "array of service keys"; both discovery producers key `services` by `CoreServiceName` members only, so the closed enum loses nothing and turns a misspelling into a parse refusal instead of a runtime skip — the contract-tightening direction. The ruling's "a misspelling skips loudly" still holds for a valid key the target does not declare. If the seat reads the ruling as an open vocabulary, it is a one-line change to `z.array(z.string())`. - **`TestResult.passed` is kept** (`false` on a skip) beside the new `status`. A consumer that counts `!passed` as a failure reads a skip as a failure — loud, never a silent pass; the changeset tells it to read `status`. - **Out of scope, unchanged:** a suite whose `scenarios` is `[]` (no `--tags`, nothing skipped) still prints `SUCCESS: All 0 scenarios passed.` and exits 0 under `--fail-on-empty` — the all-skipped posture does not widen to it. - `@objectstack/core` and `@objectstack/cli` are `minor` (additive public fields and an optional interface method; new output and exit posture); `@objectstack/spec` is `minor` with the BREAKING banner. All three share the changesets fixed group. --- _Generated by [Claude Code](https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 9e9bb46 commit 0bbe400

25 files changed

Lines changed: 1536 additions & 70 deletions

‎.changeset/20289-os-test-names-tags.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@ A Quality Protocol suite's `name`, each scenario's `name` and `description`, and
1414
- **`--tags TAG[,TAG...]`** runs only the scenarios carrying AT LEAST ONE of the listed tags (any-of, exact, case-sensitive) — the comma-list reading of Odoo's `--test-tags` and the everyday use of Playwright's `--grep @a|@b`. With the flag, an untagged scenario is left out. Left-out scenarios are **deselected**: not run, counted on the summary (`--tags smoke selected 1 of 4 scenarios; 3 deselected (not run, not counted as passed).`), never counted as passed. A requested tag that no loaded scenario carries is named on the summary. An empty entry (`--tags smoke,`) is refused before anything runs. Without the flag nothing changes: every scenario runs.
1515
- **Exit status.** A selection that matches no scenario takes the posture an empty pattern already has: exit `0` with `No scenario matched --tags …`, and exit `1` under `--fail-on-empty`, whose description now covers both cases. The `Found N test suites.` line and the `SUCCESS: All N scenarios passed.` / `FAILED: …` summary lines keep their spelling.
1616
- **`@objectstack/core`:** `QA.TestResult` gains `scenarioName` and `description` on every result, and `suiteName` on every result `runSuite` produces (absent only from a lone `runScenario` call, which has no suite).
17-
- **`@objectstack/spec`:** `TestScenario.requires` (`params`, `plugins`) is still checked by nothing — its describe() now says **NOT CHECKED** instead of reading as a guard, so a scenario that declares a plugin the target lacks still runs, and the unmet requirement surfaces only as whatever failure it causes, if any. The liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers.
17+
- **`@objectstack/spec`:** the liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. `TestScenario.requires`, the family's fifth key, is checked in this same release and has its own note: an unmet `params` or `services` entry skips the scenario with its reason, and `requires.plugins` is retired into `requires.services`.
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
'@objectstack/spec': minor
3+
'@objectstack/core': minor
4+
'@objectstack/cli': minor
5+
---
6+
7+
feat(spec,core,cli)!: a scenario's `requires` is checked before it runs — unmet `params` or `services` SKIP it with a reason; `requires.plugins` is retired into `requires.services` (#20289)
8+
9+
Clause-②: yes (narrowing)
10+
11+
**BREAKING** — shipped as `minor` under the launch-window convention
12+
(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by
13+
this banner, the `(narrowing)` arm above and the ADR-0087 disposition below,
14+
never by the level).
15+
16+
A Quality Protocol scenario's `requires` block declared preconditions —
17+
`params` (environment variables) and `plugins` (plugins that must be loaded) —
18+
that nothing checked: measured on a stub target, a scenario naming a missing
19+
plugin and an unset variable reported PASSED exactly like its no-requirements
20+
control. ADR-0049 enforce-or-remove, verdict ENFORCE (the mainstream has
21+
declared preconditions: JUnit `@EnabledIfEnvironmentVariable`, pytest `skipif`),
22+
ruled B for the shape: each key is judged against something `os test` can
23+
actually observe.
24+
25+
- **`requires.params`** — each variable must be set to a non-empty value in the
26+
environment of the process running `os test` (not the target server's, which a
27+
suite cannot see). An empty value counts as unset: an unconfigured CI secret
28+
arrives as an empty string.
29+
- **`requires.services`** (new) — each entry is a discovery service key
30+
(`CoreServiceName`: `auth`, `automation`, `analytics`, `ai`, `storage`, …; a
31+
misspelling is refused when the suite loads) that the target must declare
32+
`enabled` with status `available` in its discovery document (ADR-0076 D12).
33+
It is read from the discovery request the HTTP adapter already makes once per
34+
run; a suite that requires no service issues no extra request.
35+
- **SKIPPED.** A scenario with an unmet entry runs no step — `setup` included —
36+
and `os test` prints it with its reason, naming every unmet entry and, for a
37+
service, the services the target does declare available:
38+
`Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.`
39+
It is counted on its own — `SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` —
40+
and never as passed. Skips alone exit `0`; a run in which EVERY selected
41+
scenario was skipped prints `No scenario ran: …` instead of `SUCCESS`, exits
42+
`0`, and exits `1` under `--fail-on-empty`. With nothing skipped, the summary
43+
lines keep their spelling.
44+
- **`@objectstack/core`:** `QA.TestResult` gains `status` (`'passed' | 'failed' | 'skipped'`)
45+
and, on a skipped result, `skipped` (`reason`, `unmet[]`, `availableServices`);
46+
`passed` stays and is `false` on a skip. `TestRunner` takes an optional
47+
`{ env }` (default: this process's environment), and `TestExecutionAdapter`
48+
gains an optional `readTargetServices()` — `HttpTestAdapter` answers it from
49+
its one discovery probe. An adapter without it skips a service requirement
50+
rather than running it.
51+
52+
```
53+
FROM { "id": "ai-summary", "requires": { "plugins": ["@objectstack/service-ai"] }, "steps": [...] }
54+
-> ran anyway; the missing plugin surfaced as whatever failure it caused, or passed
55+
TO -> os test refuses the suite at load:
56+
✗ scenarios.0.requires.plugins: `scenarios[].requires.plugins` was removed in
57+
@objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: …
58+
Delete the key and name the service the scenario needs in `requires.services`, …
59+
Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, …
60+
61+
FROM { "requires": { "services": ["ai"] }, … } (new key)
62+
TO -> against a target whose discovery does not declare `ai` enabled and available:
63+
⏭️ Scenario: Summarise an account [ai-summary] (skipped)
64+
Skipped: requires.services 'ai' is not available on the target (…). The target declares available: …
65+
```
66+
67+
**Fix.** `requires.plugins: ["<package>"]` → `requires.services: ["<service>"]`,
68+
using the mapping the refusal prints (derived from `CORE_SERVICE_PROVIDER`, the
69+
provider table discovery itself reports): `@objectstack/plugin-auth` → `auth`,
70+
`@objectstack/service-analytics` → `analytics`, `@objectstack/service-automation`
71+
→ `automation`, `@objectstack/service-storage` → `storage`, and so on; the `ai`
72+
service is provided by ObjectStack Cloud/Enterprise. A plugin that fills no
73+
discovery service slot has no service to require — gate that scenario with a
74+
`params` variable or select it with `--tags`. `tsc` refuses `plugins` at a typed
75+
authoring site (its input type is `never`). A `TestResult` consumer that counted
76+
`!passed` as a failure should read `status` — a skipped result is `passed: false`
77+
and is not a failure.
78+
79+
**What does not change.** A scenario without `requires` runs exactly as before,
80+
and a suite that requires no service issues no discovery request it did not
81+
already issue.
82+
83+
### The retirement kit
84+
85+
- **Schema.** `TestScenarioSchema.requires` is a non-strict `z.object()`, so
86+
`plugins` is a `retiredKey()` tombstone carrying its prescription (a bare
87+
deletion would have stripped it in silence); `services` is new, closed over
88+
`CoreServiceName`.
89+
- **ADR-0087.** `RETIRED_KEYS_BY_MAJOR[18]` gains `qa/TestScenario:requires.plugins`.
90+
No D2 conversion: a QA suite is a loose JSON file `os test` loads, never a
91+
stack collection member or a stored row. The family's D3 entry,
92+
`qa-scenario-requires-plugins-retired`, carries the prescription to
93+
`os migrate meta` and the upgrade guide.
94+
- **Ledger and docs.** `liveness/qa.json` moves `qa.scenarios.requires` from
95+
`dead` to `live`, citing the runner's judgement and the adapter as producer;
96+
`state-counts.md` moves `qa` to 9 live / 0 dead. The `os test` section of the
97+
CLI reference documents the check, the skip line and the exit posture, and the
98+
generated `qa/testing` reference page is regenerated.
99+
100+
<!-- adr-0087: registered qa-scenario-requires-plugins-retired -->

‎content/docs/deployment/cli.mdx‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1715,9 +1715,38 @@ out are **deselected**: not run, counted on the summary
17151715
and never counted as passed. A listed tag that no loaded scenario carries is named
17161716
on the summary, since a typo narrows the run without failing it, and an empty entry
17171717
(`--tags smoke,`) is refused before anything runs. Without the flag every scenario
1718-
runs. A scenario's `requires` block is **not checked**: a scenario that names a
1719-
plugin the target does not load still runs, and the unmet requirement surfaces
1720-
only as whatever failure it causes, if any — the scenario can still pass.
1718+
runs.
1719+
1720+
**A scenario's `requires` is checked before its first step.** Two keys, each
1721+
judged against something `os test` can observe:
1722+
1723+
- `requires.params` — environment variables that must be set to a non-empty
1724+
value **in the process running `os test`** (not on the target server, which a
1725+
suite cannot see): `"params": ["BILLING_SANDBOX_KEY"]`.
1726+
- `requires.services` — service keys the target must declare in its discovery
1727+
document as `enabled` with status `available`, read from the same discovery
1728+
request the runner already makes once per run: `"services": ["ai", "analytics"]`.
1729+
The keys are the discovery service names (`auth`, `automation`, `storage`, …),
1730+
and a misspelled one is refused when the suite loads.
1731+
1732+
A scenario with an unmet entry is **skipped**: none of its steps runs — `setup`
1733+
included — and its line says why, naming every unmet entry and, for a service,
1734+
the services the target does declare available:
1735+
1736+
```text
1737+
⏭️ Scenario: Summarise an account with the AI service [ai-summary] (skipped)
1738+
Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.
1739+
```
1740+
1741+
A skipped scenario is counted on its own —
1742+
`SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — and
1743+
is never counted as passed. Skips alone do not fail a run, but a run in which
1744+
**every** selected scenario was skipped proved nothing: it prints
1745+
`No scenario ran: all 2 selected scenarios were skipped on unmet requirements.`
1746+
instead of `SUCCESS`, exits **0**, and exits **1** under `--fail-on-empty`. The
1747+
retired `requires.plugins` is refused when the suite loads, with the plugin →
1748+
service mapping in its message: no served surface lists loaded plugins, so it
1749+
was never checkable, and `requires.services` asks the question it stood for.
17211750
17221751
**A pattern that matches no suite is not a failure by default.** The run prints
17231752
`Found 0 test suites.` — the same machine-readable line a full run prints, so a

‎content/docs/references/qa/testing.mdx‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ A complete test scenario with setup, execution steps, and teardown
116116
| **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) |
117117
| **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute |
118118
| **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution |
119-
| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes |
119+
| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed |
120120

121121
### Nested Shape: `TestScenario.setup[number]`
122122

@@ -158,8 +158,9 @@ A single step in a test scenario, consisting of an action and optional assertion
158158

159159
| Property | Type | Required | Description |
160160
| :--- | :--- | :--- | :--- |
161-
| **params** | `string[]` | optional | Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs |
162-
| **plugins** | `string[]` | optional | Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs |
161+
| **params** | `string[]` | optional | Environment variables that must be set to a non-empty value in the process running `os test` — the runner's own environment, not the target server's. An unset or empty variable skips the scenario (SKIPPED, never passed) with a reason naming the variable |
162+
| **services** | `Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[]` | optional | Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), read from the discovery document `os test` already fetches once per run. An entry the target does not declare that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target does declare available |
163+
| **plugins** | `never` | optional | [REMOVED] `scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in `requires.services`, which is judged against the services the target's discovery document declares enabled and available — an unmet entry skips the scenario and says why. Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, @objectstack/service-automation → automation, @objectstack/service-cache → cache, @objectstack/service-queue → queue, @objectstack/service-job → job, @objectstack/service-realtime → realtime, @objectstack/service-storage → storage, @objectstack/service-i18n → i18n, @objectstack/service-messaging → notification, @objectstack/metadata-protocol → ui. A plugin not listed fills no discovery service slot, so there is no service to require for it. |
163164

164165

165166
---
@@ -224,7 +225,7 @@ A complete test scenario with setup, execution steps, and teardown
224225
| **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) |
225226
| **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute |
226227
| **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution |
227-
| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes |
228+
| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed |
228229

229230

230231
---

0 commit comments

Comments
 (0)