feat(spec,core,cli)!: a scenario's requires is checked before it runs — unmet params or services skip it with a reason; requires.plugins retires into requires.services

.changeset/20289-os-test-names-tags.md

@@ -1,3 +1,3 @@
 ---
 '@objectstack/cli': minor
 '@objectstack/core': minor
@@ -14,4 +14,4 @@
 - **`--tags TAG[,TAG...]`** runs only the scenarios carrying AT LEAST ONE of the listed tags (any-of, exact, case-sensitive) — the comma-list reading of Odoo's `--test-tags` and the everyday use of Playwright's `--grep @a|@b`. With the flag, an untagged scenario is left out. Left-out scenarios are **deselected**: not run, counted on the summary (`--tags smoke selected 1 of 4 scenarios; 3 deselected (not run, not counted as passed).`), never counted as passed. A requested tag that no loaded scenario carries is named on the summary. An empty entry (`--tags smoke,`) is refused before anything runs. Without the flag nothing changes: every scenario runs.
 - **Exit status.** A selection that matches no scenario takes the posture an empty pattern already has: exit `0` with `No scenario matched --tags …`, and exit `1` under `--fail-on-empty`, whose description now covers both cases. The `Found N test suites.` line and the `SUCCESS: All N scenarios passed.` / `FAILED: …` summary lines keep their spelling.
 - **`@objectstack/core`:** `QA.TestResult` gains `scenarioName` and `description` on every result, and `suiteName` on every result `runSuite` produces (absent only from a lone `runScenario` call, which has no suite).
-- **`@objectstack/spec`:** `TestScenario.requires` (`params`, `plugins`) is still checked by nothing — its describe() now says **NOT CHECKED** instead of reading as a guard, so a scenario that declares a plugin the target lacks still runs, and the unmet requirement surfaces only as whatever failure it causes, if any. The liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers.
+- **`@objectstack/spec`:** the liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. `TestScenario.requires`, the family's fifth key, is checked in this same release and has its own note: an unmet `params` or `services` entry skips the scenario with its reason, and `requires.plugins` is retired into `requires.services`.

.changeset/20289-requires-services-skip.md

@@ -0,0 +1,100 @@
+---
+'@objectstack/spec': minor
+'@objectstack/core': minor
+'@objectstack/cli': minor
+---
+
+feat(spec,core,cli)!: a scenario's `requires` is checked before it runs — unmet `params` or `services` SKIP it with a reason; `requires.plugins` is retired into `requires.services` (#20289)
+
+Clause-②: yes (narrowing)
+
+**BREAKING** — shipped as `minor` under the launch-window convention
+(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by
+this banner, the `(narrowing)` arm above and the ADR-0087 disposition below,
+never by the level).
+
+A Quality Protocol scenario's `requires` block declared preconditions —
+`params` (environment variables) and `plugins` (plugins that must be loaded) —
+that nothing checked: measured on a stub target, a scenario naming a missing
+plugin and an unset variable reported PASSED exactly like its no-requirements
+control. ADR-0049 enforce-or-remove, verdict ENFORCE (the mainstream has
+declared preconditions: JUnit `@EnabledIfEnvironmentVariable`, pytest `skipif`),
+ruled B for the shape: each key is judged against something `os test` can
+actually observe.
+
+- **`requires.params`** — each variable must be set to a non-empty value in the
+  environment of the process running `os test` (not the target server's, which a
+  suite cannot see). An empty value counts as unset: an unconfigured CI secret
+  arrives as an empty string.
+- **`requires.services`** (new) — each entry is a discovery service key
+  (`CoreServiceName`: `auth`, `automation`, `analytics`, `ai`, `storage`, …; a
+  misspelling is refused when the suite loads) that the target must declare
+  `enabled` with status `available` in its discovery document (ADR-0076 D12).
+  It is read from the discovery request the HTTP adapter already makes once per
+  run; a suite that requires no service issues no extra request.
+- **SKIPPED.** A scenario with an unmet entry runs no step — `setup` included —
+  and `os test` prints it with its reason, naming every unmet entry and, for a
+  service, the services the target does declare available:
+  `Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.`
+  It is counted on its own — `SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` —
+  and never as passed. Skips alone exit `0`; a run in which EVERY selected
+  scenario was skipped prints `No scenario ran: …` instead of `SUCCESS`, exits
+  `0`, and exits `1` under `--fail-on-empty`. With nothing skipped, the summary
+  lines keep their spelling.
+- **`@objectstack/core`:** `QA.TestResult` gains `status` (`'passed' | 'failed' | 'skipped'`)
+  and, on a skipped result, `skipped` (`reason`, `unmet[]`, `availableServices`);
+  `passed` stays and is `false` on a skip. `TestRunner` takes an optional
+  `{ env }` (default: this process's environment), and `TestExecutionAdapter`
+  gains an optional `readTargetServices()` — `HttpTestAdapter` answers it from
+  its one discovery probe. An adapter without it skips a service requirement
+  rather than running it.
+
+```
+FROM  { "id": "ai-summary", "requires": { "plugins": ["@objectstack/service-ai"] }, "steps": [...] }
+      -> ran anyway; the missing plugin surfaced as whatever failure it caused, or passed
+TO    -> os test refuses the suite at load:
+           ✗ scenarios.0.requires.plugins: `scenarios[].requires.plugins` was removed in
+             @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: …
+             Delete the key and name the service the scenario needs in `requires.services`, …
+             Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, …
+
+FROM  { "requires": { "services": ["ai"] }, … }   (new key)
+TO    -> against a target whose discovery does not declare `ai` enabled and available:
+           ⏭️  Scenario: Summarise an account [ai-summary] (skipped)
+              Skipped: requires.services 'ai' is not available on the target (…). The target declares available: …
+```
+
+**Fix.** `requires.plugins: ["<package>"]` → `requires.services: ["<service>"]`,
+using the mapping the refusal prints (derived from `CORE_SERVICE_PROVIDER`, the
+provider table discovery itself reports): `@objectstack/plugin-auth` → `auth`,
+`@objectstack/service-analytics` → `analytics`, `@objectstack/service-automation`
+→ `automation`, `@objectstack/service-storage` → `storage`, and so on; the `ai`
+service is provided by ObjectStack Cloud/Enterprise. A plugin that fills no
+discovery service slot has no service to require — gate that scenario with a
+`params` variable or select it with `--tags`. `tsc` refuses `plugins` at a typed
+authoring site (its input type is `never`). A `TestResult` consumer that counted
+`!passed` as a failure should read `status` — a skipped result is `passed: false`
+and is not a failure.
+
+**What does not change.** A scenario without `requires` runs exactly as before,
+and a suite that requires no service issues no discovery request it did not
+already issue.
+
+### The retirement kit
+
+- **Schema.** `TestScenarioSchema.requires` is a non-strict `z.object()`, so
+  `plugins` is a `retiredKey()` tombstone carrying its prescription (a bare
+  deletion would have stripped it in silence); `services` is new, closed over
+  `CoreServiceName`.
+- **ADR-0087.** `RETIRED_KEYS_BY_MAJOR[18]` gains `qa/TestScenario:requires.plugins`.
+  No D2 conversion: a QA suite is a loose JSON file `os test` loads, never a
+  stack collection member or a stored row. The family's D3 entry,
+  `qa-scenario-requires-plugins-retired`, carries the prescription to
+  `os migrate meta` and the upgrade guide.
+- **Ledger and docs.** `liveness/qa.json` moves `qa.scenarios.requires` from
+  `dead` to `live`, citing the runner's judgement and the adapter as producer;
+  `state-counts.md` moves `qa` to 9 live / 0 dead. The `os test` section of the
+  CLI reference documents the check, the skip line and the exit posture, and the
+  generated `qa/testing` reference page is regenerated.
+
+<!-- adr-0087: registered qa-scenario-requires-plugins-retired -->

content/docs/deployment/cli.mdx

@@ -1715,9 +1715,38 @@ out are **deselected**: not run, counted on the summary
 and never counted as passed. A listed tag that no loaded scenario carries is named
 on the summary, since a typo narrows the run without failing it, and an empty entry
 (`--tags smoke,`) is refused before anything runs. Without the flag every scenario
-runs. A scenario's `requires` block is **not checked**: a scenario that names a
-plugin the target does not load still runs, and the unmet requirement surfaces
-only as whatever failure it causes, if any — the scenario can still pass.
+runs.
+
+**A scenario's `requires` is checked before its first step.** Two keys, each
+judged against something `os test` can observe:
+
+- `requires.params` — environment variables that must be set to a non-empty
+  value **in the process running `os test`** (not on the target server, which a
+  suite cannot see): `"params": ["BILLING_SANDBOX_KEY"]`.
+- `requires.services` — service keys the target must declare in its discovery
+  document as `enabled` with status `available`, read from the same discovery
+  request the runner already makes once per run: `"services": ["ai", "analytics"]`.
+  The keys are the discovery service names (`auth`, `automation`, `storage`, …),
+  and a misspelled one is refused when the suite loads.
+
+A scenario with an unmet entry is **skipped**: none of its steps runs — `setup`
+included — and its line says why, naming every unmet entry and, for a service,
+the services the target does declare available:
+
+```text
+  ⏭️  Scenario: Summarise an account with the AI service [ai-summary] (skipped)
+     Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.
+```
+
+A skipped scenario is counted on its own —
+`SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — and
+is never counted as passed. Skips alone do not fail a run, but a run in which
+**every** selected scenario was skipped proved nothing: it prints
+`No scenario ran: all 2 selected scenarios were skipped on unmet requirements.`
+instead of `SUCCESS`, exits **0**, and exits **1** under `--fail-on-empty`. The
+retired `requires.plugins` is refused when the suite loads, with the plugin →
+service mapping in its message: no served surface lists loaded plugins, so it
+was never checkable, and `requires.services` asks the question it stood for.
 
 **A pattern that matches no suite is not a failure by default.** The run prints
 `Found 0 test suites.` — the same machine-readable line a full run prints, so a

content/docs/references/qa/testing.mdx

@@ -116,7 +116,7 @@ A complete test scenario with setup, execution steps, and teardown
 | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) |
 | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute |
 | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution |
-| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes |
+| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed |
 
 ### Nested Shape: `TestScenario.setup[number]`
 
@@ -158,8 +158,9 @@ A single step in a test scenario, consisting of an action and optional assertion
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **params** | `string[]` | optional | Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs |
-| **plugins** | `string[]` | optional | Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs |
+| **params** | `string[]` | optional | Environment variables that must be set to a non-empty value in the process running `os test` — the runner's own environment, not the target server's. An unset or empty variable skips the scenario (SKIPPED, never passed) with a reason naming the variable |
+| **services** | `Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[]` | optional | Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), read from the discovery document `os test` already fetches once per run. An entry the target does not declare that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target does declare available |
+| **plugins** | `never` | optional | [REMOVED] `scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in `requires.services`, which is judged against the services the target's discovery document declares enabled and available — an unmet entry skips the scenario and says why. Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, @objectstack/service-automation → automation, @objectstack/service-cache → cache, @objectstack/service-queue → queue, @objectstack/service-job → job, @objectstack/service-realtime → realtime, @objectstack/service-storage → storage, @objectstack/service-i18n → i18n, @objectstack/service-messaging → notification, @objectstack/metadata-protocol → ui. A plugin not listed fills no discovery service slot, so there is no service to require for it. |
 
 
 ---
@@ -224,7 +225,7 @@ A complete test scenario with setup, execution steps, and teardown
 | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) |
 | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute |
 | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution |
-| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes |
+| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed |
 
 
 ---