diff --git a/.changeset/20289-os-test-names-tags.md b/.changeset/20289-os-test-names-tags.md index 56d93b9abdf..f0ee66a94f7 100644 --- a/.changeset/20289-os-test-names-tags.md +++ b/.changeset/20289-os-test-names-tags.md @@ -14,4 +14,4 @@ A Quality Protocol suite's `name`, each scenario's `name` and `description`, and - **`--tags TAG[,TAG...]`** runs only the scenarios carrying AT LEAST ONE of the listed tags (any-of, exact, case-sensitive) — the comma-list reading of Odoo's `--test-tags` and the everyday use of Playwright's `--grep @a|@b`. With the flag, an untagged scenario is left out. Left-out scenarios are **deselected**: not run, counted on the summary (`--tags smoke selected 1 of 4 scenarios; 3 deselected (not run, not counted as passed).`), never counted as passed. A requested tag that no loaded scenario carries is named on the summary. An empty entry (`--tags smoke,`) is refused before anything runs. Without the flag nothing changes: every scenario runs. - **Exit status.** A selection that matches no scenario takes the posture an empty pattern already has: exit `0` with `No scenario matched --tags …`, and exit `1` under `--fail-on-empty`, whose description now covers both cases. The `Found N test suites.` line and the `SUCCESS: All N scenarios passed.` / `FAILED: …` summary lines keep their spelling. - **`@objectstack/core`:** `QA.TestResult` gains `scenarioName` and `description` on every result, and `suiteName` on every result `runSuite` produces (absent only from a lone `runScenario` call, which has no suite). -- **`@objectstack/spec`:** `TestScenario.requires` (`params`, `plugins`) is still checked by nothing — its describe() now says **NOT CHECKED** instead of reading as a guard, so a scenario that declares a plugin the target lacks still runs, and the unmet requirement surfaces only as whatever failure it causes, if any. The liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. +- **`@objectstack/spec`:** the liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. `TestScenario.requires`, the family's fifth key, is checked in this same release and has its own note: an unmet `params` or `services` entry skips the scenario with its reason, and `requires.plugins` is retired into `requires.services`. diff --git a/.changeset/20289-requires-services-skip.md b/.changeset/20289-requires-services-skip.md new file mode 100644 index 00000000000..c07c678ad14 --- /dev/null +++ b/.changeset/20289-requires-services-skip.md @@ -0,0 +1,100 @@ +--- +'@objectstack/spec': minor +'@objectstack/core': minor +'@objectstack/cli': minor +--- + +feat(spec,core,cli)!: a scenario's `requires` is checked before it runs — unmet `params` or `services` SKIP it with a reason; `requires.plugins` is retired into `requires.services` (#20289) + +Clause-②: yes (narrowing) + +**BREAKING** — shipped as `minor` under the launch-window convention +(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by +this banner, the `(narrowing)` arm above and the ADR-0087 disposition below, +never by the level). + +A Quality Protocol scenario's `requires` block declared preconditions — +`params` (environment variables) and `plugins` (plugins that must be loaded) — +that nothing checked: measured on a stub target, a scenario naming a missing +plugin and an unset variable reported PASSED exactly like its no-requirements +control. ADR-0049 enforce-or-remove, verdict ENFORCE (the mainstream has +declared preconditions: JUnit `@EnabledIfEnvironmentVariable`, pytest `skipif`), +ruled B for the shape: each key is judged against something `os test` can +actually observe. + +- **`requires.params`** — each variable must be set to a non-empty value in the + environment of the process running `os test` (not the target server's, which a + suite cannot see). An empty value counts as unset: an unconfigured CI secret + arrives as an empty string. +- **`requires.services`** (new) — each entry is a discovery service key + (`CoreServiceName`: `auth`, `automation`, `analytics`, `ai`, `storage`, …; a + misspelling is refused when the suite loads) that the target must declare + `enabled` with status `available` in its discovery document (ADR-0076 D12). + It is read from the discovery request the HTTP adapter already makes once per + run; a suite that requires no service issues no extra request. +- **SKIPPED.** A scenario with an unmet entry runs no step — `setup` included — + and `os test` prints it with its reason, naming every unmet entry and, for a + service, the services the target does declare available: + `Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.` + It is counted on its own — `SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — + and never as passed. Skips alone exit `0`; a run in which EVERY selected + scenario was skipped prints `No scenario ran: …` instead of `SUCCESS`, exits + `0`, and exits `1` under `--fail-on-empty`. With nothing skipped, the summary + lines keep their spelling. +- **`@objectstack/core`:** `QA.TestResult` gains `status` (`'passed' | 'failed' | 'skipped'`) + and, on a skipped result, `skipped` (`reason`, `unmet[]`, `availableServices`); + `passed` stays and is `false` on a skip. `TestRunner` takes an optional + `{ env }` (default: this process's environment), and `TestExecutionAdapter` + gains an optional `readTargetServices()` — `HttpTestAdapter` answers it from + its one discovery probe. An adapter without it skips a service requirement + rather than running it. + +``` +FROM { "id": "ai-summary", "requires": { "plugins": ["@objectstack/service-ai"] }, "steps": [...] } + -> ran anyway; the missing plugin surfaced as whatever failure it caused, or passed +TO -> os test refuses the suite at load: + ✗ scenarios.0.requires.plugins: `scenarios[].requires.plugins` was removed in + @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: … + Delete the key and name the service the scenario needs in `requires.services`, … + Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, … + +FROM { "requires": { "services": ["ai"] }, … } (new key) +TO -> against a target whose discovery does not declare `ai` enabled and available: + ⏭️ Scenario: Summarise an account [ai-summary] (skipped) + Skipped: requires.services 'ai' is not available on the target (…). The target declares available: … +``` + +**Fix.** `requires.plugins: [""]` → `requires.services: [""]`, +using the mapping the refusal prints (derived from `CORE_SERVICE_PROVIDER`, the +provider table discovery itself reports): `@objectstack/plugin-auth` → `auth`, +`@objectstack/service-analytics` → `analytics`, `@objectstack/service-automation` +→ `automation`, `@objectstack/service-storage` → `storage`, and so on; the `ai` +service is provided by ObjectStack Cloud/Enterprise. A plugin that fills no +discovery service slot has no service to require — gate that scenario with a +`params` variable or select it with `--tags`. `tsc` refuses `plugins` at a typed +authoring site (its input type is `never`). A `TestResult` consumer that counted +`!passed` as a failure should read `status` — a skipped result is `passed: false` +and is not a failure. + +**What does not change.** A scenario without `requires` runs exactly as before, +and a suite that requires no service issues no discovery request it did not +already issue. + +### The retirement kit + +- **Schema.** `TestScenarioSchema.requires` is a non-strict `z.object()`, so + `plugins` is a `retiredKey()` tombstone carrying its prescription (a bare + deletion would have stripped it in silence); `services` is new, closed over + `CoreServiceName`. +- **ADR-0087.** `RETIRED_KEYS_BY_MAJOR[18]` gains `qa/TestScenario:requires.plugins`. + No D2 conversion: a QA suite is a loose JSON file `os test` loads, never a + stack collection member or a stored row. The family's D3 entry, + `qa-scenario-requires-plugins-retired`, carries the prescription to + `os migrate meta` and the upgrade guide. +- **Ledger and docs.** `liveness/qa.json` moves `qa.scenarios.requires` from + `dead` to `live`, citing the runner's judgement and the adapter as producer; + `state-counts.md` moves `qa` to 9 live / 0 dead. The `os test` section of the + CLI reference documents the check, the skip line and the exit posture, and the + generated `qa/testing` reference page is regenerated. + + diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index cc9dd12178b..cc0f5769b63 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1715,9 +1715,38 @@ out are **deselected**: not run, counted on the summary and never counted as passed. A listed tag that no loaded scenario carries is named on the summary, since a typo narrows the run without failing it, and an empty entry (`--tags smoke,`) is refused before anything runs. Without the flag every scenario -runs. A scenario's `requires` block is **not checked**: a scenario that names a -plugin the target does not load still runs, and the unmet requirement surfaces -only as whatever failure it causes, if any — the scenario can still pass. +runs. + +**A scenario's `requires` is checked before its first step.** Two keys, each +judged against something `os test` can observe: + +- `requires.params` — environment variables that must be set to a non-empty + value **in the process running `os test`** (not on the target server, which a + suite cannot see): `"params": ["BILLING_SANDBOX_KEY"]`. +- `requires.services` — service keys the target must declare in its discovery + document as `enabled` with status `available`, read from the same discovery + request the runner already makes once per run: `"services": ["ai", "analytics"]`. + The keys are the discovery service names (`auth`, `automation`, `storage`, …), + and a misspelled one is refused when the suite loads. + +A scenario with an unmet entry is **skipped**: none of its steps runs — `setup` +included — and its line says why, naming every unmet entry and, for a service, +the services the target does declare available: + +```text + ⏭️ Scenario: Summarise an account with the AI service [ai-summary] (skipped) + Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata. +``` + +A skipped scenario is counted on its own — +`SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — and +is never counted as passed. Skips alone do not fail a run, but a run in which +**every** selected scenario was skipped proved nothing: it prints +`No scenario ran: all 2 selected scenarios were skipped on unmet requirements.` +instead of `SUCCESS`, exits **0**, and exits **1** under `--fail-on-empty`. The +retired `requires.plugins` is refused when the suite loads, with the plugin → +service mapping in its message: no served surface lists loaded plugins, so it +was never checkable, and `requires.services` asks the question it stood for. **A pattern that matches no suite is not a failure by default.** The run prints `Found 0 test suites.` — the same machine-readable line a full run prints, so a diff --git a/content/docs/references/qa/testing.mdx b/content/docs/references/qa/testing.mdx index f94eebcb9d2..a824479ccde 100644 --- a/content/docs/references/qa/testing.mdx +++ b/content/docs/references/qa/testing.mdx @@ -116,7 +116,7 @@ A complete test scenario with setup, execution steps, and teardown | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) | | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute | | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution | -| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes | +| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed | ### Nested Shape: `TestScenario.setup[number]` @@ -158,8 +158,9 @@ A single step in a test scenario, consisting of an action and optional assertion | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | -| **params** | `string[]` | optional | Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs | -| **plugins** | `string[]` | optional | Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs | +| **params** | `string[]` | optional | Environment variables that must be set to a non-empty value in the process running `os test` — the runner's own environment, not the target server's. An unset or empty variable skips the scenario (SKIPPED, never passed) with a reason naming the variable | +| **services** | `Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[]` | optional | Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), read from the discovery document `os test` already fetches once per run. An entry the target does not declare that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target does declare available | +| **plugins** | `never` | optional | [REMOVED] `scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in `requires.services`, which is judged against the services the target's discovery document declares enabled and available — an unmet entry skips the scenario and says why. Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, @objectstack/service-automation → automation, @objectstack/service-cache → cache, @objectstack/service-queue → queue, @objectstack/service-job → job, @objectstack/service-realtime → realtime, @objectstack/service-storage → storage, @objectstack/service-i18n → i18n, @objectstack/service-messaging → notification, @objectstack/metadata-protocol → ui. A plugin not listed fills no discovery service slot, so there is no service to require for it. | --- @@ -224,7 +225,7 @@ A complete test scenario with setup, execution steps, and teardown | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) | | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute | | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution | -| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes | +| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed | --- diff --git a/docs/qa/platform-checklist/areas/cli.json b/docs/qa/platform-checklist/areas/cli.json index d328057fed9..e47e72af417 100644 --- a/docs/qa/platform-checklist/areas/cli.json +++ b/docs/qa/platform-checklist/areas/cli.json @@ -438,7 +438,7 @@ "title": "os test: a Quality Protocol suite is validated at LOAD, executed against a booted app, and its verdict is the exit code — capture/interpolation thread state, an unevaluable assertion FAILS", "since": "v17", "status": "active", - "revision": 5, + "revision": 6, "priority": "P1", "surface": "cli", "personas": ["operator (local shell)", "suite author (writes qa/*.test.json)"], @@ -453,7 +453,7 @@ "knownGaps": [ "`run_script` has no adapter branch and fails by name (`Unsupported action type in HttpAdapter: run_script`) — the variant sweep records ONE refusal and that is the honest verdict, not a fixture gap to work around. The other seven members execute since #7848; the five record-shaped ones did not until then (see `negative`), so a sweep transcript predating that fix shows five 404s and is not comparable", "the record action types resolve the Data Protocol mount from the server's own `/discovery` (`routes.data`) since #7983, so a deployment that sets `crud.dataPrefix` IS reached; what remains out of reach is `api.apiPath`, which moves the discovery document itself out from under the probe (measured: `{apiBase}/discovery` 404s, and `/.well-known/objectstack` reports the DISPATCHER's `/api/v1/data`, not the REST mount). Against such a host the adapter falls back to the convention and says so — it names the mount it addressed, the probe that failed and the remedy, on the warning AND on every 404 — and those steps must still be written as `api_call`. The fixture boot is stock, so neither case bites here", - "scenario SELECTION is by tag only, and preconditions are NOT honoured: `os test --tags a,b` runs the scenarios carrying at least one listed tag (any-of, exact, case-sensitive — commands/test.ts `selectScenariosByTags`), counts the rest as deselected — not run, never counted as passed — and without the flag the whole glob runs; `scenario.requires` (`params`/`plugins`) is declared but NOT CHECKED — its describe() says so and its row in packages/spec/liveness/qa.json stays `dead` pending the maintainer's decision on #20289 — so a suite still cannot declare a precondition it will be skipped for, and a scenario naming a plugin the target lacks still runs, the unmet requirement surfacing only as whatever failure it causes, if any" + "scenario SELECTION is by tag only: `os test --tags a,b` runs the scenarios carrying at least one listed tag (any-of, exact, case-sensitive — commands/test.ts `selectScenariosByTags`), counts the rest as deselected — not run, never counted as passed — and without the flag the whole glob runs. Preconditions SKIP rather than select: a scenario whose `requires.params` variable is unset or empty in the `os test` process, or whose `requires.services` entry the target's discovery does not declare enabled and available, is skipped with its reason (core runner.ts `judgeRequirements`), counted apart and never passed. The fixture suite declares no `requires`, so the green path exercises no skip; a skip probe must be STAGED — a scenario requiring a service the stock showcase boot does not declare available (`search` is one: nothing in the open framework registers that slot) — and an all-skipped run exits 0, or 1 under --fail-on-empty" ] }, "steps": [ @@ -548,7 +548,7 @@ "packages/core/src/qa/runner.ts#scenario (scenario sequencing, `capture` + `{{var}}` interpolation, the assertion operators, setup/teardown semantics, the #7256 unevaluable-`contains` fix)", "packages/core/src/qa/http-adapter.ts#mount (the action-type switch — its case labels ARE the enum values; the record routes take their prefix from the one memoised `/discovery` probe per run, falling back to the RestApiConfigSchema + CrudEndpointsConfigSchema convention with a diagnostic that names the mount, #7848 / #7983)", "packages/spec/src/qa/testing.zod.ts#TestSuiteSchema (TestSuiteSchema — the shape enforced at load; TestActionTypeSchema pinned above)", - "packages/spec/liveness/qa.json#tags (the ADR-0049 ledger whose existence this item is coverage.json's mapping for — its `tags` row is live on `os test --tags`, and its `requires` row stays dead: the precondition is declared and not checked)", + "packages/spec/liveness/qa.json#tags (the ADR-0049 ledger whose existence this item is coverage.json's mapping for — its `tags` row is live on `os test --tags`, and its `requires` row is live on the runner's precondition judgement: an unmet `params` or `services` entry skips the scenario)", "packages/cli/test/qa-suite-schema-load.test.ts, packages/cli/test/resolve-glob-lazy-walk.test.ts, packages/core/src/qa/runner.test.ts (the three unit pins — cited so a run knows what is already covered, NOT a substitute for driving a booted app)", "content/docs/deployment/cli.mdx §os test (the documented command contract)", "examples/app-showcase/qa/platform-smoke.test.json (the fixture suite this item drives)" @@ -558,7 +558,8 @@ {"revision": 2, "date": "2026-08-12", "change": "the adapter was repaired, which this item's own `negative` clause declared to be a revision rather than a silent green (#7848). Item 1: the five record-shaped action types now round-trip against a stock server — the `${baseUrl}/api/data/:object` literal became a prefix DERIVED from the two schemas RestServer itself resolves from (RestApiConfigSchema `apiPath ?? {basePath}/{version}` + CrudEndpointsConfigSchema.dataPrefix), and `update_record` PATCHes where it used to PUT a route that has no PUT sibling. Re-measured on a booted showcase, one scenario per member with NO shared setup so no member's verdict is inferred from a sibling: 7 of 8 execute and assert, `run_script` still refuses by name. The `negative` clause is inverted accordingly — a 404 from a record action type is the regression now — and the knownGap it rested on is replaced by the narrower one that survives: the record types address the DEFAULT mount only, so a host that moves it with `api.apiPath`/`crud.dataPrefix` still needs `api_call`. Item 2: a zero-match glob still exits 0, deliberately (a repo that legitimately ships no suites must not start failing CI), but the posture is now DECLARED — stated in `--help`, opt out with the new `--fail-on-empty`, and `Found N test suites.` is emitted on EVERY run including `Found 0 test suites.`, which is the line this item's first acceptance clause already asks a run record to quote", "ref": "#7848"}, {"revision": 3, "date": "2026-08-17", "change": "the record action types stopped ASSUMING the mount (#7983). They now resolve it from the server: one memoised `GET {apiBase}/discovery` per run, addressing whatever `routes.data` advertises, with the RestApiConfigSchema + CrudEndpointsConfigSchema convention as the fallback — the `@objectstack/client` `getRoute` pattern, copied rather than re-invented. Measured on a booted stack (REST generator + dispatcher bridge) before and after, three configs: stock stays green; `crud.dataPrefix: '/objects'` went from `HTTP Error 404` to a created record, so that row of the gap is CLOSED; `api.apiPath: '/api/2026-01'` still 404s and is NOT closed — `apiPath` moves the discovery document itself, and the one fixed-path document (`/.well-known/objectstack`) advertises the dispatcher's `/api/v1/data` under all three configs, so trusting it would attach a false provenance to the same 404. That row is narrowed instead: the fallback is announced (a warning naming the mount, the failed probe and the remedy) and every 404 from a record action now carries the mount it addressed and where that mount came from, so the failure can no longer read as the suite author's own URL mistake. `api_call` is unchanged and probes nothing — it remains the escape hatch for a host the probe cannot reach", "ref": "#7983"}, { "revision": 4, "date": "2026-08-18", "change": "corrected the flag count from two to three. packages/cli/src/commands/test.ts declares url, token AND fail-on-empty; the same revision's own acceptance already describes --fail-on-empty, so the knownGap contradicted its own item. The gap's point is unchanged and now stated directly: none of the three flags selects scenarios (#9417)", "ref": "#9386" }, - { "revision": 5, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified by #20289 (PR #20341) and are corrected: `os test` gained `--tags` (any-of, exact, case-sensitive; deselected scenarios are counted and never passed; a selection that matches no scenario exits 0, or 1 under --fail-on-empty), and the report now heads each suite with its `name` and prints each scenario's `name` beside its id. What stays true is narrower: `scenario.requires` is declared but NOT CHECKED, pending the maintainer's decision on #20289. Steps and acceptance are unchanged — the green-path clause's scenarioId is still on every scenario line, now in brackets after the name, or bare when the name equals the id", "ref": "#20289" } + { "revision": 5, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified by #20289 (PR #20341) and are corrected: `os test` gained `--tags` (any-of, exact, case-sensitive; deselected scenarios are counted and never passed; a selection that matches no scenario exits 0, or 1 under --fail-on-empty), and the report now heads each suite with its `name` and prints each scenario's `name` beside its id. What stays true is narrower: `scenario.requires` is declared but NOT CHECKED, pending the maintainer's decision on #20289. Steps and acceptance are unchanged — the green-path clause's scenarioId is still on every scenario line, now in brackets after the name, or bare when the name equals the id", "ref": "#20289" }, + { "revision": 6, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified again by #20289's `requires` round (ruling B) and are corrected: `scenario.requires` is now CHECKED before the first step — `params` against the `os test` process environment, `services` against the target's discovery `services` (enabled and `available`) — and an unmet entry SKIPS the scenario with its reason, counted apart and never passed; an all-skipped run exits 0, or 1 under --fail-on-empty. `requires.plugins` is retired (refused at load, prescription names `requires.services`). Steps and acceptance are unchanged: the fixture suite declares no `requires`, so the green path runs every scenario exactly as before", "ref": "#20289" } ] }, { diff --git a/packages/cli/src/commands/test.ts b/packages/cli/src/commands/test.ts index 774d2436cf4..d1733563922 100644 --- a/packages/cli/src/commands/test.ts +++ b/packages/cli/src/commands/test.ts @@ -329,6 +329,18 @@ export function tagSelectionLine(selected: number, deselected: number, tags: rea return `--tags ${tags.join(',')} selected ${selected} of ${selected + deselected} scenarios; ${deselected} deselected (not run, not counted as passed).`; } +/** + * The run's closing tally. A SKIPPED scenario — one whose `requires` did not + * hold, so no step ran — is its own count: never passed, never failed. When + * nothing was skipped the line reads exactly as it did before skips existed. + */ +export function summaryLine(passed: number, failed: number, skipped: number): string { + const skippedClause = skipped > 0 ? ` ${skipped} skipped (not run, not counted as passed).` : ''; + if (failed > 0) return `FAILED: ${failed} scenarios failed. ${passed} passed.${skippedClause}`; + if (skipped > 0) return `SUCCESS: ${passed} scenarios passed.${skippedClause}`; + return `SUCCESS: All ${passed} scenarios passed.`; +} + export default class Test extends Command { /** * The empty-match posture is stated in the help text on purpose (#7848). @@ -353,7 +365,13 @@ export default class Test extends Command { 'case-sensitive). Untagged scenarios are left out whenever --tags is given. Left-out ' + 'scenarios are counted as deselected — never run and never counted as passed. A ' + 'selection that matches no scenario exits 0 like an empty pattern, and 1 under ' + - '--fail-on-empty.'; + '--fail-on-empty.\n' + + 'A scenario whose "requires" does not hold — an environment variable in "params" ' + + 'unset or empty in this process, or a service in "services" the target\'s discovery ' + + 'document does not declare enabled and available — is SKIPPED: no step runs, the ' + + 'reason is printed, and it is counted as skipped, never as passed. Skips alone do ' + + 'not fail the run, but a run in which every selected scenario was skipped is not a ' + + 'pass either: it exits 0, and 1 under --fail-on-empty.'; static override args = { files: Args.string({ description: 'Glob pattern for test files (e.g. "qa/*.test.json")', required: false, default: 'qa/*.test.json' }), @@ -364,7 +382,7 @@ export default class Test extends Command { token: Flags.string({ description: 'Authentication token' }), 'fail-on-empty': Flags.boolean({ description: - 'Exit non-zero when the pattern matches no test suite, or --tags matches no scenario (default: both exit 0)', + 'Exit non-zero when the pattern matches no test suite, --tags matches no scenario, or every selected scenario was skipped on an unmet "requires" (default: all three exit 0)', default: false, }), // A custom flag so a malformed list is refused while the invocation is @@ -387,7 +405,9 @@ export default class Test extends Command { // 1. Setup Runner const adapter = new CoreQA.HttpTestAdapter(flags.url, flags.token); - const runner = new CoreQA.TestRunner(adapter); + // `requires.params` is judged against THIS process's environment — the one + // running the suite — which is what the spec declares. + const runner = new CoreQA.TestRunner(adapter, { env: process.env }); // 2. Find test files using glob-style pattern matching const testFiles: string[] = resolveGlob(filesPattern); @@ -411,6 +431,7 @@ export default class Test extends Command { const tags = flags.tags; let totalPassed = 0; let totalFailed = 0; + let totalSkipped = 0; let totalSelected = 0; let totalDeselected = 0; const matchedTags = new Set(); @@ -444,6 +465,15 @@ export default class Test extends Command { const results = await runner.runSuite(suite); for (const result of results) { + if (result.status === 'skipped') { + // Not run, so not a pass and not a failure: its own line, + // its own count, and the reason in full — the unmet entry + // and, for a service, what the target does serve. + console.log(` ⏭️ Scenario: ${scenarioLabel(result)} (skipped)`); + console.log(chalk.yellow(` Skipped: ${result.skipped?.reason ?? 'a requirement was not met.'}`)); + totalSkipped++; + continue; + } const icon = result.passed ? '✅' : '❌'; console.log(` ${icon} Scenario: ${scenarioLabel(result)} (${result.duration}ms)`); if (!result.passed) { @@ -482,7 +512,7 @@ export default class Test extends Command { } } if (totalFailed > 0) { - console.log(chalk.red(`FAILED: ${totalFailed} scenarios failed. ${totalPassed} passed.`)); + console.log(chalk.red(summaryLine(totalPassed, totalFailed, totalSkipped))); process.exit(1); } else if (tags && totalSelected === 0) { // The --tags twin of the empty pattern above, and the same posture: a @@ -496,8 +526,19 @@ export default class Test extends Command { } console.log(chalk.dim(`Exiting 0 — a selection that matches nothing is not a failure. Pass --fail-on-empty to make it one.`)); process.exit(0); + } else if (totalSkipped > 0 && totalPassed === 0) { + // Every selected scenario was skipped, so the run proved nothing: not + // a SUCCESS, and — the same posture as an empty pattern and an empty + // --tags selection — a failure only under --fail-on-empty. + console.warn(chalk.yellow(`No scenario ran: all ${totalSkipped} selected scenarios were skipped on unmet requirements.`)); + if (flags['fail-on-empty']) { + console.error(chalk.red(`--fail-on-empty: a run in which every scenario was skipped is a failed run.`)); + process.exit(1); + } + console.log(chalk.dim(`Exiting 0 — skipped scenarios are not failures. Pass --fail-on-empty to make a run that ran nothing one.`)); + process.exit(0); } else { - console.log(chalk.green(`SUCCESS: All ${totalPassed} scenarios passed.`)); + console.log(chalk.green(summaryLine(totalPassed, totalFailed, totalSkipped))); process.exit(0); } } diff --git a/packages/cli/test/qa-requires-skip-run.test.ts b/packages/cli/test/qa-requires-skip-run.test.ts new file mode 100644 index 00000000000..2d3c06c6b5c --- /dev/null +++ b/packages/cli/test/qa-requires-skip-run.test.ts @@ -0,0 +1,197 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * PIN — what `os test` PRINTS and how it EXITS when a scenario's `requires` + * does not hold (the `qa-runner` family's `requires` key, ruled B), measured on + * a real child process against a real HTTP target. + * + * Before: `requires` was declared and checked by nothing — a scenario naming a + * service or a variable the run lacked ran anyway and reported whatever its + * steps happened to produce, PASSED included. Now core's TestRunner judges it + * before the first step and `os test` reports an unmet scenario as SKIPPED + * with its reason, counted apart and never as passed. The runner's judgement is + * held by `packages/core/src/qa/runner.test.ts`; this file holds the wiring + * those unit pins cannot see — the discovery document reaching the judgement + * through the adapter's one probe, the runner reading THIS process's + * environment, the reason reaching stdout, and the exit status a CI step reads. + * + * The fixture is built so each run's EXIT STATUS is itself evidence: every + * gated scenario asserts a health status the stub never returns, so it can + * only exit 0 if the scenario was really skipped — not run, and not merely + * hidden from the report. + * + * A real child process, because `process.exit(1)` inside a vitest worker is + * not an exit status. Spawned through `bin/run-dev.js` + tsx so the suite does + * not depend on `packages/cli/dist`. The target is a stub `node:http` server + * in this process that answers discovery the way both real producers do; no + * ObjectStack server is booted. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { execFile } from 'node:child_process'; +import { createServer, type Server } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { CLI, TSX, childEnv } from './helpers/serve-process.js'; + +/** oclif + tsx cold start; a healthy run here is a few seconds. */ +const RUN_TIMEOUT_MS = 120_000; + +/** The variable the `params`-gated scenario needs — test-only, so `OS_TEST_*`. */ +const PARAM = 'OS_TEST_QA_REQUIRES_PROBE_TOKEN'; + +interface Run { + code: number; + stdout: string; + stderr: string; +} + +function runCli(args: string[], cwd: string, env: Record = {}): Promise { + return new Promise((resolvePromise) => { + execFile( + TSX, + [CLI, ...args], + // The variable is REMOVED unless the run sets it, so an operator's shell + // cannot decide what the unset-variable run measures. + { cwd, maxBuffer: 8 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', [PARAM]: undefined, ...env }) }, + (err, stdout, stderr) => { + resolvePromise({ + // `err.code` is the real exit status; a signalled child is never read as 0. + code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0, + stdout: String(stdout), + stderr: String(stderr), + }); + }, + ); + }); +} + +/** One `api_call` step against the stub's health route, asserting `data.status`. */ +function healthStep(expectedStatus: string) { + return { + name: `status is ${expectedStatus}`, + action: { type: 'api_call', target: '/api/v1/health', payload: { method: 'GET' } }, + assertions: [{ field: 'data.status', operator: 'equals', expectedValue: expectedStatus }], + }; +} + +/** The discovery `services` map the stub serves — both halves of availability matter. */ +const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + auth: { enabled: true, status: 'available', route: '/api/v1/auth' }, + ai: { enabled: false, status: 'unavailable', message: 'No implementation ships in the open framework' }, + metadata: { enabled: true, status: 'degraded', route: '/api/v1/meta' }, +}; + +const MIXED = { + name: 'Requires probe', + scenarios: [ + { id: 'plain', name: 'No requirements', steps: [healthStep('ok')] }, + { id: 'svc-met', name: 'Needs data', requires: { services: ['data'] }, steps: [healthStep('ok')] }, + // Would FAIL if it ran: the stub never answers `degraded`. + { id: 'svc-unmet', name: 'Needs ai', requires: { services: ['ai'] }, steps: [healthStep('degraded')] }, + { id: 'param-gated', name: 'Needs a token', requires: { params: [PARAM] }, steps: [healthStep('ok')] }, + ], +}; + +const ALL_SKIPPED = { + name: 'Nothing here can run', + scenarios: [ + { id: 'ai-1', name: 'Needs ai', requires: { services: ['ai'] }, steps: [healthStep('degraded')] }, + { id: 'meta-1', name: 'Needs metadata available', requires: { services: ['metadata'] }, steps: [healthStep('degraded')] }, + ], +}; + +let dir: string; +let server: Server; +let discoveryHits = 0; +let unset: Run; +let withParam: Run; +let allSkipped: Run; +let allSkippedStrict: Run; + +beforeAll(async () => { + server = createServer((req, res) => { + res.setHeader('content-type', 'application/json'); + if (req.url === '/api/v1/discovery') { + discoveryHits += 1; + res.end(JSON.stringify({ version: 'v1', routes: { data: '/api/v1/data' }, services: SERVICES })); + return; + } + res.end(JSON.stringify({ success: true, data: { status: 'ok' }, path: req.url })); + }); + await new Promise((resolveListen) => server.listen(0, '127.0.0.1', resolveListen)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + + dir = mkdtempSync(join(tmpdir(), 'os-qa-requires-')); + mkdirSync(join(dir, 'qa')); + mkdirSync(join(dir, 'qa-skip')); + writeFileSync(join(dir, 'qa', 'requires.test.json'), JSON.stringify(MIXED), 'utf-8'); + writeFileSync(join(dir, 'qa-skip', 'skip.test.json'), JSON.stringify(ALL_SKIPPED), 'utf-8'); + + const mixed = ['test', 'qa/*.test.json', '--url', url]; + const skip = ['test', 'qa-skip/*.test.json', '--url', url]; + [unset, withParam, allSkipped, allSkippedStrict] = await Promise.all([ + runCli(mixed, dir), + runCli(mixed, dir, { [PARAM]: 'tok' }), + runCli(skip, dir), + runCli([...skip, '--fail-on-empty'], dir), + ]); +}, RUN_TIMEOUT_MS * 2); + +afterAll(async () => { + await new Promise((resolveClose) => server.close(() => resolveClose())); + rmSync(dir, { recursive: true, force: true }); +}); + +describe('`os test` — an unmet `requires` SKIPS the scenario, with its reason', () => { + it('CONTROL — a scenario without requirements and one whose service the target serves both run', () => { + expect(unset.stdout).toContain('✅ Scenario: No requirements [plain]'); + expect(unset.stdout).toContain('✅ Scenario: Needs data [svc-met]'); + }); + + it('an unmet `services` entry skips, naming the service and the services the target declares available', () => { + expect(unset.stdout).toContain('⏭️ Scenario: Needs ai [svc-unmet] (skipped)'); + expect(unset.stdout).toMatch( + /Skipped: requires\.services 'ai' is not available on the target \(enabled: false, status: unavailable\)\. The target declares available: auth, data\./, + ); + // It never ran: its step would have failed, and nothing did. + expect(unset.stdout).not.toContain('❌'); + }); + + it('an unmet `params` entry skips naming the variable; set in the `os test` process, the scenario runs', () => { + expect(unset.stdout).toContain('⏭️ Scenario: Needs a token [param-gated] (skipped)'); + expect(unset.stdout).toContain(`Skipped: requires.params '${PARAM}' is not set`); + expect(withParam.stdout).toContain('✅ Scenario: Needs a token [param-gated]'); + }); + + it('skipped scenarios are counted apart and never as passed; skips alone exit 0', () => { + expect(unset.stdout).toContain('SUCCESS: 2 scenarios passed. 2 skipped (not run, not counted as passed).'); + expect(unset.code).toBe(0); + expect(withParam.stdout).toContain('SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).'); + expect(withParam.code).toBe(0); + }); + + it('the judgement rides the ONE discovery probe per run — never one per scenario', () => { + // Four runs, each requiring a service in at least one scenario: one probe each. + expect(discoveryHits).toBe(4); + }); +}); + +describe('`os test` — a run in which every scenario was skipped is not a pass', () => { + it('prints no SUCCESS, says nothing ran, and exits 0 by default', () => { + expect(allSkipped.stdout).not.toContain('SUCCESS'); + expect(`${allSkipped.stdout}${allSkipped.stderr}`).toContain( + 'No scenario ran: all 2 selected scenarios were skipped on unmet requirements.', + ); + expect(allSkipped.code).toBe(0); + }); + + it('exits 1 under --fail-on-empty', () => { + expect(allSkippedStrict.stdout).not.toContain('SUCCESS'); + expect(allSkippedStrict.stderr).toContain('--fail-on-empty'); + expect(allSkippedStrict.code).toBe(1); + }); +}); diff --git a/packages/cli/test/qa-requires-summary.test.ts b/packages/cli/test/qa-requires-summary.test.ts new file mode 100644 index 00000000000..e5fc12ed4e2 --- /dev/null +++ b/packages/cli/test/qa-requires-summary.test.ts @@ -0,0 +1,31 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * PIN — `os test`'s closing tally once SKIPPED exists (the `qa-runner` + * family's `requires` key, ruled B). A skipped scenario never ran, so it is + * its own count: never passed, never failed. The wiring — the line reaching + * stdout and the exit status beside it — is pinned end to end in + * `qa-requires-skip-run.test.ts`; this holds the three shapes of the line, + * including the one CI transcripts already match on, unchanged when nothing + * was skipped. + */ + +import { describe, it, expect } from 'vitest'; +import { summaryLine } from '../src/commands/test'; + +describe('os test — summaryLine counts skipped scenarios apart', () => { + it('reads exactly as before when nothing was skipped — CONTROL', () => { + expect(summaryLine(3, 0, 0)).toBe('SUCCESS: All 3 scenarios passed.'); + expect(summaryLine(3, 1, 0)).toBe('FAILED: 1 scenarios failed. 3 passed.'); + }); + + it('never folds a skip into "All passed"', () => { + const line = summaryLine(3, 0, 2); + expect(line).toBe('SUCCESS: 3 scenarios passed. 2 skipped (not run, not counted as passed).'); + expect(line).not.toContain('All'); + }); + + it('carries the skip count on a failed run too', () => { + expect(summaryLine(1, 2, 3)).toBe('FAILED: 2 scenarios failed. 1 passed. 3 skipped (not run, not counted as passed).'); + }); +}); diff --git a/packages/cli/test/qa-suite-schema-load.test.ts b/packages/cli/test/qa-suite-schema-load.test.ts index 04aaec06927..402ffe8b93a 100644 --- a/packages/cli/test/qa-suite-schema-load.test.ts +++ b/packages/cli/test/qa-suite-schema-load.test.ts @@ -110,4 +110,30 @@ describe('os test — suite load is schema-checked (#6247)', () => { const file = suiteFile('not-json.test.json', '{ "name": '); expect(() => loadTestSuite(file)).toThrow(/not-json\.test\.json/); }); + + // The `qa-runner` family's `requires` key, ruled B: `requires.plugins` is a + // tombstone. A suite still carrying it is refused before anything runs, and + // the refusal is the retirement prescription — located at the scenario and + // naming the replacement — rather than the key being stripped in silence. + it('refuses the retired `requires.plugins`, located, with the prescription naming `requires.services`', () => { + const retired = structuredClone(VALID_SUITE) as Record; + retired.scenarios[0].requires = { plugins: ['@objectstack/plugin-auth'] }; + const file = suiteFile('retired-plugins.test.json', JSON.stringify(retired)); + let message = ''; + try { + loadTestSuite(file); + } catch (e) { + message = e instanceof Error ? e.message : String(e); + } + expect(message).toContain('retired-plugins.test.json'); + expect(message).toContain('scenarios.0.requires.plugins'); + expect(message).toMatch(/`scenarios\[\]\.requires\.plugins` was removed.*`requires\.services`/s); + }); + + it('loads `requires.params` and `requires.services` unchanged — CONTROL', () => { + const gated = structuredClone(VALID_SUITE) as Record; + gated.scenarios[0].requires = { params: ['OS_TEST_QA_TOKEN'], services: ['auth'] }; + const suite = loadTestSuite(suiteFile('gated.test.json', JSON.stringify(gated))); + expect(suite.scenarios[0].requires).toEqual({ params: ['OS_TEST_QA_TOKEN'], services: ['auth'] }); + }); }); diff --git a/packages/core/src/qa/adapter.ts b/packages/core/src/qa/adapter.ts index be96e4b2f00..4a133172e28 100644 --- a/packages/core/src/qa/adapter.ts +++ b/packages/core/src/qa/adapter.ts @@ -2,6 +2,23 @@ import * as QA from '@objectstack/spec/qa'; +/** + * What an adapter can say about the services its target serves — the input the + * runner judges `TestScenario.requires.services` against. + */ +export interface TargetServices { + /** + * The target's per-service status map, keyed by service name — the + * discovery document's `services` (ADR-0076 D12: advertise only what is + * mounted). Absent when the adapter could not read it; the runner then has + * nothing to judge a service requirement against, and every such requirement + * is unmet. + */ + services?: Readonly>; + /** One clause saying where `services` came from, or why it is absent. */ + source: string; +} + /** * Interface for executing test actions against a target system. * The target could be a local Kernel instance or a remote API. @@ -13,4 +30,12 @@ export interface TestExecutionAdapter { * @returns The result of the action (e.g. created record, API response) */ execute(action: QA.TestAction, context: Record): Promise; + + /** + * The services the target declares, for judging `requires.services` before a + * scenario runs. Optional: an adapter that cannot say leaves it out, and a + * scenario requiring a service is then SKIPPED with that reason — never run + * on an unverified precondition, never passed. + */ + readTargetServices?(): Promise; } diff --git a/packages/core/src/qa/http-adapter.test.ts b/packages/core/src/qa/http-adapter.test.ts index 81053d92115..32340f337e4 100644 --- a/packages/core/src/qa/http-adapter.test.ts +++ b/packages/core/src/qa/http-adapter.test.ts @@ -397,3 +397,58 @@ describe('[#7848] auth and impersonation headers still ride along', () => { expect(call.headers['X-Run-As']).toBe('alice'); }); }); + +describe('[#20289] readTargetServices hands the runner the SAME probe\'s `services` map', () => { + const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + ai: { enabled: false, status: 'unavailable' }, + }; + + it('reads `services` off the discovery document the record actions already probe — one request for both', async () => { + discoveryReply = () => jsonResponse({ version: 'v1', routes: { data: ADVERTISED_DATA_PATH }, services: SERVICES }); + const adapter = new HttpTestAdapter(BASE_URL); + const target = await adapter.readTargetServices(); + await adapter.execute(action('create_record', 'crm_account', { name: 'Acme' }), {}); + + expect(target.services).toEqual(SERVICES); + expect(target.source).toContain(EXPECTED_DISCOVERY_URL); + expect(calls.filter((c) => c.url === EXPECTED_DISCOVERY_URL)).toHaveLength(1); + // The record action still addresses the mount the same document advertised. + expect(actionCalls()[0]!.url).toBe(`${BASE_URL}${ADVERTISED_DATA_PATH}/crm_account`); + expect(warnings).toEqual([]); + }); + + it('reads the dispatcher bridge\'s `{ data: … }` envelope as well as the bare document', async () => { + discoveryReply = () => jsonResponse({ success: true, data: { routes: { data: '/api/v1/data' }, services: SERVICES } }); + const target = await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(target.services).toEqual(SERVICES); + }); + + it('sends the bearer token — a host that gates discovery still answers', async () => { + discoveryReply = () => jsonResponse({ routes: {}, services: SERVICES }); + await new HttpTestAdapter(BASE_URL, 'tok_123').readTargetServices(); + expect(calls.find((c) => c.url === EXPECTED_DISCOVERY_URL)!.headers['Authorization']).toBe('Bearer tok_123'); + }); + + /** Each row is a way the probe yields no services map; `services` is then absent and `source` says why. */ + const failures: Array<[label: string, reply: () => Response, evidence: RegExp]> = [ + ['discovery is not mounted (404)', () => jsonResponse({ error: 'Not found' }, 404), /answered 404/], + ['the document carries no services map', () => discoveryDocument({ data: ADVERTISED_DATA_PATH }), /answered 200 but carried no services map/], + ['the host cannot be reached', () => { throw new Error('ECONNREFUSED'); }, /could not be reached \(ECONNREFUSED\)/], + ]; + for (const [label, reply, evidence] of failures) { + it(`reports no services, and why, when ${label}`, async () => { + discoveryReply = reply; + const target = await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(target.services).toBeUndefined(); + expect(target.source).toMatch(evidence); + expect(target.source).toContain(EXPECTED_DISCOVERY_URL); + }); + } + + it('reading services alone does not print the data-mount warning — no record action asked for a mount', async () => { + discoveryReply = () => jsonResponse({ error: 'Not found' }, 404); + await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(warnings).toEqual([]); + }); +}); diff --git a/packages/core/src/qa/http-adapter.ts b/packages/core/src/qa/http-adapter.ts index c1daa841f91..9bd81019d69 100644 --- a/packages/core/src/qa/http-adapter.ts +++ b/packages/core/src/qa/http-adapter.ts @@ -2,7 +2,7 @@ import * as QA from '@objectstack/spec/qa'; import { RestApiConfigSchema, CrudEndpointsConfigSchema } from '@objectstack/spec/api'; -import { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; /** Memoised {@link conventionMounts} — the schemas are `lazySchema`, so build them once. */ let conventionCache: { apiBase: string; dataPath: string } | undefined; @@ -64,26 +64,109 @@ interface ResolvedDataMount { readonly why: string; } +/** + * What the run's one discovery probe returned. + * + * The probe used to be read for `routes.data` and dropped. It now keeps the + * document, because `requires.services` is judged against the SAME document's + * `services` map (#20289, ruled B) — one probe answers both questions, and a + * second request for the second question would be a second chance for the two + * answers to disagree. + */ +interface DiscoveryRead { + /** `GET {baseUrl}{apiBase}/discovery` — the URL probed. */ + readonly probeUrl: string; + /** The HTTP status, when the server answered at all. */ + readonly status?: number; + /** + * The discovery document, unwrapped from the dispatcher bridge's + * `{ data: … }` envelope; absent when the probe yielded none. + */ + readonly document?: { readonly routes?: Record; readonly services?: unknown }; + /** One clause naming the probe and its outcome. */ + readonly why: string; +} + export class HttpTestAdapter implements TestExecutionAdapter { /** * The single discovery probe of a run, memoised as the in-flight promise so - * concurrent record actions share one request rather than racing N. + * concurrent record actions — and the `requires.services` judgement — share + * one request rather than racing N. * * `os test` builds ONE adapter for the whole run (`packages/cli/src/commands/ * test.ts`) and hands it to every suite, so instance scope IS run scope. */ + private discoveryPromise?: Promise; + /** The data mount derived from that probe, memoised so its fallback warns once. */ private mountPromise?: Promise; constructor(private baseUrl: string, private authToken?: string) {} + /** The run's discovery read; probes at most once per adapter. */ + private discovery(): Promise { + if (this.discoveryPromise === undefined) { + this.discoveryPromise = this.readDiscovery(); + } + return this.discoveryPromise; + } + /** The resolved data mount; probes at most once per adapter. */ private dataMount(): Promise { if (this.mountPromise === undefined) { - this.mountPromise = this.resolveDataMount(); + this.mountPromise = this.discovery().then((read) => this.resolveDataMount(read)); } return this.mountPromise; } + /** + * The target's declared services, for the runner's `requires.services` + * judgement — the `services` map of the discovery document this adapter + * probes once per run anyway (ADR-0076 D12: discovery advertises only what + * is mounted). Asked only when a scenario declares a service requirement, so + * a suite that declares none keeps the probe count it had: zero for an + * `api_call`-only suite, one otherwise. + */ + async readTargetServices(): Promise { + const read = await this.discovery(); + const services = read.document?.services; + if (services !== null && typeof services === 'object' && !Array.isArray(services)) { + return { services: services as Record, source: `GET ${read.probeUrl} advertised services` }; + } + return { + source: read.document + ? `GET ${read.probeUrl} answered ${read.status} but carried no services map` + : read.why, + }; + } + + /** Issue the one discovery request of the run and keep what it returned. */ + private async readDiscovery(): Promise { + const { apiBase } = conventionMounts(); + const probeUrl = `${this.baseUrl}${apiBase}/discovery`; + try { + const headers: Record = {}; + if (this.authToken) { + headers['Authorization'] = `Bearer ${this.authToken}`; + } + const response = await fetch(probeUrl, { method: 'GET', headers }); + if (!response.ok) { + return { probeUrl, status: response.status, why: `GET ${probeUrl} answered ${response.status}` }; + } + const body = await response.json(); + // `@objectstack/rest` answers the document bare; the dispatcher bridge + // wraps it as `{ data: … }`. Pick the object that actually carries the + // document's own keys rather than `body.data || body` — the document's + // own `routes.data` key makes the looser test ambiguous to read here. + const isDocument = (value: unknown): value is { routes?: Record; services?: unknown } => + value !== null && typeof value === 'object' && ('routes' in value || 'services' in value); + const inner = (body as { data?: unknown } | null)?.data; + const document = isDocument(body) ? body : isDocument(inner) ? inner : undefined; + return { probeUrl, status: response.status, document, why: `GET ${probeUrl} answered ${response.status}` }; + } catch (error) { + return { probeUrl, why: `GET ${probeUrl} could not be reached (${(error as Error).message})` }; + } + } + /** * Ask the server where it serves the Data Protocol, and fall back to the * convention — loudly — when it cannot say. @@ -120,40 +203,23 @@ export class HttpTestAdapter implements TestExecutionAdapter { * the remedy. `api_call` takes the path it is given and is unaffected either * way — it stays the escape hatch for a host this cannot reach. */ - private async resolveDataMount(): Promise { - const { apiBase, dataPath } = conventionMounts(); - const probeUrl = `${this.baseUrl}${apiBase}/discovery`; + private resolveDataMount(read: DiscoveryRead): ResolvedDataMount { + const { dataPath } = conventionMounts(); + const { probeUrl } = read; let why: string; - try { - const headers: Record = {}; - if (this.authToken) { - headers['Authorization'] = `Bearer ${this.authToken}`; - } - const response = await fetch(probeUrl, { method: 'GET', headers }); - if (response.ok) { - const body = await response.json(); - // `@objectstack/rest` answers the document bare; the dispatcher bridge - // wraps it as `{ data: … }`. Pick the object that actually carries - // `routes` rather than `body.data || body` — the document's own - // `routes.data` key makes the looser test ambiguous to read here. - const doc = (body && typeof body === 'object' && 'routes' in body) - ? (body as { routes?: Record }) - : ((body as { data?: { routes?: Record } } | null)?.data); - const advertised = doc?.routes?.data; - if (typeof advertised === 'string' && advertised.length > 0) { - return { - path: advertised, - source: 'discovery', - why: `GET ${probeUrl} advertised routes.data`, - }; - } - why = `GET ${probeUrl} answered ${response.status} but carried no routes.data`; - } else { - why = `GET ${probeUrl} answered ${response.status}`; + if (read.status !== undefined && read.status >= 200 && read.status < 300) { + const advertised = read.document?.routes?.data; + if (typeof advertised === 'string' && advertised.length > 0) { + return { + path: advertised, + source: 'discovery', + why: `GET ${probeUrl} advertised routes.data`, + }; } - } catch (error) { - why = `GET ${probeUrl} could not be reached (${(error as Error).message})`; + why = `GET ${probeUrl} answered ${read.status} but carried no routes.data`; + } else { + why = read.why; } const mount: ResolvedDataMount = { path: dataPath, source: 'convention', why }; diff --git a/packages/core/src/qa/runner.test.ts b/packages/core/src/qa/runner.test.ts index 77f7697f7be..df1f00a7042 100644 --- a/packages/core/src/qa/runner.test.ts +++ b/packages/core/src/qa/runner.test.ts @@ -15,7 +15,7 @@ import { describe, it, expect } from 'vitest'; import * as QA from '@objectstack/spec/qa'; import { TestRunner } from './runner.js'; -import type { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; /** An adapter that hands the runner one fixed result — the assertion is the unit under test. */ class StubAdapter implements TestExecutionAdapter { @@ -281,3 +281,167 @@ describe('TestRunner — results carry the suite and scenario names the author w expect(result.suiteName).toBeUndefined(); }); }); + +// `TestScenario.requires` is ENFORCED (ADR-0049; the `qa-runner` family's fifth +// key, ruled B). Before any step runs — setup included — `params` is judged +// against the runner's environment and `services` against the target's +// discovery `services`; an unmet entry makes the scenario SKIPPED with a reason, +// its own verdict, never passed. These pin the runner half; `os test`'s printing +// and exit posture are pinned in packages/cli/test/qa-requires-skip-run.test.ts. +describe('TestRunner — `requires` is judged before the first step', () => { + const step: QA.TestStep = { name: 'step-1', action: { type: 'api_call', target: '/api/v1/health' } }; + + /** A discovery `services` map as both producers answer it (ADR-0076 D12). */ + const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + auth: { enabled: true, status: 'available', route: '/api/v1/auth' }, + ai: { enabled: false, status: 'unavailable', message: 'no implementation ships' }, + metadata: { enabled: true, status: 'degraded', route: '/api/v1/meta' }, + }; + + /** Records every action it runs, and answers `readTargetServices` from a canned reading. */ + class TargetAdapter implements TestExecutionAdapter { + executed: string[] = []; + serviceReads = 0; + constructor(private reading: TargetServices | (() => TargetServices)) {} + async execute(action: QA.TestAction): Promise { + this.executed.push(action.target); + return { ok: true }; + } + async readTargetServices(): Promise { + this.serviceReads += 1; + return typeof this.reading === 'function' ? this.reading() : this.reading; + } + } + + const served = () => new TargetAdapter({ services: SERVICES, source: 'GET /api/v1/discovery advertised services' }); + + const scenario = (requires: QA.TestScenario['requires']): QA.TestScenario => ({ + id: 'needs-things', + name: 'Needs things', + setup: [{ name: 'setup-1', action: { type: 'api_call', target: '/setup' } }], + steps: [step], + requires, + }); + + it('an unmet `services` entry skips — no step runs, setup included — and lists the declared services', async () => { + const adapter = served(); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['ai'] })); + + expect(result.status).toBe('skipped'); + expect(result.passed).toBe(false); + expect(result.steps).toEqual([]); + expect(adapter.executed).toEqual([]); + expect(result.skipped!.unmet).toEqual([ + { key: 'services', name: 'ai', detail: expect.stringContaining('not available on the target') }, + ]); + // `metadata` is declared but degraded, `ai` disabled: only the two that are + // enabled AND available are what the target "declares available". + expect(result.skipped!.availableServices).toEqual(['auth', 'data']); + expect(result.skipped!.reason).toContain("'ai'"); + expect(result.skipped!.reason).toContain('auth, data'); + }); + + it('a met `services` entry runs — CONTROL', async () => { + const adapter = served(); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['data', 'auth'] })); + + expect(result.status).toBe('passed'); + expect(result.passed).toBe(true); + expect(result.skipped).toBeUndefined(); + expect(adapter.executed).toEqual(['/setup', '/api/v1/health']); + }); + + it('`enabled` alone is not enough: a degraded service is unmet, and so is one the target does not declare', async () => { + const result = await new TestRunner(served(), { env: {} }).runScenario(scenario({ services: ['metadata', 'search'] })); + + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet.map((u) => [u.name, u.detail])).toEqual([ + ['metadata', expect.stringContaining('status: degraded')], + ['search', 'not declared by the target'], + ]); + }); + + it('an unmet `params` entry skips naming the variable; set and non-empty, it runs', async () => { + const needsToken = scenario({ params: ['OS_QA_TOKEN'] }); + + const unset = await new TestRunner(served(), { env: {} }).runScenario(needsToken); + expect(unset.status).toBe('skipped'); + expect(unset.skipped!.unmet).toEqual([ + { key: 'params', name: 'OS_QA_TOKEN', detail: expect.stringContaining('not set') }, + ]); + expect(unset.skipped!.reason).toContain("requires.params 'OS_QA_TOKEN'"); + // No service was required, so no service list is attached. + expect(unset.skipped!.availableServices).toBeUndefined(); + + // An unconfigured CI secret arrives as an empty string: unmet too. + const empty = await new TestRunner(served(), { env: { OS_QA_TOKEN: '' } }).runScenario(needsToken); + expect(empty.status).toBe('skipped'); + expect(empty.skipped!.unmet[0]!.detail).toContain('empty'); + + const adapter = served(); + const set = await new TestRunner(adapter, { env: { OS_QA_TOKEN: 'tok' } }).runScenario(needsToken); + expect(set.status).toBe('passed'); + expect(adapter.executed).toEqual(['/setup', '/api/v1/health']); + }); + + it('every unmet entry is reported, params before services', async () => { + const result = await new TestRunner(served(), { env: {} }).runScenario( + scenario({ params: ['A', 'B'], services: ['data', 'ai'] }), + ); + expect(result.skipped!.unmet.map((u) => `${u.key}:${u.name}`)).toEqual(['params:A', 'params:B', 'services:ai']); + }); + + it('asks the target only when a service is required', async () => { + const adapter = served(); + await new TestRunner(adapter, { env: { X: '1' } }).runScenario(scenario({ params: ['X'] })); + await new TestRunner(adapter, { env: {} }).runScenario(scenario(undefined)); + expect(adapter.serviceReads).toBe(0); + }); + + it('a target whose services cannot be read declares none: the requirement is unmet and says why', async () => { + const adapter = new TargetAdapter({ source: 'GET http://x/api/v1/discovery answered 404' }); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['data'] })); + + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet[0]!.detail).toContain('answered 404'); + expect(result.skipped!.availableServices).toEqual([]); + expect(adapter.executed).toEqual([]); + }); + + it('an adapter that cannot report its target skips a service requirement rather than running it', async () => { + const result = await new TestRunner(new StubAdapter({ ok: true }), { env: {} }).runScenario( + scenario({ services: ['data'] }), + ); + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet[0]!.detail).toContain('does not report'); + }); + + it('runSuite stamps the suite name on a skipped result too, and the verdicts stay distinct', async () => { + const results = await new TestRunner(served(), { env: {} }).runSuite({ + name: 'Mixed', + scenarios: [ + { id: 'runs', name: 'Runs', steps: [step] }, + { id: 'skips', name: 'Skips', steps: [step], requires: { services: ['ai'] } }, + ], + }); + expect(results.map((r) => [r.suiteName, r.scenarioId, r.status, r.passed])).toEqual([ + ['Mixed', 'runs', 'passed', true], + ['Mixed', 'skips', 'skipped', false], + ]); + }); + + it('a failed scenario says `failed` — skipped is not a kind of failure', async () => { + class ThrowingTarget extends TargetAdapter { + override async execute(): Promise { + throw new Error('boom'); + } + } + const result = await new TestRunner( + new ThrowingTarget({ services: SERVICES, source: 'x' }), + { env: {} }, + ).runScenario({ id: 'f', name: 'F', steps: [step], requires: { services: ['data'] } }); + expect(result.status).toBe('failed'); + expect(result.skipped).toBeUndefined(); + }); +}); diff --git a/packages/core/src/qa/runner.ts b/packages/core/src/qa/runner.ts index faf69146dca..77e3f4c5a76 100644 --- a/packages/core/src/qa/runner.ts +++ b/packages/core/src/qa/runner.ts @@ -1,7 +1,51 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. import * as QA from '@objectstack/spec/qa'; -import { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; + +/** + * A scenario's verdict. `skipped` is its own outcome, not a kind of pass or + * fail: the scenario never ran because a `requires` entry was unmet, so it + * proved nothing — a report counts it separately and never as passed. + */ +export type TestResultStatus = 'passed' | 'failed' | 'skipped'; + +/** One `TestScenario.requires` entry that did not hold. */ +export interface UnmetRequirement { + /** The `requires` key the entry sits under. */ + key: 'params' | 'services'; + /** The entry: the environment variable or the service key. */ + name: string; + /** Why it is unmet, in one clause — what was observed instead. */ + detail: string; +} + +/** Why a scenario was skipped instead of run. */ +export interface SkipReport { + /** + * The sentence a report prints: every unmet entry by key and name and, when + * a service is unmet, the services the target does declare available. + */ + reason: string; + /** Each unmet entry, in `requires` order (`params` first, then `services`). */ + unmet: UnmetRequirement[]; + /** + * The services the target declares `enabled` with status `available`, + * sorted — present whenever a `services` requirement was judged. Empty when + * the target's services could not be read. + */ + availableServices?: string[]; +} + +/** Options for {@link TestRunner}. */ +export interface TestRunnerOptions { + /** + * The environment `requires.params` is judged against. Defaults to this + * process's own environment — the process running the suite (`os test`), + * never the target server's, which no suite can observe. + */ + env?: Readonly>; +} /** * One scenario's outcome, carrying the names a report leads with. @@ -24,7 +68,19 @@ export interface TestResult { scenarioName: string; /** `TestScenario.description`, when the author wrote one. */ description?: string; + /** + * The verdict. `skipped` means a `requires` entry was unmet and no step — + * setup included — ran; {@link skipped} says which entry and why. + */ + status: TestResultStatus; + /** + * `status === 'passed'`. A skipped scenario is `passed: false` — it proved + * nothing — but it is not a failure either: read `status` to tell the two + * apart. + */ passed: boolean; + /** Present exactly when `status` is `skipped`. */ + skipped?: SkipReport; steps: StepResult[]; error?: unknown; duration: number; @@ -72,8 +128,36 @@ function containsInapplicableHint(actual: unknown): string { ); } +/** + * One discovery `services` entry is AVAILABLE exactly when the target says + * `enabled: true` and `status: 'available'` — both halves, because they answer + * different questions (ADR-0076 D12): `enabled` is "the slot is filled", + * `status` is whether what fills it is the real thing rather than a stub or a + * degraded fallback. + */ +function isAvailable(entry: unknown): boolean { + if (entry === null || typeof entry !== 'object') return false; + const info = entry as { enabled?: unknown; status?: unknown }; + return info.enabled === true && info.status === 'available'; +} + +/** Why a required service is unmet, from what the target said about it. */ +function describeUnavailable(entry: unknown, target: TargetServices): string { + if (target.services === undefined) { + return `not judgeable: the target's services could not be read (${target.source})`; + } + if (entry === undefined) return 'not declared by the target'; + if (entry === null || typeof entry !== 'object') return 'declared by the target in an unreadable shape'; + const info = entry as { enabled?: unknown; status?: unknown }; + return `not available on the target (enabled: ${String(info.enabled)}, status: ${String(info.status)})`; +} + export class TestRunner { - constructor(private adapter: TestExecutionAdapter) {} + private readonly env: Readonly>; + + constructor(private adapter: TestExecutionAdapter, options: TestRunnerOptions = {}) { + this.env = options.env ?? (typeof process === 'undefined' ? {} : process.env); + } async runSuite(suite: QA.TestSuite): Promise { const results: TestResult[] = []; @@ -85,6 +169,23 @@ export class TestRunner { async runScenario(scenario: QA.TestScenario): Promise { const startTime = Date.now(); + + // Preconditions first: a scenario whose `requires` does not hold runs no + // step at all — setup included, since setup can write records. + const skipped = await this.judgeRequirements(scenario); + if (skipped) { + return { + scenarioId: scenario.id, + scenarioName: scenario.name, + description: scenario.description, + status: 'skipped', + passed: false, + skipped, + steps: [], + duration: Date.now() - startTime, + }; + } + const context: Record = {}; // Variable context // Initialize context from initial payload if needed? Currently schema doesn't have initial context prop on Scenario @@ -100,6 +201,7 @@ export class TestRunner { scenarioId: scenario.id, scenarioName: scenario.name, description: scenario.description, + status: 'failed', passed: false, steps: [], error: `Setup failed: ${e instanceof Error ? e.message : String(e)}`, @@ -156,6 +258,7 @@ export class TestRunner { scenarioId: scenario.id, scenarioName: scenario.name, description: scenario.description, + status: scenarioPassed ? 'passed' : 'failed', passed: scenarioPassed, steps: stepResults, error: scenarioError, @@ -163,6 +266,63 @@ export class TestRunner { }; } + /** + * Judge `TestScenario.requires` (ADR-0049: declared is enforced; ruled B). + * Returns the skip report when an entry is unmet, `undefined` when every + * entry holds or the scenario declares none. + * + * - `params`: each variable must be set to a non-empty value in {@link env} + * — the process running the suite. Empty counts as unset: a CI secret that + * is not configured arrives as an empty string, and a scenario that needs + * it cannot run on it. + * - `services`: each key must be declared by the target `enabled` with + * `status === 'available'` — the discovery document's own verdict + * (ADR-0076 D12), read through the adapter's one probe of the run. A + * target whose services cannot be read declares none, so every service + * requirement is unmet and the reason says why. + * + * Every unmet entry is reported, not just the first, so one run tells the + * author everything the target is missing. + */ + private async judgeRequirements(scenario: QA.TestScenario): Promise { + const requires = scenario.requires; + if (!requires) return undefined; + const unmet: UnmetRequirement[] = []; + + for (const name of requires.params ?? []) { + const value = this.env[name]; + if (value === undefined) { + unmet.push({ key: 'params', name, detail: 'not set in the environment of the process running the suite' }); + } else if (value === '') { + unmet.push({ key: 'params', name, detail: 'set to an empty value in the environment of the process running the suite' }); + } + } + + const services = requires.services ?? []; + let availableServices: string[] | undefined; + if (services.length > 0) { + const target: TargetServices = this.adapter.readTargetServices + ? await this.adapter.readTargetServices() + : { source: 'the execution adapter does not report the target\'s services' }; + const declared = target.services ?? {}; + availableServices = Object.keys(declared).filter((key) => isAvailable(declared[key])).sort(); + for (const name of services) { + if (isAvailable(declared[name])) continue; + unmet.push({ key: 'services', name, detail: describeUnavailable(declared[name], target) }); + } + } + + if (unmet.length === 0) return undefined; + const clauses = unmet.map((u) => `requires.${u.key} '${u.name}' is ${u.detail}`); + let reason = `${clauses.join('; ')}.`; + if (availableServices !== undefined && unmet.some((u) => u.key === 'services')) { + reason += availableServices.length > 0 + ? ` The target declares available: ${availableServices.join(', ')}.` + : ' The target declares no service available.'; + } + return { reason, unmet, ...(availableServices !== undefined ? { availableServices } : {}) }; + } + private async runStep(step: QA.TestStep, context: Record): Promise { // 1. Resolve Variables with Context (Simple interpolation or just pass context?) // For now, assume adpater handles context resolution or we do basic replacement diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index 35ed7ffb701..ec677507cde 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -927,7 +927,7 @@ marker where the Notes cell goes, never a guess at what belongs there. | mapping | seeded 2026-08-01 (#4488) at 8/11 live; **0 dead since #4509** retired the three that were not. The import half (#2611) is loudly enforced — unsupported transforms/formats are 400s, `mode`/`upsertKey` default the request, the wizard picker renders `label`. RETIRED 17.0.0: `extractQuery` (authorWarn — "for export only" promised an export path no exporter implements) + `errorPolicy`/`batchSize`, which were dead AND **unwarnable** (schema defaults materialize at parse, so presence ≠ authored — `_authorWarnSkipped`, the non-boolean instance of the default(true) rule). That unwarnability is why they went out in the 17.0.0 window rather than after a deprecation cycle: removal was the only channel that could ever reach the author. Rows DELETED, not tombstoned — MappingSchema is strict, so the keys left the walked shape | | seed | seeded 2026-08-01 (#4488). Fully live via SeedLoaderService on both doors (boot/per-org replay + runtime-draft publish). `records` is the z.record walk boundary: the keys an author writes are the target object's fields, governed by that object's own definitions — recorded in the entry, not silently skipped | | translation | seeded 2026-08-01 (#4488) — after fixing the walker: the registered schema is a z.preprocess pipe (#3778 retired-dialect guard) whose transform side the unwrap always took, so the type was literally unwalkable. 11 of 12 groups live across spec resolvers, REST localization, objectui client resolvers and plugin-audit (whose composed-key `t()` calls make `messages` easy to mis-verify as dead) — `flows` was the one that was not, and was `planned` at seeding. **#14253** added the twelfth, `datasets`, seeded LIVE and DRILLED (label / description / dimensions / measures) with its reader in the same change: `translateDataset` in the dispatch table, which is what `TRANSLATABLE_METADATA_TYPES` is derived from, so the REST boundary followed with nothing else to remember. The same change gave `objects.._views..bulkActions` and `objects.._validations..message` their first keys — both beneath the walk boundary, so neither adds a row here. Dead 1 = `validationMessages` (authorWarn) at seeding: nothing resolved it, and #3778's own legacy-key migration table steered `errors:` authors into it — a shipped false signpost, the capabilities.readOnly shape. **#4667**: `validationMessages` REMOVED (row deleted) — removed from the shared translationDataShape(), so it retired at BOTH doors at once, closing the item-only asymmetry #3778's original guard had. #3778's own `errors` guidance was rewritten in the same change: it had been steering authors INTO this dead group. ⚠️ **What that left behind is this table's own worked example of the defect it warns about** (#7377): the same commit that deleted the `validationMessages` row wrote a count column of `dead 2` beside a sentence that named exactly one dead key — and that one was the key it had just removed. The real two were `name` and `label`, which the cell never mentioned. Measured at that commit, not inferred: the ledger's dead set there is `{name, label}` and `validationMessages` is absent from `props`. The number was right and the prose was false, in the same cell, on the day it was written — which is why the counts are now generated and this cell holds prose only. **#7131** (PR #7425) resolves it: `name` and `label` re-grade `dead` → `live` under the designer-previews-count-as-consumers ruling (objectui `TranslationPreview.tsx:67` reads `label` first and falls back to `name`, both rendering at `:100`), so the dead set is empty and there is no dead-set sentence left to keep true. As on `job`, the ADR-0033 docs-shaped exemption is untouched — nothing about enforce-or-remove moved. **#19620** (ruling batch #210 item 2 letter B): `settings` row DELETED — the strict-delete route, because `TranslationItemSchema` no longer declares the key and refuses it by name (the item door now takes the per-app face, as the file door has since #15178). ⚠️ The deleted row read `live`, and that verdict was TRUE and stays true of the platform: its evidence read the SERVED tree, which the platform bundle feeds, so the deletion retires the key from the application-authored item and nothing else — the capability lives on `PlatformTranslationDataSchema`, outside this ledger. **#20296**: `flows` is now half-read. `flows.screens` re-graded `planned` → `live`: objectui's FlowRunner reads each screen's `title` and each field's `label` / `placeholder` at the `.objectui-sha` pin f8a9d0fb. `flows.label` stays `planned` because nothing reads it yet (#20318), and so does the container's `authorWarn`, whose `authorHint` now names the read half and the unread one. | -| qa | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). The seeding recorded Dead 5 — `name`, `scenarios.name`, `scenarios.description`, `scenarios.tags` and `scenarios.requires` — and #20289 (family `qa-runner`, verdict ENFORCE) made FOUR of them live: the suite `name` heads the suite in `os test`'s report and is stamped as `suiteName` on every TestResult the suite produces; `scenarios.name` is printed beside the id and carried as `scenarioName`; `scenarios.description` is carried and printed under a failed scenario; and `scenarios.tags` is read by `os test --tags` (any-of, exact; deselected scenarios are counted and never passed). Dead 1 = `scenarios.requires`, which declares param/plugin preconditions nothing checks, so a suite naming a missing plugin runs anyway, and the unmet requirement surfaces only as whatever failure it causes, if any. It was put to the maintainer rather than guessed: over HTTP no server surface lists the loaded plugins, and whose environment `params` names is undefined — its describe() now says NOT CHECKED in the meantime. It does not carry `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops | +| qa | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). The seeding recorded Dead 5 — `name`, `scenarios.name`, `scenarios.description`, `scenarios.tags` and `scenarios.requires` — and #20289 (family `qa-runner`, verdict ENFORCE) made all FIVE live: the suite `name` heads the suite in `os test`'s report and is stamped as `suiteName` on every TestResult the suite produces; `scenarios.name` is printed beside the id and carried as `scenarioName`; `scenarios.description` is carried and printed under a failed scenario; `scenarios.tags` is read by `os test --tags` (any-of, exact; deselected scenarios are counted and never passed); and `scenarios.requires` is judged by the TestRunner before a scenario's first step — `params` against the environment of the process running `os test`, `services` against the target's discovery `services` (enabled and `available`) — so an unmet entry SKIPS the scenario with its reason, counted apart and never passed. `requires` was put to the maintainer first rather than guessed (ruled B): over HTTP no server surface lists the loaded plugins, so `requires.plugins` is a `retiredKey()` tombstone inside the block whose prescription names `requires.services`. It does not carry `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops | | validation | seeded 2026-08-01 (#4488). The ADR-0020 carrier: the evaluator honors active/events/priority/severity/type/condition/message (the zod header's "only reads type/condition/…" prose is STALE — trust the ledger). Dead 3 = label/description/tags, declared governance metadata, kept unmarked. Union walk boundary recorded: only base + `script` keys walked; per-variant keys are governed by the evaluator's tests, not ledger rows. **No longer a registered metadata kind** — #4509 retired it under ADR-0088 (a standalone rule had no object-binding key and every variant is `.strict()`, so it bound to nothing and gated no write; a state machine authored that way saved cleanly and did nothing). The rule VOCABULARY is untouched and fully live via `object.validations[]`, so the ledger keeps governing it through the gate's spec-only override, alongside `webhook` and `query`. The contrast with the two bridges in the same batch is the point: enforce-or-remove picked ENFORCE where the feature existed and only the wiring was missing, and REMOVE where the shape itself could not carry the feature | | api | seeded 2026-08-04 (#5271, part of #5206; PR #5312) — **not a metadata type until that same change made it one**, which is the row's point: governance and registration landed together, the treatment `datasource` did not get (#4487) and paid for with six inert keys found by hand. What #5206 measured before the fix: `api` was in neither `DEFAULT_METADATA_TYPE_REGISTRY` nor `BUILTIN_METADATA_TYPE_SCHEMAS`, so `saveMetaItem`'s `resolveOverlaySchema('api', …)` → `getMetadataTypeSchema('api')` returned `undefined` and took its own documented branch — an unregistered type is stored **unvalidated** — while `getMetaTypes()` could not enumerate the type at all, so Studio rendered neither list nor form. That issue names the shape precisely and it is the inverse of this ledger's usual one: **enforced but undeclared** (the matcher was already indexing these entries, #5089), where `dead` is declared-but-unenforced. The seeding pass classified 27 keys — live 25 / planned 2 / dead 0 — each cited `file:line` at the consumer layer that reads it: the MATCHER (`packages/metadata/src/endpoint-matcher.ts`) indexes `name`/`path`/`method`; the EXECUTOR (`packages/runtime/src/endpoint-executor.ts`) dispatches on `type` and reads `target`/`objectParams`; the POLICY chain (`packages/runtime/src/endpoint-policy.ts` + `security/inbound-rate-limit.ts`) enforces `authRequired`/`rateLimit`/`cacheTtl`; the MAPPING layer (`packages/runtime/src/api-mapping.ts`) applies `inputMapping`/`outputMapping`; and OpenAPI enrichment (`packages/rest/src/openapi-endpoints.ts`) emits `summary`/`description`. Timing was the reason it was cheap: #5040's E-series had built every one of those consumers and all of it was on main, so each key had a real evidence path rather than a promise. **Planned 2 = `inputMapping.transform` + `outputMapping.transform`, and `planned` rather than `dead` is load-bearing**: `dead` here means parsed with no consumer — a silent no-op — and these are the opposite, parsed and then LOUDLY REFUSED at publish (`endpoint-publish-gate.ts` mappingGate) and again at runtime, because no transformation-function registry exists anywhere in the platform. An author who writes one is told so and told what to do instead, so there is nothing for enforce-or-remove to chase; they stay in the vocabulary because admitting them needs a function registry **and** a sandbox ruling (#5040 §3.4), which is a design decision, not a key to quietly delete. Zero dead | | capability | seeded 2026-08-08 (#5961; PR #6540) — `CapabilityDeclarationSchema`, the DECLARATION side of ADR-0066 D1's three-way separation: packages DEFINE a capability, permission sets GRANT it via `systemPermissions`, resources REQUIRE it via `requiredPermissions`. **The gate's 12 and the seeding PR's 5 are the same measurement at two granularities** — PR #6540 call-graph-closed **5 authorable properties**, every one to a real reader in `packages/plugins/plugin-security/src/bootstrap-declared-capabilities.ts` (the one consumer that turns a declaration into a `sys_capability` row), all `live`, with no `PENDING_GOVERNANCE` debt recorded; the other 7 are the ADR-0010 protection-envelope keys the gate auto-classifies `live` and which carry `null` verdicts in the file, exactly as on `permission`/`position`. The same worked example as `api` above and PR #6540 says so in those words — **enforced but undeclared**, the mirror of the hole #5271 closed. What #5961 measured: absent from `DEFAULT_METADATA_TYPE_REGISTRY`, `BUILTIN_METADATA_TYPE_SCHEMAS` and `HAND_CRAFTED_SCHEMAS`, so `isRuntimeCreateAllowed()` took its no-static-entry fallback (permanently true) and `saveMetaItem` its no-schema branch — `PUT /api/v1/meta/capability/:name` accepted **arbitrary JSON** onto an authorization surface whose names `systemPermissions`/`requiredPermissions` resolve by string, while `/meta/types` synthesised a false `allowRuntimeCreate: true` descriptor Studio drew a raw-JSON create form from. #5870 did not open that path (the write gate reads the registry, not the item store); it only made the type visible in `getMetaTypes()`, and both the issue and this row say so to stop the next reader filing it as a regression. Landed as ruling A on ADR-0066 D1's own authority: `allowRuntimeCreate: false` **and** `allowOrgOverride: false`, the second self-judged inside the ruling's rationale and flagged for veto — a tenant overlay of a package declaration would lift `scope` from `org` to `platform`, which is the one field on this type that is an escalation rather than display. Its reverse verification is worth copying: deleting the registry entry gave 7 red / 3 green and measured something **sharper than predicted** — a garbage payload turned 422 rather than resolving, i.e. the schema binding is a real second line of defence behind the registry row, not a restatement of it; deleting the schema binding alone gave exactly 3 red. `packageId` is the one key that reads oddly: deliberately a FALLBACK, not the primary, since #5870 added `capabilities` to the ObjectQL stamped-collection list so `_packageId` now reaches a declaration and wins — it stays `live` because the fallback branch still decides materialization for any declaration arriving unstamped. Zero dead | diff --git a/packages/spec/liveness/qa.json b/packages/spec/liveness/qa.json index cc9c16e037f..f79c93a3cad 100644 --- a/packages/spec/liveness/qa.json +++ b/packages/spec/liveness/qa.json @@ -62,10 +62,12 @@ "note": "Same TestStep surface as `setup`/`steps`. Worth knowing: a teardown throw only turns the scenario red when the scenario had otherwise PASSED (:92-95), so cleanup failures behind a real failure are swallowed on purpose. 2026-08-28: RE-ANCHORED (#13003) and REPOINTED — `:86-98`'s endpoint had rotted onto the `try {` that opens the MAIN step loop. Re-closed by hand against 8cb96ec41." }, "requires": { - "status": "dead", - "verifiedAt": "2026-09-27", + "status": "live", + "evidence": "packages/core/src/qa/runner.ts#judgeRequirements (before a scenario's first step — setup included — each `requires.params` variable must be set non-empty in the runner's environment and each `requires.services` key declared `enabled` with status `available` by the target; any unmet entry returns a `skipped` TestResult with the reason and runs no step) · packages/cli/src/commands/test.ts#summaryLine (`os test` prints each skipped scenario with its reason and counts it apart — never as passed; a run in which every selected scenario was skipped is not a SUCCESS and exits 1 under `--fail-on-empty`)", + "producer": "packages/core/src/qa/http-adapter.ts#readTargetServices — hands the runner the `services` map of the discovery document the adapter already probes once per run, served by packages/metadata-protocol/src/protocol.ts#getDiscovery and packages/runtime/src/http-dispatcher.ts#getDiscoveryInfo (ADR-0076 D12). `params`: packages/cli/src/commands/test.ts#run constructs the TestRunner with `env: process.env` — the environment of the process running `os test`, the only one a suite can observe.", + "verifiedAt": "2026-09-28", "evidenceScope": "in-repo", - "note": "Declared environment preconditions — `requires.params` (environment variables or parameters) and `requires.plugins` (plugins that must be loaded) — that nothing checks. Re-measured 2026-09-27 (#20289) on a stub target: a scenario declaring a plugin that does not exist and an environment variable that is unset reported PASSED, exactly like its no-requirements control. STILL DEAD, and deliberately: #20289 enforced the other four rows of this family and put this one to the maintainer, because neither sub-key has an honest judge yet. `plugins`: `os test` reaches the target over HTTP, and no server surface lists the loaded plugins — the discovery document advertises `services` and `capabilities`, not plugins — and the plugin spelling itself (package name or `plugin.name`) is undefined. `params`: whose environment is undefined — the runner's is observable but no step can consume a runner variable (interpolation reads only captured context), and the server's is not observable at all. Guessing either would make a skip that means nothing. What changed now: the describe() of `requires` and both sub-keys says NOT CHECKED instead of reading as a guard, so an author (or an AI) is no longer told it is one. The enforce route still needs a SKIPPED result with its reason, counted separately and never passed; the remove route drops the block." + "note": "Was `dead` from the 2026-08-10 seeding until #20289's `requires` round (the family's fifth key; ruled B): declared preconditions — `params` (environment variables or parameters) and `plugins` (plugins that must be loaded) — that nothing checked, re-measured on a stub target as an unmet-requirements scenario reporting PASSED exactly like its no-requirements control. ENFORCED now, each sub-key judged against something `os test` can actually observe: `params` against the environment of the process running the suite (non-empty; an empty CI secret counts as unset), and `services` — NEW, closed over `CoreServiceName` so a misspelling is refused at parse — against the target's discovery `services` map, met only when the target declares the service `enabled` AND `status: 'available'`. Every unmet entry is reported, and a service reason lists what the target does declare available; a target whose discovery cannot be read declares nothing, so each service requirement is unmet and says why. `plugins` is REMOVED — a `retiredKey()` tombstone inside this block, whose prescription names `requires.services` and carries the plugin → service mapping derived from `CORE_SERVICE_PROVIDER`: over HTTP no served surface lists loaded plugins and the plugin spelling was never defined, so it could not join the judgement honestly (D3 `qa-scenario-requires-plugins-retired`; `RETIRED_KEYS_BY_MAJOR[18]` `qa/TestScenario:requires.plugins`; no D2 conversion, since a suite is a loose file and never a stored row). The walk classifies this row as one block, so the tombstone needs no row of its own. SKIPPED is its own outcome in core's `TestResult` (`status: 'skipped'`, `passed: false`, the `skipped` report): never counted as passed, and not a failure either — a failure-free run with skips exits 0." } } } diff --git a/packages/spec/liveness/state-counts.md b/packages/spec/liveness/state-counts.md index 33e4ce3dd0b..9e9edfb7172 100644 --- a/packages/spec/liveness/state-counts.md +++ b/packages/spec/liveness/state-counts.md @@ -56,7 +56,7 @@ for both corollaries. | `validation` | 18 | 0 | 0 | 0 | 0 | 18 | | `api` | 25 | 0 | 0 | 1 | 2 | 28 | | `capability` | 12 | 0 | 0 | 0 | 0 | 12 | -| `qa` | 8 | 0 | 0 | 1 | 0 | 9 | +| `qa` | 9 | 0 | 0 | 0 | 0 | 9 | | `manifest` | 23 | 0 | 1 | 15 | 0 | 39 | | `crud_endpoints` | 6 | 0 | 0 | 2 | 0 | 8 | | `metadata_endpoints` | 7 | 0 | 0 | 2 | 0 | 9 | @@ -67,4 +67,4 @@ for both corollaries. | `sharing_rule` | 16 | 0 | 0 | 0 | 1 | 17 | | `connector` | 29 | 0 | 0 | 30 | 1 | 60 | | `analytics_cube` | 18 | 0 | 0 | 9 | 0 | 27 | -| **total** | **940** | **5** | **1** | **148** | **9** | **1103** | +| **total** | **941** | **5** | **1** | **147** | **9** | **1103** | diff --git a/packages/spec/scripts/liveness/check-liveness.mts b/packages/spec/scripts/liveness/check-liveness.mts index 6bd1d578df8..bb77151c3d8 100644 --- a/packages/spec/scripts/liveness/check-liveness.mts +++ b/packages/spec/scripts/liveness/check-liveness.mts @@ -365,11 +365,13 @@ const PENDING_GOVERNANCE: Record = {}; // was withdrawn on 2026-08-08 in favour of enforce. Governing it here is the // half of that ruling that keeps the surface honest going forward: the runner // reads a real, measured subset of the declared keys, and a key nothing reads is -// recorded as such instead of being invisible. `suite.name` and `scenario.tags` -// were two of those and have since gained readers in `os test` (the suite -// heading; the `--tags` selection); `scenario.requires` is the one still unread — -// declared, NOT CHECKED, and its row stays dead. Like `query`, there is no -// registry to fold it back onto — the override IS its governance. +// recorded as such instead of being invisible. `suite.name`, `scenario.tags` +// and `scenario.requires` were three of those and have since gained readers +// (the suite heading and the `--tags` selection in `os test`; the TestRunner's +// precondition judgement, which skips a scenario whose `params` or `services` +// do not hold — `requires.plugins`, which nothing could judge, is a tombstone). +// Like `query`, there is no registry to fold it back onto — the override IS its +// governance. // `manifest` is the THIRD category the override has had to reach, and the one // that showed the escape hatch was load-bearing rather than a webhook special // case. `ManifestSchema` (src/kernel/manifest.zod.ts) is what an author writes diff --git a/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts b/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts new file mode 100644 index 00000000000..20a1fd99dd2 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts @@ -0,0 +1,15 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — +// `requires.plugins` could never be judged: `os test` reaches its target over +// HTTP and no served surface lists the loaded plugins. It retires into +// `requires.services`, judged against the services the target's discovery +// document declares enabled and available. Tombstoned with `retiredKey()` inside +// the live `requires` block, whose `params` and `services` keep parsing. No D2 +// conversion: a QA suite is a loose JSON file `os test` loads, never a stack +// collection member or a stored row. D3 semantic entry +// `qa-scenario-requires-plugins-retired`. Registered under 18 for the +// launch-window reason its neighbours state. +// +// Nested key of an inline block — no `authorable-surface/` line of its own. +export const entry = 'qa/TestScenario:requires.plugins'; diff --git a/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts b/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts new file mode 100644 index 00000000000..404349fda88 --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts @@ -0,0 +1,40 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +// #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — the +// D3 entry of the family (one D3 entry per retirement family). Registered key: +// `qa/TestScenario:requires.plugins`. No D2 conversion: a QA suite is a loose +// JSON file `os test` loads, never a stack collection member or a stored row, so +// there is no source for a conversion to rewrite — and the move is a judgement +// (which service did the plugin stand for?), not a mechanical edit. This entry +// is where the prescription reaches `os migrate meta`, the upgrade guide and +// `spec-changes.json`. +export const entry: SemanticMigration = { + id: 'qa-scenario-requires-plugins-retired', + surface: 'qa.scenarios[].requires.plugins', + replacement: + '`requires.services` — the discovery service keys the scenario needs (for example `auth`, `analytics`, ' + + '`automation`, `ai`), each judged against the target\'s discovery document: met only when the target declares ' + + 'the service `enabled` with status `available`. The plugin → service mapping follows the provider table ' + + 'discovery itself reports (`CORE_SERVICE_PROVIDER`): `@objectstack/plugin-auth` fills `auth`, ' + + '`@objectstack/service-analytics` fills `analytics`, `@objectstack/service-automation` fills `automation`, ' + + 'and so on. A plugin that fills no discovery service slot has no service to require.', + reason: + '`requires.plugins` was declared as a precondition and checked by nothing: `os test` reaches its target over ' + + 'HTTP, no served surface lists the loaded plugins, and the plugin spelling (package name or `plugin.name`) ' + + 'was never defined — so a scenario naming a missing plugin ran anyway and failed, or passed, on whatever the ' + + 'missing plugin caused. The block is now enforced (ADR-0049): core\'s TestRunner judges `requires` before the ' + + 'first step, and an unmet entry makes the scenario SKIPPED with a reason, counted separately and never as ' + + 'passed. `plugins` could not join that judgement honestly, so it retires into `services`, which the target\'s ' + + 'discovery document already answers (ADR-0076 D12: advertise only what is mounted). The consumer still owes ' + + 'the judgement because a plugin name does not always map to one service — a plugin that fills no discovery ' + + 'slot was never a checkable precondition, and only the suite\'s author knows what the scenario needed it for.', + acceptanceCriteria: + 'No QA suite (`qa/*.test.json`) carries `requires.plugins` — `os test` now refuses such a file at load time ' + + 'with the retirement prescription, naming the key, instead of running it; `tsc` refuses the key at a typed ' + + 'authoring site (`never`). Each scenario that declared plugins declares the services it needs in ' + + '`requires.services`, and `os test` against a target that serves them runs the scenario, while against one ' + + 'that does not it prints the scenario as skipped with the unmet service and the services the target declares ' + + 'available. A suite without `requires` runs exactly as before.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index f3d2c181696..62b321fd59b 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -13972,6 +13972,42 @@ const step18: MigrationStep = { + 'row, no lockfile pin. ⛔ Do not repair one by widening the check back — the grammar ' + 'is the contract now, on nine carriers at once.', }, + // #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — the + // D3 entry of the family (one D3 entry per retirement family). Registered key: + // `qa/TestScenario:requires.plugins`. No D2 conversion: a QA suite is a loose + // JSON file `os test` loads, never a stack collection member or a stored row, so + // there is no source for a conversion to rewrite — and the move is a judgement + // (which service did the plugin stand for?), not a mechanical edit. This entry + // is where the prescription reaches `os migrate meta`, the upgrade guide and + // `spec-changes.json`. + { + id: 'qa-scenario-requires-plugins-retired', + surface: 'qa.scenarios[].requires.plugins', + replacement: + '`requires.services` — the discovery service keys the scenario needs (for example `auth`, `analytics`, ' + + '`automation`, `ai`), each judged against the target\'s discovery document: met only when the target declares ' + + 'the service `enabled` with status `available`. The plugin → service mapping follows the provider table ' + + 'discovery itself reports (`CORE_SERVICE_PROVIDER`): `@objectstack/plugin-auth` fills `auth`, ' + + '`@objectstack/service-analytics` fills `analytics`, `@objectstack/service-automation` fills `automation`, ' + + 'and so on. A plugin that fills no discovery service slot has no service to require.', + reason: + '`requires.plugins` was declared as a precondition and checked by nothing: `os test` reaches its target over ' + + 'HTTP, no served surface lists the loaded plugins, and the plugin spelling (package name or `plugin.name`) ' + + 'was never defined — so a scenario naming a missing plugin ran anyway and failed, or passed, on whatever the ' + + 'missing plugin caused. The block is now enforced (ADR-0049): core\'s TestRunner judges `requires` before the ' + + 'first step, and an unmet entry makes the scenario SKIPPED with a reason, counted separately and never as ' + + 'passed. `plugins` could not join that judgement honestly, so it retires into `services`, which the target\'s ' + + 'discovery document already answers (ADR-0076 D12: advertise only what is mounted). The consumer still owes ' + + 'the judgement because a plugin name does not always map to one service — a plugin that fills no discovery ' + + 'slot was never a checkable precondition, and only the suite\'s author knows what the scenario needed it for.', + acceptanceCriteria: + 'No QA suite (`qa/*.test.json`) carries `requires.plugins` — `os test` now refuses such a file at load time ' + + 'with the retirement prescription, naming the key, instead of running it; `tsc` refuses the key at a typed ' + + 'authoring site (`never`). Each scenario that declared plugins declares the services it needs in ' + + '`requires.services`, and `os test` against a target that serves them runs the scenario, while against one ' + + 'that does not it prints the scenario as skipped with the unmet service and the services the target declares ' + + 'available. A suite without `requires` runs exactly as before.', + }, { id: 'record-chatter-position-vocabulary-converged', surface: @@ -19748,6 +19784,19 @@ export const RETIRED_KEYS_BY_MAJOR: Readonly> // carried on the `UpgradeResult`, never authored into a metadata document. The // semantic entry `kernel-package-lifecycle-durations-unit-in-key` carries it. 'kernel/UpgradePlan:estimatedDuration', + // #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — + // `requires.plugins` could never be judged: `os test` reaches its target over + // HTTP and no served surface lists the loaded plugins. It retires into + // `requires.services`, judged against the services the target's discovery + // document declares enabled and available. Tombstoned with `retiredKey()` inside + // the live `requires` block, whose `params` and `services` keep parsing. No D2 + // conversion: a QA suite is a loose JSON file `os test` loads, never a stack + // collection member or a stored row. D3 semantic entry + // `qa-scenario-requires-plugins-retired`. Registered under 18 for the + // launch-window reason its neighbours state. + // + // Nested key of an inline block — no `authorable-surface/` line of its own. + 'qa/TestScenario:requires.plugins', // #12497 — the RESPONSE-side face of `security/ObjectPermission:allowPurge` // (see that entry for the full rationale: ADR-0049 enforce-or-remove, // maintainer ruling 2026-08-26 accepting #1883's recommendation B; the key diff --git a/packages/spec/src/qa/requires-plugins-retirement.test.ts b/packages/spec/src/qa/requires-plugins-retirement.test.ts new file mode 100644 index 00000000000..9fd617e1d71 --- /dev/null +++ b/packages/spec/src/qa/requires-plugins-retirement.test.ts @@ -0,0 +1,318 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `scenarios[].requires.plugins` RETIRED into `requires.services` (#20289, the + * `qa-runner` family's fifth key; ruling B) — ADR-0049 enforce-or-remove on a + * block that is otherwise ENFORCED: core's TestRunner judges `requires` before + * a scenario's first step and skips it with a reason when an entry is unmet. + * `plugins` could not join that judgement — `os test` reaches its target over + * HTTP and no served surface lists the loaded plugins — so it retires into + * `services`, which the discovery document already answers. + * + * Bookkeeping shapes, pinned below: + * 1. A `retiredKey()` tombstone INSIDE the live `requires` block — a + * non-strict `z.object()`, so a bare deletion would strip the key in + * silence. The refusal itself is pinned beside the schema, in + * `testing.test.ts`. + * 2. No D2 conversion: a QA suite is a loose JSON file `os test` loads, never + * a stack collection member or a stored row. `RETIRED_KEYS_BY_MAJOR[18]` + * carries `qa/TestScenario:requires.plugins`; the family's D3 entry is + * `qa-scenario-requires-plugins-retired`. + * 3. The liveness row `qa.scenarios.requires` is `live` — the block is read — + * and its note records the tombstone (`check:liveness` judges that half). + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { MIGRATIONS_BY_MAJOR, RETIRED_KEYS_BY_MAJOR } from '../migrations/registry'; + +describe('requires.plugins retirement — ADR-0087 registration', () => { + it('declares the key under major 18 and carries one D3 entry for the family, with no D2 conversion', () => { + expect(RETIRED_KEYS_BY_MAJOR[18]).toContain('qa/TestScenario:requires.plugins'); + const step = MIGRATIONS_BY_MAJOR[18]!; + const entry = step.semantic.find((e) => e.id === 'qa-scenario-requires-plugins-retired'); + expect(entry, 'the family D3 entry must be registered in the step-18 chain').toBeDefined(); + expect(entry!.surface).toBe('qa.scenarios[].requires.plugins'); + expect(entry!.replacement).toContain('`requires.services`'); + // A QA suite has no stack or stored source for a conversion to rewrite. A + // conversion id naming the key would be a strip with nothing to strip. + expect(step.conversionIds.filter((id) => /requires-plugins/.test(id))).toEqual([]); + }); +}); + +// ─── Tree-scoped absence, with a DECLARED radius ───────────────────────────── +// +// `tsc` is the primary sweeper for TYPED authorings — `retiredKey()` types the +// key `never` on `TestScenario` — and `os test` refuses a suite file carrying it +// at load. The residue is what neither judges before a run: a suite JSON nobody +// runs in CI, YAML, MD/MDX code fences, untyped `.js`, and TS literals cast +// through `as any` / `as never`. This walk covers that residue across five +// roots, each already declared for `@objectstack/spec#test` in +// `scripts/cross-package-test-inputs.mjs` and mirrored in `turbo.json` — the +// same roots and extensions the `rest_api` and view-item retirement pins walk. +// +// ⭐ `requires` and `plugins` are ordinary words (a stack's `requires` is an +// array of capabilities; a plugin manifest's is an array of `{ pluginId }`), so +// the matcher is STRUCTURAL: an offender is an object literal (or YAML mapping) +// that is the VALUE of a `requires` key and carries a `plugins` key. Nothing +// else. +// +// The bound, stated: a block assembled by SPREAD or computed keys, and a YAML +// flow mapping (`requires: { plugins: [x] }` on one YAML line), are invisible to +// this walk; `docs/**`, `.claude/**`, `.github/**` and the repo-root files are +// outside the radius. +describe('tree-scoped absence: no scenario inside the declared radius still authors `requires.plugins`', () => { + const SPEC_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..'); + const REPO_ROOT = path.resolve(SPEC_ROOT, '../..'); + const THIS_FILE = path.relative(REPO_ROOT, fileURLToPath(import.meta.url)).split(path.sep).join('/'); + + /** The walked roots — declared in `scripts/cross-package-test-inputs.mjs` under `@objectstack/spec`. */ + const WALK_ROOTS = ['packages', 'examples', 'skills', 'content', 'scripts']; + const SCANNED_EXT = new Set(['.ts', '.mts', '.cts', '.js', '.mjs', '.cjs', '.json', '.md', '.mdx', '.yaml', '.yml']); + /** Under `examples/` only the non-code extensions are scanned AND declared. */ + const EXAMPLES_EXT = new Set(['.json', '.md', '.mdx', '.yaml', '.yml']); + const SKIPPED_DIRS = new Set(['node_modules', 'dist', '.git', '.turbo', '.cache', '.objectstack', 'coverage', '.next', '.source']); + + /** + * Structural exclusions — the retirement kit, each with its reason. ⛔ NOT an + * allowlist file (`spec-property-retirement` §4): every entry's JOB is to + * spell the retired key on a `requires` block. + */ + const EXCLUDED = new Set([ + // This pin authors the key to assert the matcher finds it. + THIS_FILE, + // The schema's refusal pins author the key to assert the SCHEMA refuses it. + 'packages/spec/src/qa/testing.test.ts', + // `os test`'s load-time refusal pin authors it to assert the CLI refuses it. + 'packages/cli/test/qa-suite-schema-load.test.ts', + ]); + const EXCLUDED_PREFIXES = [ + // Release-owned prose records the removal; never edited by a code PR. + 'content/docs/releases/', + ]; + /** tsup's own bundle of `tsup.config.ts`, written and deleted mid-build. */ + const TSUP_BUNDLED_CONFIG = /\.bundled_[^./]+\.mjs$/; + + const isOffender = (parentKey: string | undefined, keys: Set): boolean => + parentKey === 'requires' && keys.has('plugins'); + + /** + * One pass over JS/TS/JSON text: a stack of bracket frames, each `{` frame + * collecting its OWN keys — an identifier or quoted string in key position + * (after `{` or `,`) followed by `:` — and remembering the key it is the + * VALUE of (a `{` whose previous significant token is the `:` of a key). + * Strings and comments are skipped; a single- or double-quoted string never + * spans a line, so a mis-lexed quote (a regex literal) costs at most that + * line. Returns the 1-based line of each closing brace whose frame offends. + * Copied from the `rest_api` retirement pin, never imported: each pin keeps + * running standalone. + */ + const lexOffenders = (text: string): number[] => { + const out: number[] = []; + const stack: { kind: string; keys: Set; parentKey?: string }[] = []; + let lastSig = ''; + let justKey: string | undefined; + let colonKey: string | undefined; + let line = 1; + let i = 0; + const n = text.length; + const sawToken = (token: string, j: number) => { + const top = stack[stack.length - 1]; + const isKey = text[j] === ':' && top?.kind === '{' && (lastSig === '{' || lastSig === ','); + if (isKey) top!.keys.add(token); + justKey = isKey ? token : undefined; + }; + while (i < n) { + const c = text[i]!; + if (c === '\n') { line += 1; i += 1; continue; } + if (c === '/' && text[i + 1] === '/') { while (i < n && text[i] !== '\n') i += 1; continue; } + if (c === '/' && text[i + 1] === '*') { + i += 2; + while (i < n && !(text[i] === '*' && text[i + 1] === '/')) { if (text[i] === '\n') line += 1; i += 1; } + i += 2; + continue; + } + if (c === '"' || c === "'" || c === '`') { + const start = i; + i += 1; + while (i < n && text[i] !== c) { + if (text[i] === '\\') i += 1; + else if (text[i] === '\n') { if (c !== '`') break; line += 1; } + i += 1; + } + const token = text.slice(start + 1, i); + i += 1; + let j = i; + while (j < n && (text[j] === ' ' || text[j] === '\t')) j += 1; + if (c === '`') justKey = undefined; + else sawToken(token, j); + lastSig = 'str'; + continue; + } + if (/[A-Za-z_$]/.test(c)) { + const start = i; + while (i < n && /[\w$]/.test(text[i]!)) i += 1; + const token = text.slice(start, i); + let j = i; + while (j < n && (text[j] === ' ' || text[j] === '\t')) j += 1; + sawToken(token, j); + lastSig = 'id'; + continue; + } + if (c === ':') { + colonKey = justKey; + justKey = undefined; + lastSig = ':'; + i += 1; + continue; + } + if (c === '{') stack.push({ kind: c, keys: new Set(), parentKey: lastSig === ':' ? colonKey : undefined }); + else if (c === '[' || c === '(') stack.push({ kind: c, keys: new Set() }); + else if (c === '}' || c === ']' || c === ')') { + const frame = stack.pop(); + if (frame?.kind === '{' && c === '}' && isOffender(frame.parentKey, frame.keys)) out.push(line); + } + if (!/\s/.test(c)) { lastSig = c; justKey = undefined; } + i += 1; + } + return out; + }; + + /** + * YAML: a block mapping opened by `requires:` with no inline value; its OWN + * keys are the lines at its first child's column, until a line at or above + * the opener's column. Returns the 1-based line of each offending opener. + */ + const yamlOffenders = (text: string): number[] => { + const rows = text.split('\n').map((raw, idx) => { + const m = /^(\s*)(-\s+)?([A-Za-z_][\w]*)\s*:/.exec(raw); + if (!m) return null; + const opensBlock = /^\s*(-\s+)?[A-Za-z_][\w]*\s*:\s*(#.*)?$/.test(raw); + return { idx, col: m[1]!.length + (m[2]?.length ?? 0), key: m[3]!, opensBlock }; + }); + const out: number[] = []; + rows.forEach((row, at) => { + if (!row || !row.opensBlock || row.key !== 'requires') return; + const keys = new Set(); + let childCol: number | undefined; + for (let k = at + 1; k < rows.length; k += 1) { + const r = rows[k]; + if (!r) continue; + if (r.col <= row.col) break; + childCol ??= r.col; + if (r.col === childCol) keys.add(r.key); + } + if (isOffender(row.key, keys)) out.push(row.idx + 1); + }); + return out; + }; + + /** MD/MDX: only fenced code is judged — prose mentions are not authorings. */ + const markdownOffenders = (text: string): number[] => { + const out: number[] = []; + const fence = /^```([\w-]*)[^\n]*\n([\s\S]*?)^```/gm; + for (let m = fence.exec(text); m; m = fence.exec(text)) { + const lang = m[1]!.toLowerCase(); + const body = m[2]!; + const offset = text.slice(0, m.index).split('\n').length; + const hits = lang === 'yaml' || lang === 'yml' ? yamlOffenders(body) : lexOffenders(body); + for (const h of hits) out.push(offset + h); + } + return out; + }; + + const mentions = (text: string): boolean => text.includes('requires') && text.includes('plugins'); + + const offendersIn = (ext: string, text: string): number[] => { + if (!mentions(text)) return []; + if (ext === '.yaml' || ext === '.yml') return yamlOffenders(text); + if (ext === '.md' || ext === '.mdx') return markdownOffenders(text); + return lexOffenders(text); + }; + + const vanished: string[] = []; + /** Tolerates ONLY a path's disappearance mid-walk; every other fault is re-raised. */ + const readIfPresent = (full: string, rel: string): string | undefined => { + try { + return fs.readFileSync(full, 'utf-8'); + } catch (err) { + if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') throw err; + vanished.push(rel); + return undefined; + } + }; + + it('the matcher finds an authoring and ignores every neighbouring shape (anti-vacuity)', () => { + // Offenders — the retired spelling, in each syntax the walk reads. + expect(offendersIn('.json', '{ "scenarios": [ { "id": "a", "requires": { "plugins": ["crm"] } } ] }')).toEqual([1]); + expect(offendersIn('.json', '{\n "requires": {\n "params": ["X"],\n "plugins": []\n }\n}')).toEqual([5]); + expect(offendersIn('.ts', "const s = { id: 'a', requires: { services: ['ai'], plugins: ['crm'] } } as never;")).toEqual([1]); + expect(offendersIn('.yaml', 'scenarios:\n - id: a\n requires:\n plugins:\n - crm\n')).toEqual([3]); + expect(offendersIn('.md', 'Prose.\n\n```json\n{ "requires": { "plugins": ["crm"] } }\n```\n')).toEqual([4]); + // Neighbours that must NOT match. + // The replacement spelling. + expect(offendersIn('.json', '{ "requires": { "params": ["X"], "services": ["ai"] } }')).toEqual([]); + // A stack's `requires` is an array of capabilities; a manifest's is an array of `{ pluginId }`. + expect(offendersIn('.ts', "defineStack({ requires: ['ai'], plugins: [new AuthPlugin()] })")).toEqual([]); + expect(offendersIn('.json', '{ "requires": [ { "pluginId": "a", "plugins": ["b"] } ] }')).toEqual([]); + // `plugins` NESTED one level down belongs to another block. + expect(offendersIn('.ts', "({ requires: { params: ['X'], extra: { plugins: [] } } })")).toEqual([]); + // The schema declaration itself: the `{` is `z.object(`'s argument, not the key's value. + expect(offendersIn('.ts', "requires: z.object({ plugins: retiredKey('gone') })")).toEqual([]); + // Prose, comments and quoted strings are not authorings. + expect(offendersIn('.md', 'A suite once wrote `requires: { plugins: [crm] }` here.')).toEqual([]); + expect(offendersIn('.ts', '// requires: { plugins: [] }')).toEqual([]); + expect(offendersIn('.ts', 'const s = "{ requires: { plugins: [] } }";')).toEqual([]); + // A YAML `requires` mapping whose `plugins` belongs to a nested block. + expect(offendersIn('.yaml', 'requires:\n params:\n - X\n other:\n plugins: []\n')).toEqual([]); + }); + + it('a path that VANISHES mid-walk is not a finding, and every other read fault still is', () => { + const before = vanished.length; + const gone = path.join(REPO_ROOT, 'packages/spec/does-not-exist.bundled_probe.mjs'); + expect(fs.existsSync(gone)).toBe(false); + expect(readIfPresent(gone, 'probe/gone')).toBeUndefined(); + expect(vanished.slice(before)).toEqual(['probe/gone']); + expect(readIfPresent(fileURLToPath(import.meta.url), THIS_FILE)).toContain('tree-scoped absence'); + expect(() => readIfPresent(path.join(REPO_ROOT, 'packages/spec'), 'probe/dir')).toThrow(); + expect(vanished.length).toBe(before + 1); + }); + + it('no `requires` block authoring `plugins` survives inside the declared radius', () => { + const offenders: string[] = []; + let visited = 0; + let suites = 0; + const walk = (dir: string) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + const rel = path.relative(REPO_ROOT, full).split(path.sep).join('/'); + if (entry.isDirectory()) { + if (SKIPPED_DIRS.has(entry.name) || entry.name.startsWith('.')) continue; + walk(full); + continue; + } + if (!entry.isFile()) continue; + const ext = path.extname(entry.name); + if (!(rel.startsWith('examples/') ? EXAMPLES_EXT : SCANNED_EXT).has(ext)) continue; + if (entry.name === 'CHANGELOG.md') continue; // release prose records the removal + if (EXCLUDED.has(rel) || EXCLUDED_PREFIXES.some((p) => rel.startsWith(p))) continue; + if (TSUP_BUNDLED_CONFIG.test(entry.name)) continue; + visited += 1; + const text = readIfPresent(full, rel); + if (text === undefined) continue; + if (entry.name.endsWith('.test.json') && text.includes('"scenarios"')) suites += 1; + for (const lineNo of offendersIn(ext, text)) offenders.push(`${rel}:${lineNo}`); + } + }; + for (const root of WALK_ROOTS) walk(path.join(REPO_ROOT, root)); + // Anti-vacuity: the walk covered the tree, and the authored QA suites — + // the files that most plausibly hold an authoring — were really judged. + expect(visited).toBeGreaterThan(1000); + expect(suites).toBeGreaterThan(0); + expect(offenders, 'a `requires` block authoring `plugins` means the retirement is being undone').toEqual([]); + }); +}); diff --git a/packages/spec/src/qa/testing.test.ts b/packages/spec/src/qa/testing.test.ts index 1087d484e53..4bdbd06ba12 100644 --- a/packages/spec/src/qa/testing.test.ts +++ b/packages/spec/src/qa/testing.test.ts @@ -8,7 +8,9 @@ import { TestStepSchema, TestScenarioSchema, TestSuiteSchema, + type TestScenario, } from './testing.zod'; +import { CORE_SERVICE_PROVIDER } from '../system/core-services.zod'; describe('TestContextSchema', () => { it('should accept a valid context record', () => { @@ -158,7 +160,7 @@ describe('TestScenarioSchema', () => { teardown: [{ name: 'cleanup', action: { type: 'delete_record', target: 'account' } }], requires: { params: ['API_KEY'], - plugins: ['crm'], + services: ['data', 'auth'], }, }; expect(() => TestScenarioSchema.parse(scenario)).not.toThrow(); @@ -200,3 +202,93 @@ describe('TestSuiteSchema', () => { expect(() => TestSuiteSchema.parse({ name: 'Suite' })).toThrow(); }); }); + +/** + * `requires` is ENFORCED (ADR-0049; the `qa-runner` family's `requires` key, + * ruled B): core's TestRunner judges it before a scenario's first step, and an + * unmet entry makes the scenario SKIPPED with its reason. The runner half is + * pinned in `packages/core/src/qa/runner.test.ts`; this half pins the contract + * an author writes against — `services` is the target's own vocabulary, closed + * at parse, and `plugins` is a tombstone that carries its prescription. + * + * On the assertion set: a schema refusal raises a `ZodError` whose issues carry + * `code` and `path` but no ADR-0112 `status` — that envelope belongs to the API + * error surface. So these pins assert refusal, the issue `code`, the `path` + * naming the key, and the prescription. + */ +describe('TestScenarioSchema.requires', () => { + const scenario = (requires: unknown) => ({ + id: 'sc-req', + name: 'Requires', + steps: [{ name: 'step1', action: { type: 'api_call', target: '/api/v1/health' } }], + requires, + }); + + it('accepts `params` and `services` — the two judged keys — and keeps them', () => { + const r = TestScenarioSchema.safeParse(scenario({ params: ['OS_QA_TOKEN'], services: ['ai', 'analytics'] })); + expect(r.success).toBe(true); + if (!r.success) return; + expect(r.data.requires).toEqual({ params: ['OS_QA_TOKEN'], services: ['ai', 'analytics'] }); + }); + + it('closes `services` over the discovery service keys: a misspelling is refused at parse, located', () => { + const r = TestScenarioSchema.safeParse(scenario({ services: ['analytic'] })); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'requires.services.0'); + expect(issue, 'the refusal must locate `requires.services.0`').toBeDefined(); + expect(issue!.code).toBe('invalid_value'); + }); + + // Unanchored, because a thrown `ZodError`'s message is the JSON of its issues; + // the key-first house convention is asserted on the issue message itself. + const PLUGINS_PRESCRIPTION = + /`scenarios\[\]\.requires\.plugins` was removed in @objectstack\/spec 17\.5\.0 \(ADR-0049 enforce-or-remove\).*nothing ever checked it.*Delete the key and name the service the scenario needs in `requires\.services`.*Plugin → service: .*@objectstack\/plugin-auth → auth/s; + + for (const plugins of [['crm'], ['@objectstack/plugin-auth'], []]) { + it(`refuses \`requires.plugins: ${JSON.stringify(plugins)}\` at its path, with the prescription`, () => { + const r = TestScenarioSchema.safeParse(scenario({ plugins })); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'requires.plugins'); + expect(issue, 'the refusal must locate `requires.plugins`').toBeDefined(); + expect(issue!.code).toBe('invalid_type'); + expect(issue!.message).toMatch(PLUGINS_PRESCRIPTION); + expect(issue!.message.startsWith('`scenarios[].requires.plugins` was removed')).toBe(true); + }); + } + + it('the suite door refuses it THROUGH `scenarios`, located at the scenario', () => { + const r = TestSuiteSchema.safeParse({ name: 'Suite', scenarios: [scenario({ plugins: ['crm'] })] }); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'scenarios.0.requires.plugins'); + expect(issue, 'the refusal must locate `scenarios.0.requires.plugins`').toBeDefined(); + expect(issue!.message).toMatch(PLUGINS_PRESCRIPTION); + }); + + it('fails tsc at the authoring site: the input type is `never`', () => { + const authored: TestScenario = { + id: 'sc-typed', + name: 'Typed', + steps: [], + // @ts-expect-error — `plugins` is a retiredKey() tombstone: its input type is `never`. + requires: { plugins: ['crm'] }, + }; + // The parse channel agrees with the type channel on the same literal. + expect(() => TestScenarioSchema.parse(authored)).toThrow(PLUGINS_PRESCRIPTION); + }); + + it('the prescription maps every provider package discovery names to its service', () => { + const r = TestScenarioSchema.safeParse(scenario({ plugins: ['x'] })); + expect(r.success).toBe(false); + if (r.success) return; + const message = r.error.issues.find((i) => i.path.join('.') === 'requires.plugins')!.message; + for (const [slot, pkg] of Object.entries(CORE_SERVICE_PROVIDER)) { + if (pkg === null || slot === 'file-storage') continue; + expect(message, `${pkg} → ${slot} must be in the mapping`).toMatch(new RegExp(`${pkg.replace(/[/.]/g, '\\$&')} → [a-z/ -]*\\b${slot}\\b`)); + } + // The deprecated alias is not taught to a converting author. + expect(message).not.toContain('file-storage'); + }); +}); diff --git a/packages/spec/src/qa/testing.zod.ts b/packages/spec/src/qa/testing.zod.ts index 850fb87d2cc..5376f0841e8 100644 --- a/packages/spec/src/qa/testing.zod.ts +++ b/packages/spec/src/qa/testing.zod.ts @@ -5,6 +5,26 @@ import { z } from 'zod'; // --- Building Blocks --- import { lazySchema } from '../shared/lazy-schema'; +import { retiredKey } from '../shared/retired-key'; +import { CORE_SERVICE_PROVIDER, CoreServiceName } from '../system/core-services.zod'; + +/** + * The plugin → service mapping the `requires.plugins` tombstone hands an author + * converting a suite, derived from {@link CORE_SERVICE_PROVIDER} — the one table + * both discovery builders read to name a slot's provider — so the prescription + * cannot name a package that does not fill the slot it claims. `file-storage` + * is left out: it is the deprecated alias of `storage`, and a converting author + * should write the canonical slot. + */ +const PLUGIN_TO_SERVICE: string = (() => { + const byPackage = new Map(); + for (const [slot, pkg] of Object.entries(CORE_SERVICE_PROVIDER)) { + if (pkg === null || slot === 'file-storage') continue; + byPackage.set(pkg, [...(byPackage.get(pkg) ?? []), slot]); + } + return [...byPackage].map(([pkg, slots]) => `${pkg} → ${slots.join(' / ')}`).join(', '); +})(); + export const TestContextSchema = lazySchema(() => z.record(z.string(), z.unknown()).describe('Initial context or variables for the test')); // Action Types @@ -68,11 +88,42 @@ export const TestScenarioSchema = lazySchema(() => z.object({ steps: z.array(TestStepSchema).describe('Main test sequence to execute'), teardown: z.array(TestStepSchema).optional().describe('Steps to cleanup after test execution'), - // Environment requirements — declared, not yet checked by any reader. + // Preconditions, judged by core's `TestRunner` before the scenario's first + // step (ADR-0049: declared is enforced). An unmet entry makes the scenario + // SKIPPED with a reason — counted on its own, never as passed. requires: z.object({ - params: z.array(z.string()).optional().describe('Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs'), - plugins: z.array(z.string()).optional().describe('Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs') - }).optional().describe('Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes') + params: z.array(z.string()).optional().describe( + 'Environment variables that must be set to a non-empty value in the process running `os test` — the runner\'s own ' + + 'environment, not the target server\'s. An unset or empty variable skips the scenario (SKIPPED, never passed) ' + + 'with a reason naming the variable' + ), + services: z.array(CoreServiceName).optional().describe( + 'Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), ' + + 'read from the discovery document `os test` already fetches once per run. An entry the target does not declare ' + + 'that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target ' + + 'does declare available' + ), + // ADR-0049 enforce-or-remove, ruled B: `plugins` could never be judged — + // `os test` reaches its target over HTTP and no served surface lists the + // loaded plugins — so it retires into `services`, the discovery contract + // that already says what a host serves. A tombstone, not a bare deletion: + // this object is non-strict, and a deleted key would be stripped in + // silence. No D2 conversion: a QA suite is a loose JSON file `os test` + // loads, never a stack collection member or a stored row; the family's D3 + // entry is `qa-scenario-requires-plugins-retired`. + plugins: retiredKey( + '`scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ' + + 'ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a ' + + 'scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in ' + + '`requires.services`, which is judged against the services the target\'s discovery document declares enabled ' + + 'and available — an unmet entry skips the scenario and says why. Plugin → service: ' + + `${PLUGIN_TO_SERVICE}. A plugin not listed fills no discovery service slot, so there is no service to require ` + + 'for it.', + ), + }).optional().describe( + 'Preconditions judged before the scenario\'s first step (setup included). Every entry must hold, or the scenario is ' + + 'SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed' + ) }).describe('A complete test scenario with setup, execution steps, and teardown')); export const TestSuiteSchema = lazySchema(() => z.object({ diff --git a/packages/spec/vitest.repo-tests.json b/packages/spec/vitest.repo-tests.json index 5386ac6a77f..1ce4c7e6316 100644 --- a/packages/spec/vitest.repo-tests.json +++ b/packages/spec/vitest.repo-tests.json @@ -29,6 +29,7 @@ "src/identity/position-delegatable-enforcer.pin.test.ts", "src/integration/connector-connection-timeout-retirement.test.ts", "src/integration/connector-resilience-keys-retirement.test.ts", + "src/qa/requires-plugins-retirement.test.ts", "src/security/rls-tags-retirement.test.ts", "src/shared/retired-key-migrate-sentence.test.ts", "src/system/compliance-families-retirement.test.ts",