From 89edc7a0bfc136c22c107c56d28c96a4e8a90d86 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 19:21:21 +0000 Subject: [PATCH 1/4] =?UTF-8?q?feat(spec,core,cli)!:=20enforce=20scenario?= =?UTF-8?q?=20requires=20=E2=80=94=20params=20and=20services=20skip=20with?= =?UTF-8?q?=20a=20reason;=20retire=20requires.plugins=20(WIP)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ruling B on the qa-runner family's requires key: requires.params is judged against the environment of the process running os test, requires.services (new, closed over CoreServiceName) against the target's discovery services map, and an unmet entry makes the scenario SKIPPED with its reason. The retired requires.plugins is a retiredKey() tombstone whose prescription names requires.services. Claude-Session: https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH Co-authored-by: Claude --- content/docs/deployment/cli.mdx | 35 +- docs/qa/platform-checklist/areas/cli.json | 9 +- packages/cli/src/commands/test.ts | 51 ++- packages/core/src/qa/adapter.ts | 25 ++ packages/core/src/qa/http-adapter.ts | 134 ++++++-- packages/core/src/qa/runner.ts | 164 ++++++++- packages/spec/liveness/README.md | 2 +- packages/spec/liveness/qa.json | 8 +- .../spec/scripts/liveness/check-liveness.mts | 12 +- .../18.qa__TestScenario__requires.plugins.ts | 15 + ...18.qa-scenario-requires-plugins-retired.ts | 40 +++ packages/spec/src/migrations/registry.ts | 49 +++ .../qa/requires-plugins-retirement.test.ts | 318 ++++++++++++++++++ packages/spec/src/qa/testing.test.ts | 94 +++++- packages/spec/src/qa/testing.zod.ts | 59 +++- packages/spec/vitest.repo-tests.json | 1 + 16 files changed, 954 insertions(+), 62 deletions(-) create mode 100644 packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts create mode 100644 packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts create mode 100644 packages/spec/src/qa/requires-plugins-retirement.test.ts diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index cc9dd12178b..c7c6052e72c 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1715,9 +1715,38 @@ out are **deselected**: not run, counted on the summary and never counted as passed. A listed tag that no loaded scenario carries is named on the summary, since a typo narrows the run without failing it, and an empty entry (`--tags smoke,`) is refused before anything runs. Without the flag every scenario -runs. A scenario's `requires` block is **not checked**: a scenario that names a -plugin the target does not load still runs, and the unmet requirement surfaces -only as whatever failure it causes, if any — the scenario can still pass. +runs. + +**A scenario's `requires` is checked before its first step.** Two keys, each +judged against something `os test` can observe: + +- `requires.params` — environment variables that must be set to a non-empty + value **in the process running `os test`** (not on the target server, which a + suite cannot see): `"params": ["OS_QA_BILLING_TOKEN"]`. +- `requires.services` — service keys the target must declare in its discovery + document as `enabled` with status `available`, read from the same discovery + request the runner already makes once per run: `"services": ["ai", "analytics"]`. + The keys are the discovery service names (`auth`, `automation`, `storage`, …), + and a misspelled one is refused when the suite loads. + +A scenario with an unmet entry is **skipped**: none of its steps runs — `setup` +included — and its line says why, naming every unmet entry and, for a service, +the services the target does declare available: + +```text + ⏭️ Scenario: Summarise an account with the AI service [ai-summary] (skipped) + Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata. +``` + +A skipped scenario is counted on its own — +`SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — and +is never counted as passed. Skips alone do not fail a run, but a run in which +**every** selected scenario was skipped proved nothing: it prints +`No scenario ran: all 2 selected scenarios were skipped on unmet requirements.` +instead of `SUCCESS`, exits **0**, and exits **1** under `--fail-on-empty`. The +retired `requires.plugins` is refused when the suite loads, with the plugin → +service mapping in its message: no served surface lists loaded plugins, so it +was never checkable, and `requires.services` asks the question it stood for. **A pattern that matches no suite is not a failure by default.** The run prints `Found 0 test suites.` — the same machine-readable line a full run prints, so a diff --git a/docs/qa/platform-checklist/areas/cli.json b/docs/qa/platform-checklist/areas/cli.json index d328057fed9..e47e72af417 100644 --- a/docs/qa/platform-checklist/areas/cli.json +++ b/docs/qa/platform-checklist/areas/cli.json @@ -438,7 +438,7 @@ "title": "os test: a Quality Protocol suite is validated at LOAD, executed against a booted app, and its verdict is the exit code — capture/interpolation thread state, an unevaluable assertion FAILS", "since": "v17", "status": "active", - "revision": 5, + "revision": 6, "priority": "P1", "surface": "cli", "personas": ["operator (local shell)", "suite author (writes qa/*.test.json)"], @@ -453,7 +453,7 @@ "knownGaps": [ "`run_script` has no adapter branch and fails by name (`Unsupported action type in HttpAdapter: run_script`) — the variant sweep records ONE refusal and that is the honest verdict, not a fixture gap to work around. The other seven members execute since #7848; the five record-shaped ones did not until then (see `negative`), so a sweep transcript predating that fix shows five 404s and is not comparable", "the record action types resolve the Data Protocol mount from the server's own `/discovery` (`routes.data`) since #7983, so a deployment that sets `crud.dataPrefix` IS reached; what remains out of reach is `api.apiPath`, which moves the discovery document itself out from under the probe (measured: `{apiBase}/discovery` 404s, and `/.well-known/objectstack` reports the DISPATCHER's `/api/v1/data`, not the REST mount). Against such a host the adapter falls back to the convention and says so — it names the mount it addressed, the probe that failed and the remedy, on the warning AND on every 404 — and those steps must still be written as `api_call`. The fixture boot is stock, so neither case bites here", - "scenario SELECTION is by tag only, and preconditions are NOT honoured: `os test --tags a,b` runs the scenarios carrying at least one listed tag (any-of, exact, case-sensitive — commands/test.ts `selectScenariosByTags`), counts the rest as deselected — not run, never counted as passed — and without the flag the whole glob runs; `scenario.requires` (`params`/`plugins`) is declared but NOT CHECKED — its describe() says so and its row in packages/spec/liveness/qa.json stays `dead` pending the maintainer's decision on #20289 — so a suite still cannot declare a precondition it will be skipped for, and a scenario naming a plugin the target lacks still runs, the unmet requirement surfacing only as whatever failure it causes, if any" + "scenario SELECTION is by tag only: `os test --tags a,b` runs the scenarios carrying at least one listed tag (any-of, exact, case-sensitive — commands/test.ts `selectScenariosByTags`), counts the rest as deselected — not run, never counted as passed — and without the flag the whole glob runs. Preconditions SKIP rather than select: a scenario whose `requires.params` variable is unset or empty in the `os test` process, or whose `requires.services` entry the target's discovery does not declare enabled and available, is skipped with its reason (core runner.ts `judgeRequirements`), counted apart and never passed. The fixture suite declares no `requires`, so the green path exercises no skip; a skip probe must be STAGED — a scenario requiring a service the stock showcase boot does not declare available (`search` is one: nothing in the open framework registers that slot) — and an all-skipped run exits 0, or 1 under --fail-on-empty" ] }, "steps": [ @@ -548,7 +548,7 @@ "packages/core/src/qa/runner.ts#scenario (scenario sequencing, `capture` + `{{var}}` interpolation, the assertion operators, setup/teardown semantics, the #7256 unevaluable-`contains` fix)", "packages/core/src/qa/http-adapter.ts#mount (the action-type switch — its case labels ARE the enum values; the record routes take their prefix from the one memoised `/discovery` probe per run, falling back to the RestApiConfigSchema + CrudEndpointsConfigSchema convention with a diagnostic that names the mount, #7848 / #7983)", "packages/spec/src/qa/testing.zod.ts#TestSuiteSchema (TestSuiteSchema — the shape enforced at load; TestActionTypeSchema pinned above)", - "packages/spec/liveness/qa.json#tags (the ADR-0049 ledger whose existence this item is coverage.json's mapping for — its `tags` row is live on `os test --tags`, and its `requires` row stays dead: the precondition is declared and not checked)", + "packages/spec/liveness/qa.json#tags (the ADR-0049 ledger whose existence this item is coverage.json's mapping for — its `tags` row is live on `os test --tags`, and its `requires` row is live on the runner's precondition judgement: an unmet `params` or `services` entry skips the scenario)", "packages/cli/test/qa-suite-schema-load.test.ts, packages/cli/test/resolve-glob-lazy-walk.test.ts, packages/core/src/qa/runner.test.ts (the three unit pins — cited so a run knows what is already covered, NOT a substitute for driving a booted app)", "content/docs/deployment/cli.mdx §os test (the documented command contract)", "examples/app-showcase/qa/platform-smoke.test.json (the fixture suite this item drives)" @@ -558,7 +558,8 @@ {"revision": 2, "date": "2026-08-12", "change": "the adapter was repaired, which this item's own `negative` clause declared to be a revision rather than a silent green (#7848). Item 1: the five record-shaped action types now round-trip against a stock server — the `${baseUrl}/api/data/:object` literal became a prefix DERIVED from the two schemas RestServer itself resolves from (RestApiConfigSchema `apiPath ?? {basePath}/{version}` + CrudEndpointsConfigSchema.dataPrefix), and `update_record` PATCHes where it used to PUT a route that has no PUT sibling. Re-measured on a booted showcase, one scenario per member with NO shared setup so no member's verdict is inferred from a sibling: 7 of 8 execute and assert, `run_script` still refuses by name. The `negative` clause is inverted accordingly — a 404 from a record action type is the regression now — and the knownGap it rested on is replaced by the narrower one that survives: the record types address the DEFAULT mount only, so a host that moves it with `api.apiPath`/`crud.dataPrefix` still needs `api_call`. Item 2: a zero-match glob still exits 0, deliberately (a repo that legitimately ships no suites must not start failing CI), but the posture is now DECLARED — stated in `--help`, opt out with the new `--fail-on-empty`, and `Found N test suites.` is emitted on EVERY run including `Found 0 test suites.`, which is the line this item's first acceptance clause already asks a run record to quote", "ref": "#7848"}, {"revision": 3, "date": "2026-08-17", "change": "the record action types stopped ASSUMING the mount (#7983). They now resolve it from the server: one memoised `GET {apiBase}/discovery` per run, addressing whatever `routes.data` advertises, with the RestApiConfigSchema + CrudEndpointsConfigSchema convention as the fallback — the `@objectstack/client` `getRoute` pattern, copied rather than re-invented. Measured on a booted stack (REST generator + dispatcher bridge) before and after, three configs: stock stays green; `crud.dataPrefix: '/objects'` went from `HTTP Error 404` to a created record, so that row of the gap is CLOSED; `api.apiPath: '/api/2026-01'` still 404s and is NOT closed — `apiPath` moves the discovery document itself, and the one fixed-path document (`/.well-known/objectstack`) advertises the dispatcher's `/api/v1/data` under all three configs, so trusting it would attach a false provenance to the same 404. That row is narrowed instead: the fallback is announced (a warning naming the mount, the failed probe and the remedy) and every 404 from a record action now carries the mount it addressed and where that mount came from, so the failure can no longer read as the suite author's own URL mistake. `api_call` is unchanged and probes nothing — it remains the escape hatch for a host the probe cannot reach", "ref": "#7983"}, { "revision": 4, "date": "2026-08-18", "change": "corrected the flag count from two to three. packages/cli/src/commands/test.ts declares url, token AND fail-on-empty; the same revision's own acceptance already describes --fail-on-empty, so the knownGap contradicted its own item. The gap's point is unchanged and now stated directly: none of the three flags selects scenarios (#9417)", "ref": "#9386" }, - { "revision": 5, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified by #20289 (PR #20341) and are corrected: `os test` gained `--tags` (any-of, exact, case-sensitive; deselected scenarios are counted and never passed; a selection that matches no scenario exits 0, or 1 under --fail-on-empty), and the report now heads each suite with its `name` and prints each scenario's `name` beside its id. What stays true is narrower: `scenario.requires` is declared but NOT CHECKED, pending the maintainer's decision on #20289. Steps and acceptance are unchanged — the green-path clause's scenarioId is still on every scenario line, now in brackets after the name, or bare when the name equals the id", "ref": "#20289" } + { "revision": 5, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified by #20289 (PR #20341) and are corrected: `os test` gained `--tags` (any-of, exact, case-sensitive; deselected scenarios are counted and never passed; a selection that matches no scenario exits 0, or 1 under --fail-on-empty), and the report now heads each suite with its `name` and prints each scenario's `name` beside its id. What stays true is narrower: `scenario.requires` is declared but NOT CHECKED, pending the maintainer's decision on #20289. Steps and acceptance are unchanged — the green-path clause's scenarioId is still on every scenario line, now in brackets after the name, or bare when the name equals the id", "ref": "#20289" }, + { "revision": 6, "date": "2026-09-28", "change": "the knownGap and the qa.json source note were falsified again by #20289's `requires` round (ruling B) and are corrected: `scenario.requires` is now CHECKED before the first step — `params` against the `os test` process environment, `services` against the target's discovery `services` (enabled and `available`) — and an unmet entry SKIPS the scenario with its reason, counted apart and never passed; an all-skipped run exits 0, or 1 under --fail-on-empty. `requires.plugins` is retired (refused at load, prescription names `requires.services`). Steps and acceptance are unchanged: the fixture suite declares no `requires`, so the green path runs every scenario exactly as before", "ref": "#20289" } ] }, { diff --git a/packages/cli/src/commands/test.ts b/packages/cli/src/commands/test.ts index 774d2436cf4..d1733563922 100644 --- a/packages/cli/src/commands/test.ts +++ b/packages/cli/src/commands/test.ts @@ -329,6 +329,18 @@ export function tagSelectionLine(selected: number, deselected: number, tags: rea return `--tags ${tags.join(',')} selected ${selected} of ${selected + deselected} scenarios; ${deselected} deselected (not run, not counted as passed).`; } +/** + * The run's closing tally. A SKIPPED scenario — one whose `requires` did not + * hold, so no step ran — is its own count: never passed, never failed. When + * nothing was skipped the line reads exactly as it did before skips existed. + */ +export function summaryLine(passed: number, failed: number, skipped: number): string { + const skippedClause = skipped > 0 ? ` ${skipped} skipped (not run, not counted as passed).` : ''; + if (failed > 0) return `FAILED: ${failed} scenarios failed. ${passed} passed.${skippedClause}`; + if (skipped > 0) return `SUCCESS: ${passed} scenarios passed.${skippedClause}`; + return `SUCCESS: All ${passed} scenarios passed.`; +} + export default class Test extends Command { /** * The empty-match posture is stated in the help text on purpose (#7848). @@ -353,7 +365,13 @@ export default class Test extends Command { 'case-sensitive). Untagged scenarios are left out whenever --tags is given. Left-out ' + 'scenarios are counted as deselected — never run and never counted as passed. A ' + 'selection that matches no scenario exits 0 like an empty pattern, and 1 under ' + - '--fail-on-empty.'; + '--fail-on-empty.\n' + + 'A scenario whose "requires" does not hold — an environment variable in "params" ' + + 'unset or empty in this process, or a service in "services" the target\'s discovery ' + + 'document does not declare enabled and available — is SKIPPED: no step runs, the ' + + 'reason is printed, and it is counted as skipped, never as passed. Skips alone do ' + + 'not fail the run, but a run in which every selected scenario was skipped is not a ' + + 'pass either: it exits 0, and 1 under --fail-on-empty.'; static override args = { files: Args.string({ description: 'Glob pattern for test files (e.g. "qa/*.test.json")', required: false, default: 'qa/*.test.json' }), @@ -364,7 +382,7 @@ export default class Test extends Command { token: Flags.string({ description: 'Authentication token' }), 'fail-on-empty': Flags.boolean({ description: - 'Exit non-zero when the pattern matches no test suite, or --tags matches no scenario (default: both exit 0)', + 'Exit non-zero when the pattern matches no test suite, --tags matches no scenario, or every selected scenario was skipped on an unmet "requires" (default: all three exit 0)', default: false, }), // A custom flag so a malformed list is refused while the invocation is @@ -387,7 +405,9 @@ export default class Test extends Command { // 1. Setup Runner const adapter = new CoreQA.HttpTestAdapter(flags.url, flags.token); - const runner = new CoreQA.TestRunner(adapter); + // `requires.params` is judged against THIS process's environment — the one + // running the suite — which is what the spec declares. + const runner = new CoreQA.TestRunner(adapter, { env: process.env }); // 2. Find test files using glob-style pattern matching const testFiles: string[] = resolveGlob(filesPattern); @@ -411,6 +431,7 @@ export default class Test extends Command { const tags = flags.tags; let totalPassed = 0; let totalFailed = 0; + let totalSkipped = 0; let totalSelected = 0; let totalDeselected = 0; const matchedTags = new Set(); @@ -444,6 +465,15 @@ export default class Test extends Command { const results = await runner.runSuite(suite); for (const result of results) { + if (result.status === 'skipped') { + // Not run, so not a pass and not a failure: its own line, + // its own count, and the reason in full — the unmet entry + // and, for a service, what the target does serve. + console.log(` ⏭️ Scenario: ${scenarioLabel(result)} (skipped)`); + console.log(chalk.yellow(` Skipped: ${result.skipped?.reason ?? 'a requirement was not met.'}`)); + totalSkipped++; + continue; + } const icon = result.passed ? '✅' : '❌'; console.log(` ${icon} Scenario: ${scenarioLabel(result)} (${result.duration}ms)`); if (!result.passed) { @@ -482,7 +512,7 @@ export default class Test extends Command { } } if (totalFailed > 0) { - console.log(chalk.red(`FAILED: ${totalFailed} scenarios failed. ${totalPassed} passed.`)); + console.log(chalk.red(summaryLine(totalPassed, totalFailed, totalSkipped))); process.exit(1); } else if (tags && totalSelected === 0) { // The --tags twin of the empty pattern above, and the same posture: a @@ -496,8 +526,19 @@ export default class Test extends Command { } console.log(chalk.dim(`Exiting 0 — a selection that matches nothing is not a failure. Pass --fail-on-empty to make it one.`)); process.exit(0); + } else if (totalSkipped > 0 && totalPassed === 0) { + // Every selected scenario was skipped, so the run proved nothing: not + // a SUCCESS, and — the same posture as an empty pattern and an empty + // --tags selection — a failure only under --fail-on-empty. + console.warn(chalk.yellow(`No scenario ran: all ${totalSkipped} selected scenarios were skipped on unmet requirements.`)); + if (flags['fail-on-empty']) { + console.error(chalk.red(`--fail-on-empty: a run in which every scenario was skipped is a failed run.`)); + process.exit(1); + } + console.log(chalk.dim(`Exiting 0 — skipped scenarios are not failures. Pass --fail-on-empty to make a run that ran nothing one.`)); + process.exit(0); } else { - console.log(chalk.green(`SUCCESS: All ${totalPassed} scenarios passed.`)); + console.log(chalk.green(summaryLine(totalPassed, totalFailed, totalSkipped))); process.exit(0); } } diff --git a/packages/core/src/qa/adapter.ts b/packages/core/src/qa/adapter.ts index be96e4b2f00..4a133172e28 100644 --- a/packages/core/src/qa/adapter.ts +++ b/packages/core/src/qa/adapter.ts @@ -2,6 +2,23 @@ import * as QA from '@objectstack/spec/qa'; +/** + * What an adapter can say about the services its target serves — the input the + * runner judges `TestScenario.requires.services` against. + */ +export interface TargetServices { + /** + * The target's per-service status map, keyed by service name — the + * discovery document's `services` (ADR-0076 D12: advertise only what is + * mounted). Absent when the adapter could not read it; the runner then has + * nothing to judge a service requirement against, and every such requirement + * is unmet. + */ + services?: Readonly>; + /** One clause saying where `services` came from, or why it is absent. */ + source: string; +} + /** * Interface for executing test actions against a target system. * The target could be a local Kernel instance or a remote API. @@ -13,4 +30,12 @@ export interface TestExecutionAdapter { * @returns The result of the action (e.g. created record, API response) */ execute(action: QA.TestAction, context: Record): Promise; + + /** + * The services the target declares, for judging `requires.services` before a + * scenario runs. Optional: an adapter that cannot say leaves it out, and a + * scenario requiring a service is then SKIPPED with that reason — never run + * on an unverified precondition, never passed. + */ + readTargetServices?(): Promise; } diff --git a/packages/core/src/qa/http-adapter.ts b/packages/core/src/qa/http-adapter.ts index c1daa841f91..9bd81019d69 100644 --- a/packages/core/src/qa/http-adapter.ts +++ b/packages/core/src/qa/http-adapter.ts @@ -2,7 +2,7 @@ import * as QA from '@objectstack/spec/qa'; import { RestApiConfigSchema, CrudEndpointsConfigSchema } from '@objectstack/spec/api'; -import { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; /** Memoised {@link conventionMounts} — the schemas are `lazySchema`, so build them once. */ let conventionCache: { apiBase: string; dataPath: string } | undefined; @@ -64,26 +64,109 @@ interface ResolvedDataMount { readonly why: string; } +/** + * What the run's one discovery probe returned. + * + * The probe used to be read for `routes.data` and dropped. It now keeps the + * document, because `requires.services` is judged against the SAME document's + * `services` map (#20289, ruled B) — one probe answers both questions, and a + * second request for the second question would be a second chance for the two + * answers to disagree. + */ +interface DiscoveryRead { + /** `GET {baseUrl}{apiBase}/discovery` — the URL probed. */ + readonly probeUrl: string; + /** The HTTP status, when the server answered at all. */ + readonly status?: number; + /** + * The discovery document, unwrapped from the dispatcher bridge's + * `{ data: … }` envelope; absent when the probe yielded none. + */ + readonly document?: { readonly routes?: Record; readonly services?: unknown }; + /** One clause naming the probe and its outcome. */ + readonly why: string; +} + export class HttpTestAdapter implements TestExecutionAdapter { /** * The single discovery probe of a run, memoised as the in-flight promise so - * concurrent record actions share one request rather than racing N. + * concurrent record actions — and the `requires.services` judgement — share + * one request rather than racing N. * * `os test` builds ONE adapter for the whole run (`packages/cli/src/commands/ * test.ts`) and hands it to every suite, so instance scope IS run scope. */ + private discoveryPromise?: Promise; + /** The data mount derived from that probe, memoised so its fallback warns once. */ private mountPromise?: Promise; constructor(private baseUrl: string, private authToken?: string) {} + /** The run's discovery read; probes at most once per adapter. */ + private discovery(): Promise { + if (this.discoveryPromise === undefined) { + this.discoveryPromise = this.readDiscovery(); + } + return this.discoveryPromise; + } + /** The resolved data mount; probes at most once per adapter. */ private dataMount(): Promise { if (this.mountPromise === undefined) { - this.mountPromise = this.resolveDataMount(); + this.mountPromise = this.discovery().then((read) => this.resolveDataMount(read)); } return this.mountPromise; } + /** + * The target's declared services, for the runner's `requires.services` + * judgement — the `services` map of the discovery document this adapter + * probes once per run anyway (ADR-0076 D12: discovery advertises only what + * is mounted). Asked only when a scenario declares a service requirement, so + * a suite that declares none keeps the probe count it had: zero for an + * `api_call`-only suite, one otherwise. + */ + async readTargetServices(): Promise { + const read = await this.discovery(); + const services = read.document?.services; + if (services !== null && typeof services === 'object' && !Array.isArray(services)) { + return { services: services as Record, source: `GET ${read.probeUrl} advertised services` }; + } + return { + source: read.document + ? `GET ${read.probeUrl} answered ${read.status} but carried no services map` + : read.why, + }; + } + + /** Issue the one discovery request of the run and keep what it returned. */ + private async readDiscovery(): Promise { + const { apiBase } = conventionMounts(); + const probeUrl = `${this.baseUrl}${apiBase}/discovery`; + try { + const headers: Record = {}; + if (this.authToken) { + headers['Authorization'] = `Bearer ${this.authToken}`; + } + const response = await fetch(probeUrl, { method: 'GET', headers }); + if (!response.ok) { + return { probeUrl, status: response.status, why: `GET ${probeUrl} answered ${response.status}` }; + } + const body = await response.json(); + // `@objectstack/rest` answers the document bare; the dispatcher bridge + // wraps it as `{ data: … }`. Pick the object that actually carries the + // document's own keys rather than `body.data || body` — the document's + // own `routes.data` key makes the looser test ambiguous to read here. + const isDocument = (value: unknown): value is { routes?: Record; services?: unknown } => + value !== null && typeof value === 'object' && ('routes' in value || 'services' in value); + const inner = (body as { data?: unknown } | null)?.data; + const document = isDocument(body) ? body : isDocument(inner) ? inner : undefined; + return { probeUrl, status: response.status, document, why: `GET ${probeUrl} answered ${response.status}` }; + } catch (error) { + return { probeUrl, why: `GET ${probeUrl} could not be reached (${(error as Error).message})` }; + } + } + /** * Ask the server where it serves the Data Protocol, and fall back to the * convention — loudly — when it cannot say. @@ -120,40 +203,23 @@ export class HttpTestAdapter implements TestExecutionAdapter { * the remedy. `api_call` takes the path it is given and is unaffected either * way — it stays the escape hatch for a host this cannot reach. */ - private async resolveDataMount(): Promise { - const { apiBase, dataPath } = conventionMounts(); - const probeUrl = `${this.baseUrl}${apiBase}/discovery`; + private resolveDataMount(read: DiscoveryRead): ResolvedDataMount { + const { dataPath } = conventionMounts(); + const { probeUrl } = read; let why: string; - try { - const headers: Record = {}; - if (this.authToken) { - headers['Authorization'] = `Bearer ${this.authToken}`; - } - const response = await fetch(probeUrl, { method: 'GET', headers }); - if (response.ok) { - const body = await response.json(); - // `@objectstack/rest` answers the document bare; the dispatcher bridge - // wraps it as `{ data: … }`. Pick the object that actually carries - // `routes` rather than `body.data || body` — the document's own - // `routes.data` key makes the looser test ambiguous to read here. - const doc = (body && typeof body === 'object' && 'routes' in body) - ? (body as { routes?: Record }) - : ((body as { data?: { routes?: Record } } | null)?.data); - const advertised = doc?.routes?.data; - if (typeof advertised === 'string' && advertised.length > 0) { - return { - path: advertised, - source: 'discovery', - why: `GET ${probeUrl} advertised routes.data`, - }; - } - why = `GET ${probeUrl} answered ${response.status} but carried no routes.data`; - } else { - why = `GET ${probeUrl} answered ${response.status}`; + if (read.status !== undefined && read.status >= 200 && read.status < 300) { + const advertised = read.document?.routes?.data; + if (typeof advertised === 'string' && advertised.length > 0) { + return { + path: advertised, + source: 'discovery', + why: `GET ${probeUrl} advertised routes.data`, + }; } - } catch (error) { - why = `GET ${probeUrl} could not be reached (${(error as Error).message})`; + why = `GET ${probeUrl} answered ${read.status} but carried no routes.data`; + } else { + why = read.why; } const mount: ResolvedDataMount = { path: dataPath, source: 'convention', why }; diff --git a/packages/core/src/qa/runner.ts b/packages/core/src/qa/runner.ts index faf69146dca..77e3f4c5a76 100644 --- a/packages/core/src/qa/runner.ts +++ b/packages/core/src/qa/runner.ts @@ -1,7 +1,51 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. import * as QA from '@objectstack/spec/qa'; -import { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; + +/** + * A scenario's verdict. `skipped` is its own outcome, not a kind of pass or + * fail: the scenario never ran because a `requires` entry was unmet, so it + * proved nothing — a report counts it separately and never as passed. + */ +export type TestResultStatus = 'passed' | 'failed' | 'skipped'; + +/** One `TestScenario.requires` entry that did not hold. */ +export interface UnmetRequirement { + /** The `requires` key the entry sits under. */ + key: 'params' | 'services'; + /** The entry: the environment variable or the service key. */ + name: string; + /** Why it is unmet, in one clause — what was observed instead. */ + detail: string; +} + +/** Why a scenario was skipped instead of run. */ +export interface SkipReport { + /** + * The sentence a report prints: every unmet entry by key and name and, when + * a service is unmet, the services the target does declare available. + */ + reason: string; + /** Each unmet entry, in `requires` order (`params` first, then `services`). */ + unmet: UnmetRequirement[]; + /** + * The services the target declares `enabled` with status `available`, + * sorted — present whenever a `services` requirement was judged. Empty when + * the target's services could not be read. + */ + availableServices?: string[]; +} + +/** Options for {@link TestRunner}. */ +export interface TestRunnerOptions { + /** + * The environment `requires.params` is judged against. Defaults to this + * process's own environment — the process running the suite (`os test`), + * never the target server's, which no suite can observe. + */ + env?: Readonly>; +} /** * One scenario's outcome, carrying the names a report leads with. @@ -24,7 +68,19 @@ export interface TestResult { scenarioName: string; /** `TestScenario.description`, when the author wrote one. */ description?: string; + /** + * The verdict. `skipped` means a `requires` entry was unmet and no step — + * setup included — ran; {@link skipped} says which entry and why. + */ + status: TestResultStatus; + /** + * `status === 'passed'`. A skipped scenario is `passed: false` — it proved + * nothing — but it is not a failure either: read `status` to tell the two + * apart. + */ passed: boolean; + /** Present exactly when `status` is `skipped`. */ + skipped?: SkipReport; steps: StepResult[]; error?: unknown; duration: number; @@ -72,8 +128,36 @@ function containsInapplicableHint(actual: unknown): string { ); } +/** + * One discovery `services` entry is AVAILABLE exactly when the target says + * `enabled: true` and `status: 'available'` — both halves, because they answer + * different questions (ADR-0076 D12): `enabled` is "the slot is filled", + * `status` is whether what fills it is the real thing rather than a stub or a + * degraded fallback. + */ +function isAvailable(entry: unknown): boolean { + if (entry === null || typeof entry !== 'object') return false; + const info = entry as { enabled?: unknown; status?: unknown }; + return info.enabled === true && info.status === 'available'; +} + +/** Why a required service is unmet, from what the target said about it. */ +function describeUnavailable(entry: unknown, target: TargetServices): string { + if (target.services === undefined) { + return `not judgeable: the target's services could not be read (${target.source})`; + } + if (entry === undefined) return 'not declared by the target'; + if (entry === null || typeof entry !== 'object') return 'declared by the target in an unreadable shape'; + const info = entry as { enabled?: unknown; status?: unknown }; + return `not available on the target (enabled: ${String(info.enabled)}, status: ${String(info.status)})`; +} + export class TestRunner { - constructor(private adapter: TestExecutionAdapter) {} + private readonly env: Readonly>; + + constructor(private adapter: TestExecutionAdapter, options: TestRunnerOptions = {}) { + this.env = options.env ?? (typeof process === 'undefined' ? {} : process.env); + } async runSuite(suite: QA.TestSuite): Promise { const results: TestResult[] = []; @@ -85,6 +169,23 @@ export class TestRunner { async runScenario(scenario: QA.TestScenario): Promise { const startTime = Date.now(); + + // Preconditions first: a scenario whose `requires` does not hold runs no + // step at all — setup included, since setup can write records. + const skipped = await this.judgeRequirements(scenario); + if (skipped) { + return { + scenarioId: scenario.id, + scenarioName: scenario.name, + description: scenario.description, + status: 'skipped', + passed: false, + skipped, + steps: [], + duration: Date.now() - startTime, + }; + } + const context: Record = {}; // Variable context // Initialize context from initial payload if needed? Currently schema doesn't have initial context prop on Scenario @@ -100,6 +201,7 @@ export class TestRunner { scenarioId: scenario.id, scenarioName: scenario.name, description: scenario.description, + status: 'failed', passed: false, steps: [], error: `Setup failed: ${e instanceof Error ? e.message : String(e)}`, @@ -156,6 +258,7 @@ export class TestRunner { scenarioId: scenario.id, scenarioName: scenario.name, description: scenario.description, + status: scenarioPassed ? 'passed' : 'failed', passed: scenarioPassed, steps: stepResults, error: scenarioError, @@ -163,6 +266,63 @@ export class TestRunner { }; } + /** + * Judge `TestScenario.requires` (ADR-0049: declared is enforced; ruled B). + * Returns the skip report when an entry is unmet, `undefined` when every + * entry holds or the scenario declares none. + * + * - `params`: each variable must be set to a non-empty value in {@link env} + * — the process running the suite. Empty counts as unset: a CI secret that + * is not configured arrives as an empty string, and a scenario that needs + * it cannot run on it. + * - `services`: each key must be declared by the target `enabled` with + * `status === 'available'` — the discovery document's own verdict + * (ADR-0076 D12), read through the adapter's one probe of the run. A + * target whose services cannot be read declares none, so every service + * requirement is unmet and the reason says why. + * + * Every unmet entry is reported, not just the first, so one run tells the + * author everything the target is missing. + */ + private async judgeRequirements(scenario: QA.TestScenario): Promise { + const requires = scenario.requires; + if (!requires) return undefined; + const unmet: UnmetRequirement[] = []; + + for (const name of requires.params ?? []) { + const value = this.env[name]; + if (value === undefined) { + unmet.push({ key: 'params', name, detail: 'not set in the environment of the process running the suite' }); + } else if (value === '') { + unmet.push({ key: 'params', name, detail: 'set to an empty value in the environment of the process running the suite' }); + } + } + + const services = requires.services ?? []; + let availableServices: string[] | undefined; + if (services.length > 0) { + const target: TargetServices = this.adapter.readTargetServices + ? await this.adapter.readTargetServices() + : { source: 'the execution adapter does not report the target\'s services' }; + const declared = target.services ?? {}; + availableServices = Object.keys(declared).filter((key) => isAvailable(declared[key])).sort(); + for (const name of services) { + if (isAvailable(declared[name])) continue; + unmet.push({ key: 'services', name, detail: describeUnavailable(declared[name], target) }); + } + } + + if (unmet.length === 0) return undefined; + const clauses = unmet.map((u) => `requires.${u.key} '${u.name}' is ${u.detail}`); + let reason = `${clauses.join('; ')}.`; + if (availableServices !== undefined && unmet.some((u) => u.key === 'services')) { + reason += availableServices.length > 0 + ? ` The target declares available: ${availableServices.join(', ')}.` + : ' The target declares no service available.'; + } + return { reason, unmet, ...(availableServices !== undefined ? { availableServices } : {}) }; + } + private async runStep(step: QA.TestStep, context: Record): Promise { // 1. Resolve Variables with Context (Simple interpolation or just pass context?) // For now, assume adpater handles context resolution or we do basic replacement diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index 35ed7ffb701..ec677507cde 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -927,7 +927,7 @@ marker where the Notes cell goes, never a guess at what belongs there. | mapping | seeded 2026-08-01 (#4488) at 8/11 live; **0 dead since #4509** retired the three that were not. The import half (#2611) is loudly enforced — unsupported transforms/formats are 400s, `mode`/`upsertKey` default the request, the wizard picker renders `label`. RETIRED 17.0.0: `extractQuery` (authorWarn — "for export only" promised an export path no exporter implements) + `errorPolicy`/`batchSize`, which were dead AND **unwarnable** (schema defaults materialize at parse, so presence ≠ authored — `_authorWarnSkipped`, the non-boolean instance of the default(true) rule). That unwarnability is why they went out in the 17.0.0 window rather than after a deprecation cycle: removal was the only channel that could ever reach the author. Rows DELETED, not tombstoned — MappingSchema is strict, so the keys left the walked shape | | seed | seeded 2026-08-01 (#4488). Fully live via SeedLoaderService on both doors (boot/per-org replay + runtime-draft publish). `records` is the z.record walk boundary: the keys an author writes are the target object's fields, governed by that object's own definitions — recorded in the entry, not silently skipped | | translation | seeded 2026-08-01 (#4488) — after fixing the walker: the registered schema is a z.preprocess pipe (#3778 retired-dialect guard) whose transform side the unwrap always took, so the type was literally unwalkable. 11 of 12 groups live across spec resolvers, REST localization, objectui client resolvers and plugin-audit (whose composed-key `t()` calls make `messages` easy to mis-verify as dead) — `flows` was the one that was not, and was `planned` at seeding. **#14253** added the twelfth, `datasets`, seeded LIVE and DRILLED (label / description / dimensions / measures) with its reader in the same change: `translateDataset` in the dispatch table, which is what `TRANSLATABLE_METADATA_TYPES` is derived from, so the REST boundary followed with nothing else to remember. The same change gave `objects.._views..bulkActions` and `objects.._validations..message` their first keys — both beneath the walk boundary, so neither adds a row here. Dead 1 = `validationMessages` (authorWarn) at seeding: nothing resolved it, and #3778's own legacy-key migration table steered `errors:` authors into it — a shipped false signpost, the capabilities.readOnly shape. **#4667**: `validationMessages` REMOVED (row deleted) — removed from the shared translationDataShape(), so it retired at BOTH doors at once, closing the item-only asymmetry #3778's original guard had. #3778's own `errors` guidance was rewritten in the same change: it had been steering authors INTO this dead group. ⚠️ **What that left behind is this table's own worked example of the defect it warns about** (#7377): the same commit that deleted the `validationMessages` row wrote a count column of `dead 2` beside a sentence that named exactly one dead key — and that one was the key it had just removed. The real two were `name` and `label`, which the cell never mentioned. Measured at that commit, not inferred: the ledger's dead set there is `{name, label}` and `validationMessages` is absent from `props`. The number was right and the prose was false, in the same cell, on the day it was written — which is why the counts are now generated and this cell holds prose only. **#7131** (PR #7425) resolves it: `name` and `label` re-grade `dead` → `live` under the designer-previews-count-as-consumers ruling (objectui `TranslationPreview.tsx:67` reads `label` first and falls back to `name`, both rendering at `:100`), so the dead set is empty and there is no dead-set sentence left to keep true. As on `job`, the ADR-0033 docs-shaped exemption is untouched — nothing about enforce-or-remove moved. **#19620** (ruling batch #210 item 2 letter B): `settings` row DELETED — the strict-delete route, because `TranslationItemSchema` no longer declares the key and refuses it by name (the item door now takes the per-app face, as the file door has since #15178). ⚠️ The deleted row read `live`, and that verdict was TRUE and stays true of the platform: its evidence read the SERVED tree, which the platform bundle feeds, so the deletion retires the key from the application-authored item and nothing else — the capability lives on `PlatformTranslationDataSchema`, outside this ledger. **#20296**: `flows` is now half-read. `flows.screens` re-graded `planned` → `live`: objectui's FlowRunner reads each screen's `title` and each field's `label` / `placeholder` at the `.objectui-sha` pin f8a9d0fb. `flows.label` stays `planned` because nothing reads it yet (#20318), and so does the container's `authorWarn`, whose `authorHint` now names the read half and the unread one. | -| qa | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). The seeding recorded Dead 5 — `name`, `scenarios.name`, `scenarios.description`, `scenarios.tags` and `scenarios.requires` — and #20289 (family `qa-runner`, verdict ENFORCE) made FOUR of them live: the suite `name` heads the suite in `os test`'s report and is stamped as `suiteName` on every TestResult the suite produces; `scenarios.name` is printed beside the id and carried as `scenarioName`; `scenarios.description` is carried and printed under a failed scenario; and `scenarios.tags` is read by `os test --tags` (any-of, exact; deselected scenarios are counted and never passed). Dead 1 = `scenarios.requires`, which declares param/plugin preconditions nothing checks, so a suite naming a missing plugin runs anyway, and the unmet requirement surfaces only as whatever failure it causes, if any. It was put to the maintainer rather than guessed: over HTTP no server surface lists the loaded plugins, and whose environment `params` names is undefined — its describe() now says NOT CHECKED in the meantime. It does not carry `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops | +| qa | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). The seeding recorded Dead 5 — `name`, `scenarios.name`, `scenarios.description`, `scenarios.tags` and `scenarios.requires` — and #20289 (family `qa-runner`, verdict ENFORCE) made all FIVE live: the suite `name` heads the suite in `os test`'s report and is stamped as `suiteName` on every TestResult the suite produces; `scenarios.name` is printed beside the id and carried as `scenarioName`; `scenarios.description` is carried and printed under a failed scenario; `scenarios.tags` is read by `os test --tags` (any-of, exact; deselected scenarios are counted and never passed); and `scenarios.requires` is judged by the TestRunner before a scenario's first step — `params` against the environment of the process running `os test`, `services` against the target's discovery `services` (enabled and `available`) — so an unmet entry SKIPS the scenario with its reason, counted apart and never passed. `requires` was put to the maintainer first rather than guessed (ruled B): over HTTP no server surface lists the loaded plugins, so `requires.plugins` is a `retiredKey()` tombstone inside the block whose prescription names `requires.services`. It does not carry `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops | | validation | seeded 2026-08-01 (#4488). The ADR-0020 carrier: the evaluator honors active/events/priority/severity/type/condition/message (the zod header's "only reads type/condition/…" prose is STALE — trust the ledger). Dead 3 = label/description/tags, declared governance metadata, kept unmarked. Union walk boundary recorded: only base + `script` keys walked; per-variant keys are governed by the evaluator's tests, not ledger rows. **No longer a registered metadata kind** — #4509 retired it under ADR-0088 (a standalone rule had no object-binding key and every variant is `.strict()`, so it bound to nothing and gated no write; a state machine authored that way saved cleanly and did nothing). The rule VOCABULARY is untouched and fully live via `object.validations[]`, so the ledger keeps governing it through the gate's spec-only override, alongside `webhook` and `query`. The contrast with the two bridges in the same batch is the point: enforce-or-remove picked ENFORCE where the feature existed and only the wiring was missing, and REMOVE where the shape itself could not carry the feature | | api | seeded 2026-08-04 (#5271, part of #5206; PR #5312) — **not a metadata type until that same change made it one**, which is the row's point: governance and registration landed together, the treatment `datasource` did not get (#4487) and paid for with six inert keys found by hand. What #5206 measured before the fix: `api` was in neither `DEFAULT_METADATA_TYPE_REGISTRY` nor `BUILTIN_METADATA_TYPE_SCHEMAS`, so `saveMetaItem`'s `resolveOverlaySchema('api', …)` → `getMetadataTypeSchema('api')` returned `undefined` and took its own documented branch — an unregistered type is stored **unvalidated** — while `getMetaTypes()` could not enumerate the type at all, so Studio rendered neither list nor form. That issue names the shape precisely and it is the inverse of this ledger's usual one: **enforced but undeclared** (the matcher was already indexing these entries, #5089), where `dead` is declared-but-unenforced. The seeding pass classified 27 keys — live 25 / planned 2 / dead 0 — each cited `file:line` at the consumer layer that reads it: the MATCHER (`packages/metadata/src/endpoint-matcher.ts`) indexes `name`/`path`/`method`; the EXECUTOR (`packages/runtime/src/endpoint-executor.ts`) dispatches on `type` and reads `target`/`objectParams`; the POLICY chain (`packages/runtime/src/endpoint-policy.ts` + `security/inbound-rate-limit.ts`) enforces `authRequired`/`rateLimit`/`cacheTtl`; the MAPPING layer (`packages/runtime/src/api-mapping.ts`) applies `inputMapping`/`outputMapping`; and OpenAPI enrichment (`packages/rest/src/openapi-endpoints.ts`) emits `summary`/`description`. Timing was the reason it was cheap: #5040's E-series had built every one of those consumers and all of it was on main, so each key had a real evidence path rather than a promise. **Planned 2 = `inputMapping.transform` + `outputMapping.transform`, and `planned` rather than `dead` is load-bearing**: `dead` here means parsed with no consumer — a silent no-op — and these are the opposite, parsed and then LOUDLY REFUSED at publish (`endpoint-publish-gate.ts` mappingGate) and again at runtime, because no transformation-function registry exists anywhere in the platform. An author who writes one is told so and told what to do instead, so there is nothing for enforce-or-remove to chase; they stay in the vocabulary because admitting them needs a function registry **and** a sandbox ruling (#5040 §3.4), which is a design decision, not a key to quietly delete. Zero dead | | capability | seeded 2026-08-08 (#5961; PR #6540) — `CapabilityDeclarationSchema`, the DECLARATION side of ADR-0066 D1's three-way separation: packages DEFINE a capability, permission sets GRANT it via `systemPermissions`, resources REQUIRE it via `requiredPermissions`. **The gate's 12 and the seeding PR's 5 are the same measurement at two granularities** — PR #6540 call-graph-closed **5 authorable properties**, every one to a real reader in `packages/plugins/plugin-security/src/bootstrap-declared-capabilities.ts` (the one consumer that turns a declaration into a `sys_capability` row), all `live`, with no `PENDING_GOVERNANCE` debt recorded; the other 7 are the ADR-0010 protection-envelope keys the gate auto-classifies `live` and which carry `null` verdicts in the file, exactly as on `permission`/`position`. The same worked example as `api` above and PR #6540 says so in those words — **enforced but undeclared**, the mirror of the hole #5271 closed. What #5961 measured: absent from `DEFAULT_METADATA_TYPE_REGISTRY`, `BUILTIN_METADATA_TYPE_SCHEMAS` and `HAND_CRAFTED_SCHEMAS`, so `isRuntimeCreateAllowed()` took its no-static-entry fallback (permanently true) and `saveMetaItem` its no-schema branch — `PUT /api/v1/meta/capability/:name` accepted **arbitrary JSON** onto an authorization surface whose names `systemPermissions`/`requiredPermissions` resolve by string, while `/meta/types` synthesised a false `allowRuntimeCreate: true` descriptor Studio drew a raw-JSON create form from. #5870 did not open that path (the write gate reads the registry, not the item store); it only made the type visible in `getMetaTypes()`, and both the issue and this row say so to stop the next reader filing it as a regression. Landed as ruling A on ADR-0066 D1's own authority: `allowRuntimeCreate: false` **and** `allowOrgOverride: false`, the second self-judged inside the ruling's rationale and flagged for veto — a tenant overlay of a package declaration would lift `scope` from `org` to `platform`, which is the one field on this type that is an escalation rather than display. Its reverse verification is worth copying: deleting the registry entry gave 7 red / 3 green and measured something **sharper than predicted** — a garbage payload turned 422 rather than resolving, i.e. the schema binding is a real second line of defence behind the registry row, not a restatement of it; deleting the schema binding alone gave exactly 3 red. `packageId` is the one key that reads oddly: deliberately a FALLBACK, not the primary, since #5870 added `capabilities` to the ObjectQL stamped-collection list so `_packageId` now reaches a declaration and wins — it stays `live` because the fallback branch still decides materialization for any declaration arriving unstamped. Zero dead | diff --git a/packages/spec/liveness/qa.json b/packages/spec/liveness/qa.json index cc9c16e037f..f79c93a3cad 100644 --- a/packages/spec/liveness/qa.json +++ b/packages/spec/liveness/qa.json @@ -62,10 +62,12 @@ "note": "Same TestStep surface as `setup`/`steps`. Worth knowing: a teardown throw only turns the scenario red when the scenario had otherwise PASSED (:92-95), so cleanup failures behind a real failure are swallowed on purpose. 2026-08-28: RE-ANCHORED (#13003) and REPOINTED — `:86-98`'s endpoint had rotted onto the `try {` that opens the MAIN step loop. Re-closed by hand against 8cb96ec41." }, "requires": { - "status": "dead", - "verifiedAt": "2026-09-27", + "status": "live", + "evidence": "packages/core/src/qa/runner.ts#judgeRequirements (before a scenario's first step — setup included — each `requires.params` variable must be set non-empty in the runner's environment and each `requires.services` key declared `enabled` with status `available` by the target; any unmet entry returns a `skipped` TestResult with the reason and runs no step) · packages/cli/src/commands/test.ts#summaryLine (`os test` prints each skipped scenario with its reason and counts it apart — never as passed; a run in which every selected scenario was skipped is not a SUCCESS and exits 1 under `--fail-on-empty`)", + "producer": "packages/core/src/qa/http-adapter.ts#readTargetServices — hands the runner the `services` map of the discovery document the adapter already probes once per run, served by packages/metadata-protocol/src/protocol.ts#getDiscovery and packages/runtime/src/http-dispatcher.ts#getDiscoveryInfo (ADR-0076 D12). `params`: packages/cli/src/commands/test.ts#run constructs the TestRunner with `env: process.env` — the environment of the process running `os test`, the only one a suite can observe.", + "verifiedAt": "2026-09-28", "evidenceScope": "in-repo", - "note": "Declared environment preconditions — `requires.params` (environment variables or parameters) and `requires.plugins` (plugins that must be loaded) — that nothing checks. Re-measured 2026-09-27 (#20289) on a stub target: a scenario declaring a plugin that does not exist and an environment variable that is unset reported PASSED, exactly like its no-requirements control. STILL DEAD, and deliberately: #20289 enforced the other four rows of this family and put this one to the maintainer, because neither sub-key has an honest judge yet. `plugins`: `os test` reaches the target over HTTP, and no server surface lists the loaded plugins — the discovery document advertises `services` and `capabilities`, not plugins — and the plugin spelling itself (package name or `plugin.name`) is undefined. `params`: whose environment is undefined — the runner's is observable but no step can consume a runner variable (interpolation reads only captured context), and the server's is not observable at all. Guessing either would make a skip that means nothing. What changed now: the describe() of `requires` and both sub-keys says NOT CHECKED instead of reading as a guard, so an author (or an AI) is no longer told it is one. The enforce route still needs a SKIPPED result with its reason, counted separately and never passed; the remove route drops the block." + "note": "Was `dead` from the 2026-08-10 seeding until #20289's `requires` round (the family's fifth key; ruled B): declared preconditions — `params` (environment variables or parameters) and `plugins` (plugins that must be loaded) — that nothing checked, re-measured on a stub target as an unmet-requirements scenario reporting PASSED exactly like its no-requirements control. ENFORCED now, each sub-key judged against something `os test` can actually observe: `params` against the environment of the process running the suite (non-empty; an empty CI secret counts as unset), and `services` — NEW, closed over `CoreServiceName` so a misspelling is refused at parse — against the target's discovery `services` map, met only when the target declares the service `enabled` AND `status: 'available'`. Every unmet entry is reported, and a service reason lists what the target does declare available; a target whose discovery cannot be read declares nothing, so each service requirement is unmet and says why. `plugins` is REMOVED — a `retiredKey()` tombstone inside this block, whose prescription names `requires.services` and carries the plugin → service mapping derived from `CORE_SERVICE_PROVIDER`: over HTTP no served surface lists loaded plugins and the plugin spelling was never defined, so it could not join the judgement honestly (D3 `qa-scenario-requires-plugins-retired`; `RETIRED_KEYS_BY_MAJOR[18]` `qa/TestScenario:requires.plugins`; no D2 conversion, since a suite is a loose file and never a stored row). The walk classifies this row as one block, so the tombstone needs no row of its own. SKIPPED is its own outcome in core's `TestResult` (`status: 'skipped'`, `passed: false`, the `skipped` report): never counted as passed, and not a failure either — a failure-free run with skips exits 0." } } } diff --git a/packages/spec/scripts/liveness/check-liveness.mts b/packages/spec/scripts/liveness/check-liveness.mts index 6bd1d578df8..bb77151c3d8 100644 --- a/packages/spec/scripts/liveness/check-liveness.mts +++ b/packages/spec/scripts/liveness/check-liveness.mts @@ -365,11 +365,13 @@ const PENDING_GOVERNANCE: Record = {}; // was withdrawn on 2026-08-08 in favour of enforce. Governing it here is the // half of that ruling that keeps the surface honest going forward: the runner // reads a real, measured subset of the declared keys, and a key nothing reads is -// recorded as such instead of being invisible. `suite.name` and `scenario.tags` -// were two of those and have since gained readers in `os test` (the suite -// heading; the `--tags` selection); `scenario.requires` is the one still unread — -// declared, NOT CHECKED, and its row stays dead. Like `query`, there is no -// registry to fold it back onto — the override IS its governance. +// recorded as such instead of being invisible. `suite.name`, `scenario.tags` +// and `scenario.requires` were three of those and have since gained readers +// (the suite heading and the `--tags` selection in `os test`; the TestRunner's +// precondition judgement, which skips a scenario whose `params` or `services` +// do not hold — `requires.plugins`, which nothing could judge, is a tombstone). +// Like `query`, there is no registry to fold it back onto — the override IS its +// governance. // `manifest` is the THIRD category the override has had to reach, and the one // that showed the escape hatch was load-bearing rather than a webhook special // case. `ManifestSchema` (src/kernel/manifest.zod.ts) is what an author writes diff --git a/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts b/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts new file mode 100644 index 00000000000..20a1fd99dd2 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-keys/18.qa__TestScenario__requires.plugins.ts @@ -0,0 +1,15 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — +// `requires.plugins` could never be judged: `os test` reaches its target over +// HTTP and no served surface lists the loaded plugins. It retires into +// `requires.services`, judged against the services the target's discovery +// document declares enabled and available. Tombstoned with `retiredKey()` inside +// the live `requires` block, whose `params` and `services` keep parsing. No D2 +// conversion: a QA suite is a loose JSON file `os test` loads, never a stack +// collection member or a stored row. D3 semantic entry +// `qa-scenario-requires-plugins-retired`. Registered under 18 for the +// launch-window reason its neighbours state. +// +// Nested key of an inline block — no `authorable-surface/` line of its own. +export const entry = 'qa/TestScenario:requires.plugins'; diff --git a/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts b/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts new file mode 100644 index 00000000000..404349fda88 --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/18.qa-scenario-requires-plugins-retired.ts @@ -0,0 +1,40 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +// #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — the +// D3 entry of the family (one D3 entry per retirement family). Registered key: +// `qa/TestScenario:requires.plugins`. No D2 conversion: a QA suite is a loose +// JSON file `os test` loads, never a stack collection member or a stored row, so +// there is no source for a conversion to rewrite — and the move is a judgement +// (which service did the plugin stand for?), not a mechanical edit. This entry +// is where the prescription reaches `os migrate meta`, the upgrade guide and +// `spec-changes.json`. +export const entry: SemanticMigration = { + id: 'qa-scenario-requires-plugins-retired', + surface: 'qa.scenarios[].requires.plugins', + replacement: + '`requires.services` — the discovery service keys the scenario needs (for example `auth`, `analytics`, ' + + '`automation`, `ai`), each judged against the target\'s discovery document: met only when the target declares ' + + 'the service `enabled` with status `available`. The plugin → service mapping follows the provider table ' + + 'discovery itself reports (`CORE_SERVICE_PROVIDER`): `@objectstack/plugin-auth` fills `auth`, ' + + '`@objectstack/service-analytics` fills `analytics`, `@objectstack/service-automation` fills `automation`, ' + + 'and so on. A plugin that fills no discovery service slot has no service to require.', + reason: + '`requires.plugins` was declared as a precondition and checked by nothing: `os test` reaches its target over ' + + 'HTTP, no served surface lists the loaded plugins, and the plugin spelling (package name or `plugin.name`) ' + + 'was never defined — so a scenario naming a missing plugin ran anyway and failed, or passed, on whatever the ' + + 'missing plugin caused. The block is now enforced (ADR-0049): core\'s TestRunner judges `requires` before the ' + + 'first step, and an unmet entry makes the scenario SKIPPED with a reason, counted separately and never as ' + + 'passed. `plugins` could not join that judgement honestly, so it retires into `services`, which the target\'s ' + + 'discovery document already answers (ADR-0076 D12: advertise only what is mounted). The consumer still owes ' + + 'the judgement because a plugin name does not always map to one service — a plugin that fills no discovery ' + + 'slot was never a checkable precondition, and only the suite\'s author knows what the scenario needed it for.', + acceptanceCriteria: + 'No QA suite (`qa/*.test.json`) carries `requires.plugins` — `os test` now refuses such a file at load time ' + + 'with the retirement prescription, naming the key, instead of running it; `tsc` refuses the key at a typed ' + + 'authoring site (`never`). Each scenario that declared plugins declares the services it needs in ' + + '`requires.services`, and `os test` against a target that serves them runs the scenario, while against one ' + + 'that does not it prints the scenario as skipped with the unmet service and the services the target declares ' + + 'available. A suite without `requires` runs exactly as before.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index f3d2c181696..62b321fd59b 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -13972,6 +13972,42 @@ const step18: MigrationStep = { + 'row, no lockfile pin. ⛔ Do not repair one by widening the check back — the grammar ' + 'is the contract now, on nine carriers at once.', }, + // #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — the + // D3 entry of the family (one D3 entry per retirement family). Registered key: + // `qa/TestScenario:requires.plugins`. No D2 conversion: a QA suite is a loose + // JSON file `os test` loads, never a stack collection member or a stored row, so + // there is no source for a conversion to rewrite — and the move is a judgement + // (which service did the plugin stand for?), not a mechanical edit. This entry + // is where the prescription reaches `os migrate meta`, the upgrade guide and + // `spec-changes.json`. + { + id: 'qa-scenario-requires-plugins-retired', + surface: 'qa.scenarios[].requires.plugins', + replacement: + '`requires.services` — the discovery service keys the scenario needs (for example `auth`, `analytics`, ' + + '`automation`, `ai`), each judged against the target\'s discovery document: met only when the target declares ' + + 'the service `enabled` with status `available`. The plugin → service mapping follows the provider table ' + + 'discovery itself reports (`CORE_SERVICE_PROVIDER`): `@objectstack/plugin-auth` fills `auth`, ' + + '`@objectstack/service-analytics` fills `analytics`, `@objectstack/service-automation` fills `automation`, ' + + 'and so on. A plugin that fills no discovery service slot has no service to require.', + reason: + '`requires.plugins` was declared as a precondition and checked by nothing: `os test` reaches its target over ' + + 'HTTP, no served surface lists the loaded plugins, and the plugin spelling (package name or `plugin.name`) ' + + 'was never defined — so a scenario naming a missing plugin ran anyway and failed, or passed, on whatever the ' + + 'missing plugin caused. The block is now enforced (ADR-0049): core\'s TestRunner judges `requires` before the ' + + 'first step, and an unmet entry makes the scenario SKIPPED with a reason, counted separately and never as ' + + 'passed. `plugins` could not join that judgement honestly, so it retires into `services`, which the target\'s ' + + 'discovery document already answers (ADR-0076 D12: advertise only what is mounted). The consumer still owes ' + + 'the judgement because a plugin name does not always map to one service — a plugin that fills no discovery ' + + 'slot was never a checkable precondition, and only the suite\'s author knows what the scenario needed it for.', + acceptanceCriteria: + 'No QA suite (`qa/*.test.json`) carries `requires.plugins` — `os test` now refuses such a file at load time ' + + 'with the retirement prescription, naming the key, instead of running it; `tsc` refuses the key at a typed ' + + 'authoring site (`never`). Each scenario that declared plugins declares the services it needs in ' + + '`requires.services`, and `os test` against a target that serves them runs the scenario, while against one ' + + 'that does not it prints the scenario as skipped with the unmet service and the services the target declares ' + + 'available. A suite without `requires` runs exactly as before.', + }, { id: 'record-chatter-position-vocabulary-converged', surface: @@ -19748,6 +19784,19 @@ export const RETIRED_KEYS_BY_MAJOR: Readonly> // carried on the `UpgradeResult`, never authored into a metadata document. The // semantic entry `kernel-package-lifecycle-durations-unit-in-key` carries it. 'kernel/UpgradePlan:estimatedDuration', + // #20289 (family `qa-runner`, verdict ENFORCE; the `requires` key ruled B) — + // `requires.plugins` could never be judged: `os test` reaches its target over + // HTTP and no served surface lists the loaded plugins. It retires into + // `requires.services`, judged against the services the target's discovery + // document declares enabled and available. Tombstoned with `retiredKey()` inside + // the live `requires` block, whose `params` and `services` keep parsing. No D2 + // conversion: a QA suite is a loose JSON file `os test` loads, never a stack + // collection member or a stored row. D3 semantic entry + // `qa-scenario-requires-plugins-retired`. Registered under 18 for the + // launch-window reason its neighbours state. + // + // Nested key of an inline block — no `authorable-surface/` line of its own. + 'qa/TestScenario:requires.plugins', // #12497 — the RESPONSE-side face of `security/ObjectPermission:allowPurge` // (see that entry for the full rationale: ADR-0049 enforce-or-remove, // maintainer ruling 2026-08-26 accepting #1883's recommendation B; the key diff --git a/packages/spec/src/qa/requires-plugins-retirement.test.ts b/packages/spec/src/qa/requires-plugins-retirement.test.ts new file mode 100644 index 00000000000..9fd617e1d71 --- /dev/null +++ b/packages/spec/src/qa/requires-plugins-retirement.test.ts @@ -0,0 +1,318 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `scenarios[].requires.plugins` RETIRED into `requires.services` (#20289, the + * `qa-runner` family's fifth key; ruling B) — ADR-0049 enforce-or-remove on a + * block that is otherwise ENFORCED: core's TestRunner judges `requires` before + * a scenario's first step and skips it with a reason when an entry is unmet. + * `plugins` could not join that judgement — `os test` reaches its target over + * HTTP and no served surface lists the loaded plugins — so it retires into + * `services`, which the discovery document already answers. + * + * Bookkeeping shapes, pinned below: + * 1. A `retiredKey()` tombstone INSIDE the live `requires` block — a + * non-strict `z.object()`, so a bare deletion would strip the key in + * silence. The refusal itself is pinned beside the schema, in + * `testing.test.ts`. + * 2. No D2 conversion: a QA suite is a loose JSON file `os test` loads, never + * a stack collection member or a stored row. `RETIRED_KEYS_BY_MAJOR[18]` + * carries `qa/TestScenario:requires.plugins`; the family's D3 entry is + * `qa-scenario-requires-plugins-retired`. + * 3. The liveness row `qa.scenarios.requires` is `live` — the block is read — + * and its note records the tombstone (`check:liveness` judges that half). + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { MIGRATIONS_BY_MAJOR, RETIRED_KEYS_BY_MAJOR } from '../migrations/registry'; + +describe('requires.plugins retirement — ADR-0087 registration', () => { + it('declares the key under major 18 and carries one D3 entry for the family, with no D2 conversion', () => { + expect(RETIRED_KEYS_BY_MAJOR[18]).toContain('qa/TestScenario:requires.plugins'); + const step = MIGRATIONS_BY_MAJOR[18]!; + const entry = step.semantic.find((e) => e.id === 'qa-scenario-requires-plugins-retired'); + expect(entry, 'the family D3 entry must be registered in the step-18 chain').toBeDefined(); + expect(entry!.surface).toBe('qa.scenarios[].requires.plugins'); + expect(entry!.replacement).toContain('`requires.services`'); + // A QA suite has no stack or stored source for a conversion to rewrite. A + // conversion id naming the key would be a strip with nothing to strip. + expect(step.conversionIds.filter((id) => /requires-plugins/.test(id))).toEqual([]); + }); +}); + +// ─── Tree-scoped absence, with a DECLARED radius ───────────────────────────── +// +// `tsc` is the primary sweeper for TYPED authorings — `retiredKey()` types the +// key `never` on `TestScenario` — and `os test` refuses a suite file carrying it +// at load. The residue is what neither judges before a run: a suite JSON nobody +// runs in CI, YAML, MD/MDX code fences, untyped `.js`, and TS literals cast +// through `as any` / `as never`. This walk covers that residue across five +// roots, each already declared for `@objectstack/spec#test` in +// `scripts/cross-package-test-inputs.mjs` and mirrored in `turbo.json` — the +// same roots and extensions the `rest_api` and view-item retirement pins walk. +// +// ⭐ `requires` and `plugins` are ordinary words (a stack's `requires` is an +// array of capabilities; a plugin manifest's is an array of `{ pluginId }`), so +// the matcher is STRUCTURAL: an offender is an object literal (or YAML mapping) +// that is the VALUE of a `requires` key and carries a `plugins` key. Nothing +// else. +// +// The bound, stated: a block assembled by SPREAD or computed keys, and a YAML +// flow mapping (`requires: { plugins: [x] }` on one YAML line), are invisible to +// this walk; `docs/**`, `.claude/**`, `.github/**` and the repo-root files are +// outside the radius. +describe('tree-scoped absence: no scenario inside the declared radius still authors `requires.plugins`', () => { + const SPEC_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..'); + const REPO_ROOT = path.resolve(SPEC_ROOT, '../..'); + const THIS_FILE = path.relative(REPO_ROOT, fileURLToPath(import.meta.url)).split(path.sep).join('/'); + + /** The walked roots — declared in `scripts/cross-package-test-inputs.mjs` under `@objectstack/spec`. */ + const WALK_ROOTS = ['packages', 'examples', 'skills', 'content', 'scripts']; + const SCANNED_EXT = new Set(['.ts', '.mts', '.cts', '.js', '.mjs', '.cjs', '.json', '.md', '.mdx', '.yaml', '.yml']); + /** Under `examples/` only the non-code extensions are scanned AND declared. */ + const EXAMPLES_EXT = new Set(['.json', '.md', '.mdx', '.yaml', '.yml']); + const SKIPPED_DIRS = new Set(['node_modules', 'dist', '.git', '.turbo', '.cache', '.objectstack', 'coverage', '.next', '.source']); + + /** + * Structural exclusions — the retirement kit, each with its reason. ⛔ NOT an + * allowlist file (`spec-property-retirement` §4): every entry's JOB is to + * spell the retired key on a `requires` block. + */ + const EXCLUDED = new Set([ + // This pin authors the key to assert the matcher finds it. + THIS_FILE, + // The schema's refusal pins author the key to assert the SCHEMA refuses it. + 'packages/spec/src/qa/testing.test.ts', + // `os test`'s load-time refusal pin authors it to assert the CLI refuses it. + 'packages/cli/test/qa-suite-schema-load.test.ts', + ]); + const EXCLUDED_PREFIXES = [ + // Release-owned prose records the removal; never edited by a code PR. + 'content/docs/releases/', + ]; + /** tsup's own bundle of `tsup.config.ts`, written and deleted mid-build. */ + const TSUP_BUNDLED_CONFIG = /\.bundled_[^./]+\.mjs$/; + + const isOffender = (parentKey: string | undefined, keys: Set): boolean => + parentKey === 'requires' && keys.has('plugins'); + + /** + * One pass over JS/TS/JSON text: a stack of bracket frames, each `{` frame + * collecting its OWN keys — an identifier or quoted string in key position + * (after `{` or `,`) followed by `:` — and remembering the key it is the + * VALUE of (a `{` whose previous significant token is the `:` of a key). + * Strings and comments are skipped; a single- or double-quoted string never + * spans a line, so a mis-lexed quote (a regex literal) costs at most that + * line. Returns the 1-based line of each closing brace whose frame offends. + * Copied from the `rest_api` retirement pin, never imported: each pin keeps + * running standalone. + */ + const lexOffenders = (text: string): number[] => { + const out: number[] = []; + const stack: { kind: string; keys: Set; parentKey?: string }[] = []; + let lastSig = ''; + let justKey: string | undefined; + let colonKey: string | undefined; + let line = 1; + let i = 0; + const n = text.length; + const sawToken = (token: string, j: number) => { + const top = stack[stack.length - 1]; + const isKey = text[j] === ':' && top?.kind === '{' && (lastSig === '{' || lastSig === ','); + if (isKey) top!.keys.add(token); + justKey = isKey ? token : undefined; + }; + while (i < n) { + const c = text[i]!; + if (c === '\n') { line += 1; i += 1; continue; } + if (c === '/' && text[i + 1] === '/') { while (i < n && text[i] !== '\n') i += 1; continue; } + if (c === '/' && text[i + 1] === '*') { + i += 2; + while (i < n && !(text[i] === '*' && text[i + 1] === '/')) { if (text[i] === '\n') line += 1; i += 1; } + i += 2; + continue; + } + if (c === '"' || c === "'" || c === '`') { + const start = i; + i += 1; + while (i < n && text[i] !== c) { + if (text[i] === '\\') i += 1; + else if (text[i] === '\n') { if (c !== '`') break; line += 1; } + i += 1; + } + const token = text.slice(start + 1, i); + i += 1; + let j = i; + while (j < n && (text[j] === ' ' || text[j] === '\t')) j += 1; + if (c === '`') justKey = undefined; + else sawToken(token, j); + lastSig = 'str'; + continue; + } + if (/[A-Za-z_$]/.test(c)) { + const start = i; + while (i < n && /[\w$]/.test(text[i]!)) i += 1; + const token = text.slice(start, i); + let j = i; + while (j < n && (text[j] === ' ' || text[j] === '\t')) j += 1; + sawToken(token, j); + lastSig = 'id'; + continue; + } + if (c === ':') { + colonKey = justKey; + justKey = undefined; + lastSig = ':'; + i += 1; + continue; + } + if (c === '{') stack.push({ kind: c, keys: new Set(), parentKey: lastSig === ':' ? colonKey : undefined }); + else if (c === '[' || c === '(') stack.push({ kind: c, keys: new Set() }); + else if (c === '}' || c === ']' || c === ')') { + const frame = stack.pop(); + if (frame?.kind === '{' && c === '}' && isOffender(frame.parentKey, frame.keys)) out.push(line); + } + if (!/\s/.test(c)) { lastSig = c; justKey = undefined; } + i += 1; + } + return out; + }; + + /** + * YAML: a block mapping opened by `requires:` with no inline value; its OWN + * keys are the lines at its first child's column, until a line at or above + * the opener's column. Returns the 1-based line of each offending opener. + */ + const yamlOffenders = (text: string): number[] => { + const rows = text.split('\n').map((raw, idx) => { + const m = /^(\s*)(-\s+)?([A-Za-z_][\w]*)\s*:/.exec(raw); + if (!m) return null; + const opensBlock = /^\s*(-\s+)?[A-Za-z_][\w]*\s*:\s*(#.*)?$/.test(raw); + return { idx, col: m[1]!.length + (m[2]?.length ?? 0), key: m[3]!, opensBlock }; + }); + const out: number[] = []; + rows.forEach((row, at) => { + if (!row || !row.opensBlock || row.key !== 'requires') return; + const keys = new Set(); + let childCol: number | undefined; + for (let k = at + 1; k < rows.length; k += 1) { + const r = rows[k]; + if (!r) continue; + if (r.col <= row.col) break; + childCol ??= r.col; + if (r.col === childCol) keys.add(r.key); + } + if (isOffender(row.key, keys)) out.push(row.idx + 1); + }); + return out; + }; + + /** MD/MDX: only fenced code is judged — prose mentions are not authorings. */ + const markdownOffenders = (text: string): number[] => { + const out: number[] = []; + const fence = /^```([\w-]*)[^\n]*\n([\s\S]*?)^```/gm; + for (let m = fence.exec(text); m; m = fence.exec(text)) { + const lang = m[1]!.toLowerCase(); + const body = m[2]!; + const offset = text.slice(0, m.index).split('\n').length; + const hits = lang === 'yaml' || lang === 'yml' ? yamlOffenders(body) : lexOffenders(body); + for (const h of hits) out.push(offset + h); + } + return out; + }; + + const mentions = (text: string): boolean => text.includes('requires') && text.includes('plugins'); + + const offendersIn = (ext: string, text: string): number[] => { + if (!mentions(text)) return []; + if (ext === '.yaml' || ext === '.yml') return yamlOffenders(text); + if (ext === '.md' || ext === '.mdx') return markdownOffenders(text); + return lexOffenders(text); + }; + + const vanished: string[] = []; + /** Tolerates ONLY a path's disappearance mid-walk; every other fault is re-raised. */ + const readIfPresent = (full: string, rel: string): string | undefined => { + try { + return fs.readFileSync(full, 'utf-8'); + } catch (err) { + if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') throw err; + vanished.push(rel); + return undefined; + } + }; + + it('the matcher finds an authoring and ignores every neighbouring shape (anti-vacuity)', () => { + // Offenders — the retired spelling, in each syntax the walk reads. + expect(offendersIn('.json', '{ "scenarios": [ { "id": "a", "requires": { "plugins": ["crm"] } } ] }')).toEqual([1]); + expect(offendersIn('.json', '{\n "requires": {\n "params": ["X"],\n "plugins": []\n }\n}')).toEqual([5]); + expect(offendersIn('.ts', "const s = { id: 'a', requires: { services: ['ai'], plugins: ['crm'] } } as never;")).toEqual([1]); + expect(offendersIn('.yaml', 'scenarios:\n - id: a\n requires:\n plugins:\n - crm\n')).toEqual([3]); + expect(offendersIn('.md', 'Prose.\n\n```json\n{ "requires": { "plugins": ["crm"] } }\n```\n')).toEqual([4]); + // Neighbours that must NOT match. + // The replacement spelling. + expect(offendersIn('.json', '{ "requires": { "params": ["X"], "services": ["ai"] } }')).toEqual([]); + // A stack's `requires` is an array of capabilities; a manifest's is an array of `{ pluginId }`. + expect(offendersIn('.ts', "defineStack({ requires: ['ai'], plugins: [new AuthPlugin()] })")).toEqual([]); + expect(offendersIn('.json', '{ "requires": [ { "pluginId": "a", "plugins": ["b"] } ] }')).toEqual([]); + // `plugins` NESTED one level down belongs to another block. + expect(offendersIn('.ts', "({ requires: { params: ['X'], extra: { plugins: [] } } })")).toEqual([]); + // The schema declaration itself: the `{` is `z.object(`'s argument, not the key's value. + expect(offendersIn('.ts', "requires: z.object({ plugins: retiredKey('gone') })")).toEqual([]); + // Prose, comments and quoted strings are not authorings. + expect(offendersIn('.md', 'A suite once wrote `requires: { plugins: [crm] }` here.')).toEqual([]); + expect(offendersIn('.ts', '// requires: { plugins: [] }')).toEqual([]); + expect(offendersIn('.ts', 'const s = "{ requires: { plugins: [] } }";')).toEqual([]); + // A YAML `requires` mapping whose `plugins` belongs to a nested block. + expect(offendersIn('.yaml', 'requires:\n params:\n - X\n other:\n plugins: []\n')).toEqual([]); + }); + + it('a path that VANISHES mid-walk is not a finding, and every other read fault still is', () => { + const before = vanished.length; + const gone = path.join(REPO_ROOT, 'packages/spec/does-not-exist.bundled_probe.mjs'); + expect(fs.existsSync(gone)).toBe(false); + expect(readIfPresent(gone, 'probe/gone')).toBeUndefined(); + expect(vanished.slice(before)).toEqual(['probe/gone']); + expect(readIfPresent(fileURLToPath(import.meta.url), THIS_FILE)).toContain('tree-scoped absence'); + expect(() => readIfPresent(path.join(REPO_ROOT, 'packages/spec'), 'probe/dir')).toThrow(); + expect(vanished.length).toBe(before + 1); + }); + + it('no `requires` block authoring `plugins` survives inside the declared radius', () => { + const offenders: string[] = []; + let visited = 0; + let suites = 0; + const walk = (dir: string) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + const rel = path.relative(REPO_ROOT, full).split(path.sep).join('/'); + if (entry.isDirectory()) { + if (SKIPPED_DIRS.has(entry.name) || entry.name.startsWith('.')) continue; + walk(full); + continue; + } + if (!entry.isFile()) continue; + const ext = path.extname(entry.name); + if (!(rel.startsWith('examples/') ? EXAMPLES_EXT : SCANNED_EXT).has(ext)) continue; + if (entry.name === 'CHANGELOG.md') continue; // release prose records the removal + if (EXCLUDED.has(rel) || EXCLUDED_PREFIXES.some((p) => rel.startsWith(p))) continue; + if (TSUP_BUNDLED_CONFIG.test(entry.name)) continue; + visited += 1; + const text = readIfPresent(full, rel); + if (text === undefined) continue; + if (entry.name.endsWith('.test.json') && text.includes('"scenarios"')) suites += 1; + for (const lineNo of offendersIn(ext, text)) offenders.push(`${rel}:${lineNo}`); + } + }; + for (const root of WALK_ROOTS) walk(path.join(REPO_ROOT, root)); + // Anti-vacuity: the walk covered the tree, and the authored QA suites — + // the files that most plausibly hold an authoring — were really judged. + expect(visited).toBeGreaterThan(1000); + expect(suites).toBeGreaterThan(0); + expect(offenders, 'a `requires` block authoring `plugins` means the retirement is being undone').toEqual([]); + }); +}); diff --git a/packages/spec/src/qa/testing.test.ts b/packages/spec/src/qa/testing.test.ts index 1087d484e53..4bdbd06ba12 100644 --- a/packages/spec/src/qa/testing.test.ts +++ b/packages/spec/src/qa/testing.test.ts @@ -8,7 +8,9 @@ import { TestStepSchema, TestScenarioSchema, TestSuiteSchema, + type TestScenario, } from './testing.zod'; +import { CORE_SERVICE_PROVIDER } from '../system/core-services.zod'; describe('TestContextSchema', () => { it('should accept a valid context record', () => { @@ -158,7 +160,7 @@ describe('TestScenarioSchema', () => { teardown: [{ name: 'cleanup', action: { type: 'delete_record', target: 'account' } }], requires: { params: ['API_KEY'], - plugins: ['crm'], + services: ['data', 'auth'], }, }; expect(() => TestScenarioSchema.parse(scenario)).not.toThrow(); @@ -200,3 +202,93 @@ describe('TestSuiteSchema', () => { expect(() => TestSuiteSchema.parse({ name: 'Suite' })).toThrow(); }); }); + +/** + * `requires` is ENFORCED (ADR-0049; the `qa-runner` family's `requires` key, + * ruled B): core's TestRunner judges it before a scenario's first step, and an + * unmet entry makes the scenario SKIPPED with its reason. The runner half is + * pinned in `packages/core/src/qa/runner.test.ts`; this half pins the contract + * an author writes against — `services` is the target's own vocabulary, closed + * at parse, and `plugins` is a tombstone that carries its prescription. + * + * On the assertion set: a schema refusal raises a `ZodError` whose issues carry + * `code` and `path` but no ADR-0112 `status` — that envelope belongs to the API + * error surface. So these pins assert refusal, the issue `code`, the `path` + * naming the key, and the prescription. + */ +describe('TestScenarioSchema.requires', () => { + const scenario = (requires: unknown) => ({ + id: 'sc-req', + name: 'Requires', + steps: [{ name: 'step1', action: { type: 'api_call', target: '/api/v1/health' } }], + requires, + }); + + it('accepts `params` and `services` — the two judged keys — and keeps them', () => { + const r = TestScenarioSchema.safeParse(scenario({ params: ['OS_QA_TOKEN'], services: ['ai', 'analytics'] })); + expect(r.success).toBe(true); + if (!r.success) return; + expect(r.data.requires).toEqual({ params: ['OS_QA_TOKEN'], services: ['ai', 'analytics'] }); + }); + + it('closes `services` over the discovery service keys: a misspelling is refused at parse, located', () => { + const r = TestScenarioSchema.safeParse(scenario({ services: ['analytic'] })); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'requires.services.0'); + expect(issue, 'the refusal must locate `requires.services.0`').toBeDefined(); + expect(issue!.code).toBe('invalid_value'); + }); + + // Unanchored, because a thrown `ZodError`'s message is the JSON of its issues; + // the key-first house convention is asserted on the issue message itself. + const PLUGINS_PRESCRIPTION = + /`scenarios\[\]\.requires\.plugins` was removed in @objectstack\/spec 17\.5\.0 \(ADR-0049 enforce-or-remove\).*nothing ever checked it.*Delete the key and name the service the scenario needs in `requires\.services`.*Plugin → service: .*@objectstack\/plugin-auth → auth/s; + + for (const plugins of [['crm'], ['@objectstack/plugin-auth'], []]) { + it(`refuses \`requires.plugins: ${JSON.stringify(plugins)}\` at its path, with the prescription`, () => { + const r = TestScenarioSchema.safeParse(scenario({ plugins })); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'requires.plugins'); + expect(issue, 'the refusal must locate `requires.plugins`').toBeDefined(); + expect(issue!.code).toBe('invalid_type'); + expect(issue!.message).toMatch(PLUGINS_PRESCRIPTION); + expect(issue!.message.startsWith('`scenarios[].requires.plugins` was removed')).toBe(true); + }); + } + + it('the suite door refuses it THROUGH `scenarios`, located at the scenario', () => { + const r = TestSuiteSchema.safeParse({ name: 'Suite', scenarios: [scenario({ plugins: ['crm'] })] }); + expect(r.success).toBe(false); + if (r.success) return; + const issue = r.error.issues.find((i) => i.path.join('.') === 'scenarios.0.requires.plugins'); + expect(issue, 'the refusal must locate `scenarios.0.requires.plugins`').toBeDefined(); + expect(issue!.message).toMatch(PLUGINS_PRESCRIPTION); + }); + + it('fails tsc at the authoring site: the input type is `never`', () => { + const authored: TestScenario = { + id: 'sc-typed', + name: 'Typed', + steps: [], + // @ts-expect-error — `plugins` is a retiredKey() tombstone: its input type is `never`. + requires: { plugins: ['crm'] }, + }; + // The parse channel agrees with the type channel on the same literal. + expect(() => TestScenarioSchema.parse(authored)).toThrow(PLUGINS_PRESCRIPTION); + }); + + it('the prescription maps every provider package discovery names to its service', () => { + const r = TestScenarioSchema.safeParse(scenario({ plugins: ['x'] })); + expect(r.success).toBe(false); + if (r.success) return; + const message = r.error.issues.find((i) => i.path.join('.') === 'requires.plugins')!.message; + for (const [slot, pkg] of Object.entries(CORE_SERVICE_PROVIDER)) { + if (pkg === null || slot === 'file-storage') continue; + expect(message, `${pkg} → ${slot} must be in the mapping`).toMatch(new RegExp(`${pkg.replace(/[/.]/g, '\\$&')} → [a-z/ -]*\\b${slot}\\b`)); + } + // The deprecated alias is not taught to a converting author. + expect(message).not.toContain('file-storage'); + }); +}); diff --git a/packages/spec/src/qa/testing.zod.ts b/packages/spec/src/qa/testing.zod.ts index 850fb87d2cc..5376f0841e8 100644 --- a/packages/spec/src/qa/testing.zod.ts +++ b/packages/spec/src/qa/testing.zod.ts @@ -5,6 +5,26 @@ import { z } from 'zod'; // --- Building Blocks --- import { lazySchema } from '../shared/lazy-schema'; +import { retiredKey } from '../shared/retired-key'; +import { CORE_SERVICE_PROVIDER, CoreServiceName } from '../system/core-services.zod'; + +/** + * The plugin → service mapping the `requires.plugins` tombstone hands an author + * converting a suite, derived from {@link CORE_SERVICE_PROVIDER} — the one table + * both discovery builders read to name a slot's provider — so the prescription + * cannot name a package that does not fill the slot it claims. `file-storage` + * is left out: it is the deprecated alias of `storage`, and a converting author + * should write the canonical slot. + */ +const PLUGIN_TO_SERVICE: string = (() => { + const byPackage = new Map(); + for (const [slot, pkg] of Object.entries(CORE_SERVICE_PROVIDER)) { + if (pkg === null || slot === 'file-storage') continue; + byPackage.set(pkg, [...(byPackage.get(pkg) ?? []), slot]); + } + return [...byPackage].map(([pkg, slots]) => `${pkg} → ${slots.join(' / ')}`).join(', '); +})(); + export const TestContextSchema = lazySchema(() => z.record(z.string(), z.unknown()).describe('Initial context or variables for the test')); // Action Types @@ -68,11 +88,42 @@ export const TestScenarioSchema = lazySchema(() => z.object({ steps: z.array(TestStepSchema).describe('Main test sequence to execute'), teardown: z.array(TestStepSchema).optional().describe('Steps to cleanup after test execution'), - // Environment requirements — declared, not yet checked by any reader. + // Preconditions, judged by core's `TestRunner` before the scenario's first + // step (ADR-0049: declared is enforced). An unmet entry makes the scenario + // SKIPPED with a reason — counted on its own, never as passed. requires: z.object({ - params: z.array(z.string()).optional().describe('Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs'), - plugins: z.array(z.string()).optional().describe('Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs') - }).optional().describe('Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes') + params: z.array(z.string()).optional().describe( + 'Environment variables that must be set to a non-empty value in the process running `os test` — the runner\'s own ' + + 'environment, not the target server\'s. An unset or empty variable skips the scenario (SKIPPED, never passed) ' + + 'with a reason naming the variable' + ), + services: z.array(CoreServiceName).optional().describe( + 'Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), ' + + 'read from the discovery document `os test` already fetches once per run. An entry the target does not declare ' + + 'that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target ' + + 'does declare available' + ), + // ADR-0049 enforce-or-remove, ruled B: `plugins` could never be judged — + // `os test` reaches its target over HTTP and no served surface lists the + // loaded plugins — so it retires into `services`, the discovery contract + // that already says what a host serves. A tombstone, not a bare deletion: + // this object is non-strict, and a deleted key would be stripped in + // silence. No D2 conversion: a QA suite is a loose JSON file `os test` + // loads, never a stack collection member or a stored row; the family's D3 + // entry is `qa-scenario-requires-plugins-retired`. + plugins: retiredKey( + '`scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ' + + 'ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a ' + + 'scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in ' + + '`requires.services`, which is judged against the services the target\'s discovery document declares enabled ' + + 'and available — an unmet entry skips the scenario and says why. Plugin → service: ' + + `${PLUGIN_TO_SERVICE}. A plugin not listed fills no discovery service slot, so there is no service to require ` + + 'for it.', + ), + }).optional().describe( + 'Preconditions judged before the scenario\'s first step (setup included). Every entry must hold, or the scenario is ' + + 'SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed' + ) }).describe('A complete test scenario with setup, execution steps, and teardown')); export const TestSuiteSchema = lazySchema(() => z.object({ diff --git a/packages/spec/vitest.repo-tests.json b/packages/spec/vitest.repo-tests.json index 5386ac6a77f..1ce4c7e6316 100644 --- a/packages/spec/vitest.repo-tests.json +++ b/packages/spec/vitest.repo-tests.json @@ -29,6 +29,7 @@ "src/identity/position-delegatable-enforcer.pin.test.ts", "src/integration/connector-connection-timeout-retirement.test.ts", "src/integration/connector-resilience-keys-retirement.test.ts", + "src/qa/requires-plugins-retirement.test.ts", "src/security/rls-tags-retirement.test.ts", "src/shared/retired-key-migrate-sentence.test.ts", "src/system/compliance-families-retirement.test.ts", From 4f29d53d5bdd6d3ba8a612b41aa6dd06993fd6a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 19:25:44 +0000 Subject: [PATCH 2/4] test(core,cli,spec): pin requires skips, the all-skipped exit posture and the plugins refusal; changeset Claude-Session: https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH Co-authored-by: Claude --- .changeset/20289-os-test-names-tags.md | 2 +- .changeset/20289-requires-services-skip.md | 100 +++++++++ content/docs/deployment/cli.mdx | 2 +- .../cli/test/qa-requires-skip-run.test.ts | 197 ++++++++++++++++++ packages/cli/test/qa-requires-summary.test.ts | 31 +++ .../cli/test/qa-suite-schema-load.test.ts | 26 +++ packages/core/src/qa/http-adapter.test.ts | 55 +++++ packages/core/src/qa/runner.test.ts | 166 ++++++++++++++- 8 files changed, 576 insertions(+), 3 deletions(-) create mode 100644 .changeset/20289-requires-services-skip.md create mode 100644 packages/cli/test/qa-requires-skip-run.test.ts create mode 100644 packages/cli/test/qa-requires-summary.test.ts diff --git a/.changeset/20289-os-test-names-tags.md b/.changeset/20289-os-test-names-tags.md index 56d93b9abdf..f0ee66a94f7 100644 --- a/.changeset/20289-os-test-names-tags.md +++ b/.changeset/20289-os-test-names-tags.md @@ -14,4 +14,4 @@ A Quality Protocol suite's `name`, each scenario's `name` and `description`, and - **`--tags TAG[,TAG...]`** runs only the scenarios carrying AT LEAST ONE of the listed tags (any-of, exact, case-sensitive) — the comma-list reading of Odoo's `--test-tags` and the everyday use of Playwright's `--grep @a|@b`. With the flag, an untagged scenario is left out. Left-out scenarios are **deselected**: not run, counted on the summary (`--tags smoke selected 1 of 4 scenarios; 3 deselected (not run, not counted as passed).`), never counted as passed. A requested tag that no loaded scenario carries is named on the summary. An empty entry (`--tags smoke,`) is refused before anything runs. Without the flag nothing changes: every scenario runs. - **Exit status.** A selection that matches no scenario takes the posture an empty pattern already has: exit `0` with `No scenario matched --tags …`, and exit `1` under `--fail-on-empty`, whose description now covers both cases. The `Found N test suites.` line and the `SUCCESS: All N scenarios passed.` / `FAILED: …` summary lines keep their spelling. - **`@objectstack/core`:** `QA.TestResult` gains `scenarioName` and `description` on every result, and `suiteName` on every result `runSuite` produces (absent only from a lone `runScenario` call, which has no suite). -- **`@objectstack/spec`:** `TestScenario.requires` (`params`, `plugins`) is still checked by nothing — its describe() now says **NOT CHECKED** instead of reading as a guard, so a scenario that declares a plugin the target lacks still runs, and the unmet requirement surfaces only as whatever failure it causes, if any. The liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. +- **`@objectstack/spec`:** the liveness ledger (`liveness/qa.json`) moves the four keys above to `live`, citing their readers. `TestScenario.requires`, the family's fifth key, is checked in this same release and has its own note: an unmet `params` or `services` entry skips the scenario with its reason, and `requires.plugins` is retired into `requires.services`. diff --git a/.changeset/20289-requires-services-skip.md b/.changeset/20289-requires-services-skip.md new file mode 100644 index 00000000000..c07c678ad14 --- /dev/null +++ b/.changeset/20289-requires-services-skip.md @@ -0,0 +1,100 @@ +--- +'@objectstack/spec': minor +'@objectstack/core': minor +'@objectstack/cli': minor +--- + +feat(spec,core,cli)!: a scenario's `requires` is checked before it runs — unmet `params` or `services` SKIP it with a reason; `requires.plugins` is retired into `requires.services` (#20289) + +Clause-②: yes (narrowing) + +**BREAKING** — shipped as `minor` under the launch-window convention +(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by +this banner, the `(narrowing)` arm above and the ADR-0087 disposition below, +never by the level). + +A Quality Protocol scenario's `requires` block declared preconditions — +`params` (environment variables) and `plugins` (plugins that must be loaded) — +that nothing checked: measured on a stub target, a scenario naming a missing +plugin and an unset variable reported PASSED exactly like its no-requirements +control. ADR-0049 enforce-or-remove, verdict ENFORCE (the mainstream has +declared preconditions: JUnit `@EnabledIfEnvironmentVariable`, pytest `skipif`), +ruled B for the shape: each key is judged against something `os test` can +actually observe. + +- **`requires.params`** — each variable must be set to a non-empty value in the + environment of the process running `os test` (not the target server's, which a + suite cannot see). An empty value counts as unset: an unconfigured CI secret + arrives as an empty string. +- **`requires.services`** (new) — each entry is a discovery service key + (`CoreServiceName`: `auth`, `automation`, `analytics`, `ai`, `storage`, …; a + misspelling is refused when the suite loads) that the target must declare + `enabled` with status `available` in its discovery document (ADR-0076 D12). + It is read from the discovery request the HTTP adapter already makes once per + run; a suite that requires no service issues no extra request. +- **SKIPPED.** A scenario with an unmet entry runs no step — `setup` included — + and `os test` prints it with its reason, naming every unmet entry and, for a + service, the services the target does declare available: + `Skipped: requires.services 'ai' is not available on the target (enabled: false, status: unavailable). The target declares available: auth, data, metadata.` + It is counted on its own — `SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).` — + and never as passed. Skips alone exit `0`; a run in which EVERY selected + scenario was skipped prints `No scenario ran: …` instead of `SUCCESS`, exits + `0`, and exits `1` under `--fail-on-empty`. With nothing skipped, the summary + lines keep their spelling. +- **`@objectstack/core`:** `QA.TestResult` gains `status` (`'passed' | 'failed' | 'skipped'`) + and, on a skipped result, `skipped` (`reason`, `unmet[]`, `availableServices`); + `passed` stays and is `false` on a skip. `TestRunner` takes an optional + `{ env }` (default: this process's environment), and `TestExecutionAdapter` + gains an optional `readTargetServices()` — `HttpTestAdapter` answers it from + its one discovery probe. An adapter without it skips a service requirement + rather than running it. + +``` +FROM { "id": "ai-summary", "requires": { "plugins": ["@objectstack/service-ai"] }, "steps": [...] } + -> ran anyway; the missing plugin surfaced as whatever failure it caused, or passed +TO -> os test refuses the suite at load: + ✗ scenarios.0.requires.plugins: `scenarios[].requires.plugins` was removed in + @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: … + Delete the key and name the service the scenario needs in `requires.services`, … + Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, … + +FROM { "requires": { "services": ["ai"] }, … } (new key) +TO -> against a target whose discovery does not declare `ai` enabled and available: + ⏭️ Scenario: Summarise an account [ai-summary] (skipped) + Skipped: requires.services 'ai' is not available on the target (…). The target declares available: … +``` + +**Fix.** `requires.plugins: [""]` → `requires.services: [""]`, +using the mapping the refusal prints (derived from `CORE_SERVICE_PROVIDER`, the +provider table discovery itself reports): `@objectstack/plugin-auth` → `auth`, +`@objectstack/service-analytics` → `analytics`, `@objectstack/service-automation` +→ `automation`, `@objectstack/service-storage` → `storage`, and so on; the `ai` +service is provided by ObjectStack Cloud/Enterprise. A plugin that fills no +discovery service slot has no service to require — gate that scenario with a +`params` variable or select it with `--tags`. `tsc` refuses `plugins` at a typed +authoring site (its input type is `never`). A `TestResult` consumer that counted +`!passed` as a failure should read `status` — a skipped result is `passed: false` +and is not a failure. + +**What does not change.** A scenario without `requires` runs exactly as before, +and a suite that requires no service issues no discovery request it did not +already issue. + +### The retirement kit + +- **Schema.** `TestScenarioSchema.requires` is a non-strict `z.object()`, so + `plugins` is a `retiredKey()` tombstone carrying its prescription (a bare + deletion would have stripped it in silence); `services` is new, closed over + `CoreServiceName`. +- **ADR-0087.** `RETIRED_KEYS_BY_MAJOR[18]` gains `qa/TestScenario:requires.plugins`. + No D2 conversion: a QA suite is a loose JSON file `os test` loads, never a + stack collection member or a stored row. The family's D3 entry, + `qa-scenario-requires-plugins-retired`, carries the prescription to + `os migrate meta` and the upgrade guide. +- **Ledger and docs.** `liveness/qa.json` moves `qa.scenarios.requires` from + `dead` to `live`, citing the runner's judgement and the adapter as producer; + `state-counts.md` moves `qa` to 9 live / 0 dead. The `os test` section of the + CLI reference documents the check, the skip line and the exit posture, and the + generated `qa/testing` reference page is regenerated. + + diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index c7c6052e72c..cc0f5769b63 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1722,7 +1722,7 @@ judged against something `os test` can observe: - `requires.params` — environment variables that must be set to a non-empty value **in the process running `os test`** (not on the target server, which a - suite cannot see): `"params": ["OS_QA_BILLING_TOKEN"]`. + suite cannot see): `"params": ["BILLING_SANDBOX_KEY"]`. - `requires.services` — service keys the target must declare in its discovery document as `enabled` with status `available`, read from the same discovery request the runner already makes once per run: `"services": ["ai", "analytics"]`. diff --git a/packages/cli/test/qa-requires-skip-run.test.ts b/packages/cli/test/qa-requires-skip-run.test.ts new file mode 100644 index 00000000000..2d3c06c6b5c --- /dev/null +++ b/packages/cli/test/qa-requires-skip-run.test.ts @@ -0,0 +1,197 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * PIN — what `os test` PRINTS and how it EXITS when a scenario's `requires` + * does not hold (the `qa-runner` family's `requires` key, ruled B), measured on + * a real child process against a real HTTP target. + * + * Before: `requires` was declared and checked by nothing — a scenario naming a + * service or a variable the run lacked ran anyway and reported whatever its + * steps happened to produce, PASSED included. Now core's TestRunner judges it + * before the first step and `os test` reports an unmet scenario as SKIPPED + * with its reason, counted apart and never as passed. The runner's judgement is + * held by `packages/core/src/qa/runner.test.ts`; this file holds the wiring + * those unit pins cannot see — the discovery document reaching the judgement + * through the adapter's one probe, the runner reading THIS process's + * environment, the reason reaching stdout, and the exit status a CI step reads. + * + * The fixture is built so each run's EXIT STATUS is itself evidence: every + * gated scenario asserts a health status the stub never returns, so it can + * only exit 0 if the scenario was really skipped — not run, and not merely + * hidden from the report. + * + * A real child process, because `process.exit(1)` inside a vitest worker is + * not an exit status. Spawned through `bin/run-dev.js` + tsx so the suite does + * not depend on `packages/cli/dist`. The target is a stub `node:http` server + * in this process that answers discovery the way both real producers do; no + * ObjectStack server is booted. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { execFile } from 'node:child_process'; +import { createServer, type Server } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { CLI, TSX, childEnv } from './helpers/serve-process.js'; + +/** oclif + tsx cold start; a healthy run here is a few seconds. */ +const RUN_TIMEOUT_MS = 120_000; + +/** The variable the `params`-gated scenario needs — test-only, so `OS_TEST_*`. */ +const PARAM = 'OS_TEST_QA_REQUIRES_PROBE_TOKEN'; + +interface Run { + code: number; + stdout: string; + stderr: string; +} + +function runCli(args: string[], cwd: string, env: Record = {}): Promise { + return new Promise((resolvePromise) => { + execFile( + TSX, + [CLI, ...args], + // The variable is REMOVED unless the run sets it, so an operator's shell + // cannot decide what the unset-variable run measures. + { cwd, maxBuffer: 8 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', [PARAM]: undefined, ...env }) }, + (err, stdout, stderr) => { + resolvePromise({ + // `err.code` is the real exit status; a signalled child is never read as 0. + code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0, + stdout: String(stdout), + stderr: String(stderr), + }); + }, + ); + }); +} + +/** One `api_call` step against the stub's health route, asserting `data.status`. */ +function healthStep(expectedStatus: string) { + return { + name: `status is ${expectedStatus}`, + action: { type: 'api_call', target: '/api/v1/health', payload: { method: 'GET' } }, + assertions: [{ field: 'data.status', operator: 'equals', expectedValue: expectedStatus }], + }; +} + +/** The discovery `services` map the stub serves — both halves of availability matter. */ +const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + auth: { enabled: true, status: 'available', route: '/api/v1/auth' }, + ai: { enabled: false, status: 'unavailable', message: 'No implementation ships in the open framework' }, + metadata: { enabled: true, status: 'degraded', route: '/api/v1/meta' }, +}; + +const MIXED = { + name: 'Requires probe', + scenarios: [ + { id: 'plain', name: 'No requirements', steps: [healthStep('ok')] }, + { id: 'svc-met', name: 'Needs data', requires: { services: ['data'] }, steps: [healthStep('ok')] }, + // Would FAIL if it ran: the stub never answers `degraded`. + { id: 'svc-unmet', name: 'Needs ai', requires: { services: ['ai'] }, steps: [healthStep('degraded')] }, + { id: 'param-gated', name: 'Needs a token', requires: { params: [PARAM] }, steps: [healthStep('ok')] }, + ], +}; + +const ALL_SKIPPED = { + name: 'Nothing here can run', + scenarios: [ + { id: 'ai-1', name: 'Needs ai', requires: { services: ['ai'] }, steps: [healthStep('degraded')] }, + { id: 'meta-1', name: 'Needs metadata available', requires: { services: ['metadata'] }, steps: [healthStep('degraded')] }, + ], +}; + +let dir: string; +let server: Server; +let discoveryHits = 0; +let unset: Run; +let withParam: Run; +let allSkipped: Run; +let allSkippedStrict: Run; + +beforeAll(async () => { + server = createServer((req, res) => { + res.setHeader('content-type', 'application/json'); + if (req.url === '/api/v1/discovery') { + discoveryHits += 1; + res.end(JSON.stringify({ version: 'v1', routes: { data: '/api/v1/data' }, services: SERVICES })); + return; + } + res.end(JSON.stringify({ success: true, data: { status: 'ok' }, path: req.url })); + }); + await new Promise((resolveListen) => server.listen(0, '127.0.0.1', resolveListen)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + + dir = mkdtempSync(join(tmpdir(), 'os-qa-requires-')); + mkdirSync(join(dir, 'qa')); + mkdirSync(join(dir, 'qa-skip')); + writeFileSync(join(dir, 'qa', 'requires.test.json'), JSON.stringify(MIXED), 'utf-8'); + writeFileSync(join(dir, 'qa-skip', 'skip.test.json'), JSON.stringify(ALL_SKIPPED), 'utf-8'); + + const mixed = ['test', 'qa/*.test.json', '--url', url]; + const skip = ['test', 'qa-skip/*.test.json', '--url', url]; + [unset, withParam, allSkipped, allSkippedStrict] = await Promise.all([ + runCli(mixed, dir), + runCli(mixed, dir, { [PARAM]: 'tok' }), + runCli(skip, dir), + runCli([...skip, '--fail-on-empty'], dir), + ]); +}, RUN_TIMEOUT_MS * 2); + +afterAll(async () => { + await new Promise((resolveClose) => server.close(() => resolveClose())); + rmSync(dir, { recursive: true, force: true }); +}); + +describe('`os test` — an unmet `requires` SKIPS the scenario, with its reason', () => { + it('CONTROL — a scenario without requirements and one whose service the target serves both run', () => { + expect(unset.stdout).toContain('✅ Scenario: No requirements [plain]'); + expect(unset.stdout).toContain('✅ Scenario: Needs data [svc-met]'); + }); + + it('an unmet `services` entry skips, naming the service and the services the target declares available', () => { + expect(unset.stdout).toContain('⏭️ Scenario: Needs ai [svc-unmet] (skipped)'); + expect(unset.stdout).toMatch( + /Skipped: requires\.services 'ai' is not available on the target \(enabled: false, status: unavailable\)\. The target declares available: auth, data\./, + ); + // It never ran: its step would have failed, and nothing did. + expect(unset.stdout).not.toContain('❌'); + }); + + it('an unmet `params` entry skips naming the variable; set in the `os test` process, the scenario runs', () => { + expect(unset.stdout).toContain('⏭️ Scenario: Needs a token [param-gated] (skipped)'); + expect(unset.stdout).toContain(`Skipped: requires.params '${PARAM}' is not set`); + expect(withParam.stdout).toContain('✅ Scenario: Needs a token [param-gated]'); + }); + + it('skipped scenarios are counted apart and never as passed; skips alone exit 0', () => { + expect(unset.stdout).toContain('SUCCESS: 2 scenarios passed. 2 skipped (not run, not counted as passed).'); + expect(unset.code).toBe(0); + expect(withParam.stdout).toContain('SUCCESS: 3 scenarios passed. 1 skipped (not run, not counted as passed).'); + expect(withParam.code).toBe(0); + }); + + it('the judgement rides the ONE discovery probe per run — never one per scenario', () => { + // Four runs, each requiring a service in at least one scenario: one probe each. + expect(discoveryHits).toBe(4); + }); +}); + +describe('`os test` — a run in which every scenario was skipped is not a pass', () => { + it('prints no SUCCESS, says nothing ran, and exits 0 by default', () => { + expect(allSkipped.stdout).not.toContain('SUCCESS'); + expect(`${allSkipped.stdout}${allSkipped.stderr}`).toContain( + 'No scenario ran: all 2 selected scenarios were skipped on unmet requirements.', + ); + expect(allSkipped.code).toBe(0); + }); + + it('exits 1 under --fail-on-empty', () => { + expect(allSkippedStrict.stdout).not.toContain('SUCCESS'); + expect(allSkippedStrict.stderr).toContain('--fail-on-empty'); + expect(allSkippedStrict.code).toBe(1); + }); +}); diff --git a/packages/cli/test/qa-requires-summary.test.ts b/packages/cli/test/qa-requires-summary.test.ts new file mode 100644 index 00000000000..e5fc12ed4e2 --- /dev/null +++ b/packages/cli/test/qa-requires-summary.test.ts @@ -0,0 +1,31 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * PIN — `os test`'s closing tally once SKIPPED exists (the `qa-runner` + * family's `requires` key, ruled B). A skipped scenario never ran, so it is + * its own count: never passed, never failed. The wiring — the line reaching + * stdout and the exit status beside it — is pinned end to end in + * `qa-requires-skip-run.test.ts`; this holds the three shapes of the line, + * including the one CI transcripts already match on, unchanged when nothing + * was skipped. + */ + +import { describe, it, expect } from 'vitest'; +import { summaryLine } from '../src/commands/test'; + +describe('os test — summaryLine counts skipped scenarios apart', () => { + it('reads exactly as before when nothing was skipped — CONTROL', () => { + expect(summaryLine(3, 0, 0)).toBe('SUCCESS: All 3 scenarios passed.'); + expect(summaryLine(3, 1, 0)).toBe('FAILED: 1 scenarios failed. 3 passed.'); + }); + + it('never folds a skip into "All passed"', () => { + const line = summaryLine(3, 0, 2); + expect(line).toBe('SUCCESS: 3 scenarios passed. 2 skipped (not run, not counted as passed).'); + expect(line).not.toContain('All'); + }); + + it('carries the skip count on a failed run too', () => { + expect(summaryLine(1, 2, 3)).toBe('FAILED: 2 scenarios failed. 1 passed. 3 skipped (not run, not counted as passed).'); + }); +}); diff --git a/packages/cli/test/qa-suite-schema-load.test.ts b/packages/cli/test/qa-suite-schema-load.test.ts index 04aaec06927..402ffe8b93a 100644 --- a/packages/cli/test/qa-suite-schema-load.test.ts +++ b/packages/cli/test/qa-suite-schema-load.test.ts @@ -110,4 +110,30 @@ describe('os test — suite load is schema-checked (#6247)', () => { const file = suiteFile('not-json.test.json', '{ "name": '); expect(() => loadTestSuite(file)).toThrow(/not-json\.test\.json/); }); + + // The `qa-runner` family's `requires` key, ruled B: `requires.plugins` is a + // tombstone. A suite still carrying it is refused before anything runs, and + // the refusal is the retirement prescription — located at the scenario and + // naming the replacement — rather than the key being stripped in silence. + it('refuses the retired `requires.plugins`, located, with the prescription naming `requires.services`', () => { + const retired = structuredClone(VALID_SUITE) as Record; + retired.scenarios[0].requires = { plugins: ['@objectstack/plugin-auth'] }; + const file = suiteFile('retired-plugins.test.json', JSON.stringify(retired)); + let message = ''; + try { + loadTestSuite(file); + } catch (e) { + message = e instanceof Error ? e.message : String(e); + } + expect(message).toContain('retired-plugins.test.json'); + expect(message).toContain('scenarios.0.requires.plugins'); + expect(message).toMatch(/`scenarios\[\]\.requires\.plugins` was removed.*`requires\.services`/s); + }); + + it('loads `requires.params` and `requires.services` unchanged — CONTROL', () => { + const gated = structuredClone(VALID_SUITE) as Record; + gated.scenarios[0].requires = { params: ['OS_TEST_QA_TOKEN'], services: ['auth'] }; + const suite = loadTestSuite(suiteFile('gated.test.json', JSON.stringify(gated))); + expect(suite.scenarios[0].requires).toEqual({ params: ['OS_TEST_QA_TOKEN'], services: ['auth'] }); + }); }); diff --git a/packages/core/src/qa/http-adapter.test.ts b/packages/core/src/qa/http-adapter.test.ts index 81053d92115..32340f337e4 100644 --- a/packages/core/src/qa/http-adapter.test.ts +++ b/packages/core/src/qa/http-adapter.test.ts @@ -397,3 +397,58 @@ describe('[#7848] auth and impersonation headers still ride along', () => { expect(call.headers['X-Run-As']).toBe('alice'); }); }); + +describe('[#20289] readTargetServices hands the runner the SAME probe\'s `services` map', () => { + const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + ai: { enabled: false, status: 'unavailable' }, + }; + + it('reads `services` off the discovery document the record actions already probe — one request for both', async () => { + discoveryReply = () => jsonResponse({ version: 'v1', routes: { data: ADVERTISED_DATA_PATH }, services: SERVICES }); + const adapter = new HttpTestAdapter(BASE_URL); + const target = await adapter.readTargetServices(); + await adapter.execute(action('create_record', 'crm_account', { name: 'Acme' }), {}); + + expect(target.services).toEqual(SERVICES); + expect(target.source).toContain(EXPECTED_DISCOVERY_URL); + expect(calls.filter((c) => c.url === EXPECTED_DISCOVERY_URL)).toHaveLength(1); + // The record action still addresses the mount the same document advertised. + expect(actionCalls()[0]!.url).toBe(`${BASE_URL}${ADVERTISED_DATA_PATH}/crm_account`); + expect(warnings).toEqual([]); + }); + + it('reads the dispatcher bridge\'s `{ data: … }` envelope as well as the bare document', async () => { + discoveryReply = () => jsonResponse({ success: true, data: { routes: { data: '/api/v1/data' }, services: SERVICES } }); + const target = await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(target.services).toEqual(SERVICES); + }); + + it('sends the bearer token — a host that gates discovery still answers', async () => { + discoveryReply = () => jsonResponse({ routes: {}, services: SERVICES }); + await new HttpTestAdapter(BASE_URL, 'tok_123').readTargetServices(); + expect(calls.find((c) => c.url === EXPECTED_DISCOVERY_URL)!.headers['Authorization']).toBe('Bearer tok_123'); + }); + + /** Each row is a way the probe yields no services map; `services` is then absent and `source` says why. */ + const failures: Array<[label: string, reply: () => Response, evidence: RegExp]> = [ + ['discovery is not mounted (404)', () => jsonResponse({ error: 'Not found' }, 404), /answered 404/], + ['the document carries no services map', () => discoveryDocument({ data: ADVERTISED_DATA_PATH }), /answered 200 but carried no services map/], + ['the host cannot be reached', () => { throw new Error('ECONNREFUSED'); }, /could not be reached \(ECONNREFUSED\)/], + ]; + for (const [label, reply, evidence] of failures) { + it(`reports no services, and why, when ${label}`, async () => { + discoveryReply = reply; + const target = await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(target.services).toBeUndefined(); + expect(target.source).toMatch(evidence); + expect(target.source).toContain(EXPECTED_DISCOVERY_URL); + }); + } + + it('reading services alone does not print the data-mount warning — no record action asked for a mount', async () => { + discoveryReply = () => jsonResponse({ error: 'Not found' }, 404); + await new HttpTestAdapter(BASE_URL).readTargetServices(); + expect(warnings).toEqual([]); + }); +}); diff --git a/packages/core/src/qa/runner.test.ts b/packages/core/src/qa/runner.test.ts index 77f7697f7be..df1f00a7042 100644 --- a/packages/core/src/qa/runner.test.ts +++ b/packages/core/src/qa/runner.test.ts @@ -15,7 +15,7 @@ import { describe, it, expect } from 'vitest'; import * as QA from '@objectstack/spec/qa'; import { TestRunner } from './runner.js'; -import type { TestExecutionAdapter } from './adapter.js'; +import type { TargetServices, TestExecutionAdapter } from './adapter.js'; /** An adapter that hands the runner one fixed result — the assertion is the unit under test. */ class StubAdapter implements TestExecutionAdapter { @@ -281,3 +281,167 @@ describe('TestRunner — results carry the suite and scenario names the author w expect(result.suiteName).toBeUndefined(); }); }); + +// `TestScenario.requires` is ENFORCED (ADR-0049; the `qa-runner` family's fifth +// key, ruled B). Before any step runs — setup included — `params` is judged +// against the runner's environment and `services` against the target's +// discovery `services`; an unmet entry makes the scenario SKIPPED with a reason, +// its own verdict, never passed. These pin the runner half; `os test`'s printing +// and exit posture are pinned in packages/cli/test/qa-requires-skip-run.test.ts. +describe('TestRunner — `requires` is judged before the first step', () => { + const step: QA.TestStep = { name: 'step-1', action: { type: 'api_call', target: '/api/v1/health' } }; + + /** A discovery `services` map as both producers answer it (ADR-0076 D12). */ + const SERVICES = { + data: { enabled: true, status: 'available', route: '/api/v1/data' }, + auth: { enabled: true, status: 'available', route: '/api/v1/auth' }, + ai: { enabled: false, status: 'unavailable', message: 'no implementation ships' }, + metadata: { enabled: true, status: 'degraded', route: '/api/v1/meta' }, + }; + + /** Records every action it runs, and answers `readTargetServices` from a canned reading. */ + class TargetAdapter implements TestExecutionAdapter { + executed: string[] = []; + serviceReads = 0; + constructor(private reading: TargetServices | (() => TargetServices)) {} + async execute(action: QA.TestAction): Promise { + this.executed.push(action.target); + return { ok: true }; + } + async readTargetServices(): Promise { + this.serviceReads += 1; + return typeof this.reading === 'function' ? this.reading() : this.reading; + } + } + + const served = () => new TargetAdapter({ services: SERVICES, source: 'GET /api/v1/discovery advertised services' }); + + const scenario = (requires: QA.TestScenario['requires']): QA.TestScenario => ({ + id: 'needs-things', + name: 'Needs things', + setup: [{ name: 'setup-1', action: { type: 'api_call', target: '/setup' } }], + steps: [step], + requires, + }); + + it('an unmet `services` entry skips — no step runs, setup included — and lists the declared services', async () => { + const adapter = served(); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['ai'] })); + + expect(result.status).toBe('skipped'); + expect(result.passed).toBe(false); + expect(result.steps).toEqual([]); + expect(adapter.executed).toEqual([]); + expect(result.skipped!.unmet).toEqual([ + { key: 'services', name: 'ai', detail: expect.stringContaining('not available on the target') }, + ]); + // `metadata` is declared but degraded, `ai` disabled: only the two that are + // enabled AND available are what the target "declares available". + expect(result.skipped!.availableServices).toEqual(['auth', 'data']); + expect(result.skipped!.reason).toContain("'ai'"); + expect(result.skipped!.reason).toContain('auth, data'); + }); + + it('a met `services` entry runs — CONTROL', async () => { + const adapter = served(); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['data', 'auth'] })); + + expect(result.status).toBe('passed'); + expect(result.passed).toBe(true); + expect(result.skipped).toBeUndefined(); + expect(adapter.executed).toEqual(['/setup', '/api/v1/health']); + }); + + it('`enabled` alone is not enough: a degraded service is unmet, and so is one the target does not declare', async () => { + const result = await new TestRunner(served(), { env: {} }).runScenario(scenario({ services: ['metadata', 'search'] })); + + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet.map((u) => [u.name, u.detail])).toEqual([ + ['metadata', expect.stringContaining('status: degraded')], + ['search', 'not declared by the target'], + ]); + }); + + it('an unmet `params` entry skips naming the variable; set and non-empty, it runs', async () => { + const needsToken = scenario({ params: ['OS_QA_TOKEN'] }); + + const unset = await new TestRunner(served(), { env: {} }).runScenario(needsToken); + expect(unset.status).toBe('skipped'); + expect(unset.skipped!.unmet).toEqual([ + { key: 'params', name: 'OS_QA_TOKEN', detail: expect.stringContaining('not set') }, + ]); + expect(unset.skipped!.reason).toContain("requires.params 'OS_QA_TOKEN'"); + // No service was required, so no service list is attached. + expect(unset.skipped!.availableServices).toBeUndefined(); + + // An unconfigured CI secret arrives as an empty string: unmet too. + const empty = await new TestRunner(served(), { env: { OS_QA_TOKEN: '' } }).runScenario(needsToken); + expect(empty.status).toBe('skipped'); + expect(empty.skipped!.unmet[0]!.detail).toContain('empty'); + + const adapter = served(); + const set = await new TestRunner(adapter, { env: { OS_QA_TOKEN: 'tok' } }).runScenario(needsToken); + expect(set.status).toBe('passed'); + expect(adapter.executed).toEqual(['/setup', '/api/v1/health']); + }); + + it('every unmet entry is reported, params before services', async () => { + const result = await new TestRunner(served(), { env: {} }).runScenario( + scenario({ params: ['A', 'B'], services: ['data', 'ai'] }), + ); + expect(result.skipped!.unmet.map((u) => `${u.key}:${u.name}`)).toEqual(['params:A', 'params:B', 'services:ai']); + }); + + it('asks the target only when a service is required', async () => { + const adapter = served(); + await new TestRunner(adapter, { env: { X: '1' } }).runScenario(scenario({ params: ['X'] })); + await new TestRunner(adapter, { env: {} }).runScenario(scenario(undefined)); + expect(adapter.serviceReads).toBe(0); + }); + + it('a target whose services cannot be read declares none: the requirement is unmet and says why', async () => { + const adapter = new TargetAdapter({ source: 'GET http://x/api/v1/discovery answered 404' }); + const result = await new TestRunner(adapter, { env: {} }).runScenario(scenario({ services: ['data'] })); + + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet[0]!.detail).toContain('answered 404'); + expect(result.skipped!.availableServices).toEqual([]); + expect(adapter.executed).toEqual([]); + }); + + it('an adapter that cannot report its target skips a service requirement rather than running it', async () => { + const result = await new TestRunner(new StubAdapter({ ok: true }), { env: {} }).runScenario( + scenario({ services: ['data'] }), + ); + expect(result.status).toBe('skipped'); + expect(result.skipped!.unmet[0]!.detail).toContain('does not report'); + }); + + it('runSuite stamps the suite name on a skipped result too, and the verdicts stay distinct', async () => { + const results = await new TestRunner(served(), { env: {} }).runSuite({ + name: 'Mixed', + scenarios: [ + { id: 'runs', name: 'Runs', steps: [step] }, + { id: 'skips', name: 'Skips', steps: [step], requires: { services: ['ai'] } }, + ], + }); + expect(results.map((r) => [r.suiteName, r.scenarioId, r.status, r.passed])).toEqual([ + ['Mixed', 'runs', 'passed', true], + ['Mixed', 'skips', 'skipped', false], + ]); + }); + + it('a failed scenario says `failed` — skipped is not a kind of failure', async () => { + class ThrowingTarget extends TargetAdapter { + override async execute(): Promise { + throw new Error('boom'); + } + } + const result = await new TestRunner( + new ThrowingTarget({ services: SERVICES, source: 'x' }), + { env: {} }, + ).runScenario({ id: 'f', name: 'F', steps: [step], requires: { services: ['data'] } }); + expect(result.status).toBe('failed'); + expect(result.skipped).toBeUndefined(); + }); +}); From c8f50d88b69614419bdce6d1620214065239571c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 19:26:41 +0000 Subject: [PATCH 3/4] =?UTF-8?q?chore(spec):=20regenerate=20liveness=20stat?= =?UTF-8?q?e=20counts=20=E2=80=94=20qa=209=20live,=200=20dead?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH Co-authored-by: Claude --- packages/spec/liveness/state-counts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/spec/liveness/state-counts.md b/packages/spec/liveness/state-counts.md index 33e4ce3dd0b..9e9edfb7172 100644 --- a/packages/spec/liveness/state-counts.md +++ b/packages/spec/liveness/state-counts.md @@ -56,7 +56,7 @@ for both corollaries. | `validation` | 18 | 0 | 0 | 0 | 0 | 18 | | `api` | 25 | 0 | 0 | 1 | 2 | 28 | | `capability` | 12 | 0 | 0 | 0 | 0 | 12 | -| `qa` | 8 | 0 | 0 | 1 | 0 | 9 | +| `qa` | 9 | 0 | 0 | 0 | 0 | 9 | | `manifest` | 23 | 0 | 1 | 15 | 0 | 39 | | `crud_endpoints` | 6 | 0 | 0 | 2 | 0 | 8 | | `metadata_endpoints` | 7 | 0 | 0 | 2 | 0 | 9 | @@ -67,4 +67,4 @@ for both corollaries. | `sharing_rule` | 16 | 0 | 0 | 0 | 1 | 17 | | `connector` | 29 | 0 | 0 | 30 | 1 | 60 | | `analytics_cube` | 18 | 0 | 0 | 9 | 0 | 27 | -| **total** | **940** | **5** | **1** | **148** | **9** | **1103** | +| **total** | **941** | **5** | **1** | **147** | **9** | **1103** | From 75d17a115ee806c21e916caf4c3c3cbce7e32e56 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 20:04:32 +0000 Subject: [PATCH 4/4] =?UTF-8?q?docs(spec):=20regenerate=20the=20qa/testing?= =?UTF-8?q?=20reference=20=E2=80=94=20requires.services,=20the=20plugins?= =?UTF-8?q?=20tombstone?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01ARcDurZ5j34RdqsGgc4jgH Co-authored-by: Claude --- content/docs/references/qa/testing.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/docs/references/qa/testing.mdx b/content/docs/references/qa/testing.mdx index f94eebcb9d2..a824479ccde 100644 --- a/content/docs/references/qa/testing.mdx +++ b/content/docs/references/qa/testing.mdx @@ -116,7 +116,7 @@ A complete test scenario with setup, execution steps, and teardown | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) | | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute | | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution | -| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes | +| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed | ### Nested Shape: `TestScenario.setup[number]` @@ -158,8 +158,9 @@ A single step in a test scenario, consisting of an action and optional assertion | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | -| **params** | `string[]` | optional | Environment variables or parameters the scenario needs. Declared only: nothing checks them before the scenario runs | -| **plugins** | `string[]` | optional | Plugins the scenario needs loaded on the target. Declared only: nothing checks them before the scenario runs | +| **params** | `string[]` | optional | Environment variables that must be set to a non-empty value in the process running `os test` — the runner's own environment, not the target server's. An unset or empty variable skips the scenario (SKIPPED, never passed) with a reason naming the variable | +| **services** | `Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[]` | optional | Services the target must declare in its discovery document as `enabled` with status `available` (ADR-0076 D12), read from the discovery document `os test` already fetches once per run. An entry the target does not declare that way skips the scenario (SKIPPED, never passed) with a reason naming the service and the services the target does declare available | +| **plugins** | `never` | optional | [REMOVED] `scenarios[].requires.plugins` was removed in @objectstack/spec 17.5.0 (ADR-0049 enforce-or-remove) — nothing ever checked it: `os test` reaches its target over HTTP and no served surface lists the loaded plugins, so a scenario naming a missing plugin ran anyway. Delete the key and name the service the scenario needs in `requires.services`, which is judged against the services the target's discovery document declares enabled and available — an unmet entry skips the scenario and says why. Plugin → service: @objectstack/service-analytics → analytics, @objectstack/plugin-auth → auth, @objectstack/service-automation → automation, @objectstack/service-cache → cache, @objectstack/service-queue → queue, @objectstack/service-job → job, @objectstack/service-realtime → realtime, @objectstack/service-storage → storage, @objectstack/service-i18n → i18n, @objectstack/service-messaging → notification, @objectstack/metadata-protocol → ui. A plugin not listed fills no discovery service slot, so there is no service to require for it. | --- @@ -224,7 +225,7 @@ A complete test scenario with setup, execution steps, and teardown | **setup** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to run before main test (preconditions) | | **steps** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | ✅ | Main test sequence to execute | | **teardown** | `{ name: string; description?: string; action: object; assertions?: object[]; … }[]` | optional | Steps to cleanup after test execution | -| **requires** | `{ params?: string[]; plugins?: string[] }` | optional | Environment requirements for this scenario. NOT CHECKED by `os test` or the core TestRunner: the scenario runs whether or not they hold, and an unmet requirement surfaces only as the failure it causes | +| **requires** | `{ params?: string[]; services?: Enum<'metadata' \| 'data' \| 'auth' \| 'storage' \| 'file-storage' \| 'search' \| 'cache' \| …>[] }` | optional | Preconditions judged before the scenario's first step (setup included). Every entry must hold, or the scenario is SKIPPED with a reason naming each unmet entry — counted separately by `os test` and never counted as passed | ---