Merge remote-tracking branch 'origin/main' into claude/issue-21387-retire-role-arm

Author: claude

.changeset/21733-standalone-dev-self-heal.md

@@ -0,0 +1,13 @@
+---
+"@objectstack/runtime": minor
+"@objectstack/cli": patch
+---
+
+fix(runtime,cli): a plain `os dev` now self-heals safe schema drift on restart and provisions the `telemetry` sibling database, as `content/docs/deployment/cli.mdx` already says (#21733)
+
+Clause-②: yes (widening)
+
+- **What was broken.** A config with no instantiated `plugins[]` (every fresh scaffold) boots through the standalone stack. Its `default` datasource was built without `autoMigrate: 'safe'`: only the config-load fallback that a host config or `OS_MODE=off` takes carried it. So safe drift was never applied on restart. An example is a per-organization unique index that an older release left non-NULL-safe. Meanwhile the driver's drift line and `os migrate plan` both said the change was "auto-applied at boot under dev autoMigrate: 'safe'". The same boot never provisioned the `<db>.telemetry.<ext>` sibling either.
+- **The fix.** The dev self-heal decision now lives in one place, `devAutoMigrateConfig` in `@objectstack/runtime`. That is the driver kinds whose connection contract declares `autoMigrate` (sqlite, postgres, mysql), on a dev boot. The standalone stack, the CLI's config-load fallback and the telemetry sibling all read it, so no kind gains or loses the self-heal relative to the host path. The telemetry provision is one helper (`provisionTelemetryDatasource`) that both serving paths call, under the same `resolveTelemetryDbPath` rule: dev default-on for a file-backed SQLite primary, `OS_TELEMETRY_DB=0` to opt out, `OS_TELEMETRY_DB=<path>` to opt in anywhere.
+- **Only a serving boot self-heals.** The standalone stack arms the self-heal on an explicit `dev: true`. That is what `os dev` passes. It does not arm it on the `NODE_ENV=development` default that its sqlite step-down still takes. A one-shot command (`os migrate *`, `os meta resync`, …) passes no `dev`, so it never applies drift its operator did not confirm, whatever `NODE_ENV` says. Production boots are unchanged: the definition carries no `autoMigrate`, and the SQL driver refuses it under `NODE_ENV=production` anyway.
+- **Why minor.** `@objectstack/runtime` gains two exports on its only entry, `devAutoMigrateConfig` and its `DevAutoMigrateConfig` type. That is the widening: the existing decision moved out of the CLI so that the CLI reads it rather than keep a second copy. No config key, schema or accept set moves. `@objectstack/cli` is a `patch`: its fix restores documented behaviour and adds no public surface.

packages/cli/src/commands/serve.ts

@@ -8,6 +8,7 @@ import chalk, { chalkStderr } from 'chalk';
 import { bundleRequire } from 'bundle-require';
 import { loadConfig, BUNDLE_REQUIRE_EXTERNALS } from '../utils/config.js';
 import { mergeBootConfig } from '../utils/merge-boot-config.js';
+import { provisionTelemetryDatasource, standaloneTelemetryPrimary } from '../utils/telemetry-datasource.js';
 import { isHostConfig, shouldBootWithLibrary } from '../utils/plugin-detection.js';
 import { readInternalArtifactPath, readInternalConfigOutputPath } from '../utils/internal-artifact-channel.js';
 // The precedence's last rung — whether the cwd config takes part — decided by
@@ -2583,6 +2584,12 @@ export default class Serve extends Command {
     // human and may be redacted or labelled, while this one is handed to
     // `stat` — see `describeDriverSqliteFile`.
     let servedSqliteFilePath: string | undefined;
+    // #21733 — the file-backed native SQLite database a composed STANDALONE
+    // stack declared as its `default` datasource, read BEFORE the kernel boots
+    // so the `telemetry` sibling can be provisioned next to it (step 2 below).
+    // `servedSqliteFilePath` cannot serve: on this path it is only learned by
+    // probing the booted kernel. Unset for every other boot and every other kind.
+    let standaloneSqlitePrimary: string | undefined;
 
     // Resolve the kernel logger level up front. It decides more than the
     // logger's own threshold: it decides whether the boot-quiet window below
@@ -2741,7 +2748,7 @@ export default class Serve extends Command {
           // "missing artifact" error and assemble a bare kernel that
           // can later install marketplace apps at runtime.
           const { createDefaultHostConfig } = await import('@objectstack/runtime');
-          const bootResult = await createDefaultHostConfig({
+          const defaultHostInput = {
             requireArtifact: !useEmptyBoot,
             dev: isDev,
             // #8368: the already-fetched, already-verified LOCAL copy. Passing
@@ -2755,7 +2762,15 @@ export default class Serve extends Command {
             ...(pinnedArtifact
               ? { artifactPath: pinnedArtifact.localPath }
               : supervisorArtifact ? { artifactPath: supervisorArtifact } : {}),
-          });
+          };
+          const bootResult = await createDefaultHostConfig(defaultHostInput);
+          // #21733 — the `default` database this stack declared, for the
+          // telemetry sibling (step 2). The same input the host config was
+          // built from: the database resolution reads the artifact only for a
+          // declared default datasource, and the empty boot's synthesized stub
+          // declares none, so the artifact rung `createDefaultHostConfig`
+          // settles internally cannot answer differently.
+          standaloneSqlitePrimary = await standaloneTelemetryPrimary(defaultHostInput);
           // [#4002] `api` merges per key — see mergeBootConfig. A shallow spread
           // let the boot builder's two scoping keys wipe the author's whole `api`
           // block, silently dropping `requireAuth` / `enforceProjectMembership`.
@@ -2779,6 +2794,10 @@ export default class Serve extends Command {
             ...(supervisorArtifact ? { artifactPath: supervisorArtifact } : {}),
           };
           const bootResult = await createStandaloneStack(standaloneInput);
+          // #21733 — the `default` database the stack just declared, resolved
+          // by the runtime's own pre-boot resolution over the SAME input, for
+          // the telemetry sibling (step 2).
+          standaloneSqlitePrimary = await standaloneTelemetryPrimary(standaloneInput);
           // #21501 — did the standalone stack load a compiled artifact as this
           // app's bundle? Its AppPlugin over the bundle is the proof (it is
           // pushed only when the bundle loaded, and a non-host config carries
@@ -3190,6 +3209,26 @@ export default class Serve extends Command {
          }
       }
 
+      // ADR-0057 §3.6 — the `telemetry` sibling datasource, provisioned next to
+      // a file-backed SQLite primary by EVERY serving boot through ONE helper
+      // (`provisionTelemetryDatasource`, #21733): the config-load fallback in
+      // step 2 below and the standalone stack right after it. Dev default-on;
+      // `OS_TELEMETRY_DB=0` opts out, `OS_TELEMETRY_DB=<path>` opts in anywhere
+      // (incl. serve) — `resolveTelemetryDbPath` is the rule, never restated here.
+      const provisionTelemetry = async (primaryPath: string | undefined): Promise<void> => {
+        const telemetryPath = await provisionTelemetryDatasource({
+          primaryPath,
+          env: process.env,
+          dev: isDev,
+          use: (plugin) => kernel.use(plugin as any),
+          warn: (m) => console.warn(chalk.yellow(m)),
+        });
+        if (telemetryPath) {
+          trackPlugin('TelemetryDatasource');
+          printDiagnostic(chalk.dim(`  telemetry datasource: ${telemetryPath} (lifecycle-classed system data; OS_TELEMETRY_DB=0 to disable)`));
+        }
+      };
+
       // 2. Auto-register storage driver
       // Priority:
       //   1. OS_DATABASE_DRIVER env var (explicit override)
@@ -3234,7 +3273,7 @@ export default class Serve extends Command {
            // resolveStorageDefinition. The dev sqlite step-down (#2229) and
            // the loosen-only self-heal (#2186, via config.autoMigrate) now run
            // inside the factory at connect.
-           const { DriverPlugin, DefaultDatasourcePlugin } = await import('@objectstack/runtime');
+           const { DefaultDatasourcePlugin } = await import('@objectstack/runtime');
            const resolution = resolveStorageDefinition(driverType, { databaseUrl, isDev, authToken: databaseAuthToken });
            if (resolution) {
              // #5602: libSQL/Turso is the one kind the shared open-core factory
@@ -3257,52 +3296,11 @@ export default class Serve extends Command {
              resolvedDatabaseUrl = resolution.displayUrl;
              servedSqliteFilePath = resolution.sqliteFilePath;
 
-             // ADR-0057 §3.6 (#2834 ②): provision the dedicated `telemetry`
-             // datasource — a sibling SQLite file the engine routes every
-             // telemetry/event/audit-classed object to, so platform-generated
-             // growth can never again bloat the business DB. Dev default-on
-             // for file-backed primaries; `OS_TELEMETRY_DB=0` opts out,
-             // `OS_TELEMETRY_DB=<path>` opts in anywhere (incl. serve). Gated on
-             // an explicit SQLite primary (`sqliteFilePath`, unset for the mingo
-             // memory driver AND the dev-default `:memory:` store). The old
-             // `resolution.engine !== 'memory'` refinement is unknowable now
-             // that the primary connects later (#3826); the telemetry
-             // provision's own `telemetry.engine !== 'memory'` check below
-             // still guards the ABI-broken step-down case. The telemetry
-             // driver itself stays a pre-built DriverPlugin — the documented
-             // escape hatch for named auxiliary drivers.
-             if (resolution.sqliteFilePath) {
-               const { resolveTelemetryDbPath } = await import('../utils/telemetry-datasource.js');
-               const telemetryPath = resolveTelemetryDbPath({ primaryPath: resolution.sqliteFilePath, env: process.env, dev: isDev });
-               if (telemetryPath) {
-                 try {
-                   const { resolveSqliteDriver } = await import('@objectstack/service-datasource');
-                   const telemetry = await resolveSqliteDriver({
-                     filename: telemetryPath,
-                     dev: isDev,
-                     autoMigrate: isDev ? 'safe' : undefined,
-                     warn: (m) => console.warn(chalk.yellow(m)),
-                   });
-                   if (telemetry.engine !== 'memory') {
-                     // The engine keys datasources by driver name — the
-                     // lifecycle router looks this exact name up. The driver
-                     // name is the WHOLE wiring: DriverPlugin.init registers
-                     // `driver.telemetry`, ObjectQL's discovery loop adopts
-                     // it, and lifecycle-classed objects route to it. (An
-                     // options bag once also asked for `datasourceName:
-                     // 'telemetry'` metadata registration — inert since
-                     // inception, retired in #4320.)
-                     Object.defineProperty(telemetry.driver, 'name', { value: 'telemetry' });
-                     await kernel.use(new DriverPlugin(telemetry.driver));
-                     trackPlugin('TelemetryDatasource');
-                     printDiagnostic(chalk.dim(`  telemetry datasource: ${telemetryPath} (lifecycle-classed system data; OS_TELEMETRY_DB=0 to disable)`));
-                   }
-                 } catch {
-                   // Best-effort: a failed telemetry provision must never block
-                   // boot — objects simply stay on the primary datasource.
-                 }
-               }
-             }
+             // ADR-0057 §3.6 (#2834 ②): the dedicated `telemetry` datasource
+             // next to an explicit SQLite primary (`sqliteFilePath`, unset for
+             // every other kind AND the dev-default `:memory:` store). The ONE
+             // provision, shared with the standalone boot below.
+             await provisionTelemetry(resolution.sqliteFilePath);
            }
          } catch (e: any) {
            // "declared ≠ enforced" guard (#3276-class): a selection the CLI
@@ -3332,6 +3330,12 @@ export default class Serve extends Command {
            if (e?.code === 'MONGODB_MULTI_TENANT_UNSUPPORTED') throw e;
            // silent
          }
+      } else if (standaloneSqlitePrimary !== undefined) {
+        // #21733 — the standalone stack's `default` datasource is a file-backed
+        // SQLite database (resolved by the runtime's own pre-boot resolution in
+        // the boot-mode dispatch above), so it gets the same sibling the
+        // config-load fallback provisions, under the same rule.
+        await provisionTelemetry(standaloneSqlitePrimary);
       }
 
       // 3. Auto-register AppPlugin if config contains app definitions

packages/cli/src/utils/dev-self-heal-host-parity.test.ts

@@ -0,0 +1,97 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * #21733 — both boot hosts give the `default` datasource the SAME dev
+ * self-heal, kind by kind.
+ *
+ * The self-heal (`autoMigrate: 'safe'`, #2186) used to be decided inline in
+ * the CLI host (`resolveStorageDefinition`) and not at all in the runtime host
+ * (`createStandaloneStack`), which every plain `os dev` composes. Both now read
+ * `devAutoMigrateConfig` from `@objectstack/runtime`; this file drives the two
+ * REAL entry points, not the helper, so a host that stops reading it — or
+ * starts deciding for itself — fails here by kind. Same reason the driver
+ * vocabulary's own cross-host pin lives in this package
+ * (`driver-vocabulary-parity.test.ts`): it is the one that can import both.
+ *
+ * The host path's answer is the reference: no kind gains or loses `'safe'`
+ * relative to it (`storage-driver.test.ts` pins that answer on its own).
+ */
+
+import { describe, it, expect, afterEach } from 'vitest';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createStandaloneStack } from '@objectstack/runtime';
+import { resolveStorageDefinition } from './storage-driver.js';
+
+// [#10126] Pay the first transform of these dist-resolved workspace deps at
+// MODULE LOAD: `createStandaloneStack` reaches them through dynamic
+// `import()`s inside clocked `it()` bodies (`scripts/check-test-source-alias.mjs`).
+import '@objectstack/service-datasource';
+import '@objectstack/objectql';
+import '@objectstack/metadata';
+
+const ENV_KEYS = ['OS_DATABASE_URL', 'DATABASE_URL', 'TURSO_DATABASE_URL', 'OS_DATABASE_DRIVER', 'OS_HOME'] as const;
+const ORIGINAL_ENV: Record<string, string | undefined> = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
+const dirs: string[] = [];
+
+afterEach(() => {
+  for (const key of ENV_KEYS) {
+    const original = ORIGINAL_ENV[key];
+    if (original === undefined) delete process.env[key];
+    else process.env[key] = original;
+  }
+  for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true });
+});
+
+/** Each canonical kind, by a URL both hosts select it from. */
+function urlsFor(dir: string): Record<string, string> {
+  return {
+    sqlite: `file:${join(dir, 'p.db')}`,
+    'sqlite-wasm': `wasm-sqlite://${join(dir, 'w.db')}`,
+    postgres: 'postgres://u:p@localhost:5432/db',
+    mysql: 'mysql://u:p@localhost:3306/db',
+    mongodb: 'mongodb://localhost:27017/db',
+    turso: 'libsql://my-db.turso.io',
+  };
+}
+
+async function runtimeAutoMigrate(databaseUrl: string, dev: boolean, projectRoot: string): Promise<unknown> {
+  const stack = await createStandaloneStack({ databaseUrl, dev, projectRoot });
+  const plugin = stack.plugins.find((p: any) => p?.name === 'com.objectstack.runtime.default-datasource') as any;
+  expect(plugin, 'the standalone stack must carry the DefaultDatasourcePlugin').toBeDefined();
+  return plugin.def.config?.autoMigrate;
+}
+
+const BOOT_TIMEOUT = 90_000;
+
+describe('#21733 — the dev self-heal is the same on both boot hosts', () => {
+  it('every kind: the standalone stack carries exactly the `autoMigrate` the CLI host does, dev and production', async () => {
+    for (const key of ENV_KEYS) delete process.env[key];
+    const dir = mkdtempSync(join(tmpdir(), 'os-21733-parity-'));
+    dirs.push(dir);
+    const rows: string[] = [];
+    for (const [kind, url] of Object.entries(urlsFor(dir))) {
+      for (const dev of [true, false]) {
+        const cli = resolveStorageDefinition(kind, { databaseUrl: url, isDev: dev })?.config.autoMigrate;
+        const runtime = await runtimeAutoMigrate(url, dev, dir);
+        rows.push(`${kind} dev=${dev}: cli=${String(cli)} runtime=${String(runtime)}`);
+      }
+    }
+    // One assertion over the whole table, so a divergence names every kind it touches.
+    expect(rows).toEqual([
+      'sqlite dev=true: cli=safe runtime=safe',
+      'sqlite dev=false: cli=undefined runtime=undefined',
+      'sqlite-wasm dev=true: cli=undefined runtime=undefined',
+      'sqlite-wasm dev=false: cli=undefined runtime=undefined',
+      'postgres dev=true: cli=safe runtime=safe',
+      'postgres dev=false: cli=undefined runtime=undefined',
+      'mysql dev=true: cli=safe runtime=safe',
+      'mysql dev=false: cli=undefined runtime=undefined',
+      'mongodb dev=true: cli=undefined runtime=undefined',
+      'mongodb dev=false: cli=undefined runtime=undefined',
+      'turso dev=true: cli=undefined runtime=undefined',
+      'turso dev=false: cli=undefined runtime=undefined',
+    ]);
+  }, BOOT_TIMEOUT);
+});