fix(service-package, cli): os migrate plan / apply cover sys_packages, and the boot honours OS_SKIP_SCHEMA_SYNC for it

.changeset/23022-sys-packages-managed-then-flag.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/service-package': patch
+'@objectstack/cli': patch
+---
+
+`os migrate plan` / `apply` cover `sys_packages`, and the boot honours `OS_SKIP_SCHEMA_SYNC` for it
+
+Clause-②: no
+
+`sys_packages`, the package registry's table, is created by `PackageServicePlugin.start()` with its own DDL, because it keys on `(id, version)` and a declared object cannot carry that key. So it was never on the managed migration path: `os migrate plan` did not list it and `os migrate apply` did not create it. A deployment that sets `OS_SKIP_SCHEMA_SYNC=1` ("Use after running migrations manually") still had it created at boot, and had no documented way to create it otherwise.
+
+- **The migration path covers it.** `PackageServicePlugin.init()` declares the table as its `package.storage` kernel service: a read-only catalog probe (SQLite, PostgreSQL and MySQL each ask their own catalog) and the boot's own create statements. `os migrate plan` lists `sys_packages` as `create_table` when it is absent, and `os migrate apply` creates it, with the same statements the boot runs, so the table and its `idx_packages_latest` index come out exactly as a boot creates them. Whether the flag is set does not matter to either command. Where the table's presence cannot be read (a driver that names no SQL dialect), the run says it did not examine the table, and neither lists nor creates it.
+- **Then the boot honours the flag.** With the flag set, `PackageServicePlugin` issues no DDL and no existence probe at boot, and logs one `info` line naming `sys_packages` and the flag, and pointing at `os migrate apply`. The `package` service is still registered. The value read is the one `ObjectQLPlugin` resolved (the `skipSchemaSync` option, else the environment variable), as `plugin-audit` and `service-messaging` read it. On a flag deployment, run `os migrate apply` before the first boot that needs the table.
+- **What is unchanged.** With the flag unset, the boot creates the table exactly as before. An existing `sys_packages` table is not altered by either command; the plan lists nothing for it. No type or export changes.

packages/cli/src/utils/schema-migrate.package-table.integration.test.ts

@@ -0,0 +1,157 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * #23022 — `os migrate plan` lists `sys_packages` and `os migrate apply`
+ * creates it, with `OS_SKIP_SCHEMA_SYNC=1` in the environment.
+ *
+ * ## What was broken
+ *
+ * `sys_packages` is the one table outside the object set:
+ * `PackageServicePlugin.start()` creates it with raw DDL (it keys on
+ * `(id, version)`, which a declared object cannot carry). A migrate boot runs a
+ * provider's `init()` and never its `start()`, and no driver deferral sees a
+ * table no object declares, so the plan never listed it and `apply` never
+ * created it. A deployment that sets the flag — "skip the implicit `db:sync` on
+ * boot. Use after running migrations manually." — therefore had no documented
+ * way to get it, and the boot could not honour the flag without stranding the
+ * deployment. Measured on `9f4b652e3` through `os migrate plan --json` and
+ * `os migrate apply --yes --json`: no `sys_packages` in `pending`, and absent
+ * after the apply.
+ *
+ * ## What this holds, through the REAL funnel
+ *
+ * The boot is `bootSchemaStack` exactly as the two commands call it, composing
+ * the real `PackageServicePlugin` the way `os serve` mounts it (the always-on
+ * `package-registry` provider), so the two ends of the `package.storage`
+ * declaration — the plugin's `init()` and this CLI's reader — meet here:
+ *
+ *  1. plan (a never-started project): `create_table sys_packages` is pending,
+ *     with the table's seven columns, and no database file is brought into
+ *     existence;
+ *  2. apply: the boot creates no `sys_packages` before the flush (the table is
+ *     still absent while the plan is on screen), the flush creates it and
+ *     reports it, and the table is the one the boot's own DDL creates —
+ *     `PRIMARY KEY (id, version)`, `TEXT` timestamps, `idx_packages_latest`;
+ *  3. a plan over the migrated database lists nothing for it.
+ *
+ * The flag is SET for every boot here: the documented path must not depend on
+ * it. That the boot itself creates nothing under the flag is
+ * `@objectstack/service-package`'s `managed-path.test.ts`.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import Database from 'better-sqlite3';
+// Module-top side-effect loads, paid during COLLECTION rather than inside a
+// clocked test body: `bootSchemaStack` reaches these through dynamic
+// `import()`s that resolve through `dist/` (`check:test-source-alias`).
+import '@objectstack/runtime';
+import '@objectstack/objectql';
+import { bootSchemaStack, type SchemaStack } from './schema-migrate.js';
+
+const COLUMNS = ['id', 'version', 'manifest', 'metadata', 'hash', 'created_at', 'updated_at'];
+
+/** The env vars that outrank the unified project default (#6469). */
+const OVERRIDING_ENV = ['OS_DATABASE_URL', 'DATABASE_URL', 'TURSO_DATABASE_URL', 'OS_DATABASE_DRIVER', 'OS_HOME'];
+const SET_ENV: Record<string, string> = {
+  OS_SKIP_SCHEMA_SYNC: '1',
+  OS_TELEMETRY_DB: '0',
+  NODE_ENV: 'production',
+};
+
+describe('os migrate plan / apply cover sys_packages, flag set (#23022)', () => {
+  let dir: string;
+  let dbFile: string;
+  let stack: SchemaStack | null = null;
+  const savedEnv: Record<string, string | undefined> = {};
+
+  const boot = (mode: 'plan' | 'apply'): Promise<SchemaStack> => bootSchemaStack({
+    jsonOutput: false,
+    projectRoot: dir,
+    databaseUrl: `file:${dbFile}`,
+    deferSchemaDdl: true,
+    ...(mode === 'plan' ? { readOnlyProbe: true } : {}),
+    composeHostStack: true,
+    composeServedPlatform: true,
+  });
+
+  const packageWork = (s: SchemaStack) => s.pendingSchemaWork.filter((w) => w.table === 'sys_packages');
+
+  /** Read the file with a connection of OUR OWN — never the booted stack's. */
+  function readTable() {
+    const db = new Database(dbFile, { readonly: true });
+    try {
+      return {
+        present: db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'sys_packages'").all().length === 1,
+        columns: db.prepare("SELECT name, type, \"notnull\" AS not_null FROM pragma_table_info('sys_packages') ORDER BY cid").all(),
+        pk: db.prepare("SELECT name FROM pragma_table_info('sys_packages') WHERE pk > 0 ORDER BY pk").all(),
+        index: db.prepare("SELECT name FROM pragma_index_info('idx_packages_latest') ORDER BY seqno").all(),
+      };
+    } finally {
+      db.close();
+    }
+  }
+
+  beforeEach(() => {
+    dir = mkdtempSync(join(tmpdir(), 'os-23022-'));
+    dbFile = join(dir, 'app.db');
+    writeFileSync(join(dir, 'objectstack.config.ts'), [
+      'export default {',
+      "  manifest: { id: 'com.example.os23022', name: 'os23022', version: '0.0.0', type: 'app' },",
+      "  objects: [{ name: 'os23022_note', fields: { title: { type: 'text' } } }],",
+      '};',
+      '',
+    ].join('\n'));
+    for (const key of [...OVERRIDING_ENV, ...Object.keys(SET_ENV), 'OS_ARTIFACT_PATH']) savedEnv[key] = process.env[key];
+    for (const key of OVERRIDING_ENV) delete process.env[key];
+    Object.assign(process.env, SET_ENV);
+    process.env.OS_ARTIFACT_PATH = join(dir, 'dist', 'objectstack.json'); // deliberately absent
+  });
+
+  afterEach(async () => {
+    try { await stack?.shutdown(); } catch { /* torn down either way */ }
+    stack = null;
+    for (const [key, value] of Object.entries(savedEnv)) {
+      if (value === undefined) delete process.env[key];
+      else process.env[key] = value;
+    }
+    try { rmSync(dir, { recursive: true, force: true }); } catch { /* ignore */ }
+  });
+
+  it('plan lists it, apply creates the boot\'s own table, and a plan after that lists nothing for it', async () => {
+    // 1. plan — a never-started project.
+    stack = await boot('plan');
+    expect(stack.composition.servedBoot).toEqual({ mirrored: true });
+    expect(packageWork(stack)).toEqual([{ table: 'sys_packages', kind: 'create_table', columns: COLUMNS }]);
+    await stack.shutdown();
+    stack = null;
+    expect(existsSync(dbFile), 'a dry run brought the database into existence').toBe(false);
+
+    // 2. apply — nothing for it before the flush, the boot's table after it.
+    stack = await boot('apply');
+    expect(packageWork(stack)).toEqual([{ table: 'sys_packages', kind: 'create_table', columns: COLUMNS }]);
+    expect(readTable().present, 'the migrate boot created sys_packages before the operator confirmed').toBe(false);
+    const created = await stack.flushSchemaDdl();
+    expect(created.filter((w) => w.table === 'sys_packages')).toEqual([
+      { table: 'sys_packages', kind: 'create_table', columns: COLUMNS },
+    ]);
+    await stack.shutdown();
+    stack = null;
+    expect(readTable()).toEqual({
+      present: true,
+      columns: COLUMNS.map((name) => ({
+        name,
+        type: 'TEXT',
+        not_null: ['created_at', 'updated_at'].includes(name) ? 0 : 1,
+      })),
+      pk: [{ name: 'id' }, { name: 'version' }],
+      index: [{ name: 'id' }, { name: 'created_at' }],
+    });
+
+    // 3. plan again — the table is there, so nothing is pending for it.
+    stack = await boot('plan');
+    expect(packageWork(stack)).toEqual([]);
+  }, 300_000);
+});

packages/cli/src/utils/schema-migrate.ts

@@ -244,6 +244,82 @@ function sortPendingSchemaWork(work: PendingSchemaWork[]): PendingSchemaWork[] {
   return work.sort((a, b) => a.table.localeCompare(b.table) || a.kind.localeCompare(b.kind));
 }
 
+/**
+ * [#23022] The kernel service under which `PackageServicePlugin.init()`
+ * declares `sys_packages` — `@objectstack/service-package`'s
+ * `PACKAGE_STORAGE_SERVICE`, read by name because the package exports no
+ * constant for it.
+ */
+const PACKAGE_STORAGE_SERVICE = 'package.storage';
+
+/** What {@link PACKAGE_STORAGE_SERVICE} answers with (`service-package`'s `PackageStorageDeclaration`). */
+interface PackageStorageDeclaration {
+  readonly table: string;
+  readonly columns: readonly string[];
+  /** Read-only: `true` present, `false` absent, `undefined` could not tell. */
+  present(): Promise<boolean | undefined>;
+  /** The package registry's own `CREATE TABLE IF NOT EXISTS` and index. */
+  ensure(): Promise<void>;
+}
+
+/**
+ * [#23022] `sys_packages` as pending schema work, when it is absent.
+ *
+ * ## Why the plan needs this
+ *
+ * `PackageServicePlugin.start()` creates `sys_packages` with raw DDL, because
+ * the table cannot be a declared object (`service-package`'s
+ * `package-table.ts` records the measurement: the driver's managed tables key
+ * on `id` alone, and this one keys on `(id, version)`). So no driver deferral
+ * ever listed it, and a deployment that sets `OS_SKIP_SCHEMA_SYNC=1` — whose
+ * boot now creates nothing — was left with no way to get it from the documented
+ * path. The plugin declares the table in `init()`, which a migrate boot runs,
+ * and this lists it as `create_table` beside the objects' tables; `apply`'s
+ * flush then runs the plugin's own statements, so `apply` creates exactly what
+ * the boot creates.
+ *
+ * `null` when there is nothing to plan: the plugin is not composed, or the
+ * table exists. When its presence could not be READ, that is not absence: a
+ * note says the table was not examined, and nothing is planned.
+ */
+async function planPackageTable(
+  kernel: unknown,
+  notes: string[],
+): Promise<{ storage: PackageStorageDeclaration; work: PendingSchemaWork } | null> {
+  let service: unknown;
+  try {
+    service = (kernel as { getService?: (name: string) => unknown }).getService?.(PACKAGE_STORAGE_SERVICE);
+  } catch {
+    return null; // not composed: this boot's deployment serves no package registry
+  }
+  if (service === undefined || service === null) return null;
+  const storage = service as Partial<PackageStorageDeclaration>;
+  if (
+    typeof storage.table !== 'string' || !Array.isArray(storage.columns)
+    || typeof storage.present !== 'function' || typeof storage.ensure !== 'function'
+  ) {
+    throw new Error(
+      `The "${PACKAGE_STORAGE_SERVICE}" service does not have the shape this CLI reads (table, columns, present, `
+      + 'ensure), so sys_packages cannot be planned. @objectstack/cli and @objectstack/service-package must come '
+      + 'from the same release.',
+    );
+  }
+  const declared = storage as PackageStorageDeclaration;
+  const present = await declared.present();
+  if (present === true) return null;
+  if (present === undefined) {
+    notes.push(
+      `${declared.table} (the package registry's table) was NOT examined: its presence could not be read on this `
+      + 'database, so this run does not list it and does not create it.',
+    );
+    return null;
+  }
+  return {
+    storage: declared,
+    work: { table: declared.table, kind: 'create_table', columns: [...declared.columns] },
+  };
+}
+
 /**
  * Arms deferred-DDL mode on EVERY SQL driver the boot connects, before any of
  * them can schema-sync (#3917, #21391).
@@ -925,6 +1001,10 @@ export async function bootSchemaStack(
   for (const d of deferral?.drivers ?? []) {
     if (d.previewDeferredSchemaWork) pendingSchemaWork.push(...(await d.previewDeferredSchemaWork()));
   }
+  // [#23022] …and the one table outside the object set, `sys_packages`, which
+  // no driver deferral can see. Read-only here; `flushSchemaDdl` creates it.
+  let packageTable = deferral ? await planPackageTable(kernel, composition.notes) : null;
+  if (packageTable) pendingSchemaWork.push(packageTable.work);
   sortPendingSchemaWork(pendingSchemaWork);
   // [#21529] The tables the held-back sync would create: measured absent.
   const absentTables = new Set(
@@ -1008,6 +1088,12 @@ export async function bootSchemaStack(
       for (const d of deferral?.drivers ?? []) {
         if (d.flushDeferredSchemaDdl) performed.push(...(await d.flushDeferredSchemaDdl()));
       }
+      // [#23022] The package registry's own DDL — the statements its boot runs.
+      if (packageTable) {
+        await packageTable.storage.ensure();
+        performed.push(packageTable.work);
+        packageTable = null;
+      }
       // [#21529] The tables exist now; nothing is measured absent any more.
       absentTables.clear();
       deferral?.release();