fix(service-datasource)!: Import as Object saves through the metadata door's save

.changeset/21788-external-import-saves-like-meta.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/service-datasource': minor
+---
+
+fix(service-datasource)!: "Import as Object" saves the imported federated object through the metadata door's own save, so it is durable and reads from its remote table; federated validation stops reporting the platform's injected anchors as missing remote columns (#21788)
+
+**BREAKING** — the import route now answers `400` to some re-imports that used to answer `201`.
+
+Clause-②: no (narrowing)
+
+- **The import was neither durable nor mapped.** `POST /api/v1/datasources/:name/external/tables/:remote/import` held the generated object in the metadata service's memory only. No `sys_metadata` row was written, the object's storage was not synced, and the driver was never told the object's remote table. An object imported under a name that differs from its remote table answered `201` and then `500 DATABASE_ERROR` (`no such table: <name>`) on its first read. Every import was gone after a restart (`404 OBJECT_NOT_FOUND`).
+- **It now saves like `PUT /api/v1/meta/object/:name`.** The import calls `saveMetaItem` on the `protocol` service with the request that door sends for an `object`. The object becomes a `sys_metadata` row the next boot binds, it is written through to the engine registry, and it is mapped onto its `external.remoteName` table. An import under a different name and one under the remote table's own name both serve the remote rows, before and after a restart. The save door is looked up when an import runs. A deployment with no metadata save door still refuses the import with "requires a writable metadata store", before any remote introspection.
+- **What narrows.** The metadata door's refusals now apply to the import, and the route relays each one as `400 EXTERNAL_IMPORT_ERROR` with the door's message. A re-import that would drop a field the stored object already has, or change its type, is refused as a destructive change. It used to answer `201` with an in-memory overwrite that a restart discarded. An import whose `name` collides with an object the metadata door will not overwrite is refused the same way. The route's request and response shapes are unchanged.
+- **If a re-import or a name is refused:** import the table again under a new `name`. To change an object you already imported, save its new definition through `PUT /api/v1/meta/object/:name?force=true`, which accepts the destructive change on purpose. Re-submitting the import with `?force=true` does not help, because the import route reads no `force`.
+- **Federated validation compares only what the remote owns.** A stored federated object is read back with the anchors the platform injects without storage (`organization_id`, `created_by`, `updated_by`, `owner_id`, `owning_business_unit_id`). The boot validation gate and `POST …/external/validate` reported each of them as a `missing_column` at error severity. So once a datasource with the default `external.validation.onMismatch: 'fail'` held such an object, it refused to boot. Validation now skips the columns `unprovisionedInjectedColumns` names. This closes the boot abort for objects saved through `PUT /api/v1/meta/object/:name`, not only for imports. A declared column the remote lacks is still a `missing_column` at error severity, and `fail` still aborts on it.
+
+<!-- adr-0087: not-required (no-migration-prescription) a runtime refusal on one REST route: the external-table import now relays the metadata door's own save refusals (a destructive re-import, a name that door will not overwrite) as 400 where it answered 201 with an in-memory write no restart kept. No authorable metadata key, spelling, export, config field or stored shape moves: no schema changes, no sys_metadata row is read or rewritten, and which table or name an operator meant to re-import is not something a ledger entry can rewrite. The validation half refuses less, not more. The other categories are closed on facts: @objectstack/service-datasource publishes (not unpublished); no ADR-0087 id covers this route (not already-registered); and the change is a runtime verdict, not a declaration (not runtime-interface-only or type-surface-only). -->

packages/qa/dogfood/package.json

@@ -30,6 +30,7 @@
     "@objectstack/plugin-sharing": "workspace:*",
     "@objectstack/plugin-webhooks": "workspace:*",
     "@objectstack/service-analytics": "workspace:*",
+    "@objectstack/service-datasource": "workspace:*",
     "@objectstack/service-messaging": "workspace:*",
     "@objectstack/service-storage": "workspace:*",
     "@objectstack/spec": "workspace:*",

packages/qa/dogfood/test/external-import-saves-like-meta.dogfood.test.ts

@@ -0,0 +1,127 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// [#21788, ADR-0015 Addendum] "Import as Object" keeps what it answers `201`
+// for — over the real showcase composition, across a cold boot on one
+// database file.
+//
+// ## What was broken
+//
+// `POST /datasources/:name/external/tables/:remote/import` persisted the
+// generated object with `metadata.register('object', …)`, which held it in the
+// metadata service's memory only: no `sys_metadata` row, no engine schema
+// sync, no external-object registration with the driver. Measured on the
+// showcase before the fix: an import under a name that differs from its remote
+// table answered `201` and then `500 DATABASE_ERROR` on its first read (the
+// SQL driver resolved the table by the object's name — `no such table`), an
+// import whose name equals the remote table answered `200` only because the
+// two names coincide, and after a restart on the same database both answered
+// `404 OBJECT_NOT_FOUND`. `PUT /meta/object/:name` with the same binding was
+// durable and served the rows. The import now saves through that door's save.
+//
+// ## Why a booted stack
+//
+// The seam pins sit beside the plugin
+// (`packages/services/service-datasource/src/__tests__/external-import-saves-through-metadata-door.test.ts`).
+// What they cannot answer is whether, on the stack an operator runs, the save
+// maps the object onto its remote table and lands a row the next boot binds.
+// Only a read through the data door and a restart on the same file show that.
+//
+// The verify harness does not mount the federation service, so this file
+// mounts `ExternalDatasourceServicePlugin` itself, as `objectstack dev` and
+// `serve` do. The working directory is a temporary one because the showcase's
+// external datasource and its fixture both name a cwd-relative SQLite file:
+// this file's remote database is its own, not one a parallel file writes.
+
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import showcaseStack, { onEnable } from '@objectstack/example-showcase';
+import { ExternalDatasourceServicePlugin } from '@objectstack/service-datasource';
+import { bootStack, type VerifyStack } from '@objectstack/verify';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+/** The showcase's external datasource and its two remote tables (`external-fixture.ts`). */
+const DATASOURCE = 'showcase_external';
+/** Imported under a name that differs from its remote table — the defect's shape. */
+const RENAMED = 'dogfood_ext_cust_21788';
+/** Imported under the remote table's own name — the shape that hid the defect. */
+const SAME_NAME = 'orders';
+
+const importPath = (remote: string) => `/datasources/${DATASOURCE}/external/tables/${remote}/import`;
+
+function recordsOf(json: unknown): Array<Record<string, unknown>> {
+  const body = json as { records?: unknown[] } | undefined;
+  return (body?.records ?? []) as Array<Record<string, unknown>>;
+}
+
+async function boot(databaseFile: string): Promise<VerifyStack> {
+  return bootStack(showcaseStack, { databaseFile, extraPlugins: [new ExternalDatasourceServicePlugin()] });
+}
+
+describe('Import as Object persists like the metadata door (showcase, cold boot)', () => {
+  let stack: VerifyStack;
+  let token: string;
+  let prevCwd: string;
+  let dir: string;
+  let dbFile: string;
+
+  const call = async (method: string, path: string, body?: unknown) => {
+    const res = await stack.apiAs(token, method, path, body);
+    const json: unknown = await res.json().catch(() => ({}));
+    return { status: res.status, json };
+  };
+
+  beforeAll(async () => {
+    prevCwd = process.cwd();
+    dir = mkdtempSync(join(tmpdir(), 'dogfood-21788-'));
+    process.chdir(dir);
+    dbFile = join(dir, 'showcase.db');
+    // Provision the remote tables, exactly as `os dev` does at boot (the
+    // harness imports only the stack's default export, so `onEnable` never
+    // runs on its own).
+    await onEnable({ logger: { info() {}, warn() {} } } as never);
+    stack = await boot(dbFile);
+    token = await stack.signIn();
+  }, 180_000);
+
+  afterAll(async () => {
+    await stack?.stop();
+    if (prevCwd) process.chdir(prevCwd);
+    if (dir) rmSync(dir, { recursive: true, force: true });
+  });
+
+  it('an object imported under a name that differs from its remote table serves the remote rows', async () => {
+    const imported = await call('POST', importPath('customers'), { name: RENAMED });
+    expect(imported.status, JSON.stringify(imported.json)).toBe(201);
+
+    const read = await call('GET', `/data/${RENAMED}`);
+    expect(read.status, JSON.stringify(read.json)).toBe(200);
+    expect(recordsOf(read.json).map((r) => r.name)).toEqual(
+      expect.arrayContaining(['Aurora Labs', 'Borealis GmbH', 'Cyan Pacific']),
+    );
+  });
+
+  it('control: an object imported under its remote table\'s own name serves the remote rows', async () => {
+    const imported = await call('POST', importPath('orders'), { name: SAME_NAME });
+    expect(imported.status, JSON.stringify(imported.json)).toBe(201);
+
+    const read = await call('GET', `/data/${SAME_NAME}`);
+    expect(read.status, JSON.stringify(read.json)).toBe(200);
+    expect(recordsOf(read.json)).toHaveLength(4);
+  });
+
+  it('after a cold boot on the same database file, both imported objects still serve their rows', async () => {
+    await stack.stop();
+    stack = await boot(dbFile);
+    token = await stack.signIn();
+
+    // Independent facts about one restart, so each is reported on its own.
+    const renamed = await call('GET', `/data/${RENAMED}`);
+    expect.soft(renamed.status, JSON.stringify(renamed.json)).toBe(200);
+    expect.soft(recordsOf(renamed.json)).toHaveLength(3);
+
+    const sameName = await call('GET', `/data/${SAME_NAME}`);
+    expect.soft(sameName.status, JSON.stringify(sameName.json)).toBe(200);
+    expect.soft(recordsOf(sameName.json)).toHaveLength(4);
+  }, 180_000);
+});

packages/qa/dogfood/vitest.config.ts

@@ -264,6 +264,15 @@ export default defineConfig({
               find: /^@objectstack\/cloud-connection$/,
               replacement: path.resolve(__dirname, '../../cloud-connection/src/index.ts'),
             },
+            // [#21788] `external-import-saves-like-meta.dogfood.test.ts` mounts
+            // `ExternalDatasourceServicePlugin` on a real boot (the harness
+            // does not) and drives the import door. The plugin's save path is
+            // the pin's subject, so the verdict is aliased to THIS checkout's
+            // source, not to the last `pnpm build`.
+            {
+              find: /^@objectstack\/service-datasource$/,
+              replacement: path.resolve(__dirname, '../../services/service-datasource/src/index.ts'),
+            },
           ],
         },
         test: {

packages/services/service-datasource/src/__tests__/external-import-saves-through-metadata-door.test.ts

@@ -0,0 +1,152 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#21788] "Import as Object" saves through the metadata door's own save.
+ *
+ * ## The defect this pins closed
+ *
+ * `ExternalDatasourceServicePlugin` wired `persistObject` to
+ * `metadata.register('object', name, definition)`. That put the definition in
+ * the metadata service's memory and nothing else: no `sys_metadata` row, no
+ * engine schema sync, no external-object registration with the driver. So an
+ * object imported under a name that differs from its remote table answered
+ * `201` and then `500 no such table: <name>` (the SQL driver resolved the
+ * table by the object's name), and every import was gone after a restart.
+ * `PUT /api/v1/meta/object/:name` with the same body was durable and served
+ * the rows, because it saves through `saveMetaItem` on the `'protocol'`
+ * service — the one save that persists the row, writes it through to the
+ * engine registry and syncs the object's storage (`syncObjectSchema`, which
+ * for a federated object maps the remote table).
+ *
+ * ## What each case pins
+ *
+ *  - the import reaches `saveMetaItem` with the request the metadata door
+ *    sends for an `object` (env-wide: `object` is not org-overridable), and
+ *    registers through no second path;
+ *  - the save door is resolved when the import runs, not at `init()` — a
+ *    protocol registered after this plugin still receives the save;
+ *  - a save the door refuses refuses the import with the door's own error;
+ *  - with no save door at all, the import is refused before any remote
+ *    introspection and nothing is saved.
+ *
+ * The plugin is imported through a RELATIVE specifier, so these cases measure
+ * `src/` and need no build. The booted-stack half (import, read rows, restart,
+ * read again) is the dogfood pin's.
+ */
+
+import { describe, it, expect, vi } from 'vitest';
+import type { PluginContext } from '@objectstack/core';
+import type { IntrospectedSchema, IExternalDatasourceService } from '@objectstack/spec/contracts';
+import { ExternalDatasourceServicePlugin } from '../plugin.js';
+
+const remoteSchema = (): IntrospectedSchema =>
+  ({
+    dialect: 'sqlite',
+    tables: {
+      customers: {
+        name: 'customers',
+        columns: [
+          { name: 'id', type: 'text', nullable: false, primaryKey: true },
+          { name: 'name', type: 'text', nullable: true, primaryKey: false },
+        ],
+      },
+    },
+  }) as unknown as IntrospectedSchema;
+
+interface Harness {
+  ctx: PluginContext;
+  services: Map<string, unknown>;
+  introspect: ReturnType<typeof vi.fn>;
+  register: ReturnType<typeof vi.fn>;
+}
+
+/** A kernel context whose `getService` throws on an unregistered name, as the kernel's does. */
+function harness(): Harness {
+  const services = new Map<string, unknown>();
+  const introspect = vi.fn(async () => remoteSchema());
+  const register = vi.fn(async () => undefined);
+  services.set('data', { introspectDatasource: introspect });
+  services.set('metadata', {
+    get: async (type: string, name: string) =>
+      type === 'datasource' ? { name, schemaMode: 'external' } : undefined,
+    register,
+  });
+  const ctx = {
+    getService: (name: string) => {
+      if (!services.has(name)) throw new Error(`Service '${name}' not found`);
+      return services.get(name);
+    },
+    registerService: (name: string, service: unknown) => {
+      services.set(name, service);
+    },
+    trigger: async () => undefined,
+    logger: { info() {}, warn() {}, error() {}, debug() {} },
+  } as unknown as PluginContext;
+  return { ctx, services, introspect, register };
+}
+
+async function federation(h: Harness): Promise<IExternalDatasourceService> {
+  await new ExternalDatasourceServicePlugin().init(h.ctx);
+  return h.services.get('external-datasource') as IExternalDatasourceService;
+}
+
+describe('importObject saves through the metadata door (#21788)', () => {
+  it('reaches saveMetaItem with the metadata door\'s object request, and registers through no second path', async () => {
+    const h = harness();
+    const saveMetaItem = vi.fn(async () => ({ success: true }));
+    h.services.set('protocol', { saveMetaItem });
+
+    const result = await (await federation(h)).importObject('warehouse', 'customers', { name: 'ext_cust' });
+
+    expect(saveMetaItem).toHaveBeenCalledTimes(1);
+    expect(saveMetaItem).toHaveBeenCalledWith({ type: 'object', name: 'ext_cust', item: result.definition });
+    expect(result.definition).toMatchObject({
+      name: 'ext_cust',
+      datasource: 'warehouse',
+      external: { remoteName: 'customers' },
+    });
+    expect(h.register).not.toHaveBeenCalled();
+  });
+
+  it('resolves the save door when the import runs, so a protocol registered after init still receives it', async () => {
+    const h = harness();
+    const service = await federation(h);
+    const saveMetaItem = vi.fn(async () => ({ success: true }));
+    h.services.set('protocol', { saveMetaItem });
+
+    await service.importObject('warehouse', 'customers', { name: 'ext_cust' });
+
+    expect(saveMetaItem).toHaveBeenCalledTimes(1);
+    expect(h.register).not.toHaveBeenCalled();
+  });
+
+  it('refuses the import with the door\'s own refusal when the save is refused', async () => {
+    const h = harness();
+    const refusal = Object.assign(new Error('object/ext_cust failed validation'), {
+      code: 'INVALID_METADATA',
+      status: 422,
+    });
+    h.services.set('protocol', { saveMetaItem: vi.fn(async () => { throw refusal; }) });
+
+    const outcome = await (await federation(h))
+      .importObject('warehouse', 'customers', { name: 'ext_cust' })
+      .catch((e: unknown) => e);
+
+    expect(outcome).toBe(refusal);
+    expect(outcome).toMatchObject({ code: 'INVALID_METADATA', status: 422 });
+    expect(h.register).not.toHaveBeenCalled();
+  });
+
+  it('with no save door, refuses before any remote introspection and saves nothing', async () => {
+    const h = harness();
+
+    const outcome = await (await federation(h))
+      .importObject('warehouse', 'customers', { name: 'ext_cust' })
+      .catch((e: unknown) => e);
+
+    expect(outcome).toBeInstanceOf(Error);
+    expect((outcome as Error).message).toMatch(/requires a writable metadata store/);
+    expect(h.introspect).not.toHaveBeenCalled();
+    expect(h.register).not.toHaveBeenCalled();
+  });
+});