Skip to content

Commit 25eb7de

Browse files
fix(service-datasource): external validate sees a federated object saved at runtime, with no restart (#21875)
Fixes #21842 Clause-②: no ## What this changes `ExternalDatasourceServicePlugin` wired the federation service's `listObjects` and `getObject` to the `metadata` service. That service holds a copy of the engine's object registry, taken once at boot by `ObjectQLPlugin`'s startup bridge. `PUT /api/v1/meta/object/:name`, and the import that saves through it since #21837, write `sys_metadata` and the engine registry (`applyRegistryWriteThrough`), never that copy. So `POST /api/v1/datasources/:name/external/validate` did not see a runtime-saved object until the next restart. Both object reads now come from the engine's object registry on the `objectql` service (`IObjectQLEngine.registry`, its `getAllObjects` and `getObject`). The registry is looked up when validation runs, never at `init()` (AGENTS.md, "Startup registry reads"), the same way the import's save door is. There is no second copy and no hand refresh. The service's comparison is untouched, and datasource definitions, the package namespace and the catalog write still go through the `metadata` service exactly as before (see **Decision needed** below). Files: `packages/services/service-datasource/src/plugin.ts`, one new unit file beside it, one new dogfood file, and a `patch` changeset. No `packages/spec` edit, no new export, no `content/docs` sentence changed (none describes where the validate door reads its objects). ## Measured on the real composition (`objectstack dev`, showcase) | step | base `2df3d13d` | this branch | |:--|:--|:--| | validate `showcase_external`, before any save | 2 rows, `showcase_ext_customer` and `showcase_ext_order`, both ok | same | | `PUT /meta/object/dg21842_saved` (bound to remote `customers`, fields `name`, `email`, `ghost_col`), then validate | 200, then still 2 rows | 200, then 3 rows; `dg21842_saved` is `ok: false` with one diff, `missing_column ghost_col` | | import remote `orders` as `dg21842_imported`, then validate | 201, then still 2 rows | 201, then 4 rows; `dg21842_imported` ok | | re-save `dg21842_saved` without `ghost_col`, then validate | not run | `dg21842_saved` ok, no diffs | | restart on the same database, then validate | 4 rows | the same 4 rows, the same verdicts | The injected anchors (`organization_id`, the audit pair, `owner_id`, `owning_business_unit_id`) are not reported on the runtime-saved object: the registry's copy carries them, and the #21837 skip handles them. **H2 (does `getObject` already see the save?): falsified.** With only `listObjects` moved (an intermediate build), the saved and imported rows answered `unreachable` with `Object 'dg21842_saved' not found.` and `Object 'dg21842_imported' not found.`. The metadata service's `getObject` reads the same boot copy, so both reads moved. **H3 (the boot gate), on `objectstack dev`: holds.** Base and branch give the same boot gate output. On a fresh boot both log "all federated objects match their remote schema" with `objects: 2`. Booted on copies of one database that holds the stored objects, both log the same single drift warning (`dg21842_saved`, `missing_column ghost_col`). The bridge logs 109 of 109 registry objects copied, so at boot the copy and the registry hold the same objects. ## Decision needed: `objectstack start` Measured on `objectstack start` (production mode), showcase: - **Base:** the boot gate logs `objects: 0`, and validate answers `{ ok: true, results: [] }`. The code-defined federated objects are not listed at all. The plugin reads the `metadata` service at `init()`, and on `start` that service is the kernel's in-memory fallback, registered just before the start phase and after this plugin's `init()` (the log shows `Service 'external-datasource' registered`, then `Service 'metadata' registered`, then `Phase 2: Start plugins`). So the plugin holds no metadata service for its whole life. - **This branch:** objects are listed now, because the registry is read when validation runs. The datasource definition is still read from the service captured at `init()`, which is absent, so `validateObject` takes its "not federated" branch. Every row answers `ok: true` with no diffs, and nothing is compared. The boot gate logs `objects: 2`. A saved object with `ghost_col` answered `ok: true`. The boot gate's pass or abort does not move on either composition, but on `start` this branch turns "no rows" into rows that claim `ok` without a comparison. That is close to this dispatch's stop line, so the landing is the seat's call: - **A.** Land as is, and the init-time `metadata` capture becomes its own card. Cost: until that card lands, `start` answers per-object `ok: true` that it never compared. - **B.** Widen this PR: read the `metadata` service (datasource, namespace, catalog write) when it is used. Cost: on `start` the boot gate starts judging, so a deployment with drift under the default `onMismatch: 'fail'` refuses to boot where it started before. That is a boot-behaviour change outside this card. - **C.** The init-time capture becomes its own card and lands first; this PR then lands unchanged. Cost: this PR waits. Recommendation: **C**. Each landing stays honest on every composition. The boot-behaviour change gets its own changeset and its own decision, and this PR's diff and changeset stay as they are. ## Tests - New `packages/services/service-datasource/src/__tests__/external-validate-reads-live-registry.test.ts` (4 cases, relative import, measures `src/`). The fakes model the measured mechanism: `metadata` holds the boot copy, the `objectql` registry is live, and a save writes only the registry. A runtime-saved object is listed and judged by `validateDatasource` and by `validateAll`, and the code-defined one is still listed. A re-save is judged on what was saved. The registry is resolved when validation runs. The boot copy's object reads are never called. - New `packages/qa/dogfood/test/external-validate-sees-runtime-save.dogfood.test.ts` (3 cases, booted showcase): validate lists the code-defined objects, then also an object saved through `PUT /meta/object/:name`, then also an imported one. Under this harness the federation service finds no `metadata` service at `init()` (the same cause as `start`), so this file pins the listing only. The verdict half is the unit file's, and was measured on `objectstack dev` above. - At `b44c1c87bb` (after merging `origin/main`): `@objectstack/service-datasource` 38 files, 721 tests pass. `tsc --noEmit` passes, and `--listFiles` includes the new test file. Dogfood: the new file and `external-import-saves-like-meta.dogfood.test.ts`, 2 files, 6 tests pass. - Ablation, with the fix committed (`94e3056d62`), through `scripts/ablation-replace.mjs`: both readers were put back to the boot-copy reads (anchor hit 1, blob `5fd02852` to `cad2f3a6`). Unit: 4 of 4 failed, for example `expected [ 'code_cust' ] to deeply equal [ 'code_cust', 'saved_cust' ]`. Dogfood: 3 of 3 failed, `expected [] to deeply equal [ 'showcase_ext_customer', … ]`. Under the harness all three read `[]`, because there is no metadata service at `init()` there; the unit file is what separates "boot copy" from "live registry". Restore: blob equals HEAD (`5fd02852`), and `git diff HEAD` is empty. No `dist/` is on either path: the unit file imports `src/` relatively, and the dogfood config aliases `@objectstack/service-datasource` to `src/`. ## Gates At `c68487ae6c` (this head; it differs from `b44c1c87bb` by the changeset text only): `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derives 67 commands. All 67 were run and all exit 0, and `--ran` reconciles 67 run, 0 NOT-MEASURED, 0 UNRUN. That is the dispatch's 59 plus 8 the changeset brings (`check-adr-0087-registration` and `check-empty-changeset`, each with `--self-test`, `release-rehearsal-clone --self-test`, `release-pending-publish --self-test`, `check:objectui-changeset`, `check:pm-changeset-deadline-census`). `check:dual-build-cjs-loads` first answered PREREQUISITE NOT MET (8 unrelated packages had no `dist/`). Those packages were built, and it was re-run to exit 0. Narrowed eslint (`--no-inline-config`, `--format json`) on the 3 touched `.ts` files: 3 files, 0 errors, 0 warnings. The config enables no type-aware linting (no `parserOptions.project`), so this diff cannot move the verdict on any untouched file. ## Acceptance notes - The boot gate's completeness probe (`announceAllClear`, `packages/runtime`) still asks `metadata.listDiagnosed('object')`, a list the sweep no longer reads. It only shapes the all-clear sentence, never a verdict. Carrier: none. - `persistCatalog` and `getNamespace` read the same init-time `metadata` capture as the datasource read above. This was already noted on #21837. - `origin/main` was merged at `b44c1c87bb`. #21841 has not landed, and nothing on `main` since the base touches `service-datasource`. ## Seat's append: patch round 1 (written by `domain:services` seat 1 from the dev's report `5999337391`; the dev does not edit this body) ### Patch round 1 (head `9489265ae0`): `main` merged after #21887, and `objectstack start` measured again **The merge.** `origin/main` `607463d736` was merged as `80dfcc30b7`. It carries #21887 (merge `bc7747cb`) and #21874. There was one conflict, in `plugin.ts`'s object readers. Resolution: `getObject` and `listObjects` read the engine registry (`objectRegistry()`). `getDatasource`, `getNamespace` and the `persistCatalog` getter keep #21887's resolver, `metadata()`. `MetadataServiceLike` keeps only `get` and `register`. One sentence of the resolver's docblock said object reads went through it. It now says objects come from the registry. **#21887's unit file moved with it (`9489265ae0`).** At the merge commit, its 4 object-listing cases went red (`expected [] to deeply equal [ 'wh_customer', 'wh_order' ]`), because its harness served objects only from the metadata fake. The harness now also serves them from an `objectql` registry fake. The re-ask case tells the two services apart by the datasource definition: a `managed` replacement compares nothing. The no-metadata case also runs with no registry. Every other case is unchanged. **The door pin now asserts the verdict.** Under the harness the metadata service is read when it is used (#21887), so the runtime-saved object's row is a real comparison. It answers `ok: false` with one diff, `missing_column loyalty_tier`. The imported row answers `ok` with no diffs. **`objectstack start` (showcase): this branch against `origin/main` `607463d736` as the control, with the same steps.** | step | `origin/main` | this branch | |:--|:--|:--| | fresh boot, boot gate | `all federated objects match their remote schema {"objects":2}` | same | | `PUT /meta/object/dg21842_saved` (declares `loyalty_tier`, which the remote `customers` table lacks), then validate | 2 rows; the saved object is not listed | 3 rows; `dg21842_saved` is `ok: false` with `missing_column loyalty_tier` | | import `orders` as `dg21842_imported`, then validate | 2 rows | 4 rows; `dg21842_imported` is `ok: true` with no diffs | | restart on the same home, boot gate | one `external schema drift` warn: `dg21842_saved`, `missing_column loyalty_tier` | same | The question in this PR's **Decision needed** section is closed: on `start`, validate judges a runtime-saved object, and no row answers `ok` without a comparison. The seat decided option C (`5995103062`), and #21876 landed first as PR #21887. The boot gate output matches #21887's on both the fresh boot and the restart. **Measured at `9489265ae0`:** - `@objectstack/service-datasource`: 39 files, 732 tests pass. Typecheck passes, and `--listFiles` includes both unit files. - Dogfood typecheck passes. The three door pins (this one, #21887's `external-validate-start-ordering` and #21788's `external-import-saves-like-meta`): 3 files, 9 tests pass. - Ablation, both object readers put back to `metadata()` through `scripts/ablation-replace.mjs`: the unit file went 4 of 4 red. The door pin went 2 of 3 red: the saved and imported cases failed, and the code-defined listing stayed green because the metadata service now answers it. The restore proved the blob equal to HEAD. - Gates: 67 derived, all exit 0. `--ran`: 67 run, 0 NOT-MEASURED, 0 UNRUN. --- _Generated by [Claude Code](https://claude.ai/code/session_011K3zqE8Pv1Evw5hc8tZCnN)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1e18a07 commit 25eb7de

5 files changed

Lines changed: 419 additions & 33 deletions

File tree

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
'@objectstack/service-datasource': patch
3+
---
4+
5+
fix(service-datasource): `POST /api/v1/datasources/:name/external/validate` sees a federated object saved at runtime, with no restart (#21842)
6+
7+
Clause-②: no
8+
9+
- **What was wrong.** The federation service read its objects from the `metadata` service. That service holds a copy of the engine's object registry taken once at boot. `PUT /api/v1/meta/object/:name`, and the external-table import that saves through it, write `sys_metadata` and the engine registry, but never that copy. So after a federated object was saved at runtime, validate answered the code-defined objects only, and listed the saved one after a restart. An object re-saved at runtime was judged on its definition as it stood at boot.
10+
- **What it reads now.** `ExternalDatasourceServicePlugin` reads objects (`listObjects` and `getObject`) from the engine's object registry on the `objectql` service, which is the registry the save writes through to. The registry is looked up when validation runs, not when the plugin starts. A saved or imported object is listed and judged on what was saved, the moment the save answers.
11+
- **What does not move.** The comparison is unchanged: the same federation predicate, the same column and type checks, and datasource definitions read from the same place. On `objectstack dev` the boot validation gate sweeps the same objects with the same verdicts as before. No route's request or response shape changes, and no export is added.
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// [#21842, ADR-0015 §6] `POST /datasources/:name/external/validate` lists a
4+
// federated object the moment it is saved at runtime, with no restart, over
5+
// the real showcase composition.
6+
//
7+
// ## What was broken
8+
//
9+
// The federation service read its objects from the `metadata` service, which
10+
// holds a copy of the engine's object registry taken once at boot.
11+
// `PUT /meta/object/:name`, and the external-table import that saves through
12+
// it, write `sys_metadata` and the engine registry, never that copy. Measured
13+
// on the showcase (`objectstack dev`): after a federated object was saved on
14+
// `showcase_external`, validate still answered the two code-defined objects
15+
// only, and listed the saved one after a restart. The service now reads the
16+
// engine registry, resolved when validation runs.
17+
//
18+
// ## What this file pins
19+
//
20+
// Through the doors an operator uses: the code-defined objects are listed;
21+
// an object saved through `PUT /meta/object/:name` is listed with a real
22+
// verdict (it declares a column the remote lacks, and exactly that column is
23+
// reported); an object imported through `…/external/tables/:remote/import` is
24+
// listed beside them. The verify harness composes no metadata plugin, so its
25+
// `metadata` service is the kernel's fallback registered after the
26+
// federation service's `init()`, as on `objectstack start`; the service reads
27+
// it when validation runs ([#21876]), so each row here is a real comparison.
28+
// The seam pins sit beside the plugin
29+
// (`packages/services/service-datasource/src/__tests__/external-validate-reads-live-registry.test.ts`).
30+
//
31+
// The working directory is a temporary one because the showcase's external
32+
// datasource and its fixture both name a cwd-relative SQLite file: this
33+
// file's remote database is its own, not one a parallel file writes.
34+
35+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
36+
import showcaseStack, { onEnable } from '@objectstack/example-showcase';
37+
import { ExternalDatasourceServicePlugin } from '@objectstack/service-datasource';
38+
import { bootStack, type VerifyStack } from '@objectstack/verify';
39+
import { mkdtempSync, rmSync } from 'node:fs';
40+
import { tmpdir } from 'node:os';
41+
import { join } from 'node:path';
42+
43+
/** The showcase's external datasource (`showcase-external.datasource.ts`). */
44+
const DATASOURCE = 'showcase_external';
45+
/** The two objects the showcase binds to it in code (`data/objects/external/`). */
46+
const CODE_DEFINED = ['showcase_ext_customer', 'showcase_ext_order'];
47+
/** Saved at runtime through the metadata door, bound to the remote `customers` table. */
48+
const SAVED = 'dogfood_ext_cust_21842';
49+
/** A column the saved object declares and the remote `customers` table does not have. */
50+
const MISSING = 'loyalty_tier';
51+
/** Imported at runtime from the remote `orders` table. */
52+
const IMPORTED = 'dogfood_ext_ord_21842';
53+
54+
const validatePath = `/datasources/${DATASOURCE}/external/validate`;
55+
56+
interface Row {
57+
object: string;
58+
ok: boolean;
59+
diffs: Array<{ kind: string; column?: string; severity: string }>;
60+
}
61+
62+
function rowsOf(json: unknown): Row[] {
63+
const body = json as { data?: { results?: Row[] } } | undefined;
64+
return body?.data?.results ?? [];
65+
}
66+
67+
function listedObjects(json: unknown): string[] {
68+
return rowsOf(json).map((r) => String(r.object)).sort();
69+
}
70+
71+
const rowOf = (json: unknown, object: string) => rowsOf(json).find((r) => r.object === object);
72+
73+
describe('external validate lists a federated object saved at runtime, with no restart (showcase)', () => {
74+
let stack: VerifyStack;
75+
let token: string;
76+
let prevCwd: string;
77+
let dir: string;
78+
79+
const call = async (method: string, path: string, body?: unknown) => {
80+
const res = await stack.apiAs(token, method, path, body);
81+
const json: unknown = await res.json().catch(() => ({}));
82+
return { status: res.status, json };
83+
};
84+
85+
beforeAll(async () => {
86+
prevCwd = process.cwd();
87+
dir = mkdtempSync(join(tmpdir(), 'dogfood-21842-'));
88+
process.chdir(dir);
89+
// Provision the remote tables, exactly as `os dev` does at boot (the
90+
// harness imports only the stack's default export, so `onEnable` never
91+
// runs on its own).
92+
await onEnable({ logger: { info() {}, warn() {} } } as never);
93+
stack = await bootStack(showcaseStack, {
94+
databaseFile: join(dir, 'showcase.db'),
95+
extraPlugins: [new ExternalDatasourceServicePlugin()],
96+
});
97+
token = await stack.signIn();
98+
}, 180_000);
99+
100+
afterAll(async () => {
101+
await stack?.stop();
102+
if (prevCwd) process.chdir(prevCwd);
103+
if (dir) rmSync(dir, { recursive: true, force: true });
104+
});
105+
106+
it('lists the code-defined objects bound to the datasource', async () => {
107+
const validated = await call('POST', validatePath);
108+
expect(validated.status, JSON.stringify(validated.json)).toBe(200);
109+
expect(listedObjects(validated.json)).toEqual(CODE_DEFINED);
110+
});
111+
112+
it('lists an object saved through PUT /meta/object/:name with a real verdict, and still the code-defined ones', async () => {
113+
const saved = await call('PUT', `/meta/object/${SAVED}`, {
114+
name: SAVED,
115+
label: 'Dogfood External Customer',
116+
sharingModel: 'public_read_write',
117+
datasource: DATASOURCE,
118+
external: { remoteName: 'customers' },
119+
fields: {
120+
name: { type: 'text', label: 'Name' },
121+
email: { type: 'text', label: 'Email' },
122+
[MISSING]: { type: 'text', label: 'Loyalty Tier' },
123+
},
124+
});
125+
expect(saved.status, JSON.stringify(saved.json)).toBe(200);
126+
127+
const validated = await call('POST', validatePath);
128+
expect(validated.status, JSON.stringify(validated.json)).toBe(200);
129+
expect(listedObjects(validated.json)).toEqual([...CODE_DEFINED, SAVED].sort());
130+
// Compared, not waved through: exactly the declared column the remote lacks.
131+
expect(rowOf(validated.json, SAVED)).toMatchObject({
132+
ok: false,
133+
diffs: [{ kind: 'missing_column', remoteName: 'customers', column: MISSING, severity: 'error' }],
134+
});
135+
for (const name of CODE_DEFINED) expect(rowOf(validated.json, name)?.ok, name).toBe(true);
136+
});
137+
138+
it('lists an object imported from a remote table, beside the saved and code-defined ones', async () => {
139+
const imported = await call('POST', `/datasources/${DATASOURCE}/external/tables/orders/import`, { name: IMPORTED });
140+
expect(imported.status, JSON.stringify(imported.json)).toBe(201);
141+
142+
const validated = await call('POST', validatePath);
143+
expect(validated.status, JSON.stringify(validated.json)).toBe(200);
144+
expect(listedObjects(validated.json)).toEqual([...CODE_DEFINED, IMPORTED, SAVED].sort());
145+
expect(rowOf(validated.json, IMPORTED), JSON.stringify(rowOf(validated.json, IMPORTED))).toMatchObject({
146+
ok: true,
147+
diffs: [],
148+
});
149+
});
150+
});

‎packages/services/service-datasource/src/__tests__/external-metadata-read-at-use.test.ts‎

Lines changed: 28 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -136,10 +136,24 @@ interface Harness {
136136
services: Map<string, unknown>;
137137
}
138138

139-
/** A kernel context whose `getService` throws on an unregistered name, as the kernel's does. */
140-
function harness(): Harness {
139+
/**
140+
* A kernel context whose `getService` throws on an unregistered name, as the kernel's does.
141+
*
142+
* [#21842] The federated objects live in the engine's object registry on the
143+
* `objectql` service, which is where the federation service reads objects
144+
* (the registry a runtime save writes through to). The `metadata` fake still
145+
* holds its own copy, as the kernel's fallback does after the startup bridge;
146+
* the readers this file pins are the ones that stay on the metadata service.
147+
*/
148+
function harness(opts: { registry?: boolean } = {}): Harness {
141149
const services = new Map<string, unknown>();
142150
services.set('data', { introspectDatasource: vi.fn(async () => remoteSchema()) });
151+
if (opts.registry !== false) {
152+
const objects = new Map<string, unknown>([[CUSTOMER.name, CUSTOMER], [ORDER.name, ORDER]]);
153+
services.set('objectql', {
154+
registry: { getObject: (n: string) => objects.get(n), getAllObjects: () => [...objects.values()] },
155+
});
156+
}
143157
const ctx = {
144158
getService: (name: string) => {
145159
if (!services.has(name)) throw new Error(`Service '${name}' not found`);
@@ -257,18 +271,16 @@ describe('the federation service reads a metadata service registered after init
257271

258272
it('asks for the service again at each use, never remembering the first answer', async () => {
259273
const { h, service } = await startOrdering();
260-
expect(objectsOf(await service.validateAll())).toEqual(['wh_customer', 'wh_order']);
261-
262-
const replacement = metadataFake();
263-
await replacement.register('object', 'wh_extra', {
264-
name: 'wh_extra',
265-
datasource: 'warehouse',
266-
external: { remoteName: 'orders' },
267-
fields: {},
268-
});
269-
h.services.set('metadata', replacement.service);
274+
const customerDiffs = async () =>
275+
(await service.validateAll()).results.find((r) => r.object === 'wh_customer')?.diffs;
276+
expect(await customerDiffs()).toEqual([expect.objectContaining({ kind: 'missing_column', column: 'email' })]);
277+
278+
// [#21842] Objects come from the engine registry, so the replacement is
279+
// told apart by the datasource definition it answers: a `managed`
280+
// datasource is not compared against its remote at all.
281+
h.services.set('metadata', metadataFake({ name: 'warehouse', schemaMode: 'managed' }).service);
270282

271-
expect(objectsOf(await service.validateAll())).toEqual(['wh_customer', 'wh_extra', 'wh_order']);
283+
expect(await customerDiffs()).toEqual([]);
272284
});
273285
});
274286

@@ -292,7 +304,9 @@ describe('control: a metadata service registered before init (the dev ordering)
292304

293305
describe('with no metadata service at all, every reader keeps its fallback', () => {
294306
it('validate and the sweep answer an empty report, the draft a bare name, the catalog an unpersisted snapshot', async () => {
295-
const service = await federation(harness());
307+
// [#21842] No engine registry either: that is where validate's objects
308+
// come from, so with neither service the report has nothing to list.
309+
const service = await federation(harness({ registry: false }));
296310

297311
expect(await service.validateAll()).toEqual({ ok: true, results: [] });
298312
expect((await service.generateObjectDraft('warehouse', 'customers')).name).toBe('customers');

0 commit comments

Comments
 (0)