Repository navigation
Commit 25eb7de
fix(service-datasource): external validate sees a federated object saved at runtime, with no restart (#21875)
Fixes #21842
Clause-②: no
## What this changes
`ExternalDatasourceServicePlugin` wired the federation service's
`listObjects` and `getObject` to the `metadata` service. That service
holds a copy of the engine's object registry, taken once at boot by
`ObjectQLPlugin`'s startup bridge. `PUT /api/v1/meta/object/:name`, and
the import that saves through it since #21837, write `sys_metadata` and
the engine registry (`applyRegistryWriteThrough`), never that copy. So
`POST /api/v1/datasources/:name/external/validate` did not see a
runtime-saved object until the next restart.
Both object reads now come from the engine's object registry on the
`objectql` service (`IObjectQLEngine.registry`, its `getAllObjects` and
`getObject`). The registry is looked up when validation runs, never at
`init()` (AGENTS.md, "Startup registry reads"), the same way the
import's save door is. There is no second copy and no hand refresh. The
service's comparison is untouched, and datasource definitions, the
package namespace and the catalog write still go through the `metadata`
service exactly as before (see **Decision needed** below).
Files: `packages/services/service-datasource/src/plugin.ts`, one new
unit file beside it, one new dogfood file, and a `patch` changeset. No
`packages/spec` edit, no new export, no `content/docs` sentence changed
(none describes where the validate door reads its objects).
## Measured on the real composition (`objectstack dev`, showcase)
| step | base `2df3d13d` | this branch |
|:--|:--|:--|
| validate `showcase_external`, before any save | 2 rows,
`showcase_ext_customer` and `showcase_ext_order`, both ok | same |
| `PUT /meta/object/dg21842_saved` (bound to remote `customers`, fields
`name`, `email`, `ghost_col`), then validate | 200, then still 2 rows |
200, then 3 rows; `dg21842_saved` is `ok: false` with one diff,
`missing_column ghost_col` |
| import remote `orders` as `dg21842_imported`, then validate | 201,
then still 2 rows | 201, then 4 rows; `dg21842_imported` ok |
| re-save `dg21842_saved` without `ghost_col`, then validate | not run |
`dg21842_saved` ok, no diffs |
| restart on the same database, then validate | 4 rows | the same 4
rows, the same verdicts |
The injected anchors (`organization_id`, the audit pair, `owner_id`,
`owning_business_unit_id`) are not reported on the runtime-saved object:
the registry's copy carries them, and the #21837 skip handles them.
**H2 (does `getObject` already see the save?): falsified.** With only
`listObjects` moved (an intermediate build), the saved and imported rows
answered `unreachable` with `Object 'dg21842_saved' not found.` and
`Object 'dg21842_imported' not found.`. The metadata service's
`getObject` reads the same boot copy, so both reads moved.
**H3 (the boot gate), on `objectstack dev`: holds.** Base and branch
give the same boot gate output. On a fresh boot both log "all federated
objects match their remote schema" with `objects: 2`. Booted on copies
of one database that holds the stored objects, both log the same single
drift warning (`dg21842_saved`, `missing_column ghost_col`). The bridge
logs 109 of 109 registry objects copied, so at boot the copy and the
registry hold the same objects.
## Decision needed: `objectstack start`
Measured on `objectstack start` (production mode), showcase:
- **Base:** the boot gate logs `objects: 0`, and validate answers `{ ok:
true, results: [] }`. The code-defined federated objects are not listed
at all. The plugin reads the `metadata` service at `init()`, and on
`start` that service is the kernel's in-memory fallback, registered just
before the start phase and after this plugin's `init()` (the log shows
`Service 'external-datasource' registered`, then `Service 'metadata'
registered`, then `Phase 2: Start plugins`). So the plugin holds no
metadata service for its whole life.
- **This branch:** objects are listed now, because the registry is read
when validation runs. The datasource definition is still read from the
service captured at `init()`, which is absent, so `validateObject` takes
its "not federated" branch. Every row answers `ok: true` with no diffs,
and nothing is compared. The boot gate logs `objects: 2`. A saved object
with `ghost_col` answered `ok: true`.
The boot gate's pass or abort does not move on either composition, but
on `start` this branch turns "no rows" into rows that claim `ok` without
a comparison. That is close to this dispatch's stop line, so the landing
is the seat's call:
- **A.** Land as is, and the init-time `metadata` capture becomes its
own card. Cost: until that card lands, `start` answers per-object `ok:
true` that it never compared.
- **B.** Widen this PR: read the `metadata` service (datasource,
namespace, catalog write) when it is used. Cost: on `start` the boot
gate starts judging, so a deployment with drift under the default
`onMismatch: 'fail'` refuses to boot where it started before. That is a
boot-behaviour change outside this card.
- **C.** The init-time capture becomes its own card and lands first;
this PR then lands unchanged. Cost: this PR waits.
Recommendation: **C**. Each landing stays honest on every composition.
The boot-behaviour change gets its own changeset and its own decision,
and this PR's diff and changeset stay as they are.
## Tests
- New
`packages/services/service-datasource/src/__tests__/external-validate-reads-live-registry.test.ts`
(4 cases, relative import, measures `src/`). The fakes model the
measured mechanism: `metadata` holds the boot copy, the `objectql`
registry is live, and a save writes only the registry. A runtime-saved
object is listed and judged by `validateDatasource` and by
`validateAll`, and the code-defined one is still listed. A re-save is
judged on what was saved. The registry is resolved when validation runs.
The boot copy's object reads are never called.
- New
`packages/qa/dogfood/test/external-validate-sees-runtime-save.dogfood.test.ts`
(3 cases, booted showcase): validate lists the code-defined objects,
then also an object saved through `PUT /meta/object/:name`, then also an
imported one. Under this harness the federation service finds no
`metadata` service at `init()` (the same cause as `start`), so this file
pins the listing only. The verdict half is the unit file's, and was
measured on `objectstack dev` above.
- At `b44c1c87bb` (after merging `origin/main`):
`@objectstack/service-datasource` 38 files, 721 tests pass. `tsc
--noEmit` passes, and `--listFiles` includes the new test file. Dogfood:
the new file and `external-import-saves-like-meta.dogfood.test.ts`, 2
files, 6 tests pass.
- Ablation, with the fix committed (`94e3056d62`), through
`scripts/ablation-replace.mjs`: both readers were put back to the
boot-copy reads (anchor hit 1, blob `5fd02852` to `cad2f3a6`). Unit: 4
of 4 failed, for example `expected [ 'code_cust' ] to deeply equal [
'code_cust', 'saved_cust' ]`. Dogfood: 3 of 3 failed, `expected [] to
deeply equal [ 'showcase_ext_customer', … ]`. Under the harness all
three read `[]`, because there is no metadata service at `init()` there;
the unit file is what separates "boot copy" from "live registry".
Restore: blob equals HEAD (`5fd02852`), and `git diff HEAD` is empty. No
`dist/` is on either path: the unit file imports `src/` relatively, and
the dogfood config aliases `@objectstack/service-datasource` to `src/`.
## Gates
At `c68487ae6c` (this head; it differs from `b44c1c87bb` by the
changeset text only): `node scripts/pm/dispatch-gates.mjs --commands
--repo objectstack-ai/objectstack` derives 67 commands. All 67 were run
and all exit 0, and `--ran` reconciles 67 run, 0 NOT-MEASURED, 0 UNRUN.
That is the dispatch's 59 plus 8 the changeset brings
(`check-adr-0087-registration` and `check-empty-changeset`, each with
`--self-test`, `release-rehearsal-clone --self-test`,
`release-pending-publish --self-test`, `check:objectui-changeset`,
`check:pm-changeset-deadline-census`). `check:dual-build-cjs-loads`
first answered PREREQUISITE NOT MET (8 unrelated packages had no
`dist/`). Those packages were built, and it was re-run to exit 0.
Narrowed eslint (`--no-inline-config`, `--format json`) on the 3 touched
`.ts` files: 3 files, 0 errors, 0 warnings. The config enables no
type-aware linting (no `parserOptions.project`), so this diff cannot
move the verdict on any untouched file.
## Acceptance notes
- The boot gate's completeness probe (`announceAllClear`,
`packages/runtime`) still asks `metadata.listDiagnosed('object')`, a
list the sweep no longer reads. It only shapes the all-clear sentence,
never a verdict. Carrier: none.
- `persistCatalog` and `getNamespace` read the same init-time `metadata`
capture as the datasource read above. This was already noted on #21837.
- `origin/main` was merged at `b44c1c87bb`. #21841 has not landed, and
nothing on `main` since the base touches `service-datasource`.
## Seat's append: patch round 1 (written by `domain:services` seat 1
from the dev's report `5999337391`; the dev does not edit this body)
### Patch round 1 (head `9489265ae0`): `main` merged after #21887, and
`objectstack start` measured again
**The merge.** `origin/main` `607463d736` was merged as `80dfcc30b7`. It
carries #21887 (merge `bc7747cb`) and #21874. There was one conflict, in
`plugin.ts`'s object readers. Resolution: `getObject` and `listObjects`
read the engine registry (`objectRegistry()`). `getDatasource`,
`getNamespace` and the `persistCatalog` getter keep #21887's resolver,
`metadata()`. `MetadataServiceLike` keeps only `get` and `register`. One
sentence of the resolver's docblock said object reads went through it.
It now says objects come from the registry.
**#21887's unit file moved with it (`9489265ae0`).** At the merge
commit, its 4 object-listing cases went red (`expected [] to deeply
equal [ 'wh_customer', 'wh_order' ]`), because its harness served
objects only from the metadata fake. The harness now also serves them
from an `objectql` registry fake. The re-ask case tells the two services
apart by the datasource definition: a `managed` replacement compares
nothing. The no-metadata case also runs with no registry. Every other
case is unchanged.
**The door pin now asserts the verdict.** Under the harness the metadata
service is read when it is used (#21887), so the runtime-saved object's
row is a real comparison. It answers `ok: false` with one diff,
`missing_column loyalty_tier`. The imported row answers `ok` with no
diffs.
**`objectstack start` (showcase): this branch against `origin/main`
`607463d736` as the control, with the same steps.**
| step | `origin/main` | this branch |
|:--|:--|:--|
| fresh boot, boot gate | `all federated objects match their remote
schema {"objects":2}` | same |
| `PUT /meta/object/dg21842_saved` (declares `loyalty_tier`, which the
remote `customers` table lacks), then validate | 2 rows; the saved
object is not listed | 3 rows; `dg21842_saved` is `ok: false` with
`missing_column loyalty_tier` |
| import `orders` as `dg21842_imported`, then validate | 2 rows | 4
rows; `dg21842_imported` is `ok: true` with no diffs |
| restart on the same home, boot gate | one `external schema drift`
warn: `dg21842_saved`, `missing_column loyalty_tier` | same |
The question in this PR's **Decision needed** section is closed: on
`start`, validate judges a runtime-saved object, and no row answers `ok`
without a comparison. The seat decided option C (`5995103062`), and
#21876 landed first as PR #21887. The boot gate output matches #21887's
on both the fresh boot and the restart.
**Measured at `9489265ae0`:**
- `@objectstack/service-datasource`: 39 files, 732 tests pass. Typecheck
passes, and `--listFiles` includes both unit files.
- Dogfood typecheck passes. The three door pins (this one, #21887's
`external-validate-start-ordering` and #21788's
`external-import-saves-like-meta`): 3 files, 9 tests pass.
- Ablation, both object readers put back to `metadata()` through
`scripts/ablation-replace.mjs`: the unit file went 4 of 4 red. The door
pin went 2 of 3 red: the saved and imported cases failed, and the
code-defined listing stayed green because the metadata service now
answers it. The restore proved the blob equal to HEAD.
- Gates: 67 derived, all exit 0. `--ran`: 67 run, 0 NOT-MEASURED, 0
UNRUN.
---
_Generated by [Claude
Code](https://claude.ai/code/session_011K3zqE8Pv1Evw5hc8tZCnN)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 1e18a07 commit 25eb7de
5 files changed
Lines changed: 419 additions & 33 deletions
File tree
- .changeset
- packages
- qa/dogfood/test
- services/service-datasource/src
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
Lines changed: 150 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
Lines changed: 28 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
141 | 149 | | |
142 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
143 | 157 | | |
144 | 158 | | |
145 | 159 | | |
| |||
257 | 271 | | |
258 | 272 | | |
259 | 273 | | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
270 | 282 | | |
271 | | - | |
| 283 | + | |
272 | 284 | | |
273 | 285 | | |
274 | 286 | | |
| |||
292 | 304 | | |
293 | 305 | | |
294 | 306 | | |
295 | | - | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
296 | 310 | | |
297 | 311 | | |
298 | 312 | | |
| |||
0 commit comments