fix(objectql)!: an unprojected read serves the declared fields, never an orphaned column

[objectstack-fleet]

Fixes #21571

Clause-②: no (narrowing)

An unprojected read now serves the object's declared fields plus the platform's system columns. A column no metadata declares, such as a field retired in an upgrade whose column additive sync leaves behind, no longer leaves the engine. The decision is made once, in the engine (packages/objectql), so every driver and every door gets the same answer. No driver source and no metadata-protocol source is edited.

Measured first (before any fix)

The probe was a temporary, env-gated recorder at the engine's read sites (commit 920cecf, reverted by a7500b3; the net diff carries none of it). In trim mode it deleted every undeclared key at find, findOne, the by-id update result and the update/delete prior reads, a superset of this fix.

suite (tree)	files	under trim	undeclared-key hits
objectql (a7ab047)	366	8 failed, 7374 passed	50
rest (a7ab047)	258	all pass	0
plugin-auth (a7ab047)	118	all pass	1 (a mock sys_account row)
service-automation, plugin-sharing, plugin-audit (a7ab047)	all	all pass	0
runtime (778bf5c, with #21545 merged)	318	all pass	0

The 8 objectql failures, by cause:

• 2 in engine.test.ts (平台形态的迁移门禁:带自检的数据迁移 + 部署级标记 —— file-as-reference 回收与 strict 翻转都依赖它 #3617): the sys_migration double declared only id. The production object declares blocking, verified_at and last_run_at, and readMigrationFlagVerified reads those. The fixture now declares what production declares.
• 5 in engine.test.ts / plugin.integration.test.ts: red only at the prior-read probe sites (a fixture's readonlyWhen reads an undeclared locked). This fix does not touch prior reads.
• 1 in driver-fault-boundary-redaction.test.ts: the mock driver hands back live references to its store, and an in-place delete corrupted it. So the fix never mutates a driver row: a row carrying an undeclared key is copied without it.

In-process readers of undeclared columns, by call site:

• os migrate plan's unmapped_column detection introspects the table through the driver (driver-sql/src/schema-drift.ts). Unaffected.
• os migrate account-issuer reads sys_account.issuer through the driver (cli/src/commands/migrate/account-issuer.ts). Unaffected.
• The os migrate apply account preflight (cli/src/commands/migrate/apply.ts, then plugin-auth account-identity-preflight.ts) calls engine.find with an explicit projection naming the retired issuer. The engine's unknown-plain filter already drops that name today. Unchanged by this PR (see Acceptance notes).
• The metadata column migrations (packages/metadata/src/migrations/*) use driver raw DDL. Unaffected.
• cloneData copies every key of its findOne source into an insert. Before this fix, an orphaned column made the clone fail at the write door (Unknown field 'mailing_street', INVALID_FIELD); it now succeeds. This reader was hurt by the orphaned column, it did not rely on it.
• Export, search, the RPC dispatcher, getData and findData all read through find/findOne. They narrow with the door, which is intended.
• hotcrm's scripts/backfill-contact-mailing-address.ts is external and reads through REST. The changeset names its interim route.

Hypothesis verdicts:

• H1 (confirmed): find reached the driver with fields undefined, and driver-sql answered with select('*').
• H2 (confirmed, and it decided the shape): on the composed REST harness, an explicit projection naming a declared field with no column fails, and driver-sql's ladder returns the whole row, retired column included (the reach pin's ladder case, red before the fix). So the engine shapes the rows the driver returns instead of pushing a projection down. That holds whichever rung answered, and needs no driver edit.
• H3: one authority, the registry field map plus PLATFORM_PROVISIONED_COLUMNS. That list moved into declared-read-columns.ts. The explicit-projection filter (find/findOne), the new default projection and the write door's undeclaredWriteFieldErrors all read it. No second list.
• H4: the shaping runs before formulas, expand, file references and the hooks. Internal omission, credential masking and the __search strip still run after the hooks. The conformance matrix pins formula, password mask, internal, and each system column.
• H6: only find and findOne return row bodies. count returns a number. aggregate returns aggregates, and its groupBy names are gated at the door. findStream was retired in 17.0. expand re-enters this.find.

Changes

• packages/objectql/src/declared-read-columns.ts (new): PLATFORM_PROVISIONED_COLUMNS, declaredColumnSet (no opinion for an absent, array or empty field map, the rule the read and write doors already share), and the non-mutating withDeclaredColumnsOnly / rowsWithDeclaredColumnsOnly.
• packages/objectql/src/engine.ts: find and findOne shape the driver's rows right after the driver call. The explicit filter and the write door read the shared list.
• packages/objectql/src/no-operator-object-door.ts: a comment pointer to the list's new home.
• Tests: packages/rest/src/data-query-unprojected-declared-fields.test.ts (the reach pin), packages/objectql/src/unprojected-read-declared-fields-conformance.test.ts (the conformance matrix), and the sys_migration fixture in engine.test.ts.
• .changeset/21571-unprojected-read-declared-fields.md: @objectstack/objectql minor, BREAKING narrowing, ADR-0087 not-required (no-migration-prescription), and the interim route.
• content/docs/data-modeling/queries.mdx: one paragraph stating the default projection.

Pins

• Reach pin. The harness is the composed REST harness this package already uses: RestServer, then ObjectStackProtocolImplementation, then ObjectQL, then a real SqlDriver on better-sqlite3 with a file database. Boot one declares two mailing fields and writes values. Boot two retires them, the card's upgrade. Then POST /api/v1/data/rq_contact/query with no fields.
  • Before the fix: 4 failed, 2 passed. The retired columns were present on the query and on GET /data/:object/:id, every-key-declared was red, and the ladder rung was red. The system-columns pin and the explicit INVALID_FIELD / 400 pin were green.
  • After the fix: 6 passed.
• Conformance matrix. objectql, three driver behaviours (whole row, projection, ladder) × doors (find, bare find, findOne, ladder projection, retired-only projection, expand, findData, getData, cloneData), plus declared-treatment and store-not-mutated pins. A future driver shape is one entry.
• Explicit projection of a retired column still answers INVALID_FIELD / 400. The test asserts code and status.
• Pin sweep. Repo-wide git grep over tests for orphaned, unmapped, retired or undeclared column readings, plus the trim-mode runs above: no test asserted that an unprojected read returns an undeclared column. The only fallout was the sys_migration fixture, which now declares its columns. No refusal assertion was changed.

Reverse verification (fix committed first, at 9593fbd)

Each shaping call was replaced in turn with node scripts/ablation-replace.mjs (anchor hit 1 to 0, blob changed). Then objectql was rebuilt (exit 0), and ablation-dist-preflight --absent confirmed the call is absent from all 14 built files.

• find call removed: reach pin 3 failed, 3 passed (query, every-key-declared, ladder). Matrix 17 failed, 19 passed.
• findOne call removed: reach pin 1 failed, 5 passed (GET by id). Matrix 9 failed, 27 passed (findOne, getData, cloneData × 3 shapes). The clone's failure reads Unknown field 'mailing_street' on object 'rq_contact'.
• Restore: git checkout HEAD -- packages/objectql/src/engine.ts. Blob aadf4018 equals the HEAD blob, git diff HEAD is empty and git status --porcelain is clean. After a rebuild, the preflight finds both calls present in dist. Reach pin 6/6 and matrix 36/36.

Local verification, at 4bc22fe (the PR head, after merging main, which includes #21545)

• pnpm --filter @objectstack/objectql test: 367 files, 7415 passed. test:repo: 1 file, 5 passed. typecheck (tsc plus test-typecheck): exit 0.
• The reach pin: 6 passed. pnpm --filter @objectstack/rest run typecheck: exit 0.
• The full rest suite ran at 99033a9: 259 files, 4889 passed, 326 skipped. The last merge (3 commits) touched neither rest nor objectql, so that run was not repeated.
• Gates: node scripts/pm/dispatch-gates.mjs --commands (no paths) re-derived the same 96 commands at this head. All 96 were run and reconciled with --ran (96 run, 0 unrun).
  • 95 exit 0.
  • node scripts/check-engine-split-ratio.mjs --days 90 answered exit 2: NOT MEASURED. The clone is shallow inside the 90-day window, and this is a report-only metric.
• Lint, narrowed and proven:
  • Population, from eslint's own config: the 8 changed files went to eslint --no-inline-config --format json. eslint reports the .md/.mdx pair as "File ignored because no matching configuration was supplied".
  • Count, from the JSON: 8 files in the report, 6 TypeScript files linted, 0 errors on them. The 2 warnings are the two ignore notices.
  • Invariance: eslint.config.mjs enables no type-aware linting (no parserOptions.project, no typed rules), so this diff cannot move an untouched file's verdict.
• Not run locally, declared to CI: the remaining downstream consumers of @objectstack/objectql. The trim-mode measurement above covered rest, runtime, plugin-auth, plugin-sharing, plugin-audit and service-automation under a superset of this change.

Acceptance notes

• Write responses still carry orphaned columns. This is a finding for the seat, outside this card's read ruling.
  • Measured after this fix on the same harness: PATCH /api/v1/data/rq_contact/c1 with { name } answers 200, and its record carries mailing_street: "1 Retired Way".
  • Cause: the engine's by-id update returns driver-sql's readback (select *), and updateData strips only internal fields from it.
  • The create 201 and clone 201 bodies are built from driver-sql's returning('*') too. That is not measured here; on a new row the orphaned columns would read null.
  • The A-prime ruling keeps engine write results whole for privileged writers, so where to cut this is a decision. It is not taken here.
• Prior reads (the update/delete previous rows handed to hooks) still carry orphaned columns. In process only, no public door measured. Carrier: none.
• In process, engine.aggregate does not refuse an undeclared groupBy name. The data door does (assertGroupByFieldsExist). Carrier: none.
• The os migrate apply account preflight's issuers label reads (none) on the engine path, because the engine already drops the retired issuer from an explicit projection. The collision verdict keys on provider and account ids and is unaffected. os migrate account-issuer reads through the driver and shows real issuers. Carrier: none.
• migrate: a sanctioned, operator-only read of unmapped (orphaned) columns for data conversion, before --allow-destructive drops them (the coupling #21571 names) #21573 (the operator-only os migrate read of unmapped columns) is not addressed here. It is the interim route the changeset names.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/objectql, touching 13 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/objectql/src/no-operator-object-door.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

35 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a.

⛔ 11 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see

• 1 changed file(s) yielded no anchor (packages/objectql/src/no-operator-object-door.ts) — pages documenting those are invisible to this run
• 1 anchor(s) matched too much of the corpus to be a work list: created_at (literal, 36 pages)
• 1 name(s) were too generic to anchor anything (single lowercase words)
• the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4bb19c524028050cefaed2cabb52b9df3e407220 — the merge of head 4bc22fe798a9c0e0d04c35dcc9cfe3e76f03b383 into base 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4bb19c524028050cefaed2cabb52b9df3e407220 && git checkout 4bb19c524028050cefaed2cabb52b9df3e407220
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a 4bc22fe798a9c0e0d04c35dcc9cfe3e76f03b383 && git checkout -B drift-repro 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a && git merge --no-ff 4bc22fe798a9c0e0d04c35dcc9cfe3e76f03b383

node scripts/docs-audit/affected-docs.mjs --json 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 9a4182a752fd53b24a14bbcd2e1b4270e1174e7a → pass the list as
args.docs, on the commit named under Which tree this was computed on.