fix(objectql)!: a write's returned row and its prior read serve the declared fields, never an orphaned column

[objectstack-fleet]

Fixes #21613

Clause-②: no (narrowing)

A write now answers with the object's declared fields plus the platform's system columns. A column no metadata declares, such as a field retired in an upgrade whose column additive sync leaves behind, no longer leaves the engine through a write. The same holds for the prior read a write binds as a hook's previous. This completes #21571's read rule for writes. It is decided in the same place, the engine (packages/objectql), with the same helper (declared-read-columns.ts), on the rows as the driver returned them: before formulas, hooks, events and the door's own ingress strip. No driver source and no metadata-protocol source is edited. There is no per-door strip, no allow-list and no flag.

Measured first (before any fix)

The probes were two temporary tests on this branch: 9c73f3702b (rest) and b923112e06 (plugin-audit), reverted by dfbb80ec2e. The net diff carries none of them. Harness: the composed REST harness of #21571's reach pin (RestServer, then ObjectStackProtocolImplementation, then ObjectQL, then a real SqlDriver on better-sqlite3), two boots, at 5c9138b4b6.

door	before	after
PATCH /data/:object/:id	200, record.mailing_street = "1 Retired Way"	no retired key
POST /data/:object	201, both retired keys, null	no retired key
POST /data/:object/:id/clone	201, both retired keys, null	no retired key
POST /data/:object/createMany	201, every record carries both, null	none
POST /data/:object/batch update / upsert (update arm)	values	none
POST /data/:object/batch create / upsert (create arm)	null	none
POST /data/:object/updateMany	values	none
DELETE /data/:object/:id	no record in the body	unchanged
engine.update by id, engine.insert, engine.insertMany outcomes	values or null	none
data.record.created / data.record.updated events, after	13 of 13 carried them (16 events in all)	0 of 16 events carry one
hook contexts, previous or result	28 of 28 carried them	0 of 28 carry one

Bulk data.records.* events carry a count and no row, and changes carries the input patch only.

The audit ledger serves the prior read. Measured with a plugin-audit probe (real ObjectQL, a SQL-shaped store, installAuditWriters):

• a delete's old_value recorded mailing_street: "c2 Retired Way", and a create's new_value recorded both columns as null;
• an update recorded nothing of them, only because both sides of the diff carried them;
• with ONLY the write result shaped (the prior read left whole), every update of such a row recorded mailing_street: "c1 Retired Way" in old_value against null in new_value. That is a phantom change, and it puts the stored value into a served row on every write.

sys_audit_log is read back through the data door, so the prior read is a served row. That puts it in this card's class (H3), and shaping the result alone would have made the ledger worse. So the prior reads are shaped too.

Hypothesis verdicts

• H1 (confirmed): SqlDriver.update returns its select * readback, and create and bulkCreate return returning('*'). ObjectQL returned both as they came.

• H2 (confirmed): the engine publishes data.record.* from the same result variable the hooks get, after the driver. The webhook enqueuer copies the event's payload verbatim (payload: { ...payload, ... }). With the shaping placed after the driver, the event and the webhook payload are clean, with no edit in plugin-webhooks or service-realtime. The pins below drive the real AutoEnqueuer.

• H3 (served, in class, shaped): see the audit ledger above. The four prior reads are shaped at the read:

  • update by id (driver.findOne);
  • update by predicate (the matched rows the per-row hooks compose result from);
  • delete by id;
  • delete by predicate.
    The delete event reads only the tenant column off its pre-image, which is declared.

• H4 (enumerated): the write verbs that return a row are:

  • insert (one row and a batch);
  • insertMany (its ok outcomes carry rowHookContexts[i].result);
  • update by id.

  Predicate update and delete return a count, and delete by id returns the driver's boolean. The engine has no upsert verb: the batch door's upsert is update or insert. ObjectRepository and the scoped context delegate to these verbs. Every REST and protocol face above reaches one of the three, so shaping them covers every door.

• H5 (pinned): an internal: true field stays whole on the engine-level write result (the A-prime ruling) and is still stripped by the data door. The rest pin and the conformance matrix both assert it.

In-process readers of undeclared columns

The suites ran with the shaping in place: objectql at 10d7f333d9, the others at f082789802 (the probes were still on the branch):

suite	result
objectql	4 failures, all fixtures, see below
rest	260 files, all pass
metadata-protocol	208 files pass, 3 skipped
plugin-audit	39 files pass
plugin-webhooks	13 files pass
plugin-sharing	38 files pass
plugin-auth	118 files pass
service-automation	166 files pass
runtime	318 files pass

The 4 objectql failures were fixtures whose readonlyWhen or validation rule reads a field the fixture never declared (record.locked, record.limit), off the prior read. #21571 saw the same shape in trim mode. The fixtures now declare those fields (engine.test.ts × 3, plugin.integration.test.ts × 1). No assertion changed. At authoring, the formula validator's unknown field check (packages/formula/src/validate.ts) flags a production rule that names no field.

The operator reads that legitimately need a retired column's values go through the driver, so they are unaffected:

• os migrate plan's unmapped_column detection;
• os migrate account-issuer.

Changes

• packages/objectql/src/engine.ts, one shaping per site, each with declaredColumnSet(registry object):
  • insert: the driver's result rows, after the one-row-per-input guard;
  • update by id: the driver readback;
  • the update by-id prior read, and the update predicate's matched rows;
  • the delete by-id pre-image, and the delete predicate's doomed rows.
• Tests:
  • packages/rest/src/data-write-result-declared-fields.test.ts: the reach pin, on the composed harness.
  • packages/objectql/src/write-result-declared-fields-conformance.test.ts: the engine matrix.
  • packages/plugins/plugin-webhooks/src/webhook-payload-declared-fields.test.ts: the webhook payload through the real AutoEnqueuer.
  • packages/plugins/plugin-audit/src/audit-ledger-declared-fields.test.ts: the served ledger.
  • The objectql fixtures above.
• .changeset/21613-write-result-declared-fields.md: @objectstack/objectql minor, BREAKING (narrowing), ADR-0087 not-required (no-migration-prescription), and the interim route (convert before upgrading, or migrate: a sanctioned, operator-only read of unmapped (orphaned) columns for data conversion, before --allow-destructive drops them (the coupling #21571 names) #21573's operator read once it lands).

File surface beyond the claim, test-only: the webhook and audit pins live in their own packages. The composed REST package has no dependency on either plugin. Adding one would need a devDependency plus a source alias, because KNOWN_UNALIASED_TEST_IMPORTS is shrink-only. So the event both of them consume is pinned on the composed harness, and each consumer is pinned where it lives.

Pins

• Reach pin (rest, SqlDriver, two boots, 8 cases):
  • the fixture is real: a raw driver read still finds the stored values;
  • PATCH, POST and clone carry no retired key;
  • the bulk faces: createMany, batch upsert / create / update, updateMany;
  • every data.record.created / updated event's after;
  • the update and delete previous;
  • declared fields and the eight system columns are served, and every key is declared or provisioned;
  • internal: stripped at the door, whole on engine.update.
• Conformance matrix (objectql, SQL-shaped store, 12 cases):
  • insert one row, a batch, insertMany, and update by id;
  • the hooks' previous and result, for by id, per-row predicate update, and delete by id and by predicate;
  • the events;
  • createData, updateData and cloneData;
  • the declared treatment: internal whole on the engine result and stripped at the door, formula hydrated;
  • the driver's stored row is not mutated.
• Webhook pin: a real ObjectQL write, then the realtime publish, then the real AutoEnqueuer. The enqueued payload.after carries no retired key, for created and for updated.
• Audit pin: the update diff is exactly { name }, and the delete old_value and create new_value carry no retired key.
• Pin sweep: a repo-wide git grep over *.test.ts for orphaned, retired, unmapped or undeclared column readings, plus the suite runs above. No test asserted that a write's returned row or prior read carries an undeclared column, so no pin flipped. Each new pin asserts the declared values as well as the absence: ids, names and system columns.

Reverse verification (fix committed first)

Each shaping was removed with node scripts/ablation-replace.mjs: the anchor went from 1 hit to 0 and the blob changed. objectql was then rebuilt (exit 0), and ablation-dist-preflight --absent confirmed the call is absent from all 14 built files (exit 0 each). The pristine build carries every marker. Final run at bfbeffed43:

removed	conformance (12)	reach (8)	webhook (1)	audit (3)
insert result	5 red: insert, batch, insertMany, events, doors	4 red: POST, clone, bulk faces, events	red	create red
update by-id result	4 red: update, hooks, events, doors	4 red: PATCH, bulk faces, events, system columns	red	update red (inverse phantom diff)
update prior reads	2 red: by-id and per-row hooks	1 red: previous	green	update red (the phantom diff)
delete prior reads	1 red: delete hooks	1 red: previous	green	delete red

Restore: ablation-replace restored with git checkout HEAD -- PATH each time. The blob 6008f2bc equals the HEAD blob, git diff HEAD is empty and git status --porcelain is clean. After a rebuild, the preflight finds all five markers present, and the pins are 12/12, 8/8, 1/1 and 3/3. Each mutation was replaced with a type-valid spelling so the DTS step builds. A first C1 run with an arity-breaking spelling failed the DTS step and was redone; its red set was the same.

Local verification

• objectql: test 368 files, 7427 passed, and test:repo 5 passed, at bfb7b28a49 (after merging origin/main e367002e11). typecheck (tsc, scripts and test-typecheck) exits 0 at bfbeffed43.
• rest:
  • test 260 files, 4897 passed, 326 skipped;
  • test:repo 5 files, 177 passed;
  • typecheck exits 0;
  • all at bfb7b28a49. The rest test file is unchanged since.
• plugin-audit and plugin-webhooks:
  • full suites at bfb7b28a49: 39 files / 621 tests and 14 / 161;
  • typecheck exits 0 at bfbeffed43;
  • the two later commits touched only the new pin files, which were rerun green.
• Gates: node scripts/pm/dispatch-gates.mjs --commands (no paths) at bfbeffed43 derived 71 commands. All 71 were run on that head and each exited 0. --ran reconciles with "71 derived, 71 run, 0 NOT-MEASURED, 0 UNRUN", with every exit code recorded.
  • Added beyond the dispatch's list, by this diff's paths:
    • check-adr-0087-registration ×2, check-empty-changeset ×2 and check-tenant-audit-census ×2;
    • the release-rehearsal-clone and release-pending-publish self-tests;
    • check:engine-double-contract, check:i18n, check:i18n-stale-fill and check:objectql-double-limit;
    • check:objectui-changeset, check:pm-changeset-deadline-census and check:query-options-erasure;
    • check:type-check-coverage, check:type-check-debt and check:where-matcher.
  • check:objectql-double-limit first flagged the three new store doubles as limit-blind. They now apply the caller's bound (bfbeffed43).
  • check:dual-build-cjs-loads and check:i18n first answered PREREQUISITE NOT MET (exit 3), which is not a measurement. Both were rerun after the build they name, and both exit 0.
• Lint, narrowed and proven:
  • Population, from eslint's own config: the 8 changed files went to eslint --no-inline-config --format json. eslint reports the changeset as "File ignored because no matching configuration was supplied".
  • Count, from the JSON: 8 files in the report, 7 TypeScript files linted, 0 errors. The 1 warning is that ignore notice.
  • Invariance: eslint.config.mjs enables no type-aware linting (no parserOptions.project, no typed rules), so this diff cannot move an untouched file's verdict.
• Declared to CI: the remaining downstream consumers of @objectstack/objectql. The suites above are the ones that read write results (rest, metadata-protocol, runtime, plugin-auth, plugin-sharing, plugin-audit, plugin-webhooks, service-automation).

Acceptance notes

• declared-read-columns.ts's header still describes the read side only. The write-side seats carry their own comments in engine.ts. It was not edited, because the claim names it only for a new entry point and none was needed. Carrier: none.
• content/docs/data-modeling/queries.mdx states the default projection for reads. A write now follows the same rule, but no sentence says so. check-affected-docs is green. Carrier: none.
• engine.aggregate's undeclared groupBy name in process is unchanged (already noted on An unprojected REST data query returns ORPHANED columns that no metadata declares (fields retired in an upgrade), outside any field-level rule, until os migrate apply --allow-destructive #21571). Carrier: none.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 1 anchor(s) matched too much of the corpus to be a work list: ObjectQL (symbol, 71 pages)
• 3 name(s) were too generic to anchor anything (single lowercase words)
• the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 045b946256d988653fdca185c7fd33d6d86bd78d → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 71e15b61d1b8a6207bbb93cad7de57b3ed158b8f — the merge of head bfbeffed43bf59c7ce97b6049961e6068216aa3a into base 045b946256d988653fdca185c7fd33d6d86bd78d, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 71e15b61d1b8a6207bbb93cad7de57b3ed158b8f && git checkout 71e15b61d1b8a6207bbb93cad7de57b3ed158b8f
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 045b946256d988653fdca185c7fd33d6d86bd78d bfbeffed43bf59c7ce97b6049961e6068216aa3a && git checkout -B drift-repro 045b946256d988653fdca185c7fd33d6d86bd78d && git merge --no-ff bfbeffed43bf59c7ce97b6049961e6068216aa3a

node scripts/docs-audit/affected-docs.mjs --json 045b946256d988653fdca185c7fd33d6d86bd78d

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[objectstack-fleet]

ACCEPT — PR #21631 at head bfbeffed43

domain:engine#1 · session_017ErfyP2Rx7XWHJA27QjyUi · read at 2026-10-03T20:32Z. The os-dev report is on #21613. Judged against GitHub and the branch, not against the report.

• Shape: draft, base main, assignee os-project-manager.
  • The first lines are Fixes #21613 and Clause-②: no (narrowing): the claim's no plus the narrowing arm, as the order asked.
  • The closing-keyword scan finds #21613 only.
• Scope: 8 files, +963/-6. check-governed-merges.mjs --pr 21631: NOT governed, 969 changed lines. Nothing under packages/spec/src/**, no driver source, no metadata-protocol source and no content/docs/releases/.

  • Beyond the claim, test-only, both named in the PR body:

    • plugin-webhooks' webhook-payload-declared-fields.test.ts;
    • plugin-audit's audit-ledger-declared-fields.test.ts.

    Each pins its consumer in its own package, because the composed REST package depends on neither.

  • The fixture fallout (engine.test.ts ×3, plugin.integration.test.ts ×1) declares the locked / limit fields those rules read. No assertion changed.

• The diff, read: six sites in engine.ts, each the existing An unprojected REST data query returns ORPHANED columns that no metadata declares (fields retired in an upgrade), outside any field-level rule, until os migrate apply --allow-destructive #21571 helper on the row as the driver returned it.
  • The insert result, after the one-row-per-input guard and before the boolean coercion, the formulas, the hooks and the events.
  • The by-id update readback.
  • The update's by-id prior read and its predicate's matched rows.
  • The delete's by-id pre-image and its predicate's doomed rows.
  • Schemaless objects: declaredColumnSet answers undefined for an object with no declared fields, so those rows pass untouched.
  • Nulls: a null readback stays null.
• Deviation 1 (prior reads shaped too) — accepted, in class. The dev measured that previous reaches a served row: the audit ledger's delete old_value. Shaping only the result would make every update record a retired column as changed, carrying its stored value. That is the same defect class, in the same file, with the same helper. The ablation of the prior-read leg turns the audit update pin red with the phantom diff.
• Clause-②: no (narrowing) — accepted on the seat's own read. There is no path leg. The value is no: nothing widens, and no error code, export or accepted key is added. So no contract review is owed. The same door's read half, PR fix(objectql)!: an unprojected read serves the declared fields, never an orphaned column #21612 (An unprojected REST data query returns ORPHANED columns that no metadata declares (fields retired in an upgrade), outside any field-level rule, until os migrate apply --allow-destructive #21571), landed with the same declaration.
• Changeset, checked sentence by sentence against the diff:
  • '@objectstack/objectql': minor, the BREAKING (narrowing) banner, Clause-②: no (narrowing) and the adr-0087: not-required (no-migration-prescription) marker, in An unprojected REST data query returns ORPHANED columns that no metadata declares (fields retired in an upgrade), outside any field-level rule, until os migrate apply --allow-destructive #21571's shape.
  • The table's five rows match the six sites. Where they need the dev's measured "before" (the PATCH stored values, the create and clone nulls, the events, the hook contexts and the audit old_value), the report carries the counts.
  • "Unchanged":
    • internal: true stays whole on the engine result, because shaping is by declared field set, and declared internal fields are in it;
    • formulas are still hydrated, because the shaping comes before them in both verbs;
    • "No driver changed" holds, because no driver file is in the diff.
  • "If you still read a retired column's values off a write": it names the interim route the order asked for (convert before upgrading, or migrate: a sanctioned, operator-only read of unmapped (orphaned) columns for data conversion, before --allow-destructive drops them (the coupling #21571 names) #21573 once it lands), and states that a rule or hook must name a declared field.
• Evidence:
  • Before and after on the composed REST harness: every record-answering write door, 13 of 13 events and 28 of 28 hook contexts carried retired keys before. After, every count is 0.
  • Four-leg ablation: one leg for each site group, with a dist preflight --absent on every leg. Each leg went red in its predicted set, and the restore was proved by blob equality.
  • Downstream: rest, metadata-protocol, plugin-audit, plugin-webhooks, plugin-sharing, plugin-auth, service-automation and runtime were all exit 0 with the shaping in place.
  • Gates: dispatch-gates --ran 71 of 71, 0 NOT-MEASURED. check:objectql-double-limit was red on the first pass. The dev fixed the three new store doubles, which were limit-blind.
• CI on bfbeffed, at this read: 15 check runs are in progress, and check-expected-skips is NOT MEASURED until they finish. The seat lands only once every check is green or an expected skip.
• Base: origin/main has moved to 045b946256 since the branch's merge of e367002e11. That commit (fix(runtime,cloud-connection)!: install-local refuses a hook with no body and a job body that does not bind, and withholds such a hook on rehydrate (#21585) #21615) touches runtime and cloud-connection only. The seat merges main only if mergeable_state reads dirty.

Out-of-scope: three items, noted and not filed, all doc drift with no a/b/c defect: declared-read-columns.ts's read-only header, the queries.mdx projection sentence, and aggregate's in-process groupBy (already in #21571's acceptance notes).

---

Generated by Claude Code