Skip to content

Commit 568dc0b

Browse files
fix(data): record-change payloads apply the write-response credential mask and internal-field omission (#21866)
Fixes #21830 Clause-②: yes (widening) Record-change payloads apply the same credential mask and internal-field omission as write responses. ## What changed One rule, one helper: `omitInternalFieldsFromWriteResponse` (`@objectstack/core`, the helper PR #21816 landed for write responses). No second copy of the rule anywhere. - **Engine record-change publish** (`packages/objectql/src/engine.ts`, `publishDataEvent`). The `after` and `changes` bodies of `data.record.created` / `data.record.updated` are projected through the helper on a fresh shallow copy: credential-class fields carry `SECRET_MASK` (or `null` when unset), `internal: true` fields are omitted. The engine's own write result is untouched, so privileged in-process callers are unchanged. - **Defence in depth** at the downstream producers that store or forward a record body, each through the same helper or its collectors: - approval request snapshot (`plugin-approvals`, `payload_json` when the request is opened); - outbound webhook body and its stored delivery row (`plugin-webhooks` auto-enqueuer, `after` and `changes`); - knowledge index documents (`service-knowledge` `recordToDocument` takes the object definition as an optional fourth argument and skips credential-class and `internal` fields, under `'*'` and when named explicitly). - The audit trail already masked these fields and is unchanged. No accept set changes; the one public-surface change is additive (see Review round 1). - Changeset: `.changeset/record-change-payload-credential-mask.md` (minor for `@objectstack/service-knowledge`, patch for the other three). ## Verification (head d6e0eae, after merging origin/main) - Dependency closure build of the four touched packages: `VERDICT command-exit 0`. - `@objectstack/objectql` full suite: `Test Files 376 passed (376)`, `Tests 7486 passed (7486)`. - `@objectstack/plugin-approvals`: 60 files / 895 tests passed. `@objectstack/plugin-webhooks`: 14 / 163 passed. `@objectstack/service-knowledge`: 4 / 49 passed. - `typecheck` for objectql, plugin-approvals, plugin-webhooks: green (`check:test-typecheck: OK`). service-knowledge has no typecheck script. - Dogfood, real boot: `test/approval-snapshot-credential-field.dogfood.test.ts` 1 passed, after a build of the dogfood dependency closure. - Gates: `node scripts/pm/dispatch-gates.mjs --commands` derived 75 families; all 75 run, all exit 0 (plus `check-nul-bytes`); `--ran` reconciliation: 75 run, 0 NOT-MEASURED, a derived zero with exit codes recorded. - Ablation (one-shot, via `scripts/ablation-replace.mjs`, from the committed state): removing the helper call in the engine's event-body projection turns `src/engine-realtime-credential-mask.test.ts` red (3 failed, 1 passed); restore proven by blob hash equal to HEAD (48b8210) and an empty `git diff HEAD`. - Not run locally, declared to CI: repo-wide lint and the wide-population gate families. ## Acceptance notes - Payload projections are idempotent over an already-projected body, so the engine projection and the downstream defence-in-depth layers compose. - A producer whose engine exposes no `getSchema` (or an unregistered object) projects nothing, matching the helper's own posture; the engine-side projection still applies upstream. --- _Generated by [Claude Code](https://claude.ai/code/session_018zT8d8NpiQ1ExhuNd5TxY6)_ ## Review round 1 (PM seat) - `@objectstack/service-knowledge` is now `minor`: `recordToDocument`, a published export, gains an optional fourth argument (the object definition); three-argument calls behave as before. - The outbound webhook projection also covers `before` (no producer fills it today). - The changeset now says receivers see masked values, and that rows written before this change are not rewritten (reindexing a knowledge source refreshes its documents). - Head `c87404e8c4`: `@objectstack/plugin-webhooks` 163/163; the changeset gates green against `origin/main`. --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 149153c commit 568dc0b

10 files changed

Lines changed: 779 additions & 12 deletions

File tree

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
'@objectstack/objectql': patch
3+
'@objectstack/plugin-approvals': patch
4+
'@objectstack/plugin-webhooks': patch
5+
'@objectstack/service-knowledge': minor
6+
---
7+
8+
Record-change payloads apply the same credential mask and internal-field omission as write responses.
9+
10+
Clause-②: yes (widening)
11+
12+
- **`data.record.created` / `data.record.updated` events.** The engine projects the event's `after` and `changes` bodies through `omitInternalFieldsFromWriteResponse` (`@objectstack/core`), the helper every external write response already uses: credential-class fields (`secret`, and `password` outside the exempt `managedBy` buckets) carry `SECRET_MASK` (or `null` when unset), and `internal: true` fields are omitted. The engine's own write result is unchanged, so a privileged in-process caller that reads the stored value back off `insert` / `update` still sees it.
13+
- **Approval request snapshot.** The record snapshot an approval request stores (`payload_json`) applies the same rule when the request is opened.
14+
- **Outbound webhook body.** The delivered body, and the delivery row that stores it, apply the same rule to `before`, `after` and `changes`.
15+
- **Knowledge index documents.** `recordToDocument` takes the object definition as an optional fourth argument and skips credential-class and `internal` fields, under `'*'` and when a source names one explicitly. `KnowledgeService` passes the definition from the bound engine.
16+
- **New public surface of `@objectstack/service-knowledge` (additive):** `recordToDocument` accepts the object definition as an optional fourth argument; existing three-argument calls behave as before.
17+
- **Receivers see masked values.** Webhook receivers and realtime clients now get `SECRET_MASK` (or `null` when unset) for credential-class fields and no key for `internal` fields.
18+
- **Existing rows are not rewritten.** Approval snapshots, webhook delivery rows and knowledge documents written before this change keep their stored bodies; reindexing a knowledge source refreshes its documents.
19+
- The audit trail already masked these fields and is unchanged. No other accept set or public schema changes.
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* A `data.record.*` event is an external exit of the write it describes: its
5+
* subscribers store or forward the record bodies (`after`, `changes`) to
6+
* readers below the write boundary. So both bodies carry what a write
7+
* RESPONSE carries — credential-class fields masked, `internal: true` fields
8+
* omitted, by the one helper every write response uses — while the engine's
9+
* own write result, read by privileged in-process callers, stays whole.
10+
*/
11+
12+
import { describe, it, expect, beforeEach, vi } from 'vitest';
13+
import type { IRealtimeService, RealtimeEventPayload } from '@objectstack/spec/contracts';
14+
import { SECRET_MASK } from '@objectstack/spec/data';
15+
import { ObjectQL } from './engine.js';
16+
17+
const CREDENTIAL = 'synthetic-credential-7c1e';
18+
const INTERNAL = 'synthetic-internal-0b42';
19+
20+
const widget = {
21+
name: 'widget',
22+
label: 'Widget',
23+
fields: {
24+
id: { name: 'id', type: 'text' as const, primaryKey: true },
25+
title: { name: 'title', type: 'text' as const },
26+
passphrase: { name: 'passphrase', type: 'password' as const },
27+
lookup_digest: { name: 'lookup_digest', type: 'text' as const, internal: true },
28+
},
29+
};
30+
31+
function makeStubDriver() {
32+
const stores = new Map<string, Map<string, Record<string, unknown>>>();
33+
const storeFor = (o: string) => {
34+
let s = stores.get(o);
35+
if (!s) { s = new Map(); stores.set(o, s); }
36+
return s;
37+
};
38+
let nextId = 0;
39+
const driver: any = {
40+
name: 'memory', version: '0.0.0', supports: {},
41+
async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; },
42+
async find(o: string) { return Array.from(storeFor(o).values()); },
43+
findStream() { throw new Error('ns'); },
44+
async findOne(o: string, ast: any) {
45+
const where = ast?.where ?? {};
46+
for (const r of storeFor(o).values()) {
47+
if (Object.entries(where).every(([k, v]) => k.startsWith('$') || (r[k] ?? null) === ((v as any)?.$eq ?? v ?? null))) return r;
48+
}
49+
return null;
50+
},
51+
async create(o: string, data: Record<string, unknown>) {
52+
nextId += 1;
53+
const id = (data.id as string) ?? `r_${nextId}`;
54+
const row = { ...data, id };
55+
storeFor(o).set(id, row);
56+
return row;
57+
},
58+
async update(o: string, id: string, data: Record<string, unknown>) {
59+
const s = storeFor(o); const cur = s.get(id);
60+
if (!cur) throw new Error(`nf ${o}/${id}`);
61+
const up = { ...cur, ...data, id }; s.set(id, up); return up;
62+
},
63+
async delete(o: string, id: string) { return storeFor(o).delete(id); },
64+
async count(o: string) { return (await this.find(o)).length; },
65+
async bulkCreate(o: string, rows: Record<string, unknown>[]) { return Promise.all(rows.map((r) => this.create(o, r))); },
66+
async updateMany() { return 0; }, async deleteMany() { return 0; },
67+
async bulkUpdate() { return []; }, async bulkDelete() {},
68+
async upsert(o: string, data: Record<string, unknown>) { return this.create(o, data); },
69+
async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; },
70+
async commit() {}, async rollback() {},
71+
};
72+
return { driver };
73+
}
74+
75+
describe('record-change event bodies apply the write-response non-exposure rules', () => {
76+
let engine: ObjectQL;
77+
let published: RealtimeEventPayload[];
78+
79+
beforeEach(async () => {
80+
published = [];
81+
const realtime: IRealtimeService = {
82+
publish: vi.fn(async (event: RealtimeEventPayload) => { published.push(event); }),
83+
subscribe: vi.fn(async () => 'sub-1'),
84+
unsubscribe: vi.fn(async () => undefined),
85+
};
86+
engine = new ObjectQL();
87+
const { driver } = makeStubDriver();
88+
engine.registerDriver(driver, true);
89+
await engine.init();
90+
engine.registry.registerObject(widget);
91+
engine.setRealtimeService(realtime);
92+
vi.spyOn((engine as any).logger, 'warn').mockImplementation(() => undefined);
93+
vi.spyOn((engine as any).logger, 'debug').mockImplementation(() => undefined);
94+
});
95+
96+
it('a create event masks the credential field and omits the internal field from `after`', async () => {
97+
await engine.insert('widget', { id: 'w1', title: 'One', passphrase: CREDENTIAL, lookup_digest: INTERNAL });
98+
expect(published).toHaveLength(1);
99+
const payload = published[0].payload as Record<string, any>;
100+
expect(payload.type).toBe('data.record.created');
101+
expect(payload.after.title).toBe('One');
102+
expect(payload.after.passphrase).toBe(SECRET_MASK);
103+
expect('lookup_digest' in payload.after).toBe(false);
104+
expect(JSON.stringify(payload)).not.toContain(CREDENTIAL);
105+
expect(JSON.stringify(payload)).not.toContain(INTERNAL);
106+
});
107+
108+
it('an update event projects both `after` and `changes`', async () => {
109+
await engine.insert('widget', { id: 'w2', title: 'Two', passphrase: 'first', lookup_digest: 'first' });
110+
published.length = 0;
111+
await engine.update('widget', { id: 'w2', title: 'Two b', passphrase: CREDENTIAL, lookup_digest: INTERNAL });
112+
expect(published).toHaveLength(1);
113+
const payload = published[0].payload as Record<string, any>;
114+
expect(payload.type).toBe('data.record.updated');
115+
expect(payload.changes.title).toBe('Two b');
116+
expect(payload.changes.passphrase).toBe(SECRET_MASK);
117+
expect('lookup_digest' in payload.changes).toBe(false);
118+
expect(payload.after.passphrase).toBe(SECRET_MASK);
119+
expect('lookup_digest' in payload.after).toBe(false);
120+
expect(JSON.stringify(payload)).not.toContain(CREDENTIAL);
121+
expect(JSON.stringify(payload)).not.toContain(INTERNAL);
122+
});
123+
124+
it('an unset credential field rides the event as null, not as the mask', async () => {
125+
await engine.insert('widget', { id: 'w3', title: 'Three', passphrase: null });
126+
const payload = published[0].payload as Record<string, any>;
127+
expect(payload.after.passphrase).toBeNull();
128+
});
129+
130+
it("the engine's own write result is unchanged for the privileged in-process caller", async () => {
131+
const created = await engine.insert('widget', {
132+
id: 'w4', title: 'Four', passphrase: CREDENTIAL, lookup_digest: INTERNAL,
133+
}) as Record<string, unknown>;
134+
expect(created.passphrase).toBe(CREDENTIAL);
135+
expect(created.lookup_digest).toBe(INTERNAL);
136+
137+
const updated = await engine.update('widget', { id: 'w4', lookup_digest: `${INTERNAL}-b` }) as Record<string, unknown>;
138+
expect(updated.passphrase).toBe(CREDENTIAL);
139+
expect(updated.lookup_digest).toBe(`${INTERNAL}-b`);
140+
// ...while the events those writes published carried neither value.
141+
expect(JSON.stringify(published.map((e) => e.payload))).not.toContain(CREDENTIAL);
142+
expect(JSON.stringify(published.map((e) => e.payload))).not.toContain(INTERNAL);
143+
});
144+
});

‎packages/objectql/src/engine.ts‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,11 @@ import {
119119
// The data door's object-existence 404, shared for the same reason: an
120120
// in-process verb refuses a name the registry does not resolve with it.
121121
objectNotFoundError,
122+
// The one write-response non-exposure helper (credential mask + `internal`
123+
// omission); a record-change event body is an external exit of the same
124+
// write, so it is projected through the same rule — see
125+
// {@link projectEventRecordBody}.
126+
omitInternalFieldsFromWriteResponse,
122127
} from '@objectstack/core';
123128
import { WriteEpoch, isWriteEpochOperation } from './write-epoch.js';
124129
import { bridgeAuthzInvalidation } from './authz-invalidation-bridge.js';
@@ -3365,6 +3370,33 @@ function redactEventMetadataBody(
33653370
return rest;
33663371
}
33673372

3373+
/**
3374+
* Project a `data.record.*` event body (`after` / `changes`) through the
3375+
* generic-data-path non-exposure rules — credential-class fields MASKED,
3376+
* `internal: true` fields OMITTED — by the ONE helper every external write
3377+
* response already goes through (`omitInternalFieldsFromWriteResponse`,
3378+
* `@objectstack/core`). ⛔ Never a second copy of that rule here.
3379+
*
3380+
* The event is an external exit of the write: its subscribers (outbound
3381+
* deliveries, search indexes, flows that snapshot the record) store or forward
3382+
* what they receive to readers below the write boundary, so the body carries
3383+
* what a read of the same row would answer — never the stored value the
3384+
* engine's own write result keeps whole for the privileged in-process caller.
3385+
*
3386+
* Pure with respect to the write result: the helper deletes and assigns in
3387+
* place, so it runs on a fresh shallow copy and the caller's record (the
3388+
* engine's return value) is never touched.
3389+
*/
3390+
function projectEventRecordBody(
3391+
schema: unknown,
3392+
body: Record<string, unknown> | undefined,
3393+
): Record<string, unknown> | undefined {
3394+
if (body === undefined) return body;
3395+
const copy = { ...body };
3396+
omitInternalFieldsFromWriteResponse(schema, copy);
3397+
return copy;
3398+
}
3399+
33683400
/** `DataEvent.userId` — the acting user, when the execution context names one. */
33693401
function eventUserId(execCtx?: ExecutionContext): string | undefined {
33703402
const userId = execCtx?.userId;
@@ -7572,17 +7604,23 @@ export class ObjectQL implements IObjectQLEngine {
75727604

75737605
try {
75747606
const timestamp = new Date().toISOString();
7575-
const changes = redactEventMetadataBody(object, eventRecordBody(input.changes), input.after);
7576-
const after = redactEventMetadataBody(object, eventRecordBody(input.after), input.after);
7607+
const schema = this._registry.getObject(object);
7608+
// Credential mask + `internal` omission on both bodies, the same rule
7609+
// the write response gets ({@link projectEventRecordBody}).
7610+
const changes = projectEventRecordBody(
7611+
schema,
7612+
redactEventMetadataBody(object, eventRecordBody(input.changes), input.after),
7613+
);
7614+
const after = projectEventRecordBody(
7615+
schema,
7616+
redactEventMetadataBody(object, eventRecordBody(input.after), input.after),
7617+
);
75777618
const userId = eventUserId(input.context);
75787619
// [#14970] The RECORD's organization, off the row itself — ⛔ never
75797620
// `input.context.tenantId`, which is the CALLER's. See
75807621
// {@link eventOrganizationId}; omitted, never `''`/`undefined`, because
75817622
// absence has exactly one spelling in the schema.
7582-
const organizationId = eventOrganizationId(
7583-
this._registry.getObject(object),
7584-
input.organizationRow,
7585-
);
7623+
const organizationId = eventOrganizationId(schema, input.organizationRow);
75867624
const event: DataEvent = DataEventSchema.parse({
75877625
id: generateEventUuid(),
75887626
type: `data.record.${action}`,

‎packages/plugins/plugin-approvals/src/approval-service.ts‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ import { isFileIdToken, referenceTargetOf } from '@objectstack/spec/data';
6363
// mechanism for a second producer, and commit aa5994e17 landed this service's key
6464
// (`approval_recall_not_submitter`) into it ahead of this consumer half.
6565
import { renderOperationMessage, type ValidationMessageTranslator } from '@objectstack/spec/system';
66-
import { isGrantActive } from '@objectstack/core';
66+
import { isGrantActive, omitInternalFieldsFromWriteResponse } from '@objectstack/core';
6767
import {
6868
filterApproversWhoCanRead,
6969
resolveApproverDirectoryOrg,
@@ -3038,7 +3038,7 @@ export class ApprovalService implements IApprovalService {
30383038
current_step: input.nodeId,
30393039
current_step_index: 0,
30403040
pending_approvers: approvers.join(','),
3041-
payload_json: input.record != null ? JSON.stringify(input.record) : null,
3041+
payload_json: input.record != null ? JSON.stringify(this.snapshotRecord(input.object, input.record)) : null,
30423042
flow_run_id: input.runId,
30433043
flow_node_id: input.nodeId,
30443044
node_config_json: JSON.stringify(configSnapshot),
@@ -5915,6 +5915,37 @@ export class ApprovalService implements IApprovalService {
59155915
}
59165916
}
59175917

5918+
// ── Record snapshot ──────────────────────────────────────────
5919+
5920+
/**
5921+
* The subject record as `payload_json` stores it: credential-class fields
5922+
* MASKED and `internal: true` fields OMITTED, by the one helper every
5923+
* external write response goes through (`omitInternalFieldsFromWriteResponse`,
5924+
* `@objectstack/core`) — the same answer a read of the row gives.
5925+
*
5926+
* The record arrives from the flow's `$record`, which the record-change
5927+
* trigger builds off the engine's own write result, and that result keeps the
5928+
* stored row whole for privileged in-process callers. The snapshot is the
5929+
* opposite case: it is stored, and served to approvers and submitters below
5930+
* the write boundary (the serve-time redaction in `payload-redaction.ts`
5931+
* narrows by field-level security, and cannot know a value is a stored
5932+
* credential). Defence in depth beside the engine's own event-body projection.
5933+
*
5934+
* Pure: projects a shallow copy, never the caller's record. No schema (an
5935+
* engine double without `getSchema`, an unregistered object) projects
5936+
* nothing — the same posture as the helper itself.
5937+
*/
5938+
private snapshotRecord(object: string, record: unknown): unknown {
5939+
if (!record || typeof record !== 'object' || Array.isArray(record)) return record;
5940+
let schema: unknown;
5941+
try {
5942+
schema = this.engine.getSchema?.(object);
5943+
} catch { /* schema unavailable — nothing to project against */ }
5944+
const copy = { ...(record as Record<string, unknown>) };
5945+
omitInternalFieldsFromWriteResponse(schema, copy);
5946+
return copy;
5947+
}
5948+
59185949
// ── Display enrichment ───────────────────────────────────────
59195950

59205951
/**

0 commit comments

Comments
 (0)