fix(metadata-protocol): the default /diff from side is the nearest earlier version whose body differs (#20451)

.changeset/20397-diff-default-range-labels.md

@@ -1,3 +1,3 @@
 ---
 '@objectstack/metadata-protocol': patch
 ---
@@ -6,6 +6,6 @@
 
 With no `toVersion`, the to side is the current active `sys_metadata` row. Its body was compared, but `toVersion` came from the newest `sys_metadata_history` row, which is a draft save whenever a draft is pending: every draft save appends a history row. The labels and the bodies then named different rows. Measured on the real REST stack, an app with one active save and two draft saves answered `fromVersion 2 → toVersion 3` over its version-1 body, and a view with one active save and one draft save answered "no changes" labelled `1 → 2` while version 2 differs.
 
-- **Now:** `toVersion` is the active row's own `version`, read in the same read as its body. The default `fromVersion` is still the history version immediately before that label. An item whose active row is version 2 with a draft pending answers `1 → 2`, the same answer as `?from=1&to=2`.
+- **Now:** `toVersion` is the active row's own `version`, read in the same read as its body. The default `fromVersion` rule is not changed by this entry (#20451, in the same release, then moves it to the nearest earlier version whose body differs from the to side's). An item whose active row is version 2 with a draft pending answers `1 → 2`, the same answer as `?from=1&to=2`.
 - **No active row** (a draft-only item, or a deleted one): the to side is absent, and both labels are `null` with empty buckets, as the response schema declares for an absent side. Before, a draft-only item was labelled with its newest draft save, and its from side could be an earlier draft save's body. A deleted item was labelled `N-1 → N` up to its tombstone. That deletion is still read by naming its versions (`?from=N-1&to=N`).
 - Unchanged: the response shape, explicit `from` / `to` ranges, and the default range of an item with no draft pending.

.changeset/20451-diff-default-from-differs.md

@@ -0,0 +1,25 @@
+---
+'@objectstack/metadata-protocol': patch
+'@objectstack/rest': patch
+---
+
+fix(metadata-protocol): `GET /meta/:type/:name/diff` with no `from` compares against the nearest earlier version whose body differs, so the default diff right after a publish shows what the publish changed (#20451)
+
+Clause-②: no — no key, export, route, parameter or response field moves; only which version the default `from` side names.
+
+Every draft save appends a `sys_metadata_history` row, and publishing the draft appends the same body again as the next row. The default `from` side was the history row immediately before the `to` side, so right after a publish it was the draft save the publish came from, and the default diff answered "no changes". The change the publish carried was reachable only by naming `?from=`.
+
+- **Now:** with no `from`, `diffMetaItem` walks back from the `to` side over the history rows it already reads and takes the nearest earlier row whose body differs, by the diff's own equality (all three buckets empty means equal). A body-less row, a delete's, compares as an empty body, so the walk stops on it and the answer names the deletion. With no earlier row that differs, the `from` side is absent: `fromVersion: null`, everything added.
+- **Measured on the real REST stack**, before → after:
+
+| history | default range before | default range now |
+|:--|:--|:--|
+| v1 active, v2 draft save, v3 publish | `2 → 3`, no changes | `1 → 3`, the change the publish carried |
+| the same with a v4 draft pending | `2 → 3`, no changes | `1 → 3` |
+| create, delete, draft save, publish | `3 → 4`, no changes | `2 → 4`, everything added |
+| create, delete, active recreate | `2 → 3`, everything added | unchanged |
+| a new item draft-saved, then published | `1 → 2`, no changes | `null → 2`, everything added |
+| a single version | `null → 1`, everything added | unchanged |
+
+- **Unchanged:** an explicit `?from=` / `?to=` names exactly its versions (`?from=2&to=3` over the first row still answers "no changes"); the default `to` side is the active version; the response shape; the one history read, with no cap. The walk compares the stored bodies before redaction, as the diff itself does, so a credential-only change still stops it and its values are still not served.
+- `@objectstack/rest`: the route's OpenAPI summary states the new default.

packages/client/src/index.ts

@@ -2192,8 +2192,9 @@ export class ObjectStackClient {
     },
 
     /**
-     * Structural diff between two history versions (`from`/`to`); omit both
-     * for previous-vs-current.
+     * Structural diff between two history versions (`from`/`to`). Omit `to`
+     * for the active version; omit `from` for the nearest earlier version
+     * whose body differs from the `to` side's.
      */
     diffItem: async (type: string, name: string, opts?: { from?: number; to?: number }): Promise<DiffMetaItemResponse> => {
         const route = this.getRoute('metadata');

packages/metadata-protocol/src/protocol.diff-dead-history-read.test.ts

@@ -283,6 +283,46 @@ describe('#8798 — a history-table outage now answers the same way for every ty
     });
 });
 
+type Seed = { version: number; op: string; body: Record<string, unknown> | null };
+/**
+ * History rows in version order, plus the active and draft rows they left
+ * behind. Shared by the #20397 and #20451 blocks below, both about the
+ * DEFAULT range; the REST file names the real writes each lineage mirrors.
+ */
+function seedLineage(
+    tables: Record<string, Array<Record<string, unknown>>>,
+    type: string,
+    name: string,
+    history: Seed[],
+    rows: { active?: Seed; draft?: Seed },
+) {
+    const base = { organization_id: null, type, name };
+    history.forEach((h, i) => {
+        tables.sys_metadata_history!.push({
+            ...base,
+            id: `h_${h.version}`,
+            version: h.version,
+            event_seq: i + 1,
+            operation_type: h.op,
+            metadata: h.body == null ? null : JSON.stringify(h.body),
+            checksum: h.body == null ? null : hashSpec(h.body),
+            recorded_at: new Date(i + 1).toISOString(),
+        });
+    });
+    for (const state of ['active', 'draft'] as const) {
+        const row = rows[state];
+        if (!row) continue;
+        tables.sys_metadata!.push({
+            ...base,
+            id: `m_${state}`,
+            state,
+            version: row.version,
+            metadata: JSON.stringify(row.body),
+            checksum: hashSpec(row.body!),
+        });
+    }
+}
+
 /**
  * [#20397] The DEFAULT range (no `toVersion`) labels its to side with the
  * version whose body it compares.
@@ -297,42 +337,6 @@ describe('#8798 — a history-table outage now answers the same way for every ty
  * real REST stack are `packages/rest/src/meta-diff-default-range-labels.test.ts`.
  */
 describe('#20397 — the default range labels the to side with the active row\'s own version', () => {
-    type Seed = { version: number; op: string; body: Record<string, unknown> | null };
-    /** History rows in version order, plus the active and draft rows they left behind. */
-    function seedLineage(
-        tables: Record<string, Array<Record<string, unknown>>>,
-        type: string,
-        name: string,
-        history: Seed[],
-        rows: { active?: Seed; draft?: Seed },
-    ) {
-        const base = { organization_id: null, type, name };
-        history.forEach((h, i) => {
-            tables.sys_metadata_history!.push({
-                ...base,
-                id: `h_${h.version}`,
-                version: h.version,
-                event_seq: i + 1,
-                operation_type: h.op,
-                metadata: h.body == null ? null : JSON.stringify(h.body),
-                checksum: h.body == null ? null : hashSpec(h.body),
-                recorded_at: new Date(i + 1).toISOString(),
-            });
-        });
-        for (const state of ['active', 'draft'] as const) {
-            const row = rows[state];
-            if (!row) continue;
-            tables.sys_metadata!.push({
-                ...base,
-                id: `m_${state}`,
-                state,
-                version: row.version,
-                metadata: JSON.stringify(row.body),
-                checksum: hashSpec(row.body!),
-            });
-        }
-    }
-
     for (const type of [ORDINARY_TYPE, 'app']) {
         it(`${type}: with a draft pending, toVersion is the active row's version and the answer is the explicit range's`, async () => {
             const { engine, tables } = makeStubEngine();
@@ -436,3 +440,211 @@ describe('#20397 — the default range labels the to side with the active row\'s
         });
     });
 });
+
+/**
+ * [#20451] With no `fromVersion`, the from side is the NEAREST EARLIER history
+ * row whose body DIFFERS from the to side's, by the diff's own equality:
+ * `diffShallow`'s three buckets all empty means equal, and a body-less row (a
+ * delete's tombstone) compares as `{}` — so the walk stops on it.
+ *
+ * A draft save appends a history row and a publish appends the promoted body
+ * again as the next one, so the row immediately before a published version
+ * repeats its body. The old rule (the row immediately before) answered "no
+ * changes" right after every publish. The lineages below are the ones the REST
+ * pins in `packages/rest/src/meta-diff-default-range-labels.test.ts` write
+ * through the real routes; here they are seeded, beside this file's
+ * read-counting double.
+ */
+describe('#20451 — the default from side is the nearest earlier row whose body differs from the to side\'s', () => {
+    const everythingAdded = (body: Record<string, unknown>) =>
+        Object.entries(body).map(([path, value]) => ({ path, value }));
+
+    it('v1 active, a v2 draft save, a v3 publish: 1 → 3, the change the publish carried, in ONE history read', async () => {
+        const { engine, tables, historyReads } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const b = { name: 'item', label: 'B' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'create', body: b },
+            { version: 3, op: 'publish', body: b },
+        ], { active: { version: 3, op: 'publish', body: b } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+
+        expect(historyReads()).toBe(1);
+        expect(res).toEqual({
+            type: ORDINARY_TYPE,
+            name: 'item',
+            fromVersion: 1,
+            toVersion: 3,
+            added: [],
+            removed: [],
+            changed: [{ path: 'label', from: 'A', to: 'B' }],
+        });
+        expect(res).toEqual(await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item', fromVersion: 1, toVersion: 3 }));
+    });
+
+    it('the same lineage with a v4 draft pending still answers 1 → 3: the draft is neither side', async () => {
+        const { engine, tables } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const b = { name: 'item', label: 'B' };
+        const c = { name: 'item', label: 'C pending' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'create', body: b },
+            { version: 3, op: 'publish', body: b },
+            { version: 4, op: 'create', body: c },
+        ], { active: { version: 3, op: 'publish', body: b }, draft: { version: 4, op: 'create', body: c } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+
+        expect(res.fromVersion).toBe(1);
+        expect(res.toVersion).toBe(3);
+        expect(res.changed).toEqual([{ path: 'label', from: 'A', to: 'B' }]);
+        expect(JSON.stringify(res)).not.toContain('C pending');
+    });
+
+    it('an explicit range names exactly its versions: 2 → 3 over that lineage is still "no changes"', async () => {
+        const { engine, tables } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const b = { name: 'item', label: 'B' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'create', body: b },
+            { version: 3, op: 'publish', body: b },
+        ], { active: { version: 3, op: 'publish', body: b } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item', fromVersion: 2, toVersion: 3 });
+
+        expect(res).toEqual({
+            type: ORDINARY_TYPE, name: 'item', fromVersion: 2, toVersion: 3, added: [], removed: [], changed: [],
+        });
+    });
+
+    it('an explicit `toVersion` alone keeps its version, and the from side walks back from ITS body', async () => {
+        const { engine, tables } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const b = { name: 'item', label: 'B' };
+        const c = { name: 'item', label: 'C' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'create', body: b },
+            { version: 3, op: 'publish', body: b },
+            { version: 4, op: 'update', body: c },
+        ], { active: { version: 4, op: 'update', body: c } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item', toVersion: 3 });
+
+        expect(res.fromVersion).toBe(1);
+        expect(res.toVersion).toBe(3);
+        expect(res.changed).toEqual([{ path: 'label', from: 'A', to: 'B' }]);
+    });
+
+    it('create, delete, draft save, publish: 2 → 4, everything added — the body-less delete row differs, so the walk stops on it', async () => {
+        const { engine, tables } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const a2 = { name: 'item', label: 'A2', columns: ['name'] };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'delete', body: null },
+            { version: 3, op: 'create', body: a2 },
+            { version: 4, op: 'publish', body: a2 },
+        ], { active: { version: 4, op: 'publish', body: a2 } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+
+        expect(res).toEqual({
+            type: ORDINARY_TYPE,
+            name: 'item',
+            fromVersion: 2,
+            toVersion: 4,
+            added: everythingAdded(a2),
+            removed: [],
+            changed: [],
+        });
+    });
+
+    it('create, delete, active recreate: 2 → 3, everything added — the answer the immediately-previous rule gave', async () => {
+        const { engine, tables } = makeStubEngine();
+        const a = { name: 'item', label: 'A' };
+        const b = { name: 'item', label: 'B' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: a },
+            { version: 2, op: 'delete', body: null },
+            { version: 3, op: 'create', body: b },
+        ], { active: { version: 3, op: 'create', body: b } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+        // Naming the tombstone as the to side: an absent body is `{}` there
+        // too, so the walk passes nothing and lands on v1, as it always did.
+        const deletion: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item', toVersion: 2 });
+
+        expect(res).toEqual({
+            type: ORDINARY_TYPE, name: 'item', fromVersion: 2, toVersion: 3, added: everythingAdded(b), removed: [], changed: [],
+        });
+        expect(deletion).toEqual({
+            type: ORDINARY_TYPE, name: 'item', fromVersion: 1, toVersion: 2, added: [], removed: everythingAdded(a), changed: [],
+        });
+    });
+
+    it('a brand-new item draft-saved then published: null → 2, everything added — no earlier row differs', async () => {
+        const { engine, tables } = makeStubEngine();
+        const n = { name: 'item', label: 'New' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: n },
+            { version: 2, op: 'publish', body: n },
+        ], { active: { version: 2, op: 'publish', body: n } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+
+        expect(res).toEqual({
+            type: ORDINARY_TYPE, name: 'item', fromVersion: null, toVersion: 2, added: everythingAdded(n), removed: [], changed: [],
+        });
+    });
+
+    it('a single version: null → 1, everything added', async () => {
+        const { engine, tables } = makeStubEngine();
+        const only = { name: 'item', label: 'Only' };
+        seedLineage(tables, ORDINARY_TYPE, 'item', [
+            { version: 1, op: 'create', body: only },
+        ], { active: { version: 1, op: 'create', body: only } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: ORDINARY_TYPE, name: 'item' });
+
+        expect(res).toEqual({
+            type: ORDINARY_TYPE, name: 'item', fromVersion: null, toVersion: 1, added: everythingAdded(only), removed: [], changed: [],
+        });
+    });
+
+    it('the walk compares RAW bodies: a credential-only rotation stops it, and the served values stay redacted', async () => {
+        // The #8671 ruling (diff raw, redact what is emitted) applies to the
+        // walk's comparison too. Compared redacted, the two bodies below are
+        // equal and the walk would pass the rotation by.
+        const { engine, tables } = makeStubEngine();
+        const config = (password: string) => ({ host: 'db.internal', username: 'reporting', password });
+        const old = { name: 'warehouse', label: 'Warehouse', driver: 'postgres', config: config('hunter2-old') };
+        const rotated = { name: 'warehouse', label: 'Warehouse', driver: 'postgres', config: config('hunter3-new') };
+        seedLineage(tables, 'datasource', 'warehouse', [
+            { version: 1, op: 'create', body: old },
+            { version: 2, op: 'create', body: rotated },
+            { version: 3, op: 'publish', body: rotated },
+        ], { active: { version: 3, op: 'publish', body: rotated } });
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const res: any = await protocol.diffMetaItem({ type: 'datasource', name: 'warehouse' });
+
+        expect(res.fromVersion).toBe(1);
+        expect(res.toVersion).toBe(3);
+        expect(res.changed.map((e: { path: string }) => e.path)).toEqual(['config']);
+        expect(JSON.stringify(res)).not.toContain('hunter2-old');
+        expect(JSON.stringify(res)).not.toContain('hunter3-new');
+    });
+});