fix(metadata-protocol)!: with no crypto provider, version tokens are keyed under a process-scoped ephemeral key, never empty (#21207)

The first cut served an empty version token on a host with no crypto
provider. Every save then handed out the same empty token, and a client
that sends no pin for an empty token turned every pinned reset into an
unpinned one: the optimistic lock failed open. CI's real reset-door pin
caught it.

The doors now key under the provider when one is registered and, while
none is, under 32 random bytes drawn once per process and never written
anywhere. A token is always served, differs when the content differs, is
never the unkeyed stored hash, and no empty or withheld token equals it.
A token held across a restart, or across a provider's registration, is
refused once with 409. The no-provider refusal branch is gone, and the
batch-publish conformance pin is back to its base bytes: it passes as it
was written.

Claude-Session: https://claude.ai/code/session_01VvcEokUG1tvVxkceYfR5XB
Co-authored-by: Claude <noreply@anthropic.com>

Author: claude

.changeset/21207-keyed-served-content-hash.md

@@ -11,14 +11,14 @@ fix(metadata-protocol)!: a metadata body's stored content hash is served and com
 
 Clause-②: yes (narrowing)
 
-<!-- adr-0087: not-required (no-migration-prescription) the stored content hash of a metadata body stays the canonical hash at rest and no metadata body, authorable key, spelling or export moves; what changes is the form a door serves the hash in (the crypto provider's keyed digest), the form an inbound version token is compared in, and which query shapes the doors accept over the two hash columns, so `objectstack migrate meta` has nothing to rewrite. The operator-run rewrite this release asks for is of audit, activity and decision-audit copies, not of metadata. The other categories are closed on facts: every package here publishes (not `unpublished`); no ADR-0087 id covers a served version token or a refused query shape (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+<!-- adr-0087: not-required (no-migration-prescription) the stored content hash of a metadata body stays the canonical hash at rest and no metadata body, authorable key, spelling or export moves; what changes is the form a door serves the hash in (a keyed digest: the crypto provider's, or a process-scoped ephemeral key's when none is registered), the form an inbound version token is compared in, and which query shapes the doors accept over the two hash columns, so `objectstack migrate meta` has nothing to rewrite. The operator-run rewrite this release asks for is of audit, activity and decision-audit copies, not of metadata. The other categories are closed on facts: every package here publishes (not `unpublished`); no ADR-0087 id covers a served version token or a refused query shape (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
 
 **BREAKING**: this narrows what the metadata doors serve and accept for the stored content hash of a metadata body — a hash over the whole stored body, withheld credential material included. Served beside the projected body it let a reader confirm a guess at that material offline; filtered on, it confirmed one online. It ships as `minor` under the launch-window convention for accept-set narrowings.
 
 **Three things change for callers and operators.**
 
-1. **A held version token gets one `409 METADATA_CONFLICT`.** Every door that hands out a metadata version token — the save, publish, package-publish and rollback receipts and the history read — now hands out the crypto provider's keyed digest of the stored hash instead of the hash itself, and the save and reset doors compare a token they are sent in that same form. A token a client held from before the upgrade is refused once; take the token from the next read or receipt and retry. A deployment that registers no crypto provider issues no token — the receipts' version is empty and the history read serves none — and refuses a sent token with the same `409`; send the write without one, or register a provider.
+1. **A held version token gets one `409 METADATA_CONFLICT`.** Every door that hands out a metadata version token — the save, publish, package-publish and rollback receipts and the history read — now hands out a keyed digest of the stored hash instead of the hash itself, and the save and reset doors compare a token they are sent in that same form. The key is the crypto provider's; a host that registers none keys under a process-scoped ephemeral key instead, so a token is always issued and never empty. A token a client held from before the upgrade is refused once; take the token from the next read or receipt and retry. On a host with no provider the same happens after a restart, and on any host when a provider is first registered. An empty, withheld, raw or stale token is refused with the same `409`; it is never read as "no pin".
 2. **Filter, sort and group on the two stored content-hash columns, and on the version history's change note, now answer `400 INVALID_FIELD`** — on the generic data door, the MCP stdio reader and the analytics door, before the engine runs. The change note is included because a draft promotion that stated no message of its own recorded the draft's stored hash in it; the publish door now always states a hash-free message, and a note written before this release is served with the quoted hash in keyed form. A data-door search over the two stored-metadata tables no longer scans those columns or the stored body column, and an explicit search-field list naming one answers the same `400`. Every other column of the two tables is served, filtered, sorted and grouped as before, and every other object is unchanged.
 3. **Operators run `os migrate audit-metadata-bodies` once after upgrading, dry run first.** The audit ledger, the activity feed and the metadata decision-audit trail no longer copy the stored hash. The extended command drops it from the copies already written and withholds it in the decision-audit notes and their copies: a dry run by default, `--apply` to rewrite, idempotent. The version history stays the lineage.
 
-**What else changes.** The data door and the MCP stdio reader serve the two hash columns of the stored-metadata tables in keyed form, or omit them with no provider. A `409` conflict refusal carries keyed values or none. The ObjectQL engine gains a read accessor for the registered provider's keyed digest; it is additive. A member's read of these tables is refused as before.
+**What else changes.** The data door serves the two hash columns of the stored-metadata tables in keyed form, under the same key as the version tokens. The MCP stdio reader serves them keyed under the crypto provider's key, and omits them on a host with no provider. A `409` conflict refusal carries keyed values or none. The ObjectQL engine gains a read accessor for the registered provider's keyed digest; it is additive. A member's read of these tables is refused as before.

packages/metadata-protocol/src/metadata-redaction.ts

@@ -66,6 +66,7 @@
  * belongs on that list first; that file is `packages/spec`'s to change.
  */
 
+import { createHmac, randomBytes } from 'node:crypto';
 import { getMetadataTypeRedactor } from '@objectstack/spec/kernel';
 import type { MetadataTypeRedactor } from '@objectstack/spec/kernel';
 // [#21120] The family-wide stored-metadata-body primitives — the object set,
@@ -829,8 +830,8 @@ const QUOTED_STORED_HASH = /(?<![\w-])sha256:[0-9a-f]{64}/g;
 
 /**
  * Free text with every quoted stored content hash replaced by its served form:
- * the keyed digest, or `(withheld)` with no crypto provider. Text that quotes
- * none is returned as is.
+ * the keyed digest, or `(withheld)` when the caller holds no digest. Text that
+ * quotes none is returned as is.
  */
 export async function serveStoredHashTokens(text: string, digest: StoredHashDigest | undefined): Promise<string> {
     const quoted = text.match(QUOTED_STORED_HASH);
@@ -843,12 +844,46 @@ export async function serveStoredHashTokens(text: string, digest: StoredHashDige
 /** The keyed-digest primitive of the registered crypto provider (`ICryptoProvider.keyedDigest`). */
 export type StoredHashDigest = (plain: string) => Promise<string>;
 
+/**
+ * [#21207] The process key {@link ephemeralStoredHashDigest} keys under: 32
+ * random bytes, drawn on first use, held only in this module, never written,
+ * logged or served.
+ */
+let ephemeralDigestKey: Buffer | undefined;
+
+/**
+ * [#21207] The keyed digest the `/meta` doors serve and compare metadata
+ * version tokens under while NO crypto provider is registered: `hmac-sha256:`
+ * plus hex, the provider contract's own output shape, under a process-scoped
+ * ephemeral key.
+ *
+ * Why a key and not "serve nothing": a version token is an optimistic lock.
+ * Serving none hands every caller the same empty token, and a client that
+ * (rightly) sends no pin for an empty token turns every pinned write into an
+ * unpinned one, so the lock fails OPEN without a word. A token keyed under a
+ * secret nobody outside this process holds keeps the three properties the lock
+ * needs: it differs when the content differs; it is never the unkeyed stored
+ * hash, so it confirms no guess at withheld material offline; and no empty or
+ * withheld value ever equals it.
+ *
+ * What it costs: the key dies with the process. A token held across a restart,
+ * or across the moment a host registers a real provider (the doors read the
+ * provider per use), names no current version and is refused once with
+ * `409 METADATA_CONFLICT`; the next read or receipt serves the current one.
+ * Every protocol in one process shares this key, so per-environment protocols
+ * answer one another's tokens.
+ */
+export const ephemeralStoredHashDigest: StoredHashDigest = async (plain: string): Promise<string> => {
+    ephemeralDigestKey ??= randomBytes(32);
+    return `hmac-sha256:${createHmac('sha256', ephemeralDigestKey).update(plain, 'utf8').digest('hex')}`;
+};
+
 /**
  * The form a stored content hash is SERVED in: the keyed digest of the stored
- * value under the provider's server-held key; `null` when nothing is stored
- * (a delete event, a first version's parent); `undefined` — WITHHELD — when no
- * provider is registered, or when the stored value is not a string this
- * function can judge. ⛔ Never the stored value itself.
+ * value under a server-held key; `null` when nothing is stored (a delete
+ * event, a first version's parent); `undefined` (WITHHELD) when the caller
+ * holds no digest, or when the stored value is not a string this function can
+ * judge. ⛔ Never the stored value itself.
  *
  * A failing digest is not caught: a provider that cannot compute it fails the
  * read rather than serving what it exists to replace.
@@ -865,9 +900,9 @@ export async function servedContentHash(
 /**
  * Serve one row of a stored-metadata table with its content-hash columns in
  * their served form ({@link servedContentHash}): keyed, `null` kept `null`, and
- * the column OMITTED when the value is withheld — and, with no crypto provider,
- * both columns omitted outright. A row of any other object, and a row carrying
- * neither column, is returned by reference.
+ * the column OMITTED when the value is withheld — and, when the caller holds no
+ * digest, both columns omitted outright. A row of any other object, and a row
+ * carrying neither column, is returned by reference.
  */
 export async function serveStoredMetadataHashColumns<T>(
     object: string,
@@ -879,7 +914,7 @@ export async function serveStoredMetadataHashColumns<T>(
     const out: Record<string, unknown> = { ...row };
     for (const column of STORED_METADATA_HASH_COLUMNS) {
         if (!(column in out)) continue;
-        // No provider: the column is not served at all — a `null` included, so
+        // No digest: the column is not served at all — a `null` included, so
         // a reader cannot tell a withheld hash from an absent one either.
         const served = digest ? await servedContentHash(out[column], digest) : undefined;
         if (served === undefined) delete out[column];

packages/metadata-protocol/src/protocol.data-door-stored-content-hash.test.ts

@@ -14,7 +14,7 @@
  *  - SERVED: each column is the crypto provider's keyed digest of the stored
  *    value — neither the stored value nor a recomputation, stable across reads,
  *    moving on any body change including a credential-only one; a `null` stays
- *    `null`; with no provider both columns are omitted;
+ *    `null`; with no provider the key is the process-scoped ephemeral one;
  *  - EVALUATED: a filter, sort or grouping on either column is refused before
  *    the engine is asked — `INVALID_FIELD` / 400, the shape the body column's
  *    refusals already answer — and a search never scans them (nor the body
@@ -156,20 +156,23 @@ describe('[#21207] data door — the content-hash columns are served keyed', ()
         expect(got.record.previous_checksum).toMatch(KEYED);
     });
 
-    it('no provider: both columns are omitted on every read, the other columns untouched', async () => {
+    it('no provider: both columns keyed under the process key, never stored, stable across reads', async () => {
         const { p } = makeProtocol({ provider: false });
         const list: any = await p.findData({ object: 'sys_metadata', query: {} });
-        for (const row of list.records) {
-            expect('checksum' in row).toBe(false);
-            expect(typeof row.name).toBe('string');
+        const again: any = await p.findData({ object: 'sys_metadata', query: {} });
+        for (const row of SYS_METADATA_ROWS) {
+            const served = byId(list.records, row.id);
+            expect(served.checksum).toMatch(KEYED);
+            expect(served.checksum).not.toBe(row.checksum);
+            expect(served.checksum).not.toBe(await keyedDigest(row.checksum));
+            expect(byId(again.records, row.id).checksum).toBe(served.checksum);
+            expect(typeof served.name).toBe('string');
         }
         const hist: any = await p.findData({ object: 'sys_metadata_history', query: {} });
-        for (const row of hist.records) {
-            expect('checksum' in row).toBe(false);
-            expect('previous_checksum' in row).toBe(false);
-        }
+        expect(byId(hist.records, 'h_1').previous_checksum).toBeNull();
+        for (const row of hist.records) expect(row.checksum).toMatch(KEYED);
         const got: any = await p.getData({ object: 'sys_metadata', id: 'm_view' });
-        expect('checksum' in got.record).toBe(false);
+        expect(got.record.checksum).toBe(byId(list.records, 'm_view').checksum);
         expect(got.record.name).toBe('all_tasks');
     });
 
@@ -261,17 +264,20 @@ describe('[#21207] data door — the history change note that quotes a stored ha
     const QUOTED = hashSpec({ quoted: true });
     const NOTE_ROWS = [{ id: 'h_note', type: 'view', name: 'all_tasks', version: 3, operation_type: 'publish', metadata: '{}', checksum: QUOTED, previous_checksum: null, change_note: `publish draft (hash ${QUOTED})` }];
 
-    it('is served with the quote keyed, and withheld with no provider', async () => {
+    it('is served with the quote keyed, under the provider\'s key or the process key', async () => {
         const saved = [...HISTORY_ROWS];
         HISTORY_ROWS.push(...(NOTE_ROWS as any));
         try {
             const { p } = makeProtocol();
             const got: any = await p.getData({ object: 'sys_metadata_history', id: 'h_note' });
             expect(got.record.change_note).toBe(`publish draft (hash ${await keyedDigest(QUOTED)})`);
             const bare = makeProtocol({ provider: false });
-            const withheld: any = await bare.p.findData({ object: 'sys_metadata_history', query: {} });
-            expect(byId(withheld.records, 'h_note').change_note).toBe('publish draft (hash (withheld))');
-            expect(JSON.stringify(withheld.records)).not.toContain(QUOTED);
+            const keyed: any = await bare.p.findData({ object: 'sys_metadata_history', query: {} });
+            const note = byId(keyed.records, 'h_note');
+            // The quote is served as the row's own served hash column.
+            expect(note.change_note).toBe(`publish draft (hash ${note.checksum})`);
+            expect(note.checksum).toMatch(KEYED);
+            expect(JSON.stringify(keyed.records)).not.toContain(QUOTED);
         } finally {
             HISTORY_ROWS.length = 0;
             HISTORY_ROWS.push(...saved);