Skip to content

plugin-email: the boot sweep rewrites every effective email template on every boot (4 serial statements and a full sys_email_template scan each); since 08adfeade the default org's overlays are in that loop #22062

Description

@objectstack-fleet

Filing gate: ① a product defect, with its reach measured locally (cloud's real kernel factory on the hosted Turso face). The staging attribution is pending one log read (see Reach).

Filed by the repo:cloud seat (repo:cloud#1, session session_01Wxo1xhh2bU66T73q23jzE4, R44), from objectstack-ai/cloud#2637 (p1), round 2 (os-dev-report 6031983341 on that card). ⛔ Not a claim. The maintainer ranked cloud#2637 first today.

The step

bootstrapEffectiveEmailTemplates in packages/plugins/plugin-email/src/bootstrap-declared-email-templates.ts (loop at about :412, called from email-plugin.ts:1105 at EmailServicePlugin start) calls upsertDeclaredEmailTemplate (about :276-323) for every template of the effective list, on every boot. Each call:

  1. finds the row with find(..., { where: { name, locale }, limit: 1 }). That is SCAN sys_email_template, because the declared unique index is (COALESCE(organization_id,'__global__'), name, locale) and cannot serve (name, locale);
  2. UPDATEs it unconditionally, even when nothing changed;
  3. runs the engine's two read-backs by id.

That is 4 serial round trips per template per boot, and K × T rows visited.

08adfeade (#21818, Fixes #21785) reads the list through protocol.getMetaItems in tenancy.defaultOrgId() (about :234), instead of the registry. That is correct for #21785: an org-scoped template edit must survive a boot. But it means the default organization's email_template overlays now enter this per-template loop. The unconditional per-template rewrite itself predates that commit.

Reach (measured)

All figures are from cloud's ArtifactKernelFactory on the hosted Turso remote face, with byte-identical database copies per pin, steady-state boots, and 36 ms injected per round trip.

shape 7d078148 4e4e881427 (17.7.0)
HotCRM, 79 templates (every hosted HotCRM environment) 26.2 s 27.0 s
+ 1,000 default-org template overlays 26.2 s 183.0 s
+ 5,000 default-org template overlays 26.3 s 812.6 s (first build 814.5 s, past the 600 s hard timeout)
  • On the plain HotCRM shape, the sweep is 316 of the phase's ~830 serial round trips.
  • 8832655a matches 4e4e881427 on every shape.
  • Staging (cloud#2637). Since cloud moved to 8832655a, every build of the large environment 1ac85ba2 has passed the 600 s hard timeout, with a phase of about 14 minutes. Whether 1ac85ba2 holds thousands of default-org email templates is not yet known. One staging log line decides it, and the maintainer is asked for it on cloud#2637.
  • Independent of that reading, every hosted HotCRM environment pays the rewrite on every boot.

Proposed fix (at the producer; keeps #21785's contract)

  • Bulk-read the stored rows for the declared names, in $in pages of 200, instead of one find per template. The first row by id wins, as before.
  • Compare before write: a managed_by: 'package' row that already holds the projected columns is not rewritten.
  • Fall back: if the bulk read fails, use the per-template lookup.

The live doors keep their own lookup. One source file (+72/-1) and its test (+18/-1). No spec, schema, contract or migration change.

A rig-only trial at 4e4e881427 (reverted, never pushed) measured:

  • 5,000 default-org templates: 812.6 s → 20.1 s at 36 ms; round trips 20,717 → 427 at 0 ms;
  • the plain HotCRM shape: 27.0 s → 15.1 s at 36 ms (1.59 s → 1.28 s at 0 ms);
  • plugin-email tests: 44 passed. The new test fails on the unpatched source: expected { seeded: 450, skipped: 0 } to deeply equal { seeded: 0, skipped: 450 }.

The draft patch below is against 4e4e881427; the file pair is identical at 8832655a. It is a starting point for the implementer, not a reviewed change.

--- a/packages/plugins/plugin-email/src/bootstrap-declared-email-templates.ts
+++ b/packages/plugins/plugin-email/src/bootstrap-declared-email-templates.ts
@@ -266,4 +266,48 @@
 }
 
+/** [cloud#2637] The sweep's row key: a template is unique per `(name, locale)`. */
+function templateKey(name: unknown, locale: unknown): string {
+  return JSON.stringify([String(name), String(locale)]);
+}
+
+/**
+ * [cloud#2637] True when every column the projection writes already holds the
+ * projected value: the same column set an update would write. Booleans are
+ * stored as 0/1 and an absent optional column reads as null, so both sides are
+ * normalized before the compare.
+ */
+function projectionHeld(row: { [column: string]: unknown }, projected: { [column: string]: unknown }): boolean {
+  const norm = (v: unknown) => (v === undefined ? null : typeof v === 'boolean' ? (v ? 1 : 0) : v);
+  return Object.entries(projected).every(([k, v]) => norm(row[k]) === norm(v));
+}
+
+/**
+ * [cloud#2637] The stored rows for the declared names, read in `$in` pages of
+ * 200 instead of one lookup per template. The first row by id wins, as the
+ * per-template `find(..., { limit: 1 })` it replaces resolved it. Undefined
+ * when the read fails: the sweep then looks up per template, as before.
+ */
+async function prefetchTemplateRows(engine: IDataEngine, declared: unknown[], object: string) {
+  const names = [...new Set(declared.map((d) => (d as { name?: unknown })?.name))].filter((n) => typeof n === 'string');
+  const byKey = new Map();
+  try {
+    for (let i = 0; names.length > i; i += 200) {
+      const found = await (engine as any).find(object, {
+        where: { name: { $in: names.slice(i, i + 200) } },
+        context: SYSTEM_CTX,
+      });
+      const rows: any[] = Array.isArray(found) ? found : ((found as any)?.data ?? []);
+      for (const row of rows) {
+        const key = templateKey(row?.name, row?.locale);
+        const held = byKey.get(key);
+        if (!held || String(held.id) > String(row.id)) byKey.set(key, row);
+      }
+    }
+  } catch {
+    return undefined;
+  }
+  return byKey;
+}
+
 /**
  * Materialize ONE declared template into `sys_email_template`, honouring
@@ -289,5 +333,22 @@
   });
   const row: any = Array.isArray(existing) ? existing[0] : (existing as any)?.data?.[0];
+  return writeTemplateRow(engine, tpl, row, now, object, logger);
+}
 
+/**
+ * [cloud#2637] Write one parsed template against the row already read for its
+ * `(name, locale)`: by the live doors' own lookup above, or by the boot sweep's
+ * bulk read. Seed-not-clobber as before, and a row that already holds the
+ * projection is not rewritten (before, every boot rewrote every template: one
+ * UPDATE plus its two read-backs each).
+ */
+async function writeTemplateRow(
+  engine: IDataEngine,
+  tpl: EmailTemplateDefinition,
+  row: any,
+  now: string,
+  object: string,
+  logger?: Logger,
+) {
   if (row?.id) {
     // Admin owns a same-named row, or has edited this seeded one — never
@@ -301,4 +362,5 @@
     }
     if (row.customized === true) return false;
+    if (row.managed_by === 'package' && projectionHeld(row, mapTemplateToRow(tpl))) return false;
     await (engine as any).update(object, {
       id: row.id,
@@ -410,7 +472,16 @@
   let skipped = 0;
 
+  const prefetched = await prefetchTemplateRows(engine, declared, object);
+
   for (const raw of declared) {
     try {
-      const written = await upsertDeclaredEmailTemplate(engine, raw, object, logger);
+      let written: boolean;
+      if (prefetched) {
+        const tpl = EmailTemplateDefinitionSchema.parse(raw);
+        const row = prefetched.get(templateKey(tpl.name, tpl.locale));
+        written = await writeTemplateRow(engine, tpl, row, new Date().toISOString(), object, logger);
+      } else {
+        written = await upsertDeclaredEmailTemplate(engine, raw, object, logger);
+      }
       if (written) seeded += 1;
       else skipped += 1;
--- a/packages/plugins/plugin-email/src/bootstrap-declared-email-templates.test.ts
+++ b/packages/plugins/plugin-email/src/bootstrap-declared-email-templates.test.ts
@@ -56,3 +56,5 @@
     if (!cond) return true;
-    return Object.entries(cond).every(([k, v]) => row[k] === v);
+    // `$in` as the real engine reads it: the bulk read the boot sweep issues.
+    return Object.entries(cond).every(([k, v]) =>
+      v && typeof v === 'object' && Array.isArray((v as any).$in) ? (v as any).$in.includes(row[k]) : row[k] === v);
   }
@@ -173,2 +175,17 @@
   });
+
+  it('[cloud#2637] a boot over unchanged templates writes nothing and reads in bulk, not per template', async () => {
+    const declared = Array.from({ length: 450 }, (_, i) => declaredTemplate({ name: `tpl.n${i}` }));
+    const engine = new FakeEngine({ declared: { email_template: declared } });
+    await bootstrapDeclaredEmailTemplates(engine as any, undefined);
+
+    const find = vi.spyOn(engine, 'find');
+    const update = vi.spyOn(engine, 'update');
+    const result = await bootstrapDeclaredEmailTemplates(engine as any, undefined);
+
+    expect(result).toEqual({ seeded: 0, skipped: 450 });
+    expect(update).not.toHaveBeenCalled();
+    expect(find).toHaveBeenCalledTimes(3);
+    expect(rowsOf(engine)).toHaveLength(450);
+  });
 

⛔ Not the fix: reverting or narrowing 08adfeade back to the registry. That reopens #21785: an org-scoped template edit lost on the next boot.

Timing

Cloud stays on 17.x until its own v18 ceremony (objectstack-ai/cloud#1979; ruling recorded on #15193). So this fix reaches cloud only if it lands on objectstack main before the v18 opening. Its size fits that window.

Done when

  • A steady boot over unchanged templates issues no sys_email_template write, and one read per 200 template names (the pin test above).
  • If cloud#2637's staging reading attributes the time to this step: 1ac85ba2's first kernel build reaches kernel ready inside the 600 s hard timeout on a cloud pin that carries the fix. That half is verified on cloud#2637.

Also measured, not filed (costs, not defect classes)

  • 18c2ddc1e adds one index-served sys_metadata read per stored active permission set per boot (overlayLockLayerAt beside getMetaItem's findOne): about +22 s at 36 ms for 600 sets.
  • backfillOrgAdminGrants (plugin-security) reconciles every member on every boot, 5 serial statements per member up to 5,000. That is about 25,900 round trips per boot at 20,000 members, at every pin.

Reader

The objectstack lane that owns plugin-email. The repo:cloud seat verifies the cloud half on cloud#2637 after the pin carries the fix.


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratebugSomething isn't workingdomain:servicespriority:p1High: required for production / M2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions