Skip to content

Commit 05e3b8e

Browse files
feat(service-mailbox): store an ingested message's attachments, and each forwarded message as an .eml, through file-attachments (#23484)
Fixes #23307 Clause-②: no ## What this does `@objectstack/service-mailbox`'s ingest now stores every part of a stored message besides its text and HTML bodies as a file of that message's `sys_mail_message` row. It goes through the server-side verb `IFileAttachmentService.attachFile` (the kernel service `file-attachments`, `FILE_ATTACHMENT_SERVICE`, landed in #23386). Each attached file is one part. Each forwarded `message/rfc822` or `message/global` part is stored as the `.eml` file it travels as. - **Parent:** the row the message wrote (`parentObject: 'sys_mail_message'`, `parentId` = the new row's id). - **Organization:** the mailbox's own organization (`sys_mailbox.organization_id`), never `null`. This follows the seat pointer from #23386's contract review: the verb applies no per-user check, so this caller states its parent and its organization, and both are pinned below. - **Order:** the row first, because it is the parent. Then one `attachFile` per part, in message order. Then the association links and `associated`, as before. A message therefore leaves the restricted pool with its attachments already in place. - **Only on the stored path:** a duplicate (stage 1's dedupe, or the unique key settling a concurrent pair) and a suppressed message return before any `attachFile`. A re-delivery adds nothing, and `rematch` never reads a part. - ⛔ There is no second representation: no column of the row names a part, and no byte is stored in a column. The landing package is the producer, as the claim's file surface says. `@objectstack/service-storage` and `packages/spec/**` are not touched. ## H1: how the writer is resolved (measured) - **How ingest reaches a kernel service today.** The plugin hands `MailIngestService` a resolver closure for the engine (`engine: () => ctx.getService('objectql')`), which is read on each call. The attach verb follows the same shape. The constructor gains the optional `attachments: () => IFileAttachmentService | undefined`, and `MailboxServicePlugin` supplies `ctx.getService(FILE_ATTACHMENT_SERVICE)`, answering `undefined` when it is not registered. The kernel's `getService` throws for a name nothing registered (`core/src/kernel.ts`, the context accessor), so the plugin catches that. The resolver runs per message that carries a part, and nothing is cached, so plugin order does not matter. - **No new runtime dependency.** `FILE_ATTACHMENT_SERVICE` and `IFileAttachmentService` come from `@objectstack/spec/contracts`, which the package already depends on. `@objectstack/service-storage` is added as a **devDependency** only, for the real-verb pins (see "Surface beyond the claim's file list"). - **Choice when no `file-attachments` service is composed:** the message is stored without its parts, and a `warn` says so once per instance, naming the remedy (compose the storage capability). This is a composition gap, not a write that failed: AGENTS.md's degradation rule puts an `if (!service)` composition branch at `warn`. `storage` is on `PLATFORM_ALWAYS_ON_CAPABILITIES`, so this branch is reached only on a kernel whose slate leaves it out. - **Before `kernel:bootstrapped`**, the verb answers `503 SERVICE_UNAVAILABLE`. That is handled as a refusal (next section), not as absence. The IMAP sync's interval job first fires one interval after `kernel:ready` (`interval-job-adapter.ts`, `setInterval`), so the sync does not reach that window in practice. ## H2: what postal-mime 4.0.5 hands over (measured) I probed `PostalMime.parse` with the package's own options (`forceRfc822Attachments: true`, `attachmentEncoding: 'arraybuffer'`) on one message carrying each kind of part: - **Every non-body leaf part** arrives in `email.attachments` with `filename` (or `null`), `mimeType`, `disposition` (`attachment`, `inline` or `null`), and `content` as an `ArrayBuffer`. - **A forwarded `message/rfc822` part** arrives as an attachment whose bytes are **byte-identical** to the forwarded message. A `message/global` part arrives with its transfer encoding decoded, and its UTF-8 bytes are identical. The parts inside a forwarded message are not listed on their own, because `forceRfc822Attachments` keeps the part whole. They travel inside its `.eml`. - **An inline image** (`Content-Disposition: inline` with a `Content-ID`, inside `multipart/related`) arrives as an attachment with `disposition: 'inline'` and `related: true`. - **An Apple-Mail-style attached file** marked `Content-Disposition: inline` arrives with `disposition: 'inline'` and no `related`. - **A second `text/plain` part with no disposition** is folded into the text body. It is not an attachment. **Choice: inline parts count.** Every part postal-mime lists is stored, whatever its disposition. A disposition filter would drop the files Apple Mail attaches, because it marks them `inline`. It would also drop the `cid:` images the stored `body_html` references, which are bytes the message carries and which nothing else keeps. The line is postal-mime's own split between bodies and attachments. **Naming:** - A file keeps the name the message gives it. A part with no name is `attachment-N`, where N is its position among the message's parts. - A forwarded part is named by its own file name, else its Subject, else `forwarded`. The name always ends in `.eml`. - Control characters and path separators become `_`. A name over `sys_attachment.file_name`'s 255 is cut in its stem, keeping the extension. - The part's own media type is kept: a forwarded `message/global` part stays `message/global`, not `message/rfc822`. A type over `mime_type`'s 128 is stored as `application/octet-stream`. ## H3: order and failure (pinned) - **A refused part loses neither the message nor its other parts.** It is caught per part and logged at **`error`**, once, naming the part (number, file name, type, size), the message and the mailbox, and quoting the refusal's `CODE (status): sentence`. The verb's sentences name their own remedy, for example the `max_upload_mb` setting for `413`. - Why `error` and not `warn`: the message looks complete while a part it carried is gone, and nothing else will report it, because a re-delivery is a duplicate. That is the durability class in AGENTS.md's degradation rule. The package's IMAP sync already logs a message it skips at `error` for the same reason. - This required `MailIngestLogger` to gain `error`. See "For the seat". - **A re-delivery stops at stage 1's dedupe** before any `attachFile`. This is pinned on the double (call count) and on the real verb (`sys_attachment` and `sys_file` counts unchanged). - **A re-match never re-attaches**, and **a suppressed message attaches nothing** (pinned). ## H4: organization (pinned) A mailbox in organization A stores both rows (`sys_attachment`, `sys_file`) in A. A mailbox in organization B stores them in B. Each pin is the other's control. Both are pinned on the real verb, and the double records `organizationId` per call. ## Pins and their controls `src/mailbox-attachments-kernel.test.ts` boots a real `ObjectKernel` composing `ObjectQLPlugin`, `StorageServicePlugin` (local adapter in a temp directory, removed after the run, with a settings service carrying `max_upload_mb: 0.001` = 1048 bytes) and `MailboxServicePlugin`, over sqlite `:memory:`. The seam is reached as the `mailbox` service and finds the verb on its own. | Pin | Control | |---|---| | One forwarded message with one file gives 1 row and 2 `sys_attachment` rows on it (`quote.pdf`, `question.eml`). Each points at a committed `sys_file`, both in the mailbox's organization, and the stored bytes equal the file's and the forwarded message's. | A message without parts has no attachment. | | A re-delivered copy answers `duplicate`, and the `sys_attachment` and `sys_file` counts are unchanged. | The same parts under another Message-ID are another message's: +2. | | A mailbox in ORG_B stores both rows in ORG_B. | The first pin: ORG_A. | | A 1,304-byte part is refused `PAYLOAD_TOO_LARGE (413)` and leaves no `sys_file`, committed or pending. The message and its other two parts are stored, and one `error` names `big.bin`. | The first pin: parts within the limit are stored. | | On a kernel without storage, two messages are stored without parts, and one `warn` is said. | The first pin: with storage, no such line. | `src/ingest/mail-ingest-service.test.ts` (real engine, a recording double for the verb) pins what the seam hands the verb and when: - the parent row exists, in the mailbox's organization, at each call; - the calls follow message order, carrying the bytes and a `Buffer`; - `organizationId` is never `null`; - the dedupe, re-match and suppressed paths make no call; - the naming and type rules; - a `FILES_DISABLED (403)` refusal and an envelope-less throw are each said at `error`, and the other parts are still sent; - the no-verb path, for a resolver answering `undefined` and for no resolver at all, says one `warn`. `src/ingest/parse-message.test.ts` pins the H2 readings on the parser: every part listed in order with its bytes, a forwarded part byte-identical, parts past `MAX_FORWARDED_PARTS` still listed (their addresses are not read), and as controls, a plain message lists none and a second text part joins the body. ## Ablation (one, on the `attachFile` call) Run at `3a2884edb` with `node scripts/ablation-replace.mjs`. The anchor ` await verb.attachFile({` was replaced by ` void ({`, so the input is built and never sent. The tool's own log: anchor 1 → 0, replacement 0 → 1, blob `f8fc21733a6c` → `b86bc69395ad`. The subject resolves to `src/` through relative imports, so no `dist/` is involved. - **Result:** `Tests 12 failed | 33 passed (45)` over the two attachment suites. - Failed: every unit pin that expects a call (8), and the 4 real-verb pins (acceptance, re-delivery, ORG_B, 413). - Stayed green: the controls (no parts gives no call or no attachment, no refusal says nothing), the no-verb `warn` pins on both suites, and `arms`. - **Restore:** blob equals HEAD (`f8fc21733a6c`), and `git diff HEAD` is empty. The tool proved both. ## Tests and gates Local, at `2f293b262`, after merging `origin/main` (`cb9841aa2`) and rebuilding the closure: - `turbo run build --filter='@objectstack/service-mailbox...'`: 19/19 tasks. - `pnpm --filter @objectstack/service-mailbox typecheck`: exit 0. The package's `tsconfig.json` includes `src` with no test exclusion, so the new tests are type-checked. - `vitest run`: 10 files, 152 tests passed. - Reading, `@objectstack/service-storage` `src/file-attachment-service.test.ts`: 17 passed. Unchanged by this PR. - Gate families derived by `node scripts/pm/dispatch-gates.mjs --commands`: 79 derived, 77 exit 0. Two are NOT MEASURED locally, because they exited 3 with `PREREQUISITE NOT MET` (they read built output of the whole workspace): `check:dual-build-cjs-loads` and `check:i18n`. Lint & Repo Gates runs both in CI. `--ran` reconciliation: 79 accounted, 77 run, 2 NOT-MEASURED (derived from the recorded exit 3), 0 unrun. Also run, though not derived: `check:startup-registry-verdict` exit 0, `check:durability-log-level` exit 0. The families CI runs are read from this PR's check-runs, not re-run locally. ## Surface beyond the claim's file list (declared) - `packages/services/service-mailbox/package.json` and `pnpm-lock.yaml` (+3 lines, importer entry only): the `@objectstack/service-storage` **devDependency** for the real-verb pins, as the dispatch's suggested route asked. - `packages/services/service-mailbox/vitest.config.ts`: source aliases for `@objectstack/service-storage` and for what it reads (`@objectstack/observability`, `@objectstack/platform-objects/audit` and `/system`, `@objectstack/spec/api`, `/contracts`, `/kernel` and `/system`). With them, a verdict here is about the source, not a stale `dist/` (`check:test-source-alias`). - `src/sync/mailbox-sync-service.test.ts`: one line, so the logger it hands `MailIngestService` has `error`. ## For the seat - **Exported-type change.** `MailIngestLogger` (exported) gains a **required** `error(message)`, and `MailIngestService`'s options gain the optional `attachments`. No export is added or removed. `MailIngestService` first shipped in `18.0.0-next.1`, a prerelease, and no stable release carries it. The plugin passes `ctx.logger`, which has `error`. A host that constructs the seam itself with a warn-only logger now fails to type-check. Whether this moves the claim's `Clause-②: no` is the seat's call. The line above is copied from the claim unchanged. - **#22871:** this is the attachment item of #22871, which is in state `closed` (`completed`). Nothing else keeps it open, and this PR carries no keyword for it. ## Acceptance notes - An inline image is stored by its file name. Its `Content-ID` is not recorded, because `sys_attachment` has no column for it and a metadata-only column is ruled out here. A renderer cannot yet resolve a `cid:` reference in `body_html` to its stored file. That is a later capability, not a defect of this change. Carrier: none. - A forwarded part's own attachments are not stored as separate files. They travel inside its `.eml`, as the seam reads the forward the user made and not the forwards inside it. --- _Generated by [Claude Code](https://claude.ai/code/session_01YRp5z4kK6vgYkrdH97ztsy)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 8731510 commit 05e3b8e

13 files changed

Lines changed: 901 additions & 32 deletions
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/service-mailbox': minor
3+
---
4+
5+
Inbound mail stores its attachments: each attachment of an ingested message, and each message it forwards as an attachment as an `.eml` file, is stored through the `file-attachments` service as a file of the message's `sys_mail_message` row
6+
7+
Clause-②: no
8+
9+
- **What is stored, and where.** Every part of a stored message besides its text and HTML bodies (inline parts included, and each forwarded `message/rfc822` or `message/global` part as the `.eml` file it travels as) becomes one `sys_file` and one `sys_attachment` whose parent is the message's own `sys_mail_message` row, written by `IFileAttachmentService.attachFile` in the mailbox's organization after the row, and only for a message the call stored, so a re-delivered copy adds no attachment.
10+
- **Without the storage capability.** A kernel that composes no `file-attachments` service (registered by `@objectstack/service-storage`) stores each message without its attachments, and says so once in the log.
11+
- **Refused attachments.** A part the service refuses, such as one over the `max_upload_mb` upload size limit, is logged at `error` with its file name and the refusal, and the message and its other attachments are still stored.
12+
- **For a host that constructs `MailIngestService` itself.** It takes the service as the optional `attachments` resolver (`MailboxServicePlugin` supplies it), and its `MailIngestLogger` now needs `error`, as `MailSyncLogger` already does.

‎packages/services/service-mailbox/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record
44

55
> ⚠️ **Mail arrives only where an IMAP account source is composed.** The capability boot (`requires: ['mailbox']`) constructs the plugin with no options, so it mounts the objects, their read gates and the ingest seam, and syncs no mailbox — it says so once at boot. Compose the plugin yourself with `imap.accountFor` (below) to poll mailboxes.
66
>
7-
> ⚠️ **Attachments are not stored.** Nothing on the server writes `sys_attachment` rows today, so a message's attachments — and the body of a message forwarded as an attachment — are not kept; only its participants are read.
7+
> ⚠️ **Attachments are stored only where the storage capability is composed.** A message's attachments are written through the `file-attachments` service that `@objectstack/service-storage` registers (the `storage` capability, on the always-on slate). A kernel that does not compose it stores each message without its attachments, and says so once.
88
99
## What this package provides
1010

@@ -18,15 +18,15 @@ Inbound mail for ObjectStack: the mirrored message, its links to an app's record
1818
- **`manage_mailboxes`** — an organization capability declared in `@objectstack/spec`. Platform administrators and organization administrators hold it by default; every other position only by explicit grant. Object grants and field-level security still apply to a holder.
1919
- **The ingest seam** — `MailIngestService`, registered as the kernel service `mailbox` (`MAILBOX_SERVICE`):
2020
- **`ingest({ mailboxId, raw, receivedAt? }, context)`** reads one raw RFC 822 message and writes at most one `sys_mail_message` row, answering `stored`, `duplicate` or `suppressed`:
21-
- **participants** are the addresses of From, Sender, Reply-To, To and Cc, plus the same headers of a message forwarded as an attachment (a `message/rfc822` part, or its internationalized twin `message/global`, RFC 6532). At most `MAX_FORWARDED_PARTS` (20) forwarded parts of one message are read; the rest, and any part the MIME parser refuses, take no part in matching, and the message is stored without them (a warning names how many). The mailbox's own address is left out. A forward quoted inline in the body is read too, for an organization user's forward only: the "Forwarded message" block Gmail, Outlook (on the web and Microsoft 365) or Apple Mail writes into the text/plain body, with its English labels, adds its From, Reply-To, To and Cc — when the message's From is a user (`sys_user`) who is a member (`sys_member`) of the mailbox's organization. A block the parser does not recognise is not guessed at, and a forward from anyone else is not read: the message is matched on its other participants. The participants are stored on the row in `match_addresses`, each with its source (`outer`, `attachment` or `inline`);
21+
- **participants** are the addresses of From, Sender, Reply-To, To and Cc, plus the same headers of a message forwarded as an attachment (a `message/rfc822` part, or its internationalized twin `message/global`, RFC 6532). At most `MAX_FORWARDED_PARTS` (20) forwarded parts of one message are read; the rest, and any part the MIME parser refuses, take no part in matching (a warning names how many), and are still stored as attachments. The mailbox's own address is left out. A forward quoted inline in the body is read too, for an organization user's forward only: the "Forwarded message" block Gmail, Outlook (on the web and Microsoft 365) or Apple Mail writes into the text/plain body, with its English labels, adds its From, Reply-To, To and Cc — when the message's From is a user (`sys_user`) who is a member (`sys_member`) of the mailbox's organization. A block the parser does not recognise is not guessed at, and a forward from anyone else is not read: the message is matched on its other participants. The participants are stored on the row in `match_addresses`, each with its source (`outer`, `attachment` or `inline`);
2222
- **association** — every record of every object whose email field declares `mailAssociation: true` and holds one of those addresses (ASCII case-insensitive) is linked to the message, inside the mailbox's organization only, with one `sys_mail_association` row each (`matched_by: 'ingest'`). A message no record matches waits in the pool (`association_state: 'unassociated'`);
2323
- **dedupe** — one row per message per mailbox, keyed on the Message-ID (or, for a message without one, on its header block as received), so a second fetch answers `duplicate` and writes nothing;
2424
- **threading** — `message_id`, `in_reply_to` and `reference_ids` hold the ids of RFC 5322 §3.6.4, angle brackets stripped;
2525
- **suppression** — an auto-reply (`Auto-Submitted` other than `no`, `Precedence: bulk|junk|list`, `X-Autoreply`, `X-Autorespond`) or a bounce (`multipart/report; report-type=delivery-status`) is not stored at all.
2626

2727
Only the system context may ingest: a message enters a mailbox from its mail source, never from a user.
2828
- **`rematch({ mailboxId, messageIds? }, context)`** replays each message's stored participant set against the records that exist now and adds the links that are missing (`matched_by: 'rematch'`), moving a pool message to `associated` once it has one. It never removes a link. The system context may run it, and so may the mailbox's owner and a `manage_mailboxes` holder of the mailbox's organization; anyone else is refused (`PERMISSION_DENIED`), and another organization's mailbox is not found (`RECORD_NOT_FOUND`).
29-
- Attachments are not stored yet, so a message forwarded as an attachment is stored with the forwarder's own text, and the forwarded message's body is not kept.
29+
- **attachments** — each part of a stored message besides its text and HTML bodies is stored as a file of its `sys_mail_message` row: one `sys_file` and one `sys_attachment` (parent `sys_mail_message`, the row's id), through `IFileAttachmentService.attachFile`, the kernel service `file-attachments` (`FILE_ATTACHMENT_SERVICE`). Both rows land in the mailbox's organization, and the deployment's upload size limit (`max_upload_mb`) applies. That covers every file the message carries, a part marked `Content-Disposition: inline` included (Apple Mail marks attached files that way, and an image the HTML body shows by `cid:` reference is part of the message), and every message forwarded as an attachment (`message/rfc822` or `message/global`), stored as the `.eml` file it travels as: named by the part's own file name, else by its Subject, else `forwarded.eml`. A part with no file name is `attachment-<n>`, its position among the message's parts. Parts are stored only for a message this call stored, after its row: a duplicate or a suppressed message stores none, so a re-delivery adds no attachment, and a re-match never reads a part. A part the service refuses (over the upload size limit, for instance) is logged at `error` with its name and the refusal, and the message and its other parts are still stored. No column of the row names a part, and no byte is kept in a column;
3030
- **The IMAP sync** — `MailboxSyncService`, composed with `MailboxServicePlugin({ imap })` and run as the interval job `mailbox:imap-sync` on the composed job service (`requires: ['job']`):
3131
- for every mailbox your `accountFor` answers, it opens a **read-only** session on its IMAP account through `@objectstack/connector-imap` — the folder is opened with `EXAMINE` and bodies are fetched with `BODY.PEEK[]`, so the mailbox is never altered — and hands each message above the stored cursor, in UID order, to `ingest` under the system context. Dedupe, threading, association, the pool and suppression are the seam's, unchanged;
3232
- **the cursor** is kept on the mailbox row — `imap_folder`, `imap_uid_validity`, `imap_last_uid`, `imap_synced_at` — and written by the sync only. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; every message already stored then answers `duplicate`;

‎packages/services/service-mailbox/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
"@objectstack/driver-sql": "workspace:*",
3636
"@objectstack/lint": "workspace:*",
3737
"@objectstack/objectql": "workspace:*",
38+
"@objectstack/service-storage": "workspace:*",
3839
"@types/node": "^26.6.3",
3940
"typescript": "^6.0.3",
4041
"vitest": "^4.1.11"

0 commit comments

Comments
 (0)