Repository navigation
Commit 05e3b8e
feat(service-mailbox): store an ingested message's attachments, and each forwarded message as an .eml, through file-attachments (#23484)
Fixes #23307
Clause-②: no
## What this does
`@objectstack/service-mailbox`'s ingest now stores every part of a
stored message besides its text and HTML bodies as a file of that
message's `sys_mail_message` row. It goes through the server-side verb
`IFileAttachmentService.attachFile` (the kernel service
`file-attachments`, `FILE_ATTACHMENT_SERVICE`, landed in #23386). Each
attached file is one part. Each forwarded `message/rfc822` or
`message/global` part is stored as the `.eml` file it travels as.
- **Parent:** the row the message wrote (`parentObject:
'sys_mail_message'`, `parentId` = the new row's id).
- **Organization:** the mailbox's own organization
(`sys_mailbox.organization_id`), never `null`. This follows the seat
pointer from #23386's contract review: the verb applies no per-user
check, so this caller states its parent and its organization, and both
are pinned below.
- **Order:** the row first, because it is the parent. Then one
`attachFile` per part, in message order. Then the association links and
`associated`, as before. A message therefore leaves the restricted pool
with its attachments already in place.
- **Only on the stored path:** a duplicate (stage 1's dedupe, or the
unique key settling a concurrent pair) and a suppressed message return
before any `attachFile`. A re-delivery adds nothing, and `rematch` never
reads a part.
- ⛔ There is no second representation: no column of the row names a
part, and no byte is stored in a column.
The landing package is the producer, as the claim's file surface says.
`@objectstack/service-storage` and `packages/spec/**` are not touched.
## H1: how the writer is resolved (measured)
- **How ingest reaches a kernel service today.** The plugin hands
`MailIngestService` a resolver closure for the engine (`engine: () =>
ctx.getService('objectql')`), which is read on each call. The attach
verb follows the same shape. The constructor gains the optional
`attachments: () => IFileAttachmentService | undefined`, and
`MailboxServicePlugin` supplies
`ctx.getService(FILE_ATTACHMENT_SERVICE)`, answering `undefined` when it
is not registered. The kernel's `getService` throws for a name nothing
registered (`core/src/kernel.ts`, the context accessor), so the plugin
catches that. The resolver runs per message that carries a part, and
nothing is cached, so plugin order does not matter.
- **No new runtime dependency.** `FILE_ATTACHMENT_SERVICE` and
`IFileAttachmentService` come from `@objectstack/spec/contracts`, which
the package already depends on. `@objectstack/service-storage` is added
as a **devDependency** only, for the real-verb pins (see "Surface beyond
the claim's file list").
- **Choice when no `file-attachments` service is composed:** the message
is stored without its parts, and a `warn` says so once per instance,
naming the remedy (compose the storage capability). This is a
composition gap, not a write that failed: AGENTS.md's degradation rule
puts an `if (!service)` composition branch at `warn`. `storage` is on
`PLATFORM_ALWAYS_ON_CAPABILITIES`, so this branch is reached only on a
kernel whose slate leaves it out.
- **Before `kernel:bootstrapped`**, the verb answers `503
SERVICE_UNAVAILABLE`. That is handled as a refusal (next section), not
as absence. The IMAP sync's interval job first fires one interval after
`kernel:ready` (`interval-job-adapter.ts`, `setInterval`), so the sync
does not reach that window in practice.
## H2: what postal-mime 4.0.5 hands over (measured)
I probed `PostalMime.parse` with the package's own options
(`forceRfc822Attachments: true`, `attachmentEncoding: 'arraybuffer'`) on
one message carrying each kind of part:
- **Every non-body leaf part** arrives in `email.attachments` with
`filename` (or `null`), `mimeType`, `disposition` (`attachment`,
`inline` or `null`), and `content` as an `ArrayBuffer`.
- **A forwarded `message/rfc822` part** arrives as an attachment whose
bytes are **byte-identical** to the forwarded message. A
`message/global` part arrives with its transfer encoding decoded, and
its UTF-8 bytes are identical. The parts inside a forwarded message are
not listed on their own, because `forceRfc822Attachments` keeps the part
whole. They travel inside its `.eml`.
- **An inline image** (`Content-Disposition: inline` with a
`Content-ID`, inside `multipart/related`) arrives as an attachment with
`disposition: 'inline'` and `related: true`.
- **An Apple-Mail-style attached file** marked `Content-Disposition:
inline` arrives with `disposition: 'inline'` and no `related`.
- **A second `text/plain` part with no disposition** is folded into the
text body. It is not an attachment.
**Choice: inline parts count.** Every part postal-mime lists is stored,
whatever its disposition. A disposition filter would drop the files
Apple Mail attaches, because it marks them `inline`. It would also drop
the `cid:` images the stored `body_html` references, which are bytes the
message carries and which nothing else keeps. The line is postal-mime's
own split between bodies and attachments.
**Naming:**
- A file keeps the name the message gives it. A part with no name is
`attachment-N`, where N is its position among the message's parts.
- A forwarded part is named by its own file name, else its Subject, else
`forwarded`. The name always ends in `.eml`.
- Control characters and path separators become `_`. A name over
`sys_attachment.file_name`'s 255 is cut in its stem, keeping the
extension.
- The part's own media type is kept: a forwarded `message/global` part
stays `message/global`, not `message/rfc822`. A type over `mime_type`'s
128 is stored as `application/octet-stream`.
## H3: order and failure (pinned)
- **A refused part loses neither the message nor its other parts.** It
is caught per part and logged at **`error`**, once, naming the part
(number, file name, type, size), the message and the mailbox, and
quoting the refusal's `CODE (status): sentence`. The verb's sentences
name their own remedy, for example the `max_upload_mb` setting for
`413`.
- Why `error` and not `warn`: the message looks complete while a part it
carried is gone, and nothing else will report it, because a re-delivery
is a duplicate. That is the durability class in AGENTS.md's degradation
rule. The package's IMAP sync already logs a message it skips at `error`
for the same reason.
- This required `MailIngestLogger` to gain `error`. See "For the seat".
- **A re-delivery stops at stage 1's dedupe** before any `attachFile`.
This is pinned on the double (call count) and on the real verb
(`sys_attachment` and `sys_file` counts unchanged).
- **A re-match never re-attaches**, and **a suppressed message attaches
nothing** (pinned).
## H4: organization (pinned)
A mailbox in organization A stores both rows (`sys_attachment`,
`sys_file`) in A. A mailbox in organization B stores them in B. Each pin
is the other's control. Both are pinned on the real verb, and the double
records `organizationId` per call.
## Pins and their controls
`src/mailbox-attachments-kernel.test.ts` boots a real `ObjectKernel`
composing `ObjectQLPlugin`, `StorageServicePlugin` (local adapter in a
temp directory, removed after the run, with a settings service carrying
`max_upload_mb: 0.001` = 1048 bytes) and `MailboxServicePlugin`, over
sqlite `:memory:`. The seam is reached as the `mailbox` service and
finds the verb on its own.
| Pin | Control |
|---|---|
| One forwarded message with one file gives 1 row and 2 `sys_attachment`
rows on it (`quote.pdf`, `question.eml`). Each points at a committed
`sys_file`, both in the mailbox's organization, and the stored bytes
equal the file's and the forwarded message's. | A message without parts
has no attachment. |
| A re-delivered copy answers `duplicate`, and the `sys_attachment` and
`sys_file` counts are unchanged. | The same parts under another
Message-ID are another message's: +2. |
| A mailbox in ORG_B stores both rows in ORG_B. | The first pin: ORG_A.
|
| A 1,304-byte part is refused `PAYLOAD_TOO_LARGE (413)` and leaves no
`sys_file`, committed or pending. The message and its other two parts
are stored, and one `error` names `big.bin`. | The first pin: parts
within the limit are stored. |
| On a kernel without storage, two messages are stored without parts,
and one `warn` is said. | The first pin: with storage, no such line. |
`src/ingest/mail-ingest-service.test.ts` (real engine, a recording
double for the verb) pins what the seam hands the verb and when:
- the parent row exists, in the mailbox's organization, at each call;
- the calls follow message order, carrying the bytes and a `Buffer`;
- `organizationId` is never `null`;
- the dedupe, re-match and suppressed paths make no call;
- the naming and type rules;
- a `FILES_DISABLED (403)` refusal and an envelope-less throw are each
said at `error`, and the other parts are still sent;
- the no-verb path, for a resolver answering `undefined` and for no
resolver at all, says one `warn`.
`src/ingest/parse-message.test.ts` pins the H2 readings on the parser:
every part listed in order with its bytes, a forwarded part
byte-identical, parts past `MAX_FORWARDED_PARTS` still listed (their
addresses are not read), and as controls, a plain message lists none and
a second text part joins the body.
## Ablation (one, on the `attachFile` call)
Run at `3a2884edb` with `node scripts/ablation-replace.mjs`. The anchor
` await verb.attachFile({` was replaced by ` void ({`, so the input is
built and never sent. The tool's own log: anchor 1 → 0, replacement 0 →
1, blob `f8fc21733a6c` → `b86bc69395ad`. The subject resolves to `src/`
through relative imports, so no `dist/` is involved.
- **Result:** `Tests 12 failed | 33 passed (45)` over the two attachment
suites.
- Failed: every unit pin that expects a call (8), and the 4 real-verb
pins (acceptance, re-delivery, ORG_B, 413).
- Stayed green: the controls (no parts gives no call or no attachment,
no refusal says nothing), the no-verb `warn` pins on both suites, and
`arms`.
- **Restore:** blob equals HEAD (`f8fc21733a6c`), and `git diff HEAD` is
empty. The tool proved both.
## Tests and gates
Local, at `2f293b262`, after merging `origin/main` (`cb9841aa2`) and
rebuilding the closure:
- `turbo run build --filter='@objectstack/service-mailbox...'`: 19/19
tasks.
- `pnpm --filter @objectstack/service-mailbox typecheck`: exit 0. The
package's `tsconfig.json` includes `src` with no test exclusion, so the
new tests are type-checked.
- `vitest run`: 10 files, 152 tests passed.
- Reading, `@objectstack/service-storage`
`src/file-attachment-service.test.ts`: 17 passed. Unchanged by this PR.
- Gate families derived by `node scripts/pm/dispatch-gates.mjs
--commands`: 79 derived, 77 exit 0. Two are NOT MEASURED locally,
because they exited 3 with `PREREQUISITE NOT MET` (they read built
output of the whole workspace): `check:dual-build-cjs-loads` and
`check:i18n`. Lint & Repo Gates runs both in CI. `--ran` reconciliation:
79 accounted, 77 run, 2 NOT-MEASURED (derived from the recorded exit 3),
0 unrun. Also run, though not derived: `check:startup-registry-verdict`
exit 0, `check:durability-log-level` exit 0.
The families CI runs are read from this PR's check-runs, not re-run
locally.
## Surface beyond the claim's file list (declared)
- `packages/services/service-mailbox/package.json` and `pnpm-lock.yaml`
(+3 lines, importer entry only): the `@objectstack/service-storage`
**devDependency** for the real-verb pins, as the dispatch's suggested
route asked.
- `packages/services/service-mailbox/vitest.config.ts`: source aliases
for `@objectstack/service-storage` and for what it reads
(`@objectstack/observability`, `@objectstack/platform-objects/audit` and
`/system`, `@objectstack/spec/api`, `/contracts`, `/kernel` and
`/system`). With them, a verdict here is about the source, not a stale
`dist/` (`check:test-source-alias`).
- `src/sync/mailbox-sync-service.test.ts`: one line, so the logger it
hands `MailIngestService` has `error`.
## For the seat
- **Exported-type change.** `MailIngestLogger` (exported) gains a
**required** `error(message)`, and `MailIngestService`'s options gain
the optional `attachments`. No export is added or removed.
`MailIngestService` first shipped in `18.0.0-next.1`, a prerelease, and
no stable release carries it. The plugin passes `ctx.logger`, which has
`error`. A host that constructs the seam itself with a warn-only logger
now fails to type-check. Whether this moves the claim's `Clause-②: no`
is the seat's call. The line above is copied from the claim unchanged.
- **#22871:** this is the attachment item of #22871, which is in state
`closed` (`completed`). Nothing else keeps it open, and this PR carries
no keyword for it.
## Acceptance notes
- An inline image is stored by its file name. Its `Content-ID` is not
recorded, because `sys_attachment` has no column for it and a
metadata-only column is ruled out here. A renderer cannot yet resolve a
`cid:` reference in `body_html` to its stored file. That is a later
capability, not a defect of this change. Carrier: none.
- A forwarded part's own attachments are not stored as separate files.
They travel inside its `.eml`, as the seam reads the forward the user
made and not the forwards inside it.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01YRp5z4kK6vgYkrdH97ztsy)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 8731510 commit 05e3b8e
13 files changed
Lines changed: 901 additions & 32 deletions
File tree
- .changeset
- packages/services/service-mailbox
- src
- ingest
- sync
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
0 commit comments