Skip to content

Commit 3d830b7

Browse files
feat(connector-imap, service-mailbox): the IMAP mail source — a read-only IMAP connector and the mailbox sync that feeds the ingest seam (#23303)
Fixes #22871 Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; the `mailAssociation` liveness row goes `live`) ## Stage 2 of #22871: `connector-imap` feeds the stage-1 ingest seam Stage 1 (PR #23250, `82581819b`) shipped the ingest seam: `MailIngestService.ingest` turns one raw RFC 822 message into one `sys_mail_message` row, with dedupe, threading, association, the pool and suppression. Nothing fed it. This PR adds the mail source: a new package, `@objectstack/connector-imap`, and an IMAP sync in `@objectstack/service-mailbox` that polls each mailbox's folder above a stored UID cursor and hands every message to the seam. Stage 1's dedupe, threading, matching, re-match and suppression are reused unchanged (ruling 6116489638: Q1 A, Q3 C phase 1, Q4 A and Q5 A are not touched). Attachment extraction, the card's last scope item, moves to #23307 (`Blocked-by: #23306`, the server-side `sys_attachment` writer), under the seat's ruling B (`6151023438`). ### `@objectstack/connector-imap` (new): protocol and auth only - `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes? })` opens a **read-only** IMAP4rev1 session: - `examine(folder)` answers `uidValidity`, `uidNext` and `exists`. It uses `EXAMINE` and never `SELECT`. A folder opened without a UIDVALIDITY is refused. - `listMessages(fromUid)` and `fetchMessages(uids)` fetch with `BODY.PEEK[]`, so no message is ever marked `\Seen`. - `logout()` and `close()` end the session. - **The account and credential abstraction is shaped for the per-user mailbox (decision 4).** - `MailAccount` is `{ id, username, imap: { host, port?, security? } }`, one identity per account, never a process-wide setting. - `MailCredentialProvider.credentialFor(account)` is resolved each time a session opens and never cached. A per-user provider can therefore read a user's stored credential and refresh an OAuth token without changing the connector. - The endpoint sits under `imap`, so the same account can later name a calendar endpoint. The design record shares one account between the mail and calendar engines. - `staticCredentials(credential)` is the shared mailbox's implementor: a fixed credential the host hands over. - **Auth:** - a `password` uses `AUTHENTICATE PLAIN` (with `SASL-IR` when offered), otherwise `LOGIN`, with a synchronizing literal when quoting cannot carry the password; - an `oauth2` token uses `AUTHENTICATE XOAUTH2`, including its error-continuation flow; - a server that announces `LOGINDISABLED` and offers no `PLAIN` is refused before anything is sent. - **Transport:** implicit TLS by default, with the certificate verified. `security: 'none'` is refused to any host but loopback, before a socket opens. - **Shape:** it mirrors the existing connectors' package shape: manifest, `LICENSE`, `tsconfig.json`, `vitest.config.ts` and co-located tests. It has no runtime dependencies. It registers **no** `connector_action` handler and ships no plugin, because nothing in a flow calls IMAP; the mailbox service does. - `@objectstack/connector-imap/testing` exports `ImapServerDouble`, the in-process server the tests run against. It is a separate entry with its own `tsup.config.ts`. ### `@objectstack/service-mailbox`: the sync half - **Composition:** `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service at `kernel:ready`. `destroy()` cancels the job and closes any open session. - The capability boot constructs the plugin with no options, and then no mailbox is synced. The boot says so once, at info. - With `imap` but no job service, the boot warns. `plugin.syncService.syncAll({ isSystem: true })` is the manual path. - A run in which a mailbox could not be synced resolves `{ outcome: 'degraded' }` for the job service (the `JobRunOutcome` contract). - **`MailboxSyncService`, per mailbox:** - it asks `accountFor(mailbox)` for the account and credential (`null` means not synced); - it opens the session, reads the folder's UIDVALIDITY, and lists the UIDs above the cursor; - it fetches them `batchSize` at a time and hands each to `ingest` under the system context, in UID order. - **The cursor** lives on `sys_mailbox` in four new readonly fields, written by the sync only: `imap_folder`, `imap_uid_validity`, `imap_last_uid` and `imap_synced_at`. That is a widening of a published object, declared in the changeset. - A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start. Messages already stored answer `duplicate`, so nothing doubles. - The cursor moves after each batch, past every message stored, already stored, suppressed or skipped. A store failure stops the run below the failed message, so the next run retries it. - A message above `maxMessageBytes` (32 MiB by default), or one the seam refuses as unreadable, is skipped so it cannot stall the mailbox. It is logged at `error` once, naming its UID, with the consequence and the fix. - One mailbox is never synced twice at once in one process. Two processes waste a fetch and store nothing twice, because the dedupe key is unique. - **No credential is stored on any object.** The account and credential come from the host's `accountFor`, the way every open connector takes its credential (measured below). `system-context.mdx` gains row 47c for the sync's two system-only entry points. ### Stage 1's four carried items (`6145459294`) 1. **`mailAssociation` → `live`** (`packages/spec/liveness/field.json`). `authorWarn` and `authorHint` are dropped. The `authorWarnedProperties('field')` pin becomes `['conditionalRequired']`, and the `field` state counts go from 94 live / 4 planned to 95 / 3. The evidence anchor is `match-records.ts#MailRecordMatcher` rather than the dispatch's `#MailRecordMatcher.match`, because `check:liveness` refuses a dotted anchor ("an anchor must be one identifier"). The prose names `match`, through `flaggedFieldsOf`. This flip is its own first commit (`610e112bd`). Under the seat answer `6151023438` it lands first on its own PR, #23314. This PR keeps the commit, which becomes a no-op once #23314 is on `main` and `main` is merged here. 2. **Attachment storage:** not built here. It moves to #23307 under ruling B (`6151023438`). See Measurements, H5. 3. **`message/global` (RFC 6532):** it is now read as an `attachment` source, like `message/rfc822`, including a base64-encoded part. It is pinned in `parse-message.test.ts`. 4. **`MAX_FORWARDED_PARTS = 20`** is named in the service README, with what happens past it. ### Measurements (Zone 2), at the merge base `175e8d414` - **H1, the IMAP client: an own minimal implementation, zero runtime dependencies.** - `imapflow@2.4.0` (MIT, ESM and CJS, 1.75 MB unpacked) pins 8 exact-version dependencies: `pino` with its worker-thread transport, `socks`, `iconv-lite`, `libmime`, `libqp`, `libbase64`, `encoding-japanese` and `@zone-eu/mailsplit`. That is a lot for the five commands a read-only mirror sends. - `imap@0.8.19` (node-imap) is unmaintained. - The subset this needs is `CAPABILITY`, `LOGIN` / `AUTHENTICATE`, `EXAMINE`, `UID FETCH` and `LOGOUT`, plus literal framing. That is about 1,000 lines with the wire reader and the account types, and MIME stays with stage 1's `postal-mime`. - **H1, the server double: an own in-process double, plus a one-off interop probe against an independent strict server.** - `hoodiecrow-imap` was dormant from 2.1.0 (2016) until 3.x on 2026-10-06/07, and was deprecated hours later as "Renamed to imapkit". - `imapkit` was created 2026-10-07 and had 11 releases in two days; 5.0.1 is five days old. - `wildduck` needs MongoDB and Redis. - So no third-party package is added, even as a dev dependency. `ImapServerDouble` shares no parser or constant with the client. - The client was probed once, outside the tree, against **both** `imapkit@5.0.1` and `hoodiecrow-imap@3.3.1`. Both are strict RFC 3501 servers by the postal-mime maintainers. 18 checks, **all PASS on both**: - `LOGIN`, `AUTHENTICATE PLAIN` with and without `SASL-IR`, a UTF-8 password, and a password carrying a double quote and a backslash; - `XOAUTH2` success and its error-continuation refusal, a wrong password as `authentication`, and `LOGINDISABLED` refused before sending; - `EXAMINE` with UIDVALIDITY and UIDNEXT, `UID FETCH` sizes, the last-UID edge case of a range ending in the star, byte-exact 7-bit and 8-bit bodies, and `INTERNALDATE` with an offset; - a modified UTF-7 folder name, `LOGOUT`, and no `\Seen` left after a fetch. That last check is read through imapkit's control API; hoodiecrow has none, so it is NOT MEASURED there. - **H2, credentials:** - All four existing connectors take a static credential from the host composition: `connector-slack` `token`, `connector-rest` `authentication`, `connector-mcp` env/headers, and the showcase's `process.env.SLACK_BOT_TOKEN`. "OAuth2 install/refresh, credential vaulting, and multi-tenant connection lifecycle are the enterprise tier" is the slack connector's own header. - The sync reuses that path (`accountFor` plus a credential provider). No secret store is added, and no `secret` field either. The engine's encrypted `secret` channel (`resolveSecretField`) is the natural store for the per-user phase. - **H3, cursor state:** `sys_mailbox` held no cursor. Its own header said the account, credential and cursor "are added by the ingest half". Four fields are added (the widening above). The account and credential are not stored, by H2. - **H4, scheduling:** this reuses `IJobService.schedule(name, { type: 'interval' }, handler)` resolved at `kernel:ready`, plus `cancel` in `destroy()`. That is plugin-approvals' SLA clock pattern. No timer of its own; `check:plugin-teardown-shape` is green. - **H5, attachment storage: no server-side writer exists.** - `sys_attachment` (platform-objects) has 0 non-test `insert` call sites in the tree. - `sys_file` is written only by service-storage's `StorageMetadataStore.createFile`, reached from the upload routes. - The kernel `storage` service (`IStorageService`) is a blob store with no file-row or attachment-link verb. - Storing a mail attachment needs a new platform capability in service-storage: bytes, the `sys_file` row, and the `sys_attachment` link, with quota, organization and parent-access semantics. The dispatch forbids building that here. The seat ruled B (`6151023438`): the writer is #23306, and the mail half is #23307. ### Hardening round (seat note `6151344448`), commit `4488d1171` 1. **Bounded accumulation.** The session refuses, as the new `ImapError` kind `oversized`, and closes the connection when: - one response carries more than `IMAP_MAX_RESPONSE_LITERALS` (32) literals; - the untagged responses to one command total more than `maxCommandBytes` (default 128 MiB; each response is charged 256 bytes beyond its own size). The existing line and literal refusals moved from `protocol` to `oversized`. The package is unreleased, and the changeset lists the kinds. 2. **`maxLiteralBytes` follows `maxMessageBytes`.** The sync opens the session with `maxLiteralBytes = maxMessageBytes + SYNC_LITERAL_ENVELOPE_BYTES` (64 KiB), and `maxCommandBytes = max(128 MiB, batch budget + maxMessageBytes + envelope)`. - An `oversized` refusal during a batch fetch reopens the session (checking UIDVALIDITY), re-reads the batch one UID at a time, and skips the offender as `too_large`. - A body that arrives within the envelope but above the bound is skipped too. - The cursor moves past a skipped message, and the next message lands. 3. **A batch byte budget.** Batches are bounded by `maxBatchBytes` (default 64 MiB, never below one message's bound) as well as by count. A message the listing says is oversized costs 0, because it is never fetched; an unsized one costs the whole bound. 4. **The `\Seen` sentence** in `testing.ts`'s header and in the connector README now says what the code does: only `BODY[]` under `SELECT` marks a message seen, and `EXAMINE` never changes state. 5. **The changeset** drops `'@objectstack/spec': patch` and the spec bullet, because #23314 carries them. It also drops the liveness clause from its own `Clause-②: yes` line. The `llms.txt` package count (70 to 71) rides the fixed-group release. `main` was merged in via `os-regen-merge.sh` (`c33c6dc67`), because GitHub reported a census conflict. `main`'s side was taken for the two census files, which were then regenerated. ### Fix round after the delta review FAIL (`6154709411`), commits `a11f7c57e`, `2725aacca`, `5e075ec2f` 1. **The cursor passes every skipped message.** `skip()` now advances `result.lastUid` itself, so all three `too_large` paths pass the cursor, including a message that is the last of its batch and run. Four offender-last pins cover: listed above the bound; refused alone with the size understated; refused alone with the size omitted; above the bound once read. The second run fetches nothing and logs nothing new. 2. **A literal is charged at its announcement.** At each `{n}` announcement, before any octet is read, the reader charges the running response against `maxCommandBytes`. One hostile response now holds at most `maxCommandBytes` plus one line. A pin uses the double's new `withheld-literal` mode. 3. **The changeset's `openImapSession` line** gains `maxCommandBytes?`. 4. **The service README names the listing ceiling.** At the default bound, a folder with more than about 450,000 messages above the cursor is refused on the listing. A host raises the bound through `maxBatchBytes`; paging is the per-user phase's. Carried, not fixed: a run's time is unbounded under a server that trickles a literal. Memory stays bounded. Carrier: the per-user mailbox phase of #22871. `main` was merged in again (`671693ad6`) after PR #23314 landed as `777eed2bc`. The one conflict, the `mailAssociation` row's `note`, resolved to `main`'s side. This PR's liveness and lint-test files now equal `main`'s, so `610e112bd` is a no-op. ### Size and landing 47 files, +4105 / -63 = **4168 changed lines against the merge base** (`LICENSE` is 202 of them). That is over the 3,000-line threshold, so under the seat answer `6151023438` this PR takes the Tier H route: the seat's ACCEPT and an at-tier contract review, then it waits for a human merge. The seat does not arm auto-merge. The diff touches no governed surface. ### Verification at `671693ad6` (merged with `origin/main` at `430e5f5ad`) The figures at `671693ad6` come first, then those at `44aabc877`, then the reading at `27aa15166`, on which the first contract review PASSed (`6151327338`). - **At `671693ad6`** (under `os-verify-lock`): - a full build, 75 of 75; - `connector-imap` 42 passed; `service-mailbox` 101 passed; both typechecks exit 0; - `spec` `check:liveness` exit 0, and the lint liveness pin 100 passed; - ablations F1 (no cursor advance in `skip()`: 4 red) and F2 (no per-announcement charge: 1 red), each restored to blob == HEAD; - the interop run on the new dist: `imapkit@5.0.1` and `hoodiecrow-imap@3.3.1`, 18 of 18 each; - `dispatch-gates --ran`: 130 derived, 130 run, all exit 0. - **At `44aabc877`** (under `os-verify-lock`): - a full turbo build, 75 of 75 (the first post-merge build was OOM-killed under contention; only the green re-run is cited); - `spec` `check:generated`, all 14 current; - `@objectstack/connector-imap`: 2 files, 41 passed; `@objectstack/service-mailbox`: 8 files, 97 passed; both typechecks exit 0; - six more ablations (N1 to N6: the literals bound, the command bound, `maxLiteralBytes` not passed, no one-at-a-time re-read, no byte budget, no post-fetch length check), each red as predicted and restored to blob == HEAD; - the interop run on the hardened dist: `imapkit@5.0.1` and `hoodiecrow-imap@3.3.1`, 18 of 18 each; - `dispatch-gates --ran`: 132 derived, 132 run; 131 exit 0, and `check:platform-checklist` exits 1 with `main`'s 8 findings; - ESLint over the hardening round's 12 changed lintable files: 0 errors, 0 warnings; - `spec` `test:repo` was not re-run, because the hardening touches no spec file. - **At `27aa15166`:** - **Suites:** - `@objectstack/connector-imap`: 2 files, 37 tests passed. - `@objectstack/service-mailbox`: 8 files, 91 tests passed (stage 1 had 64). These run on a real ObjectQL over driver-sql and better-sqlite3 `:memory:`, plus a booted `ObjectKernel`. - `@objectstack/lint` `lint-liveness-properties.test.ts`: 100 passed. - `pnpm --filter @objectstack/spec test:repo`: 58 files, 1007 passed. - **Typecheck:** both packages exit 0. `tsc --listFiles` counts 2 of 2 and 8 of 8 test files. - **Ablations:** 11, each through `scripts/ablation-replace.mjs`. Each anchor hit once, each blob change was proven on disk, each file was restored to blob == HEAD with `git diff HEAD` empty, and the tests ran on mutated source (vitest aliases every workspace import to `src`). All 11 went red as predicted: - clause 1: the sync hands nothing to ingest — 14 tests red; - clause 2: the cursor is never applied (5), the dedupe key differs per fetch (2), and a changed UIDVALIDITY keeps the old cursor (1); - clause 3: re-match adds no link — 1; - clause 4: suppression removed — 1; - read-only: `BODY[]` instead of `BODY.PEEK[]` (1), and `SELECT` instead of `EXAMINE` (1); - plaintext to a non-loopback host accepted — 1; - `message/global` not read — 2; - the cursor never saved — 5. - **Gates:** `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derives 132 families on this head. All 132 were run, one log each, and `--ran` reconciles: "132 derived, 132 run, 0 NOT-MEASURED, 0 UNRUN". - 131 exit 0, among them `check:dual-build-cjs-loads`, `check:published-files`, `check:i18n`, `check:system-context-census`, `check:tenant-audit-census`, `check:lockstep-package-count`, `check:changeset-no-major` and `check:generated` after a real spec build. - `pnpm check:platform-checklist` exits 1 with 8 findings, the same 8 as on the merge base (Acceptance notes). - Extra, not derived: `pnpm check:i18n-coverage` exits 0, with none new. - The tree is 3 commits behind `origin/main` `9c7cfbf0a`. Those commits touch none of this diff's files. One of them changes `scripts/engine-double-contract.pinned.json`, which the derivation reads; its gate ran green on this tree's copy. - **Lint (narrowed):** `eslint --no-inline-config --format json` over the 33 changed lintable files gives 33 files, 0 errors, 0 warnings, 0 fatal. - The population is `eslint.config.mjs` files `**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}`; the rest of the diff is md, mdx and json. The count is read from the JSON. - Invariance: the config enables no type-aware linting (no `parserOptions.project`), so no untouched file's verdict can move. - `turbo ls --affected` is not used for any narrowing claim here. ### Acceptance notes - **`check:platform-checklist` is red on the merge base itself** (`175e8d414`, run in a throwaway worktree): the same 8 ABSENT SYMBOL / SYMBOL ANCHORS LOST findings in `docs/qa/platform-checklist/areas/{access-security,attachments-storage,search}.json`. This diff touches none of its inputs. Not fixed here. - **The stage-1 changeset** (`.changeset/22871-mail-ingest-seam.md`) still says no mail source feeds the seam. It is not edited; this PR's changeset says it supersedes that line. - **The `sys_mailbox` row itself** (organization, address, owner) still has no administration door. Server code writes it under the system context, as after stage 1. The README says so. - **`pnpm-lock.yaml`:** `pnpm install` added the two workspace importer entries and also flipped one unrelated `vitest` peer variant (esbuild 0.28.2 ↔ 0.28.1), the normalization stage 1 met in the other direction. The lockfile delta was rebuilt from the base lockfile with only the two importer entries (15 lines), and `pnpm install --frozen-lockfile` validates it. ## 维护者速读(草稿) **改了什么** - 新增发布包 `@objectstack/connector-imap`。它只做协议和认证:以只读方式连接一个 IMAP 邮箱,读出邮件原文。用 EXAMINE 打开文件夹,用 BODY.PEEK 读取,所以不会把邮件标成已读,也不改动邮箱。 - 支持密码(PLAIN / LOGIN)和 OAuth 令牌(XOAUTH2)。 - 默认 TLS 并校验证书。明文连接只允许连本机。 - 对外部服务器设了内存上限:单行、单个邮件体、单个响应、单条命令的累计响应,超过就拒绝并断开。一个恶意或出错的服务器拖不垮本进程。 - 账号与凭据按"每个用户一个邮箱"的形状设计,共享邮箱是第一个使用者。以后接个人邮箱是扩展,不是重写。 - 无任何运行时依赖。 - `@objectstack/service-mailbox` 新增邮箱同步。 - 部署方组合插件时提供"哪个邮箱用哪个 IMAP 账号和凭据"(`imap.accountFor`),插件把同步注册成定时任务,定期把新邮件交给阶段 1 已上线的接收入口。 - 去重、线程、按邮箱字段自动关联到客户/线索记录、未匹配进池、自动回复与退信过滤,全部沿用阶段 1,没有改动。 - `sys_mailbox` 新增 4 个只读字段,记录同步进度:文件夹、UIDVALIDITY、最后同步的 UID、最后同步时间。凭据不存数据库。 - 顺带完成阶段 1 的两项遗留: - 转发件为 `message/global`(国际化邮件头)时也能识别发件人; - README 写明一封邮件最多读 20 个转发件。 - `mailAssociation` 的 liveness 改为 live 这一项,已拆到单独的小 PR #23314 先行落地。 **为什么改** - 阶段 1 只解决了"拿到一封邮件怎么入库",没有邮件来源,部署上实际收不到邮件。本 PR 补上来源:销售把客户邮件转发或密送到共享邮箱后,邮件会自动挂到对应的客户、线索记录上。这是 CRM 评估者最先会试的功能,属于设计记录 #8998(维护者已「同意」,`5311578770`)的第一阶段。 - 附件提取按席位裁决 B 拆到 #23307,该卡等服务端附件写入能力 #23306 落地后再做。#22871 在本 PR 落地时关闭。 **风险与代价(含回滚)** - 规模:约 4,170 行变更(含 202 行 LICENSE),超过 3,000 行门槛,需要人工合并。 - 只有显式配置了 `imap.accountFor` 的部署才会连接外部邮箱。仅通过 `requires: ['mailbox']` 自动装配的部署不同步任何邮箱,行为与今天一致,启动时会打印一行说明。 - IMAP 客户端是自研的最小实现,没有引入 imapflow(它带 8 个锁定版本的依赖)。主要风险是与真实邮件服务器的兼容性: - 已用两个独立的严格 IMAP 测试服务器(imapkit、hoodiecrow)做过一次性互通验证,18 项全部通过; - 尚未对 Gmail、Microsoft 365 做真机验证。 - 附件暂不保存,转发附件里的原邮件正文也不保留,只读取其中的参与人。 - 单封邮件超过上限(默认 32 MiB;服务器少报或不报大小时同样适用)、或无法解析时,会跳过并记 error 日志,邮件仍留在原邮箱,同步继续往下走,不会卡住。 - 回滚:直接还原本 PR 即可。 - 新增的 4 个字段只是只读的同步进度,还原后残留的列无害。 - 不涉及数据迁移,已入库的邮件保留。 - liveness 那一项在单独的 PR #23314 里,不受影响。 **席位意见** (终稿与席位意见见 PR 评论 `6157331354`。) **你要做的** - 在 PR #23303 上提交一次 Approve 审阅。批准后由席位把它转为 ready 并入队落地;席位在你批准之前不会这样做。 --- _Generated by [Claude Code](https://claude.ai/code/session_012uFegdfR6K6uqvLzt7XsdZ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 6d96721 commit 3d830b7

47 files changed

Lines changed: 4105 additions & 63 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
'@objectstack/connector-imap': minor
3+
'@objectstack/service-mailbox': minor
4+
---
5+
6+
Inbound mail has a mail source: the new `@objectstack/connector-imap` reads a mailbox over IMAP, and `@objectstack/service-mailbox` polls each mailbox's folder into its ingest seam, keeping a UID cursor on `sys_mailbox`
7+
8+
Clause-②: yes (a new published package, `@objectstack/connector-imap`, and its exports; new exports and plugin options in `@objectstack/service-mailbox`, and four new fields on its published `sys_mailbox` object)
9+
10+
- **`@objectstack/connector-imap` (new) — protocol and authentication only.**
11+
- `openImapSession({ account, credentials, timeoutMs?, maxLiteralBytes?, maxCommandBytes? })` opens a read-only IMAP4rev1 session: `examine(folder)` (`EXAMINE`, never `SELECT`; answers `uidValidity`, `uidNext`, `exists`), `listMessages(fromUid)`, `fetchMessages(uids)` (`BODY.PEEK[]`, so no message is marked `\Seen`), `logout()` and `close()`. A folder opened without a UIDVALIDITY is refused.
12+
- The account and credential abstraction is shaped for the per-user mailbox: a `MailAccount` (`{ id, username, imap: { host, port?, security? } }`) and a `MailCredentialProvider` that resolves the account's credential each time a session opens. `staticCredentials(credential)` answers one fixed credential.
13+
- A `password` credential is sent with `AUTHENTICATE PLAIN` when the server offers it, otherwise `LOGIN`; an `oauth2` access token with `AUTHENTICATE XOAUTH2`. A server that refuses `LOGIN` and offers no `PLAIN` is refused before anything is sent.
14+
- `security: 'tls'` (the default) is implicit TLS with the certificate verified. A plaintext connection (`security: 'none'`) is refused to any host but loopback, before a socket opens.
15+
- Failures are `ImapError` with a `kind`: `configuration`, `connection`, `timeout`, `authentication`, `refused`, `protocol` or `oversized`.
16+
- An IMAP server is treated as an untrusted peer. A line above 4 MiB, a literal above `maxLiteralBytes` (64 MiB by default), a response of more than `IMAP_MAX_RESPONSE_LITERALS` (32) literals, and more than `maxCommandBytes` of responses to one command (128 MiB by default) are refused as `oversized` before the excess is kept, and the connection is closed.
17+
- `@objectstack/connector-imap/testing` exports `ImapServerDouble`, an in-process IMAP server on loopback for tests.
18+
- No runtime dependencies.
19+
- **`@objectstack/service-mailbox` — the IMAP sync.**
20+
- `new MailboxServicePlugin({ imap: { accountFor, pollIntervalMs?, batchSize?, maxMessagesPerRun?, maxMessageBytes?, maxBatchBytes? } })` schedules the interval job `mailbox:imap-sync` on the composed job service. For each mailbox `accountFor` answers (asked per mailbox per run; `null` for a mailbox the host does not sync), it fetches the messages above the stored cursor, in UID order, and hands each one to `ingest` under the system context. Dedupe, association, the pool and auto-reply suppression are unchanged.
21+
- Without `imap` — the capability boot constructs the plugin with no options — no mailbox is synced, and the boot says so once. With `imap` but no job service, the boot warns, and `plugin.syncService.syncAll({ isSystem: true })` runs a pass on the host's own schedule. A run in which a mailbox could not be synced reports `degraded` to the job service.
22+
- `sys_mailbox` gains the cursor, written by the sync only: `imap_folder`, `imap_uid_validity`, `imap_last_uid` and `imap_synced_at`. A cursor taken under another UIDVALIDITY, or in another folder, is discarded and the folder is read from its start; the messages already stored answer `duplicate`. No credential is stored on the object: the account and its credential come from `accountFor`.
23+
- The cursor moves after each batch past every message stored, already stored, suppressed or skipped. A store failure stops the run below the failed message, so the next run retries it. A message above `maxMessageBytes` (32 MiB by default) or one the seam refuses as unreadable is skipped and logged at `error` with its UID.
24+
- A batch is bounded by count (`batchSize`, 25) and by listed bytes (`maxBatchBytes`, 64 MiB). The session's literal bound is `maxMessageBytes` plus a 64 KiB framing envelope, so a server that omits or understates a message's size cannot hand over more. That message is skipped as `too_large`, after the batch is read again one message at a time on a fresh session, and the rest of the mailbox keeps arriving.
25+
- A forwarded `message/global` part (RFC 6532, the internationalized `message/rfc822`) is now read as an `attachment` source, the same as `message/rfc822`.
26+
- New exports: `MailboxSyncService`, `MAILBOX_SYNC_JOB`, `DEFAULT_MAILBOX_POLL_INTERVAL_MS`, `DEFAULT_SYNC_FOLDER`, `DEFAULT_SYNC_BATCH_SIZE`, `DEFAULT_SYNC_MAX_MESSAGES_PER_RUN`, `DEFAULT_SYNC_MAX_MESSAGE_BYTES`, `DEFAULT_SYNC_MAX_BATCH_BYTES`, `SYNC_LITERAL_ENVELOPE_BYTES`, and the types `MailboxServicePluginOptions`, `MailboxImapSyncOptions`, `MailboxAccountResolver`, `MailboxImapAccount`, `MailboxSyncTarget`, `MailboxSyncResult`, `MailboxSyncAllResult`, `MailboxSyncSkipReason`, `MailboxSyncServiceOptions`, `MailSyncEngine` and `MailSyncLogger`.
27+
- This supersedes the ingest seam entry's "no mail source feeds the seam": the IMAP sync does. Attachments are still not stored.

‎.changeset/config.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
"@objectstack/trigger-record-change",
5353
"@objectstack/trigger-api",
5454
"@objectstack/trigger-schedule",
55+
"@objectstack/connector-imap",
5556
"@objectstack/connector-mcp",
5657
"@objectstack/connector-rest",
5758
"@objectstack/connector-slack",

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -387,6 +387,7 @@ control plane.
387387
| [`@objectstack/connector-openapi`](packages/connectors/connector-openapi) | Connector actions generated from an OpenAPI document |
388388
| [`@objectstack/connector-mcp`](packages/connectors/connector-mcp) | Any MCP server's tools as connector actions |
389389
| [`@objectstack/connector-slack`](packages/connectors/connector-slack) | Slack Web API connector |
390+
| [`@objectstack/connector-imap`](packages/connectors/connector-imap) | Read-only IMAP session and mail account abstraction for the mailbox sync |
390391
| [`@objectstack/trigger-record-change`](packages/triggers/trigger-record-change) | Launch flows on insert / update / delete |
391392
| [`@objectstack/trigger-schedule`](packages/triggers/trigger-schedule) | Launch flows on a cron, interval, or one-off schedule |
392393
| [`@objectstack/trigger-api`](packages/triggers/trigger-api) | Inbound HTTP / webhook flow trigger with HMAC verification |

‎content/docs/permissions/system-context.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,7 @@ One of the largest consumers of the flag;
170170
| 46 | Knowledge search returns hits unfiltered | service-knowledge | Lose: the permission filter over search results | `packages/services/service-knowledge/src/knowledge-service.ts#applyPermissionFilter` |
171171
| 47 | The inbound-mail read gates return early (the read AST) | service-mailbox | Get: every `sys_mail_message` and `sys_mail_association` row on `find` / `findOne` — the ingest writer and each gate's own pre-scan, which read the association and mailbox rows a decision needs. Lose: the narrowing of a message to callers who can read a record it is associated with or who own its mailbox (so the unassociated pool is restricted), and of an association to callers who can read the record it names (on `count` and `aggregate` too, so a total agrees with the rows) | `packages/services/service-mailbox/src/mail-read-visibility.ts#gateMiddleware` |
172172
| 47b | The inbound-mail ingest seam: `ingest` admits the system context ALONE, and `rematch` admits it without the organization, owner or `manage_mailboxes` check | service-mailbox | Get: a mail source hands a mailbox its messages, and server code re-matches any mailbox. Lose: nothing a user held — every other caller of `ingest` is refused (`PERMISSION_DENIED`), because an ingested message is shown on every record its correspondents' addresses match and a user who could hand the seam a message could write one in a correspondent's name; every other caller of `rematch` must be in the mailbox's organization and own the mailbox or hold `manage_mailboxes` | `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts#ingest`, `#authorizeRematch` |
173+
| 47c | The inbound-mail IMAP sync: `syncMailbox` and `syncAll` admit the system context ALONE | service-mailbox | Get: the sync job reads every mailbox row (its organization, address and IMAP cursor), writes the cursor back, and hands each fetched message to `ingest` under the system context. Lose: nothing a user held — every other caller is refused (`PERMISSION_DENIED`), because a sync stores a mailbox's mail and moves its cursor | `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts#syncMailbox`, `#syncAll` |
173174

174175
### 5. Actions, metadata plane, provenance, the organization wall
175176

‎content/docs/permissions/tenant-audit-census.mdx‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way.
122122

123123
The same holds twice over for the context. An options argument spelled as a
124124
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
125-
forwarding shim cannot, and **53 of the 225 sites are spelled that way**. A
125+
forwarding shim cannot, and **53 of the 226 sites are spelled that way**. A
126126
context resolved from an inline literal or a local `const` can be tested for
127127
`isSystem`; one arriving from a helper call cannot.
128128

@@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
187187

188188
| carried figure | where it survives | this census |
189189
| :--- | :--- | ---: |
190-
| 175 write call sites | quoted in the merged changeset | **225** |
190+
| 175 write call sites | quoted in the merged changeset | **226** |
191191
| 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **32** more whose options argument is unreadable |
192-
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 225** decidable, **72** undecidable |
193-
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 120 decidably elevated, 0 decidably not, 97 undecidable |
192+
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **153 of 226** decidable, **73** undecidable |
193+
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 121 decidably elevated, 0 decidably not, 97 undecidable |
194194
| 141 and 132, two independent re-derivations | the card that filed this work | — |
195195

196196
**The differences are not reconciled, and deliberately so.** The old census's
@@ -207,11 +207,11 @@ would report a smaller number and would not say so.
207207

208208
The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
209209
figure has no surviving corroboration anywhere in the tree.** This census reads
210-
120 of 225 (53%) as decidably elevated, with 97 more whose elevation is a
210+
121 of 226 (54%) as decidably elevated, with 97 more whose elevation is a
211211
run-time fact — so the claim is neither confirmed nor refuted, and the honest
212212
answer is that a static reading cannot settle it.
213213

214-
⇒ **Cite `2 / 225`, and say what it is**: the sites whose options argument was
214+
⇒ **Cite `2 / 226`, and say what it is**: the sites whose options argument was
215215
READ and holds no tenant context, against a decidably tenancy-enabled object.
216216
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
217217
without tenant context" — **32 further sites** have an options argument this
@@ -223,31 +223,31 @@ cannot read, and they are neither in nor out.
223223

224224
| what | count |
225225
| :--- | ---: |
226-
| write call sites on the application surface | **225** |
226+
| write call sites on the application surface | **226** |
227227
| …whose object name is statically decidable | 153 |
228-
| …whose object name is chosen at run time | 72 |
228+
| …whose object name is chosen at run time | 73 |
229229
| …against an object with tenancy ENABLED | 151 |
230230
| …against an object that declares tenancy off | 1 |
231-
| threading a tenant context | 164 |
231+
| threading a tenant context | 165 |
232232
| PROVABLY carrying none (options read, no context key) | **8** |
233233
| …of those, against a decidably tenancy-enabled object | **2** |
234234
| options argument UNREADABLE — may or may not carry one | 53 |
235235
| …of those, against a decidably tenancy-enabled object | 32 |
236-
| threading a decidably ELEVATED (`isSystem`) context | 120 |
236+
| threading a decidably ELEVATED (`isSystem`) context | 121 |
237237
| threading a context that is decidably NOT elevated | 0 |
238238
| threading a context whose elevation is a run-time fact | 97 |
239239

240240
| how the instrument reached the site | count |
241241
| :--- | ---: |
242-
| receiver carried a readable engine type | 185 |
242+
| receiver carried a readable engine type | 186 |
243243
| receiver erased, placed by the object NAME | 27 |
244244
| receiver erased, placed by an `object: string` PARAMETER | 7 |
245245
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 6 |
246246

247247
| object name spelled inline | 103 |
248248
| object name spelled through a `const` | 50 |
249249
| object name is an `object: string` parameter | 9 |
250-
| object name is some other run-time expression | 63 |
250+
| object name is some other run-time expression | 64 |
251251

252252
### Subtractions the census could NOT defend — enforced
253253

@@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
297297
their values are not compared. The reasoning, and the measurement behind it,
298298
are in `scripts/check-tenant-audit-census.mjs`.
299299

300-
Measured on 2026-10-13 at `66949b772`.
300+
Measured on 2026-10-13 at `c33c6dc67`.
301301

302302
| corpus scale (not enforced) | count |
303303
| :--- | ---: |
304-
| tracked non-test sources scanned | 648 |
305-
| engine-shaped types recognised | 72 |
304+
| tracked non-test sources scanned | 650 |
305+
| engine-shaped types recognised | 73 |
306306
| declared objects in the registry | 116 |
307-
| same-named calls subtracted as non-engine | 160 |
307+
| same-named calls subtracted as non-engine | 162 |
308308

309309
{/* END GENERATED: tenant-audit-census */}

‎content/docs/protocol/backward-compatibility.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ Read the **Breaking?** column, not the version number: in the 17.x minors every
114114
1. **RFC (Request for Comments)** — Breaking changes are proposed as GitHub issues with the `protocol:breaking` label.
115115
2. **Deprecation** — The old behavior is deprecated in a MINOR release (see timeline above).
116116
3. **Migration Guide** — A detailed migration guide is published before the removal lands, in the release notes for the version that carries it.
117-
4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 71 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line).
117+
4. **Release** — In the 17.x minors the breaking change shipped in the next **MINOR** version, carrying a changeset entry marked `**BREAKING**`, and `scripts/check-changeset-no-major.mjs` failed a pull request that declared a `major` bump, because under lockstep one `major` would promote all 72 published packages. On the v18 line it ships in the next `18.0.0-next.N`: the guard stands aside in pre mode, so a pull request may declare a major bump. See [how a breaking stage lands on the v18 line](/docs/releases/v18#how-a-breaking-stage-lands-on-this-line).
118118

119119
---
120120

@@ -192,7 +192,7 @@ The `@objectstack/spec` package provides additional stability guarantees:
192192

193193
### Which surfaces this covers
194194

195-
**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **71** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt.
195+
**All of them.** It was not scoped to an experimental corner or a pre-release channel: all **72** packages published from this repository belong to a single Changesets `fixed` group, so they share one version number and one policy. No published surface was exempt.
196196

197197
The convention was enforced rather than informal — `scripts/check-changeset-no-major.mjs` failed a pull request that introduced a `major` bump, because under lockstep a single `major` on one package would promote the entire stack. Changesets is now in pre mode with the tag `next`, and the guard stands aside in pre mode, so a pull request on the v18 line may declare a major bump.
198198

‎docs/audits/2026-08-tenant-audit-write-call-sites.counts.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.
3333

3434
| Measure | Value |
3535
|---|---:|
36-
| Write call sites | 225 |
36+
| Write call sites | 226 |
3737
| Object name statically decidable | 153 |
38-
| Object name chosen at run time | 72 |
38+
| Object name chosen at run time | 73 |
3939
| Against a tenancy-enabled object | 151 |
4040
| Against an object declaring tenancy off | 1 |
41-
| Threading a tenant context | 164 |
41+
| Threading a tenant context | 165 |
4242
| Provably carrying none | 8 |
4343
| …and decidably tenancy-enabled | 2 |
4444
| Options argument unreadable | 53 |
4545
| …and decidably tenancy-enabled | 32 |
46-
| Threading a decidably elevated context | 120 |
46+
| Threading a decidably elevated context | 121 |
4747
| Threading a decidably non-elevated context | 0 |
4848
| Threading a context of undecidable elevation | 97 |
4949

@@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
9090
their values are not compared. The reasoning, and the measurement behind it,
9191
are in `scripts/check-tenant-audit-census.mjs`.
9292

93-
Measured on 2026-10-13 at `66949b772`.
93+
Measured on 2026-10-13 at `c33c6dc67`.
9494

9595
| corpus scale (not enforced) | count |
9696
| :--- | ---: |
97-
| tracked non-test sources scanned | 648 |
98-
| engine-shaped types recognised | 72 |
97+
| tracked non-test sources scanned | 650 |
98+
| engine-shaped types recognised | 73 |
9999
| declared objects in the registry | 116 |
100-
| same-named calls subtracted as non-engine | 160 |
100+
| same-named calls subtracted as non-engine | 162 |
101101

102102
## Every site
103103

@@ -208,6 +208,7 @@ Measured on 2026-10-13 at `66949b772`.
208208
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_ASSOCIATION_OBJECT` | undecidable | elevated | 1 |
209209
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `insert` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
210210
| `packages/services/service-mailbox/src/ingest/mail-ingest-service.ts` | `update` | `MAIL_MESSAGE_OBJECT` | undecidable | elevated | 1 |
211+
| `packages/services/service-mailbox/src/sync/mailbox-sync-service.ts` | `update` | `MAILBOX_OBJECT` | undecidable | elevated | 1 |
211212
| `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
212213
| `packages/services/service-messaging/src/inbox-channel.ts` | `insert` | `receiptObject` | undecidable | context, elevation undecidable | 1 |
213214
| `packages/services/service-messaging/src/messaging-service.ts` | `insert` | `RECEIPT_OBJECT` | undecidable | context, elevation undecidable | 1 |

0 commit comments

Comments
 (0)