diff --git a/apps/backend/docs/concepts-caching.md b/apps/backend/docs/concepts-caching.md new file mode 100644 index 0000000..377cf48 --- /dev/null +++ b/apps/backend/docs/concepts-caching.md @@ -0,0 +1,293 @@ +# Backend caching reference + +The backend caches exactly one thing in Redis: the conversation list a signed-in user sees +when the app opens. That is the query that runs on every cold start and every foreground +resume, it fans out into several joins and two aggregate subqueries, and its result changes +only when something in the conversation actually changes. Everything else the backend puts in +Redis — presence, rate-limit buckets, replay markers, pub/sub fan-out — is coordination state +rather than a cache, and is covered elsewhere. + +This document is the reference for that cache: what is stored, under what key, for how long, +every path that invalidates it, what happens when Redis is not there, and the one scoping +hazard the key format has to respect. + +## Contents + +- [What is cached](#what-is-cached) +- [The cache key](#the-cache-key) +- [Read and write paths](#read-and-write-paths) +- [Invalidation](#invalidation) +- [Degraded behaviour when Redis is unavailable](#degraded-behaviour-when-redis-is-unavailable) +- [The per-device scoping hazard](#the-per-device-scoping-hazard) +- [Other Redis key namespaces (not caches)](#other-redis-key-namespaces-not-caches) +- [Implementation references](#implementation-references) + +## What is cached + +| Property | Value | +| ----------- | -------------------------------------------------------------------------------- | +| Key | `conversations:{userId}` — built by `convCacheKey(userId)` in `src/lib/redis.ts` | +| Value | The JSON body of `GET /conversations`, as a string (`JSON.stringify(result)`) | +| TTL | `CONV_CACHE_TTL` = **30 seconds**, applied with `SETEX` | +| Written by | `GET /conversations` (default view only) | +| Read by | `GET /conversations` (default view only) | +| Invalidated | `invalidateConversationCaches(userIds)` and one direct `DEL` — see below | + +The payload is the complete response array, one entry per conversation the user belongs to: + +```jsonc +[ + { + "id": "…", + "type": "dm", + "name": null, + "avatarUrl": null, + "createdAt": "…", + "messages": [ + /* the single most recent message, with the ciphertext envelope + addressed to the *requesting device* */ + ], + "isMuted": false, + "isArchived": false, + "messageCount": 42, + "unreadCount": 3, + }, +] +``` + +That embedded `messages[0]` is the conversation-list preview, and it is the reason the rest of +this document is more careful than a 30-second cache would normally justify: the preview is +**device-specific**, because the envelope it carries is the one encrypted for the requesting +device and no other device can decrypt it. + +The TTL is deliberately short. Thirty seconds is long enough to absorb the burst of list +requests an app makes while starting up and reconnecting, and short enough that any +invalidation this document has missed self-heals within half a minute instead of persisting +until the user acts. + +## The cache key + +```ts +export function convCacheKey(userId: string): string { + return `conversations:${userId}`; +} +``` + +Everything that touches the cache goes through this function rather than formatting the +string inline, so the format has exactly one definition. Two consequences of the current +shape are worth stating explicitly: + +- **It is scoped to the user, not to the request.** The archived view is a different result + set, so it is neither read from nor written to the cache at all — `?archived=true` skips + both branches rather than using a second key. See + [the per-device scoping hazard](#the-per-device-scoping-hazard) for the dimension this key + does _not_ currently carry. +- **It is a plain string key with a TTL, not a hash or a set.** Invalidation is `DEL`, never a + partial update: no path rewrites part of a cached list. A change invalidates the whole entry + and the next read rebuilds it from Postgres. + +## Read and write paths + +Both live in `GET /conversations` (`src/routes/conversations.ts`) and both are guarded twice — +once on `redis` being non-null, and once by a `try`/`catch`: + +```ts +// Read — skipped entirely for the archived view +if (!showArchived && redis) { + try { + const cached = await redis.get(key); + if (cached) { + res.json(JSON.parse(cached) as unknown); + return; + } + } catch { + // Fall through to the database on any Redis error + } +} + +// … build `result` from Postgres … + +// Write — same two conditions +if (!showArchived && redis) { + try { + await redis.setex(key, CONV_CACHE_TTL, JSON.stringify(result)); + } catch { + // Ignore — the response is already computed + } +} +``` + +A cache miss, a Redis error, and a Redis that was never configured all converge on the same +path: query Postgres and answer from it. The write is best-effort and happens after the +response body exists, so a failure to cache cannot fail a request. + +## Invalidation + +`invalidateConversationCaches(userIds)` (`src/lib/conversationCache.ts`) is the single +invalidation helper: + +```ts +export async function invalidateConversationCaches(userIds: string[]): Promise { + if (!redis || userIds.length === 0) return; + const client = redis; + await Promise.allSettled([...new Set(userIds)].map((userId) => client.del(convCacheKey(userId)))); +} +``` + +Three properties: it de-duplicates the id list, it deletes in parallel, and it uses +`Promise.allSettled` so one failing `DEL` does not reject the whole call or abandon the +remaining users. Callers pass **every member of the affected conversation**, because a change +to one conversation changes every member's list. + +### Every call site + +| # | Trigger (the event a user would describe) | Call site | Users invalidated | +| --- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------- | +| 1 | Members added to a conversation (`POST /conversations/:id/members`) | `src/routes/conversations.ts:419` | All members after the change | +| 2 | Conversation metadata updated — name, avatar (`PATCH /conversations/:id`) | `src/routes/conversations.ts:529` | All members | +| 3 | The last member leaves, so the conversation row is deleted (`DELETE /conversations/:id/leave`) | `src/routes/conversations.ts:872` | The departing member | +| 4 | A member leaves a conversation that still has others (`DELETE /conversations/:id/leave`) | `src/routes/conversations.ts:900` | All members, including the one leaving | +| 5 | Per-member settings changed — mute, archive (`PATCH /conversations/:id/settings`) | `src/routes/conversations.ts:715` | **Direct `redis.del(convCacheKey(userId))`** — only the caller's own entry changes | +| 6 | A device is added or revoked, emitting a `device_added` / `device_revoked` system message into each of that user's conversations | `src/routes/devices.ts:735` (`emitDeviceChangeEvent`) | All members of every conversation the user belongs to | +| 7 | A message is sent over REST (`POST /messages`) | `src/routes/messages.ts:229` | All members | +| 8 | A message is deleted over REST (`DELETE /messages/:id`) | `src/routes/messages.ts:277` | All members | +| 9 | A message is sent over the socket (`send_message`) | `src/socket/messaging.ts:312` | All members | +| 10 | A message is edited over the socket (`edit_message`) | `src/socket/messaging.ts:441` | All members | +| 11 | A file message is sent over the socket (`send_file_message`) | `src/socket/messaging.ts:623` | All members | +| 12 | A conversation is created over the socket (`create_conversation`) | `src/socket/messaging.ts:1003` | Every member of the new conversation | +| 13 | The assistant replies (`ask_assistant`) | `src/socket/messaging.ts:1225` | All members | + +Case 5 is the one direct `DEL` outside the helper, and it is correct as written: mute and +archive are per-member columns, so no other member's list changes. Every other write path uses +the helper. + +The unifying rule, and the one to apply when adding a path: **anything that changes what +`GET /conversations` would return for a user must invalidate that user's key in the same +request.** In practice that means any write to `conversations`, `conversation_members`, or +`messages`, and any change to the device set that produces a system message. The usual shape +is a `findMany` over `conversationMembers` for the affected conversation followed by +`invalidateConversationCaches(members.map((m) => m.userId))`. + +A path that forgets to invalidate does not corrupt anything — it produces a list that is stale +for at most the 30-second TTL. That is a real bug (a sent message that does not appear in the +list preview for half a minute reads as data loss to a user) but a self-healing one, which is +why the TTL is short. + +## Degraded behaviour when Redis is unavailable + +`src/lib/redis.ts` creates the client only when `REDIS_URL` is set, with `lazyConnect: true` +and an `error` listener that deliberately swallows connection errors: + +```ts +export let redis: Redis | null = null; + +if (process.env['REDIS_URL']) { + redis = new Redis(process.env['REDIS_URL'], { lazyConnect: true }); + redis.on('error', () => { + // Graceful degradation: cache misses fall through to DB + }); +} +``` + +Without the listener, ioredis would emit an unhandled `error` event and crash the process on a +Redis blip. With it, every caching call site sees either `redis === null` or a rejected +promise, and both are already handled. + +**The cache is an optimisation, not a correctness dependency.** With Redis down or absent: + +- `GET /conversations` reads and writes nothing and answers from Postgres. The response is + byte-for-byte what the cached version would have been — the cache stores the finished body, + so there is no second code path that could diverge. +- `invalidateConversationCaches` returns immediately on the `!redis` guard, and per-user `DEL` + failures are absorbed by `allSettled`. Nothing upstream sees an error. +- No write is ever gated on the cache. No path reads the cache to make a decision; it is only + ever read to answer a `GET`. Nothing is stored in Redis that is not reconstructible from + Postgres. + +The cost of losing Redis is throughput on one endpoint, plus the effects on the _other_ +Redis-backed subsystems listed below — not correctness or data loss here. Local development +without a `REDIS_URL` is a supported configuration, and the test suite runs with `redis` mocked +to `null` for exactly this reason. + +## The per-device scoping hazard + +`GET /conversations` builds its preview through +`getConversationRelations(req.auth!.deviceId)`, whose message relation filters envelopes to the +requesting device: + +```ts +envelopes: { + where: eq(messageEnvelopes.recipientDeviceId, deviceId), + limit: 1, +} +``` + +**The response is therefore device-specific, not merely user-specific.** Each recipient device +gets its own envelope, encrypted for that device alone (see +[`concepts-protocol-negotiation.md`](./concepts-protocol-negotiation.md) for why one message +produces one ciphertext per device). Two devices belonging to the same user get _different_ +ciphertext for the same preview message, and neither can decrypt the other's. + +That makes the interaction between the payload and the key format the sharpest edge in this +subsystem: + +- A key that includes the device (`conversations:{userId}:{deviceId}`) is safe: each device + gets its own entry, and each entry holds the ciphertext addressed to that device. +- A key that omits the device serves whichever device populated the entry first to every other + device of the same user for the rest of the TTL. The second device receives an envelope it + has no key for. The failure does not look like a cache bug: it looks like a decryption + failure, or a preview stuck on an older message, on one device only, intermittently, for up + to 30 seconds after every send — the shape of bug that gets attributed to the crypto layer + and chased for days. + +**Today `convCacheKey` is `conversations:{userId}` and does not carry the device.** The +consequence above is a live hazard for a multi-device user, bounded by the 30-second TTL, and +`convCacheKey` is the single place a fix belongs — every read, write, and invalidation already +routes through it. Note that a device-scoped key changes invalidation too: +`invalidateConversationCaches` takes user ids, so it would have to delete every device's entry +for each user (a `SCAN`/`DEL` over `conversations:{userId}:*`, or a per-user set of that user's +device keys), rather than one `DEL` per user. + +The general rule this is an instance of: **a cache key must name every input the cached value +depends on.** Here the value depends on the user _and_ the device _and_ the archived flag. The +archived flag is handled by not caching that view at all; the device is the dimension to watch. +The same rule applies to any future per-device response — a sync cursor, a device-filtered +history page — that someone is tempted to cache. + +## Other Redis key namespaces (not caches) + +For orientation, so these are not mistaken for cache entries and cleared as if they were. Only +the first is invalidated by anything in this document. + +| Prefix | Purpose | Reference | +| ---------------------- | ------------------------------------------ | ------------------------------------------------------------------------ | +| `conversations:` | The conversation-list cache described here | this document | +| `replay:` | `eventId` replay markers, TTL'd | [`concepts-replay-protection.md`](./concepts-replay-protection.md) | +| `rl:` | Rate-limit buckets | [rate limits](../../../docs/security/rate-limits.md) | +| `presence:` | Device and user presence, socket mappings | [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md) | +| Socket.IO adapter keys | Cross-node pub/sub fan-out | [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md) | + +Deleting a `conversations:` key is always safe. Deleting keys in the other namespaces is not +equally harmless — dropping `rl:` keys resets live rate-limit budgets, and dropping `presence:` +keys makes online users appear offline until their next heartbeat. + +## Implementation references + +| Concern | File | +| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | +| Client, key format, TTL constant | [`src/lib/redis.ts`](../src/lib/redis.ts) | +| Invalidation helper | [`src/lib/conversationCache.ts`](../src/lib/conversationCache.ts) | +| Read, write, and the device-scoped preview | [`src/routes/conversations.ts`](../src/routes/conversations.ts) | +| Message-write invalidations | [`src/routes/messages.ts`](../src/routes/messages.ts), [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| Device-change invalidation | [`src/routes/devices.ts`](../src/routes/devices.ts) | +| Tests | `src/__tests__/conversations.cache.test.ts` | + +## Related documents + +- [Conversations API](./api-conversations.md) — the endpoint whose response is cached. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — the other + subsystem that treats Redis as an optimisation and fails open. +- [Gateway architecture](./concepts-gateway-architecture.md) — presence and pub/sub, the rest + of what Redis carries. +- [Backend testing guide](./testing.md) — mocking `lib/redis.js`, including the `null` form + that exercises the degraded path. diff --git a/apps/backend/docs/concepts-protocol-negotiation.md b/apps/backend/docs/concepts-protocol-negotiation.md new file mode 100644 index 0000000..37ee061 --- /dev/null +++ b/apps/backend/docs/concepts-protocol-negotiation.md @@ -0,0 +1,309 @@ +# Device capability and E2EE protocol negotiation + +Clicked encrypts every message once per recipient device, and not every device speaks the +same encryption protocol. A device that shipped before the Signal work landed understands +only the Phase-1 sealed box; a current one also understands the Double Ratchet; an MLS +group member understands a third construction. Negotiation is how the system picks, for +each pair of devices, the strongest construction both of them can actually use — without a +flag day, and without a newer client silently breaking an older one. + +This document covers the negotiation layer itself: +[`src/lib/capabilities.ts`](../src/lib/capabilities.ts) (what a device advertises and how a +protocol is chosen) and [`src/services/e2eeProtocol.ts`](../src/services/e2eeProtocol.ts) +(what the server does when a sender claims a protocol on an envelope). + +The migration this machinery exists to serve — the rollout order, the per-pair cutover +timeline, and the client-side ratchet work — is documented separately in +[`signal-migration.md`](./signal-migration.md). Read this document for the mechanism, that +one for the plan. + +## Contents + +- [The capability payload](#the-capability-payload) +- [Advertising capabilities at registration](#advertising-capabilities-at-registration) +- [`normalizeCapabilities` and the baseline default](#normalizecapabilities-and-the-baseline-default) +- [Picking a mutually supported protocol](#picking-a-mutually-supported-protocol) +- [The per-envelope `protocol` column](#the-per-envelope-protocol-column) +- [`checkEnvelopeProtocols` — enforcement on the way in](#checkenvelopeprotocols--enforcement-on-the-way-in) +- [The `protocol_mismatch` rejection, and what a client does with it](#the-protocol_mismatch-rejection-and-what-a-client-does-with-it) +- [Why this enables a staged rollout](#why-this-enables-a-staged-rollout) +- [Implementation references](#implementation-references) + +## The capability payload + +`devices.capabilities` is a `jsonb` column holding a small document the device publishes +about itself. It is validated by `DeviceCapabilitiesSchema`, and every field is optional: + +```jsonc +{ + "protocols": ["sealed_box", "signal"], + "ciphersuites": ["MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519"], + "fileTransfer": ["file-v1"], +} +``` + +| Field | Type | Meaning | +| -------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `protocols` | `string[]` | Messaging encryption protocols this device can **decrypt**. Known values are `sealed_box`, `signal`, `mls` (`KNOWN_PROTOCOLS`). `sealed_box` is the Phase-1 ECDH + HKDF + AES-256-GCM envelope. | +| `ciphersuites` | `string[]` | MLS/Signal ciphersuite identifiers. Only consulted when `mls` is present in `protocols`. | +| `fileTransfer` | `string[]` | File-encryption scheme versions, e.g. `file-v1`. Independent of the messaging protocol; queried through `supportsFileTransfer(capabilities, version)`. | + +Two properties of the shape are deliberate and should be preserved: + +- **Everything is optional, and the whole document is optional.** A missing field is not an + error; it means "assume the default for this field". +- **Unrecognised values are preserved, not rejected.** A device may advertise a protocol + name this server has never heard of. `selectProtocol` ignores it — it only ever matches + against names in `PROTOCOL_PRIORITY` — but the value survives in the column. This is what + makes negotiation compatible in both directions: a newer client against an older server + degrades to a protocol they share instead of failing to register. + +`capabilities` is `NOT NULL` with the sealed-box baseline as its schema-level default +(`src/db/schema.ts`), so a device row can never exist without one. + +## Advertising capabilities at registration + +A device advertises its capabilities on the two paths that create or refresh a device row. +Both accept `capabilities` as an optional member of the device object (`DeviceSchema` in +`src/schemas/auth.schemas.ts`): + +| Path | Behaviour | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `POST /auth/verify` | New device → the value is normalised and inserted. Existing device (matched on identity key) → the value is normalised and **updated in place**. | +| `POST /devices/link/verify` | The device-linking registration path. Same treatment. | + +The update-in-place case is the **upgrade path**. A client that gains Signal support does +not re-register a new identity: it re-verifies with a larger `protocols` array, and the next +negotiation for that device picks the stronger protocol. Omitting `capabilities` entirely on +a re-verify leaves the stored document untouched — the update only sets the field when the +client actually sent one, so an older build of the same client cannot accidentally downgrade +a device's advertised capabilities to its own. + +Capabilities are read back by senders from three places, all of which run the stored value +through `normalizeCapabilities` first: + +| Endpoint | What it returns | +| ---------------------------------- | ------------------------------------------------------------------------------------- | +| `GET /devices` | The caller's own devices, each with its normalised `capabilities`. | +| `GET /user-devices/:id/public-key` | A single peer device's identity key plus its normalised capabilities. | +| `GET /conversations/:id/devices` | Every active member device, with `capabilities` **and** a `negotiatedProtocol` field. | + +`negotiatedProtocol` is `selectProtocol(callerDevice, thatDevice).protocol` computed +server-side. It exists so a client does not have to re-implement the preference order to +agree with the server about the answer — a client that computes its own and disagrees is +exactly the client whose sends get rejected. + +## `normalizeCapabilities` and the baseline default + +```ts +export const BASELINE_PROTOCOL: KnownProtocol = 'sealed_box'; +``` + +`normalizeCapabilities(raw)` turns anything — `null`, `undefined`, a partial object, a +malformed one — into a concrete `DeviceCapabilities`: + +1. The value is parsed with `DeviceCapabilitiesSchema.safeParse(raw ?? {})`. **A parse + failure is not an error**: it returns a copy of `DEFAULT_CAPABILITIES`. +2. An empty or missing `protocols` array becomes `[BASELINE_PROTOCOL]`. +3. `ciphersuites` and `fileTransfer` default to empty arrays. + +The result is that **every device supports `sealed_box`, whether it said so or not**. This is +the single assumption the whole design rests on, and it is what makes clients predating the +`capabilities` field work unchanged: + +- Rows written before the column existed carry the schema default, which is the baseline. +- A client that never sends `capabilities` gets the baseline. +- A client that sends a document this server cannot parse gets the baseline rather than a + `500` at registration. + +Because the baseline is universal, `selectProtocol` never has to return `null` and no send +path needs a "these two devices cannot talk to each other" branch. + +## Picking a mutually supported protocol + +`selectProtocol(a, b)` normalises both capability documents, then walks a fixed preference +order and returns the first protocol present in both sets: + +```ts +const PROTOCOL_PRIORITY = ['mls', 'signal', 'sealed_box']; +``` + +The order is strongest-first, so an overlap of `{sealed_box, signal}` resolves to `signal`, +not to the weaker option the two happen to share. If the loop finds nothing — only possible +when a device advertises a protocol set that excludes the baseline — the function falls back +to `{ protocol: 'sealed_box', ciphersuite: null }`. + +For `mls`, and only for `mls`, a ciphersuite is negotiated alongside the protocol: +`selectCiphersuite` returns the first entry of `a.ciphersuites` that also appears in +`b.ciphersuites`, preserving the caller's preference order, or `null` when there is no +overlap. + +**Negotiation is per device pair, not per conversation.** The envelope model already +encrypts once per recipient device, so a Signal-capable pair inside a group is not held back +by a third member still running an old client: that pair uses Signal today and the laggard +keeps the sealed box until it upgrades. +`protocolsForRecipients(senderDeviceId, recipientDeviceIds)` is the batch form used by +callers that need the answer for a whole fan-out at once; device ids that do not resolve map +to `BASELINE_PROTOCOL`. + +## The per-envelope `protocol` column + +`message_envelopes.protocol` is an `e2ee_protocol` enum column, +`NOT NULL DEFAULT 'sealed_box'`, whose values mirror `KNOWN_PROTOCOLS`. It is written by +`insertMessageEnvelopes` (`src/lib/messageFanout.ts`), which is shared by the REST send path +and both socket send paths so the default cannot drift between them. + +**Why per envelope rather than per device.** The two columns answer different questions, and +only one of them is stable over time: + +- `devices.capabilities.protocols` says what a device can decrypt **right now**. It is + mutable — that is the entire point of the upgrade path above. +- `message_envelopes.protocol` says what a particular ciphertext **was actually built + with**. It has to stay true forever. + +If the protocol were recorded only on the device, then the moment a device advertised +`signal`, every envelope ever written for it would look like a Signal ciphertext. All of its +sealed-box history would become undecryptable — not because the key material is gone, but +because the reader would pick the wrong construction. Recording it per envelope is what lets +pre-cutover history keep decrypting on the Phase-1 path indefinitely, and it is why the +migration that added the column used a defaulted `NOT NULL`: the default backfills every +pre-existing row with the construction those rows really used. + +It is also per envelope rather than per **message** because one message fans out to many +devices, and those devices are not all on the same side of the cutover. The same plaintext +can legitimately be a Signal ciphertext for one recipient device and a sealed box for +another, within a single send. + +The column is read back on the paths that hand a ciphertext to a client — the delivery +pipeline (`src/services/deliveryPipeline.ts`) and the sync endpoint (`src/routes/sync.ts`) — +so the recipient selects its decryption path from the data rather than guessing from the +bytes. + +## `checkEnvelopeProtocols` — enforcement on the way in + +Negotiation tells an honest sender what to use. It does not stop a patched, buggy, or +compromised one from claiming something else, so the server re-derives the answer on every +send. `checkEnvelopeProtocols(senderDeviceId, envelopes)` loads the sender device's +capabilities and every named recipient device's capabilities in two queries, then checks each +envelope for two distinct failures: + +| Reason | Status | Condition | +| -------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------- | +| `unsupported_by_recipient` | `400` | The declared protocol is **not** in the recipient's advertised `protocols`. The ciphertext would be undecryptable. | +| `downgrade` | `409` | The declared protocol is supported, but is not the one `selectProtocol` picks for this pair — both sides can do better. | + +Details worth knowing: + +- **Envelopes naming a device that does not resolve are skipped**, not rejected. The send + paths already drop those envelopes before persisting, and failing a whole batch because a + device was revoked between the client's device-list fetch and its send would turn a routine + race into a user-visible error. +- **Every violation in the batch is reported**, not just the first, so a client can fix one + send rather than discovering the problem one device at a time. +- **When a batch contains both kinds, the status is `400`.** Undecryptability is the more + specific and more serious failure, so it decides the code. +- The check runs **before anything is persisted**, on `POST /messages` + (`src/routes/messages.ts`) and on the `send_message` socket handler + (`src/socket/messaging.ts`). Envelopes with no explicit `protocol` default to + `BASELINE_PROTOCOL` in the schema (`EnvelopeSchema`), so an older client that never sends + the field is checked as a sealed-box sender — which is exactly what it is. + +## The `protocol_mismatch` rejection, and what a client does with it + +Over REST the failure is the status code from the table above with a body of: + +```jsonc +{ + "error": "Envelope protocol is not supported by the recipient device", + "violations": [ + { + "recipientDeviceId": "…", + "declared": "signal", + "expected": "sealed_box", + "reason": "unsupported_by_recipient", + }, + ], +} +``` + +Over the socket the same information arrives as an `error` event with +`event: 'protocol_mismatch'`, carrying `code` (`400` or `409`), `message`, and the identical +`violations[]` array. See [`contracts-error-catalog.md`](./contracts-error-catalog.md) for +where this sits among the other socket error events. + +**`protocol_mismatch` is not retryable as sent.** Resending the same envelope set produces +the identical rejection. The client should: + +1. **Re-fetch the recipient device list.** `GET /conversations/:id/devices` is the one call + that returns both `capabilities` and the server's own `negotiatedProtocol` per device. The + most common cause of a mismatch is a stale device list: the peer upgraded, or a new device + joined, after the sender last looked. +2. **Re-encrypt each envelope with the protocol the server names.** `violations[].expected` + already says what should have been used for that specific device, so a client can act on + the response without a second round trip if it trusts its own key state. +3. **Resend the whole message.** Nothing was persisted, so this is a fresh send rather than a + retry of a partial one. Reuse the same `messageId` — the send paths are idempotent on it, + see [`concepts-replay-protection.md`](./concepts-replay-protection.md). +4. **Do not fall back to a weaker protocol.** A `409 downgrade` means precisely that the + client already tried that. Downgrading again in response to a downgrade rejection is a + retry loop whose only successful outcome is the thing the check exists to prevent. + +A `downgrade` violation naming a device the client believes cannot do better is a signal that +the client's cached capability document is stale, not that the server is wrong — the server +read the row a moment ago. + +## Why this enables a staged rollout + +Together the four pieces give a sealed-box → Signal rollout with no coordinated cutover: + +1. **Old clients keep working with no change at all.** They never send `capabilities`, so + they are read as sealed-box-only. They never send `protocol`, so their envelopes default + to `sealed_box`. Both defaults are applied server-side; nothing about an old client has to + know this feature exists. +2. **A new client is useful before anyone else upgrades.** It advertises + `["sealed_box", "signal"]`. Against an old peer, `selectProtocol` finds only `sealed_box` + in common and the pair keeps working exactly as before. Against another new peer, the same + code picks `signal` — with no server config change, no feature flag, and no + per-conversation gate. +3. **Progress is monotonic per pair, and independent across pairs.** A pair cuts over the + moment both sides advertise the stronger protocol, and the `downgrade` check stops them + from silently sliding back. One user's laptop can be on Signal with one contact and on the + sealed box with another, in the same conversation, at the same time. +4. **History is never re-encrypted.** Every envelope carries the construction that produced + it, so a cutover changes what is written next and never what was written before. There is + no migration job to run, and no window during which old messages are unreadable. +5. **The same machinery carries the next protocol.** Adding MLS meant adding a name to + `KNOWN_PROTOCOLS`, a value to the enum, and an entry at the top of `PROTOCOL_PRIORITY`. The + negotiation, enforcement, and recording paths did not change. + +The cost of all this is one rule for contributors: **a new encryption construction must be +added to `KNOWN_PROTOCOLS`, to the `e2ee_protocol` enum, and to `PROTOCOL_PRIORITY` in the +same change.** A protocol missing from the enum cannot be recorded; a protocol missing from +the priority list can be advertised but will never be selected, which presents as "both +devices support Signal and it is still sending sealed boxes". + +## Implementation references + +| Concern | File | +| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Capability shape, defaults, selection | [`src/lib/capabilities.ts`](../src/lib/capabilities.ts) | +| Enforcement and batch negotiation | [`src/services/e2eeProtocol.ts`](../src/services/e2eeProtocol.ts) | +| `capabilities` and `protocol` columns | [`src/db/schema.ts`](../src/db/schema.ts) | +| Registration input schema | [`src/schemas/auth.schemas.ts`](../src/schemas/auth.schemas.ts) | +| Envelope input schema and `protocol` default | [`src/schemas/message.schemas.ts`](../src/schemas/message.schemas.ts) | +| Writing the per-envelope protocol | [`src/lib/messageFanout.ts`](../src/lib/messageFanout.ts) | +| REST send enforcement | [`src/routes/messages.ts`](../src/routes/messages.ts) | +| Socket send enforcement | [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| Capability read-back endpoints | [`src/routes/devices.ts`](../src/routes/devices.ts), [`src/routes/userDevices.ts`](../src/routes/userDevices.ts), [`src/routes/conversations.ts`](../src/routes/conversations.ts) | +| Tests | `src/__tests__/e2eeProtocol.test.ts`, `src/__tests__/signalMigration.routes.test.ts`, `src/__tests__/signalInvariants.socket.test.ts` | + +## Related documents + +- [Phase-1 → Signal migration](./signal-migration.md) — the rollout plan this mechanism + serves. +- [Error code and response catalog](./contracts-error-catalog.md) — `protocol_mismatch` + alongside every other error the backend returns. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — why resending + a rejected message with the same `messageId` is safe. +- [Delivery fan-out and receipts](./concepts-delivery-fanout.md) — how the envelopes this + document validates reach devices. diff --git a/apps/backend/docs/concepts-replay-protection.md b/apps/backend/docs/concepts-replay-protection.md new file mode 100644 index 0000000..aa7281a --- /dev/null +++ b/apps/backend/docs/concepts-replay-protection.md @@ -0,0 +1,287 @@ +# Replay protection and event idempotency + +A realtime client retries. It reconnects mid-send, it resumes after a suspend, it fires the +same action twice because a tap registered twice — and a hostile client replays a captured +frame on purpose. The backend answers this in **two independent layers**, at two different +levels of the stack, and confusing them is the usual source of "why did my duplicate still +create a row" questions. + +- **Transport level:** every enveloped socket event carries an `eventId`, and + [`src/services/replay-protection.service.ts`](../src/services/replay-protection.service.ts) + drops the second and later arrivals of that id from the same device inside a TTL window. +- **Message level:** every message carries a client-generated `messageId`, and the send paths + refuse to insert a second row for an id that already exists, acknowledging the original + instead. + +This document covers both, and the dispatcher path that ties them together. + +## Contents + +- [Why both layers exist](#why-both-layers-exist) +- [Layer 1 — transport-level `eventId` dedup](#layer-1--transport-level-eventid-dedup) + - [The device-scoped key](#the-device-scoped-key) + - [The TTL](#the-ttl) + - [Fail-open when Redis is unavailable](#fail-open-when-redis-is-unavailable) + - [`dispatch_ack`](#dispatch_ack) +- [Layer 2 — message-level `messageId` idempotency](#layer-2--message-level-messageid-idempotency) +- [Every event goes through the dispatcher](#every-event-goes-through-the-dispatcher) +- [Operational notes](#operational-notes) +- [Implementation references](#implementation-references) + +## Why both layers exist + +They protect different things and neither subsumes the other. + +| | Transport layer (`eventId`) | Message layer (`messageId`) | +| ------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------- | +| Scope | Every socket event type — reads, typing, receipts, joins, sends | Message-creating sends only | +| Identity | Per delivery attempt of an event | Per message, stable across attempts and across transports | +| Storage | Redis key with a TTL | The `messages` primary key — permanent | +| Window | `REPLAY_PROTECTION_TTL_SECONDS` (default 5 minutes) | Forever | +| Behaviour on repeat | Handler is not run at all; `dispatch_ack { duplicate: true }` | Handler runs, finds the row, re-acknowledges the original `createdAt` | +| Fails | Open (allows the event through when Redis is down) | Closed (the row either exists or it does not) | + +**The transport layer cannot be the only one.** Its window is finite and its state is in +Redis. A retry after the TTL has expired, a retry from a different device, a retry that +arrives over REST instead of the socket, or any retry at all while Redis is down, gets past +it. If duplicate suppression for messages lived only here, any of those would produce a +second copy of a message in the conversation. + +**The message layer cannot be the only one either.** Most events are not message sends. +`message_read`, `typing_start`, `join_room`, `heartbeat` and the rest have no durable id to +be idempotent on, and re-running them on a replayed frame costs real work — extra fan-out, +extra receipts, extra queries — even where the end state happens to be the same. The +transport layer is also the only one that sees a _deliberate_ replay of a captured frame, +because a captured frame replays the whole envelope, `messageId` included; the message layer +would treat that as a retry and cheerfully re-ack it. + +The practical rule: **`eventId` is per attempt and must be freshly generated for every +emit; `messageId` is per message and must be reused across retries of the same message.** +Getting these the wrong way round is what produces "my resend was silently ignored" +(reused `eventId`) or "my retry created a duplicate message" (fresh `messageId`). + +## Layer 1 — transport-level `eventId` dedup + +`isReplay(redis, deviceId, eventId)` performs an atomic check-and-mark: + +```ts +const result = await redis.set(key, '1', 'EX', ttl, 'NX'); +return result === null; // null → the key already existed → this is a replay +``` + +`SET … NX` is one round trip and it is atomic, so two frames racing on the same node — or on +two different gateway nodes sharing one Redis — cannot both observe "not seen yet". There is +no read-then-write window to lose. + +Return values are stated in the positive: `false` means _not_ a replay, process it; `true` +means drop it. The function also returns `false` — process it — when `deviceId` or `eventId` +is missing or blank, since there is nothing to key on. In practice the envelope schema +requires a non-empty `eventId`, so a blank one is rejected as a malformed envelope before the +replay check is reached. + +`markSeen(redis, deviceId, eventId)` exists to mark an id without consuming the check; it is +a testing and debugging affordance, not part of the live path. `isReplay` already marks +atomically. + +### The device-scoped key + +``` +replay:{deviceId}:{eventId} +``` + +`getReplayProtectionRedisKey()` builds it and is exported so tests and debugging can compute +the same string rather than duplicating the format. + +**The `deviceId` component is load-bearing, not decoration.** `eventId` is generated by the +client, and nothing forces two clients to agree on a generator. A user's phone and laptop can +legitimately produce the same id — a counter, a low-entropy uuid, a `test-evt-1` left in a +build. With a global `replay:{eventId}` key, whichever device emitted first would win and the +other device's genuine, first-time event would be silently dropped as a "replay", producing a +handler that never runs and no error anywhere. Scoping the key to the device makes each +device's id space its own: one device's replayed id cannot block another device's legitimate +event. + +It also matches the threat being defended. A replay attack is a frame captured from one +device and re-sent; the attacker cannot change the authenticated `deviceId` on the socket, +which is taken from the JWT (`socket.auth.deviceId`), not from the payload. Cross-device +collisions are noise, not attacks. + +### The TTL + +| Setting | Value | +| -------------- | --------------------------------------------------------- | +| Env var | `REPLAY_PROTECTION_TTL_SECONDS` | +| Default | `300` (5 minutes) | +| Accepted range | `1` … `86400` seconds (1 day) | +| Invalid values | Non-numeric, out-of-range, or empty → the default is used | +| Read | Per call, not cached at import | + +The TTL is what keeps the key space bounded: entries expire on their own, so there is no +sweeper job and no unbounded growth from a device that sends a million events. It also bounds +the protection — an event replayed after the window is not detected here, which is why the +message layer exists. + +The default is tuned against the dispatcher's own freshness check rather than picked +arbitrarily. `SOCKET_EVENT_MAX_AGE_MS` (default `300000`, also 5 minutes) rejects an envelope +whose `timestamp` is older than the window, so a frame old enough to have fallen out of the +replay set is already stale enough to be refused for that reason instead. **If you raise +`SOCKET_EVENT_MAX_AGE_MS`, raise `REPLAY_PROTECTION_TTL_SECONDS` with it**, or you open a gap +in which a frame is accepted as fresh but no longer remembered as seen. + +An out-of-range value degrades to the default instead of throwing, in keeping with how the +rest of the backend reads tuning env vars: a typo in the environment must not stop the +gateway from booting. + +### Fail-open when Redis is unavailable + +`isReplay` returns `false` — process the event — in two cases: + +- `redis` is `null`, meaning `REDIS_URL` was never configured, and +- the `SET` throws, which covers a connection loss, a timeout, or a Redis in a failed state. + The error is logged once per call at `warn` level and the event proceeds. + +This is deliberate. Replay protection is a **hardening** layer, not a correctness dependency: +the durable guarantees for messages come from the `messageId` primary key, which is in +Postgres and does not care what Redis is doing. Failing closed would mean that losing Redis +takes the entire realtime surface offline — nobody can send, read, or type — in exchange for +closing a replay window that requires an attacker to already hold a captured, authenticated +frame. Trading total availability for that is the wrong side of the trade for this system. + +The consequence to be aware of when reading tests: **with Redis mocked to `null`, the dedup +does nothing.** A test that reuses an `eventId` passes against a `null` Redis and starts +failing the moment someone gives the suite an `ioredis-mock` instance. Generate a fresh +`eventId` per emit regardless of what Redis the test has — see +[`testing.md`](./testing.md#driving-socket-handlers). + +## `dispatch_ack` + +Every event that reaches the dispatcher gets exactly one `dispatch_ack` back, and the flag +says which layer answered it: + +```jsonc +{ "eventId": "…", "duplicate": false } // first occurrence — the handler ran +{ "eventId": "…", "duplicate": true } // replay — the handler was not run +``` + +`duplicate: true` is **an acknowledgement, not an error.** The event was already processed, +so the client's intent is satisfied and it should clear the item from its outbox exactly as +it would on `duplicate: false`. Treating it as a failure and retrying produces a loop that +gets the same answer until the TTL expires, at which point the retry is processed for real — +which is the one outcome the client was trying to avoid. + +Note what does _not_ produce an ack: a malformed envelope, an unknown event type, a stale +timestamp, and an unauthenticated socket all emit an `error` envelope instead, and a handler +that throws is logged server-side with no ack at all. A client waiting on `dispatch_ack` +therefore needs a timeout, and must treat `error` as terminal for that event. + +## Layer 2 — message-level `messageId` idempotency + +`messageId` is generated by the client and is the primary key of the `messages` row. Every +send path performs the same check before inserting: + +```ts +const existing = await db.query.messages.findFirst({ + where: eq(messages.id, messageId), + columns: { createdAt: true }, +}); +if (existing) { + /* acknowledge the original, insert nothing */ +} +``` + +| Path | On an id that already exists | +| ---------------------------- | ------------------------------------------------------------- | +| `POST /messages` | `200` with `{ messageId, createdAt }` (a fresh send is `201`) | +| `send_message` (socket) | `message_ack` with the **original** `createdAt` | +| `edit_message` (socket) | `message_ack` with the original `createdAt` | +| `send_file_message` (socket) | `message_ack` with the original `createdAt` | + +Three properties matter: + +- **It spans transports.** A client that sends over the socket, loses the connection before + the ack, and retries over REST with the same `messageId` gets the original message back + rather than a duplicate. The transport layer cannot do this — the REST request has no + `eventId` at all. +- **It has no window.** The check is against the durable row, so it holds a day later as + readily as a second later. +- **The distinction between `200` and `201`, and the returned `createdAt`, are the client's + signal that the retry was absorbed** — the original timestamp is returned, not the retry's, + so message ordering never shifts because of a retry. + +Because the check is `SELECT` then `INSERT` rather than an upsert, two genuinely simultaneous +sends of the same `messageId` can both pass the check; the second insert then fails on the +primary key and the request returns a `500` rather than corrupting anything. That is a +tolerable outcome for a case that requires a client to race itself, and the row count stays +correct either way. + +## Every event goes through the dispatcher + +The order of operations in `EventDispatcher.listen()` is fixed, and the replay check sits +late in it on purpose — there is no point spending a Redis round trip on a frame that is not +going to be processed anyway: + +1. **Authenticated?** Otherwise `error` — the socket must be authenticated before any event. +2. **Valid envelope?** `EventEnvelopeSchema` requires `eventId`, `type`, and a positive + integer `timestamp`. Otherwise `error`. +3. **Known event type?** Unknown types are discarded with an `error`, so an unrecognised name + cannot reach a handler. +4. **Fresh timestamp?** Within `SOCKET_EVENT_MAX_AGE_MS` in the past and + `SOCKET_EVENT_MAX_FUTURE_SKEW_MS` (default `30000`) in the future. Otherwise `error`. +5. **Replay?** `isReplay(redis, socket.auth.deviceId, envelope.eventId)`. If so, emit + `dispatch_ack { duplicate: true }` and stop. +6. **Dispatch** to the registered handler, then `dispatch_ack { duplicate: false }`. + +`dispatcher.register(type, handler)` puts the handler into a map that only step 6 can reach. +**There is no raw `socket.on(type, …)` fallback for registered types** — `listen()` attaches +exactly one listener, for the `dispatch` event — so a handler cannot be reached without +passing every check above. That is why the dedup can be described as covering every event +rather than as something each handler opts into, and it is why a test must drive handlers by +emitting a `dispatch` envelope rather than by triggering a raw event name. + +**One handler is not on this path today:** `send_file_message` is still attached with a raw +`socket.on` in `src/socket/messaging.ts`. It therefore gets no envelope validation, no +timestamp freshness check, and no `eventId` dedup; its duplicate suppression comes entirely +from the `messageId` check in layer 2. Anything that moves it onto `dispatcher.register` +inherits all of the above for free, and any _new_ handler must be registered through the +dispatcher. + +## Operational notes + +- **Keys are ephemeral and safe to drop.** `replay:*` keys carry no data beyond "this id was + seen", and losing them fails open by construction. A Redis flush costs at most a window in + which a replayed frame would be accepted; it never loses a message. +- **Sizing.** One key per event per device for the TTL window. At `n` events per second + across the fleet and a 300-second TTL, the steady-state key count is roughly `300n`, each a + short string with a one-byte value. +- **A cluster shares the state.** All gateway nodes talk to the same Redis, so a frame + replayed against a different node than the original is still caught. This is the same + Redis the gateway uses for pub/sub fan-out and presence — see + [`concepts-gateway-architecture.md`](./concepts-gateway-architecture.md). +- **Debugging a dropped event.** A replay logs at `debug` with `deviceId`, `eventId`, and + `type`. If a handler "never ran" and no error came back, check for a `dispatch_ack` with + `duplicate: true`; the usual cause is a client reusing an `eventId` across retries rather + than generating a fresh one. + +## Implementation references + +| Concern | File | +| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `isReplay`, `markSeen`, key and TTL | [`src/services/replay-protection.service.ts`](../src/services/replay-protection.service.ts) | +| Dispatch order and `dispatch_ack` | [`src/socket/dispatcher.ts`](../src/socket/dispatcher.ts) | +| Envelope schema and event registry | [`src/lib/eventEnvelope.ts`](../src/lib/eventEnvelope.ts) | +| Socket send idempotency | [`src/socket/messaging.ts`](../src/socket/messaging.ts) | +| REST send idempotency | [`src/routes/messages.ts`](../src/routes/messages.ts) | +| Redis client and degradation | [`src/lib/redis.ts`](../src/lib/redis.ts) | +| Tests | `src/services/replay-protection.service.spec.ts`, `src/socket/dispatcher.spec.ts`, `src/__tests__/dispatcher.test.ts` | + +## Related documents + +- [Gateway architecture](./concepts-gateway-architecture.md) — the socket lifecycle the + dispatcher sits inside, and what else Redis carries. +- [WebSocket events](./api-websocket-events.md) — every event type, including `dispatch_ack`. +- [Error code and response catalog](./contracts-error-catalog.md) — the `error` envelopes the + checks above emit, and which are retryable. +- [Backend testing guide](./testing.md) — driving handlers through `dispatch` without + tripping the dedup. +- [Backend caching reference](./concepts-caching.md) — the other place Redis is treated as an + optimisation rather than a correctness dependency. diff --git a/apps/backend/docs/testing.md b/apps/backend/docs/testing.md new file mode 100644 index 0000000..2c7d0da --- /dev/null +++ b/apps/backend/docs/testing.md @@ -0,0 +1,472 @@ +# Backend testing guide + +The backend-specific companion to the cross-app +[testing strategy and conventions](../../../docs/testing.md). That document sets the rules +every suite in the repository follows — no test starts Redis, Postgres, or S3; the approved +substitutes; the runners and commands. This one is the working reference for the backend +suite specifically: the mock set a route test needs, the socket pattern, and the four or five +ways a test in this suite fails confusingly rather than clearly. + +Read the cross-app document first if you have not. Everything here assumes it. + +The conventions below are not stylistic. Each one exists because the alternative produces a +failure that does not name its cause: `undefined is not a function` from a mock that is missing +one export, a handler that "never ran" because an id was reused, a `429` that only appears when +the file runs in a particular order. Getting them right is mostly a matter of copying the +skeleton and knowing which traps are there. + +## Contents + +- [Commands and layout](#commands-and-layout) +- [The standard route-test skeleton](#the-standard-route-test-skeleton) +- [Why `await import`, always](#why-await-import-always) +- [Drizzle chain mocking, and its traps](#drizzle-chain-mocking-and-its-traps) +- [Driving socket handlers](#driving-socket-handlers) +- [Service mocks: add one whenever a route gains a dependency](#service-mocks-add-one-whenever-a-route-gains-a-dependency) +- [Resetting shared in-process state](#resetting-shared-in-process-state) +- [Checklist for a new backend test](#checklist-for-a-new-backend-test) + +## Commands and layout + +```bash +pnpm --filter backend test # the whole suite +pnpm --filter backend test -- messages.routes # one file, by substring +pnpm --filter backend test:watch +pnpm --filter backend test:coverage +``` + +| Fact | Value | +| ---------------- | ---------------------------------------------------------------------------------------------------------------- | +| Runner | Vitest, `environment: 'node'`, `testTimeout: 15000` (`vitest.config.ts`) | +| Collected | `src/**/*.{test,spec}.ts`; `node_modules` and **`dist`** excluded | +| Where tests live | `src/__tests__/*.test.ts`, plus a few `*.spec.ts` co-located with their module (`src/socket/dispatcher.spec.ts`) | +| Global setup | `src/__tests__/setup.ts` — sets `JWT_SECRET`, `DATABASE_URL`, and the `OBJECT_STORE_*` placeholders | +| Lint and format | `pnpm --filter backend lint`, `pnpm --filter backend format:check` — both cover `src/`, tests included | + +`dist` is excluded deliberately: `pnpm build` emits a compiled copy of every spec, and without +the exclusion each test would run twice, the second time against stale output. + +If a new module validates a new required environment variable at import time, add a +placeholder to `src/__tests__/setup.ts`. Do not mock the config module in each test file that +happens to import the new one transitively. + +## The standard route-test skeleton + +Almost every route test mocks the same five modules. Copy this, delete what the route under +test does not touch, and add service mocks as needed. + +```ts +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import request from 'supertest'; +import express from 'express'; + +// ── 1. Database ───────────────────────────────────────────────────────────── +// Only the query methods the route actually calls need to exist. Anything the +// route calls that is missing here fails as "not a function", which reads as a +// bug in the route rather than a gap in the mock — so add the method, do not +// change the route. +const mockFindFirst = vi.fn(); +const mockFindMany = vi.fn(); +const mockInsert = vi.fn(); +const mockUpdate = vi.fn(); +const mockDelete = vi.fn(); + +vi.mock('../db/index.js', () => ({ + db: { + query: { + conversationMembers: { findFirst: mockFindFirst, findMany: mockFindMany }, + messages: { findFirst: vi.fn() }, + }, + insert: mockInsert, + update: mockUpdate, + delete: mockDelete, + }, +})); + +// ── 2. Schema ─────────────────────────────────────────────────────────────── +// Column references become inert sentinels. Their only job is to be +// distinguishable inside an assertion — the mocked operators below never +// interpret them. Every table the router imports must appear here, even one it +// only references in a code path this test does not exercise: the import itself +// is what fails otherwise. +vi.mock('../db/schema.js', () => ({ + conversations: { id: 'id', type: 'type' }, + conversationMembers: { conversationId: 'conversationId', userId: 'userId' }, + messages: { id: 'id', conversationId: 'conversationId', createdAt: 'createdAt' }, + messageEnvelopes: { messageId: 'messageId', recipientDeviceId: 'recipientDeviceId' }, + tokenTransfers: {}, +})); + +// ── 3. Drizzle operators ──────────────────────────────────────────────────── +// Identity-ish stubs, so a test asserts on the shape a call site built rather +// than on generated SQL. Every operator the module under test imports must be +// exported here or the import throws. +vi.mock('drizzle-orm', () => ({ + and: vi.fn((...args: unknown[]) => args), + eq: vi.fn((col: unknown, val: unknown) => ({ col, val })), + ne: vi.fn(), + desc: vi.fn(), + lt: vi.fn(), + inArray: vi.fn(), + count: vi.fn(), + sql: vi.fn(), +})); + +// ── 4. Redis ──────────────────────────────────────────────────────────────── +// The getter form matters: it lets a test swap the instance (or set it to null) +// between cases without re-registering the mock. `null` exercises the degraded +// path, which is the default choice unless the test is about caching itself. +vi.mock('../lib/redis.js', () => ({ + get redis() { + return null; + }, + CONV_CACHE_TTL: 30, + convCacheKey: (userId: string) => `conversations:${userId}`, +})); + +// ── 5. Auth middleware ────────────────────────────────────────────────────── +// Inject `req.auth` rather than minting a real JWT. Include `deviceId`: several +// routes read it, and a missing one surfaces as a confusing 500 or an empty +// result set rather than a 401. +vi.mock('../middleware/auth.js', () => ({ + requireAuth: (req: express.Request, _res: express.Response, next: express.NextFunction) => { + (req as express.Request & { auth: { userId: string; deviceId: string } }).auth = { + userId: 'user-1', + deviceId: 'device-1', + }; + next(); + }, +})); + +// ── Import the module under test AFTER the mocks ──────────────────────────── +const { conversationsRouter } = await import('../routes/conversations.js'); + +function makeApp() { + const app = express(); + app.use(express.json()); + app.use('/conversations', conversationsRouter); + return app; +} + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe('GET /conversations', () => { + it('…', async () => { + const res = await request(makeApp()).get('/conversations'); + expect(res.status).toBe(200); + }); +}); +``` + +Notes on the pieces that most often go wrong: + +- **Mount only the router under test**, not `app.ts`, unless the test is specifically about + middleware ordering. A `supertest` app built from one router keeps the failure local. +- **Do not mock `../middleware/validate.js`.** Request-body validation is behaviour a route + test should exercise; mocking it hides a whole class of `400`s the real route returns. +- **Auth tests use the real middleware.** The mock above is for tests that are about something + else. +- **`vi.clearAllMocks()` in `beforeEach` clears calls and implementations of mock functions + only.** It does not touch module-level state inside the code under test — see + [resetting shared in-process state](#resetting-shared-in-process-state). + +## Why `await import`, always + +`vi.mock` calls are hoisted to the top of the file, but the factories close over `vi.fn()` +handles declared below them. A static `import` of the module under test binds the real `db` +before those handles exist. Every backend suite that mocks the database therefore imports the +subject with a top-level `await import(...)`: + +```ts +const { messagesRouter } = await import('../routes/messages.js'); +``` + +The symptom of getting this wrong is a test that tries to open a real Postgres connection and +either hangs to the 15-second timeout or fails inside a driver stack trace with no mention of +your route. + +## Drizzle chain mocking, and its traps + +The cross-app guide covers the general pattern. These are the specific chains this suite hits. + +### `.values()` must be both thenable and expose `.returning()` + +Drizzle's insert builder is a thenable. `db.insert(t).values(rows)` is itself awaitable and +executes the statement, **and** `.returning()` can be chained onto it to execute and get the +inserted rows back. Both forms are used in this codebase, sometimes inside a single +transaction: the message insert needs the generated `id` and `createdAt` so it calls +`.returning()`, while the envelope batch insert (`insertMessageEnvelopes`) only cares that the +rows landed and just awaits `.values(...)`. + +The two wrong stubs fail in opposite, equally unhelpful ways: + +| Stub returns | What breaks | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `{ returning }` alone | The awaited `.values(...)` resolves to a plain object and records nothing. **The test passes** while asserting on an insert that never happened. | +| A bare promise | `.returning is not a function`, thrown from inside the route. | + +So a shared insert stub has to satisfy both shapes: + +```ts +function insertStub(table: string) { + return { + values: (vals: unknown) => ({ + returning: async () => recordInsert(table, vals), + then: (resolve: (value: unknown) => void) => resolve(recordInsert(table, vals)), + }), + }; +} +``` + +`src/__tests__/e2ee.integration.test.ts` is the canonical version. Two cautions: + +- **Do not add `then` to a stub whose call sites never await the builder directly.** A thenable + is awaited implicitly whenever it is returned from an `async` function, which can fire the + recording side effect a second time. +- **If both forms run against the same stub, make the recorder idempotent** — or assert on call + counts you have actually verified, rather than assuming one insert equals one recorded row. + +### Transactions + +A route that writes more than one table wraps the work in `db.transaction`. Mock it as a +function that invokes its callback with an object exposing the same stubbed builders, plus +whatever `query.*` methods the transaction body uses (`insertMessageEnvelopes`, for example, +calls `tx.query.devices.findMany`): + +```ts +const mockTransaction = vi.fn(async (cb: (tx: unknown) => Promise) => + cb({ + insert: insertStub, + query: { devices: { findMany: mockDeviceFindMany } }, + }), +); +``` + +A transaction body that throws must propagate, so do not wrap the callback in a `try`/`catch` +in the mock — the route's own error handling is usually the thing under test. + +### Read chains and raw SQL + +`db.select().from().where().groupBy()` needs each link to return the next one, innermost first: + +```ts +const mockGroupBy = vi.fn().mockResolvedValue([]); +const mockWhere = vi.fn(() => ({ groupBy: mockGroupBy })); +const mockFrom = vi.fn(() => ({ where: mockWhere })); +const mockSelect = vi.fn(() => ({ from: mockFrom })); +``` + +`db.execute(sql\`…\`)`is used for the aggregate subqueries (unread counts). Mock it as a plain`vi.fn()`resolving to the row array; the route spreads the result, so resolve to an array and +not to a`{ rows }` object. + +When a module uses `sql` as both a tag and a namespace (`sql.join(...)`), the `drizzle-orm` +mock has to provide both: + +```ts +const sqlMock = Object.assign( + vi.fn(() => 'sql'), + { join: vi.fn(() => 'joined') }, +); +``` + +### The two import-time failures + +Both present as an error in a file you did not touch, so recognise them by shape: + +- **A table missing from the `db/schema.js` mock** → the router's import of that name is + `undefined`, and the first use is a `Cannot read properties of undefined` far from the cause. + Add the table to the mock with the columns the route names. +- **An operator missing from the `drizzle-orm` mock** → an ESM named-export error naming the + operator (`ne`, `isNull`, `inArray`, `count`, `desc`, `lt`, `gte`, `sql`). Add it as + `vi.fn()`. + +Both happen when a route gains a new query, which is the same trigger as the service-mock rule +below. + +## Driving socket handlers + +Every client-to-server socket event goes through one enveloped `dispatch` event +(`src/socket/dispatcher.ts`). `dispatcher.register(type, handler)` stores the handler in a map +and `listen()` attaches exactly one `socket.on('dispatch', ...)` listener, which checks +authentication, validates the envelope, rejects unknown types and stale timestamps, and applies +`eventId` replay protection before any handler is reached. + +**Grabbing a raw listener no longer works.** A test that reaches into the emitter looking for +a listener registered for `'send_message'` finds nothing, because none is registered. Worse, a +test written that way against an older revision _passed_ — while bypassing envelope validation, +idempotency, and the auth gate, which are exactly the checks those events need to be tested +through. `src/socket/dispatcher.spec.ts` has a test asserting that a raw emit does **not** reach +the handler; treat it as the specification. + +Drive handlers by emitting a well-formed envelope: + +```ts +let envelopeSeq = 0; + +function dispatchEvent(socket: EventEmitter, type: string) { + return async (payload: unknown) => { + envelopeSeq += 1; + // EventEmitter.prototype.emit.call bypasses the fake socket's emit override, + // so this delivers to the listener instead of being captured as an outbound + // server -> client emit. + EventEmitter.prototype.emit.call(socket, 'dispatch', { + eventId: `test-evt-${envelopeSeq}`, + type, + timestamp: Date.now(), + payload, + }); + await new Promise((resolve) => setTimeout(resolve, 10)); + }; +} +``` + +Preserve all five of these when copying it: + +1. **A fresh `eventId` per emit.** Replay protection is keyed by + `replay:{deviceId}:{eventId}`, so a reused value turns the second and later events into + no-ops and produces a "the handler never ran" failure with no error attached. The check + _fails open_ when Redis is `null`, so a suite that mocks Redis away will not catch a reused + id — and the same test starts failing the day someone gives the suite an `ioredis-mock` + instance. Generate a fresh id regardless. See + [`concepts-replay-protection.md`](./concepts-replay-protection.md). +2. **A current `timestamp`.** Envelopes outside `SOCKET_EVENT_MAX_AGE_MS` (5 minutes) in the + past or `SOCKET_EVENT_MAX_FUTURE_SKEW_MS` (30 seconds) in the future are rejected before the + handler runs. With `vi.useFakeTimers()`, keep the envelope timestamp inside the window. +3. **Set `socket.auth` first** (`{ userId, deviceId }`). An unauthenticated socket gets an + `error` envelope back and the handler is never reached. +4. **Await a tick.** The dispatch listener is `async` and `emit` returns synchronously, so + assert after the `setTimeout` above. +5. **Register through the real registrar** — `registerMessagingHandlers(io, socket)` — rather + than pulling a handler function out of the module, so whatever the registrar installs stays + in the path. + +Assertions usually read the socket's captured emits. `dispatch_ack` is the dispatcher's own +acknowledgement: `{ eventId, duplicate: false }` on a first occurrence and +`{ eventId, duplicate: true }` on a replay, which is the cleanest way to assert that an event +was processed exactly once. + +**The one exception:** `send_file_message` is still attached with a raw +`socket.on('send_file_message', ...)` in `src/socket/messaging.ts`, so it is triggered by +emitting that event name directly. It is the only handler for which that is correct. If it +moves onto the dispatcher, its tests move with it. + +Worked examples: `src/__tests__/dispatcher.test.ts` and `src/socket/dispatcher.spec.ts` for the +dispatcher itself, `src/__tests__/askAssistant.test.ts` for a handler driven through +`dispatch`. + +## Service mocks: add one whenever a route gains a dependency + +Route tests mock the database, but a route also calls services — and an unmocked service runs +for real, which means it reaches for the database _it_ imported, or Redis, or the object store. +The failures are indirect: a timeout, a `500` from a service several frames down, or a passing +test that quietly performed a real side effect. + +**When a route or handler gains a new service dependency, every existing test file for that +route needs the new `vi.mock`, in the same change.** This is the single most common way a +change breaks unrelated backend tests, and the error message never names the new import. + +Frequently mocked services, with what they stand in for: + +| Module | Why a route test mocks it | +| --------------------------------- | ------------------------------------------------------------------------------------------------- | +| `../services/pushNotification.js` | Would attempt a real web-push send | +| `../services/deliveryPipeline.js` | Envelope delivery and per-device fan-out | +| `../services/deviceDelivery.js` | Opens a duplicated Redis subscriber connection | +| `../services/e2eeProtocol.js` | Queries `devices` for capabilities; mock when the test is not about protocol negotiation | +| `../services/mlsGroups.js` | Group epoch state | +| `../services/presence.js` | Redis presence keys | +| `../services/deviceRevocation.js` | Revocation broadcast | +| `../services/roomManager.js` | Socket room membership | +| `../services/auditLog.js` | Writes an audit row on many routes | +| `../services/fileCleanup.js` | Soft-delete plus object-store work | +| `../lib/socket.js` | `getSocketServer()` — return `null`, or `{ to: () => ({ emit }) }` when the test asserts on emits | + +Mock at the seam the route imports, and give the mock the same shape the real export has +(`vi.fn().mockResolvedValue(undefined)` for a fire-and-forget async service). A service mocked +as a synchronous `vi.fn()` where the route awaits it works by accident and stops working the +moment the route checks the result. + +When the test _is_ about the service, mock the service's own dependencies instead and exercise +the real thing — `src/__tests__/e2eeProtocol.test.ts` mocks only `db`, `db/schema`, and +`drizzle-orm`, and runs the real `checkEnvelopeProtocols`. + +## Resetting shared in-process state + +Several modules keep module-level state that survives across tests inside a Vitest worker. +`vi.clearAllMocks()` does not touch it, because it is not a mock — it is the real module's +memory. + +Rate limiting is the one that bites most often. `services/rateLimiter.ts` keeps a +`localCounters` map used whenever Redis is unavailable — which, in a suite that mocks `redis` +to `null`, is always. The map is keyed by bucket, window, and subject, and the window comes +from wall-clock time, so several tests in one file hitting the same endpoint as the same +subject are all charged against **one** budget. The symptom is a test that passes alone and +returns `429` when the file runs in order, or a failure that moves when you reorder the file. + +```ts +const { resetRateLimitBucket, clearLocalRateLimitCounters } = + await import('../services/rateLimiter.js'); + +beforeEach(async () => { + vi.clearAllMocks(); + clearLocalRateLimitCounters(); // the process-local fallback map + await resetRateLimitBucket('auth_challenge'); // that bucket's Redis keys too + await resetRateLimitBucket('auth_verify'); + await resetRateLimitBucket('global_ip'); +}); +``` + +`clearLocalRateLimitCounters()` clears only the in-process map. `resetRateLimitBucket(bucket)` +clears the matching local keys **and** scans and deletes the bucket's keys in Redis, real or +`ioredis-mock`. Where a suite shares one `ioredis-mock` instance, `await sharedRedis.flushall()` +in `beforeEach` is the blunter equivalent for the Redis half. + +The same shape of leak exists elsewhere. Each affected module exports its own hook — use it +rather than reaching into the module: + +| Module | State it keeps | Reset | +| ----------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------- | +| `services/rateLimiter.ts` | Fallback counters plus Redis buckets | `clearLocalRateLimitCounters()`, `resetRateLimitBucket(bucket)` | +| `services/rateLimit.ts` | Per-socket repeat-violation counts | `clearViolations(socketId)` | +| `services/prekeyLowSignal.ts` | One-shot alert latches, so a second low-prekey event does not re-alert | `__resetPrekeyLowLatches()` | +| `services/presence.ts` | Offline-broadcast dedupe set | `__resetOfflineBroadcastsForTesting()` | +| `services/heartbeat.ts` | Per-socket heartbeat timers | `clearHeartbeatTimer(socketId)` | +| `lib/objectStore.ts` | Memoised S3 client | `resetObjectStoreForTests()` | + +Timers are a second form of leaked state: a test that leaves a `setTimeout` pending (typing +indicators, heartbeats) can fire it during a later test. Clear timers the handler created, or +use `vi.useFakeTimers()` and dispose of them in `afterEach`. + +The general rule: **if a module keeps state outside a function so that production behaves +correctly across requests, it needs a test-visible reset, and every suite that touches it calls +that reset in `beforeEach`.** When you add such state, export the reset in the same commit. + +## Checklist for a new backend test + +1. File in `src/__tests__/` as `*.test.ts` (or co-located `*.spec.ts` for a single module). +2. Mocks declared before the subject; subject imported with `await import(...)`. +3. `db/index.js`, `db/schema.js`, `drizzle-orm`, `lib/redis.js`, `middleware/auth.js` mocked — + with every table and operator the module imports present. +4. Every service the route calls mocked, unless the service is what is under test. +5. Socket events driven through `dispatch`, with a fresh `eventId` and a current `timestamp`. +6. Shared in-process state reset in `beforeEach`. +7. Assertions on behaviour — status, body, emitted events, rows handed to `insert`/`update` — + never on generated SQL. +8. No security invariant weakened to make a test pass. The guards in + `src/__tests__/security.regression.test.ts` have their own CI job; if a change trips them, + the change is wrong. +9. `pnpm --filter backend test`, `lint`, and `format:check` all clean. + +## Related documents + +- [Testing strategy and conventions](../../../docs/testing.md) — the cross-app rules this + document sits under. +- [Replay protection and event idempotency](./concepts-replay-protection.md) — why `eventId` + must be fresh per emit and `messageId` must not be. +- [Backend caching reference](./concepts-caching.md) — what the `lib/redis.js` mock is standing + in for, and the degraded path a `null` Redis exercises. +- [Gateway architecture](./concepts-gateway-architecture.md) — the socket lifecycle the + dispatcher tests model. +- [Database migration workflow](./migrations.md) — why the suite never runs migrations. diff --git a/docs/README.md b/docs/README.md index 6be7af6..34fa642 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,14 +38,14 @@ it is enough to make a scoped change and open a pull request. You want orientation and the shortest path to a working local environment. -| Document | What it gives you | -| --- | --- | -| [Root README](../README.md) | Project pitch, tech stack, prerequisites, install, run, and test commands. | -| [System architecture overview](architecture-overview.md) | The single diagram of all four apps and every external service, with two traced end-to-end paths. | -| [Runbook](runbook.md) | Day-two operations: what to do when a service is unhealthy, and how to restart pieces safely. | -| [Observability](observability.md) | Which metrics, logs, and traces exist and where they are emitted, so you can see what your change did. | -| [Testing strategy and conventions](testing.md) | The per-app test runners and commands, the rule that tests never start Redis, Postgres, or S3, and the conventions every new test must follow. | -| [Security policy](../SECURITY.md) | How to report a vulnerability privately, what is in scope, and the response windows you can expect. | +| Document | What it gives you | +| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| [Root README](../README.md) | Project pitch, tech stack, prerequisites, install, run, and test commands. | +| [System architecture overview](architecture-overview.md) | The single diagram of all four apps and every external service, with two traced end-to-end paths. | +| [Runbook](runbook.md) | Day-two operations: what to do when a service is unhealthy, and how to restart pieces safely. | +| [Observability](observability.md) | Which metrics, logs, and traces exist and where they are emitted, so you can see what your change did. | +| [Testing strategy and conventions](testing.md) | The per-app test runners and commands, the rule that tests never start Redis, Postgres, or S3, and the conventions every new test must follow. | +| [Security policy](../SECURITY.md) | How to report a vulnerability privately, what is in scope, and the response windows you can expect. | ### Backend developer @@ -54,48 +54,52 @@ listener. **Architecture and concepts** -| Document | What it gives you | -| --- | --- | -| [Gateway architecture](../apps/backend/docs/concepts-gateway-architecture.md) | Socket.IO connection lifecycle, room semantics, and how the gateway scales horizontally over Redis pub/sub. | -| [Delivery fan-out and receipts](../apps/backend/docs/concepts-delivery-fanout.md) | How one sent message reaches every recipient device, how receipts flow back, and which services are not actually wired into the live path. | -| [Storage and push jobs](../apps/backend/docs/concepts-storage-push-jobs.md) | Object storage layout and the background jobs that expire files, devices, and envelopes. | -| [Testing strategy and conventions](testing.md) | The Drizzle mocking pattern, driving socket handlers through the `dispatch` envelope, and the in-process counters that leak between tests. | +| Document | What it gives you | +| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| [Gateway architecture](../apps/backend/docs/concepts-gateway-architecture.md) | Socket.IO connection lifecycle, room semantics, and how the gateway scales horizontally over Redis pub/sub. | +| [Delivery fan-out and receipts](../apps/backend/docs/concepts-delivery-fanout.md) | How one sent message reaches every recipient device, how receipts flow back, and which services are not actually wired into the live path. | +| [Storage and push jobs](../apps/backend/docs/concepts-storage-push-jobs.md) | Object storage layout and the background jobs that expire files, devices, and envelopes. | +| [Device capability and E2EE protocol negotiation](../apps/backend/docs/concepts-protocol-negotiation.md) | What a device advertises at registration, how a sender and recipient agree on a protocol, and why the protocol is recorded per envelope. | +| [Replay protection and event idempotency](../apps/backend/docs/concepts-replay-protection.md) | The device-scoped `eventId` dedup, the message-level `messageId` idempotency, the TTL, and the `dispatch_ack` duplicate flag. | +| [Backend caching reference](../apps/backend/docs/concepts-caching.md) | The conversation-list cache: key, TTL, payload, every invalidation site, and the behaviour when Redis is down. | +| [Testing strategy and conventions](testing.md) | The Drizzle mocking pattern, driving socket handlers through the `dispatch` envelope, and the in-process counters that leak between tests. | +| [Backend testing guide](../apps/backend/docs/testing.md) | The standard route-test mock set with a copyable skeleton, the Drizzle chain traps, and the state a suite has to reset between tests. | **API reference** -| Document | What it gives you | -| --- | --- | -| [Auth API](../apps/backend/docs/api-auth.md) | Wallet-signature login, JWT issuance, and session refresh endpoints. | -| [Users API](../apps/backend/docs/api-users.md) | User profile read and update routes. | -| [Devices and prekeys API](../apps/backend/docs/api-devices.md) | Every `/devices` and `/user-devices` route: ownership checks, prekey upload contract, and revocation side effects. | -| [Conversations API](../apps/backend/docs/api-conversations.md) | Creating conversations, managing membership, and reading history. | -| [Messages and sync API](../apps/backend/docs/api-messages-sync.md) | Message history pagination and the cross-device sync cursor. | -| [Files and uploads API](../apps/backend/docs/api-files-uploads.md) | Encrypted attachment upload, download, and lifecycle. | -| [Push API](../apps/backend/docs/api-push.md) | Push subscription registration and notification dispatch. | -| [Treasury API](../apps/backend/docs/api-treasury.md) | REST routes for treasury proposals and votes, plus how they relate to the on-chain contracts. | -| [WebSocket events](../apps/backend/docs/api-websocket-events.md) | Every Socket.IO event the gateway emits and accepts, with direction and payload. | +| Document | What it gives you | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | +| [Auth API](../apps/backend/docs/api-auth.md) | Wallet-signature login, JWT issuance, and session refresh endpoints. | +| [Users API](../apps/backend/docs/api-users.md) | User profile read and update routes. | +| [Devices and prekeys API](../apps/backend/docs/api-devices.md) | Every `/devices` and `/user-devices` route: ownership checks, prekey upload contract, and revocation side effects. | +| [Conversations API](../apps/backend/docs/api-conversations.md) | Creating conversations, managing membership, and reading history. | +| [Messages and sync API](../apps/backend/docs/api-messages-sync.md) | Message history pagination and the cross-device sync cursor. | +| [Files and uploads API](../apps/backend/docs/api-files-uploads.md) | Encrypted attachment upload, download, and lifecycle. | +| [Push API](../apps/backend/docs/api-push.md) | Push subscription registration and notification dispatch. | +| [Treasury API](../apps/backend/docs/api-treasury.md) | REST routes for treasury proposals and votes, plus how they relate to the on-chain contracts. | +| [WebSocket events](../apps/backend/docs/api-websocket-events.md) | Every Socket.IO event the gateway emits and accepts, with direction and payload. | **Contracts and schemas** -| Document | What it gives you | -| --- | --- | -| [JWT auth contract](../apps/backend/docs/contracts-jwt-auth.md) | Token claim shape, signing algorithm, and expiry rules. | -| [REST schemas](../apps/backend/docs/contracts-rest-schemas.md) | Request and response body schemas shared across the REST surface. | +| Document | What it gives you | +| ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [JWT auth contract](../apps/backend/docs/contracts-jwt-auth.md) | Token claim shape, signing algorithm, and expiry rules. | +| [REST schemas](../apps/backend/docs/contracts-rest-schemas.md) | Request and response body schemas shared across the REST surface. | | [Error code and response catalog](../apps/backend/docs/contracts-error-catalog.md) | Every error the backend can return on either transport: the REST status/`error` table, the socket `error` payload shapes, the rate-limit response, and which errors are retryable. | -| [WebSocket payloads](../apps/backend/docs/contracts-websocket-payloads.md) | Payload shapes for each WebSocket event, as validated on the wire. | +| [WebSocket payloads](../apps/backend/docs/contracts-websocket-payloads.md) | Payload shapes for each WebSocket event, as validated on the wire. | **Encryption and migrations** -| Document | What it gives you | -| --- | --- | -| [Database migration workflow](../apps/backend/docs/migrations.md) | The drizzle-kit loop from `schema.ts` to applied SQL, the `drizzle/` layout, and how to resolve the colliding-migration merge conflict that has already broken this history once. | -| [E2EE onboarding](../apps/backend/docs/e2ee-onboarding.md) | Device registration and prekey upload flow for first-contact DM setup. | -| [MLS key packages](../apps/backend/docs/mls-key-packages.md) | Key package publication, consumption, and replenishment. | -| [MLS group membership](../apps/backend/docs/mls-group-membership.md) | Adding and removing members from an MLS group and the resulting epoch changes. | -| [MLS group files](../apps/backend/docs/mls-group-files.md) | How file keys are distributed to an MLS group. | -| [Message encryption migration](../apps/backend/docs/message-encryption-migration.md) | Migrating stored messages onto the current encryption scheme. | -| [Signal migration](../apps/backend/docs/signal-migration.md) | Moving the double-ratchet implementation onto the Signal protocol. | -| [Backend security hardening](../apps/backend/docs/security-hardening.md) | Backend-specific hardening measures and the threats each one closes. | +| Document | What it gives you | +| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Database migration workflow](../apps/backend/docs/migrations.md) | The drizzle-kit loop from `schema.ts` to applied SQL, the `drizzle/` layout, and how to resolve the colliding-migration merge conflict that has already broken this history once. | +| [E2EE onboarding](../apps/backend/docs/e2ee-onboarding.md) | Device registration and prekey upload flow for first-contact DM setup. | +| [MLS key packages](../apps/backend/docs/mls-key-packages.md) | Key package publication, consumption, and replenishment. | +| [MLS group membership](../apps/backend/docs/mls-group-membership.md) | Adding and removing members from an MLS group and the resulting epoch changes. | +| [MLS group files](../apps/backend/docs/mls-group-files.md) | How file keys are distributed to an MLS group. | +| [Message encryption migration](../apps/backend/docs/message-encryption-migration.md) | Migrating stored messages onto the current encryption scheme. | +| [Signal migration](../apps/backend/docs/signal-migration.md) | Moving the double-ratchet implementation onto the Signal protocol. | +| [Backend security hardening](../apps/backend/docs/security-hardening.md) | Backend-specific hardening measures and the threats each one closes. | ### Frontend developer @@ -104,95 +108,95 @@ is about encryption and local state. **Concepts** -| Document | What it gives you | -| --- | --- | -| [Web app README](../apps/web/README.md) | Running the Next.js client on its own, its scripts, and its environment variables. | -| [E2EE architecture](../apps/web/docs/concepts-e2ee-architecture.md) | Where keys live in the browser, how sessions are established, and what never leaves the device. | -| [Message pipeline](../apps/web/docs/concepts-message-pipeline.md) | The client-side path from typed text to an encrypted envelope on the wire, and back. | -| [Auth and device lifecycle](../apps/web/docs/concepts-auth-device-lifecycle.md) | Wallet connection, device registration, session persistence, and revocation handling. | -| [File encryption](../apps/web/docs/concepts-file-encryption.md) | How attachments are encrypted client-side before upload. | -| [Local search](../apps/web/docs/concepts-local-search.md) | The on-device search index over decrypted message content. | -| [Push subscription](../apps/web/docs/concepts-push-subscription.md) | Service worker registration and push permission handling. | -| [Service worker and offline behaviour](../apps/web/docs/concepts-service-worker.md) | `sw.js` registration/update lifecycle, the content-free push handler, notification click routing, and what works offline today. | -| [Error handling and user feedback](../apps/web/docs/concepts-error-handling.md) | Toasts vs. inline error state, mapping backend errors to user-facing messages, and the rule that decryption failures never render as a generic crash. | -| [Accessibility guide](../apps/web/docs/accessibility.md) | The WCAG 2.1 AA target, keyboard navigation, modal focus management, live-region announcements, and colour contrast. | -| [Wallet and treasury UI](../apps/web/docs/concepts-wallet-treasury-ui.md) | How the wallet and treasury screens are composed and what they read from chain versus the backend. | -| [Testing strategy and conventions](testing.md) | The web Vitest setup, the `fake-indexeddb` and WebCrypto substitutes, and the include pattern that quietly skips `.tsx` test files. | +| Document | What it gives you | +| ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Web app README](../apps/web/README.md) | Running the Next.js client on its own, its scripts, and its environment variables. | +| [E2EE architecture](../apps/web/docs/concepts-e2ee-architecture.md) | Where keys live in the browser, how sessions are established, and what never leaves the device. | +| [Message pipeline](../apps/web/docs/concepts-message-pipeline.md) | The client-side path from typed text to an encrypted envelope on the wire, and back. | +| [Auth and device lifecycle](../apps/web/docs/concepts-auth-device-lifecycle.md) | Wallet connection, device registration, session persistence, and revocation handling. | +| [File encryption](../apps/web/docs/concepts-file-encryption.md) | How attachments are encrypted client-side before upload. | +| [Local search](../apps/web/docs/concepts-local-search.md) | The on-device search index over decrypted message content. | +| [Push subscription](../apps/web/docs/concepts-push-subscription.md) | Service worker registration and push permission handling. | +| [Service worker and offline behaviour](../apps/web/docs/concepts-service-worker.md) | `sw.js` registration/update lifecycle, the content-free push handler, notification click routing, and what works offline today. | +| [Error handling and user feedback](../apps/web/docs/concepts-error-handling.md) | Toasts vs. inline error state, mapping backend errors to user-facing messages, and the rule that decryption failures never render as a generic crash. | +| [Accessibility guide](../apps/web/docs/accessibility.md) | The WCAG 2.1 AA target, keyboard navigation, modal focus management, live-region announcements, and colour contrast. | +| [Wallet and treasury UI](../apps/web/docs/concepts-wallet-treasury-ui.md) | How the wallet and treasury screens are composed and what they read from chain versus the backend. | +| [Testing strategy and conventions](testing.md) | The web Vitest setup, the `fake-indexeddb` and WebCrypto substitutes, and the include pattern that quietly skips `.tsx` test files. | **Client APIs and types** -| Document | What it gives you | -| --- | --- | -| [REST client](../apps/web/docs/api-rest-client.md) | The typed wrapper around the backend REST surface. | -| [WebSocket client](../apps/web/docs/api-websocket-client.md) | Socket lifecycle, reconnection, and event subscription on the client. | -| [Soroban client](../apps/web/docs/api-soroban-client.md) | How the web app builds, signs, and submits Soroban contract invocations. | +| Document | What it gives you | +| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | +| [REST client](../apps/web/docs/api-rest-client.md) | The typed wrapper around the backend REST surface. | +| [WebSocket client](../apps/web/docs/api-websocket-client.md) | Socket lifecycle, reconnection, and event subscription on the client. | +| [Soroban client](../apps/web/docs/api-soroban-client.md) | How the web app builds, signs, and submits Soroban contract invocations. | | [Backend error catalog](../apps/backend/docs/contracts-error-catalog.md) | Every error the client can receive from the backend, both transports, and which ones are worth retrying. | -| [Auth session contract](../apps/web/docs/contracts-auth-session.md) | The shape of the persisted session and what invalidates it. | -| [IndexedDB schemas](../apps/web/docs/contracts-indexeddb-schemas.md) | Every IndexedDB object store, its keys, and its migration history. | -| [Response types](../apps/web/docs/contracts-response-types.md) | Shared TypeScript response types used across the client. | -| [MLS integration notes](../apps/web/src/lib/mls-integration.md) | Implementation notes co-located with the MLS integration code. | -| [Search module README](../apps/web/src/lib/search/README.md) | Implementation notes for the local search module. | +| [Auth session contract](../apps/web/docs/contracts-auth-session.md) | The shape of the persisted session and what invalidates it. | +| [IndexedDB schemas](../apps/web/docs/contracts-indexeddb-schemas.md) | Every IndexedDB object store, its keys, and its migration history. | +| [Response types](../apps/web/docs/contracts-response-types.md) | Shared TypeScript response types used across the client. | +| [MLS integration notes](../apps/web/src/lib/mls-integration.md) | Implementation notes co-located with the MLS integration code. | +| [Search module README](../apps/web/src/lib/search/README.md) | Implementation notes for the local search module. | ### Contract developer The Soroban workspace in `contracts/`: `token_transfer`, `group_treasury`, and `proposals`. -| Document | What it gives you | -| --- | --- | -| [Contracts README](../contracts/README.md) | Workspace layout, toolchain, and how to build and test the contracts. | -| [Contract testing guide](../contracts/docs/testing.md) | Soroban test scaffolding, auth mocking vs. asserting real auth, testing expiry via the virtual ledger clock, and the proposals → group_treasury cross-contract test setup. | -| [Deployment and invocation](../contracts/docs/api-deployment-invocation.md) | Deploying each contract, initialising it, and invoking it from the CLI, including required environment variables. | -| [Contract events reference](../contracts/docs/contracts-events.md) | Every published event across all three contracts, its topic and data shape, the state change it signals, and whether the backend listener consumes it. | -| [Proposals API](../contracts/docs/api-proposals.md) | The `proposals` contract surface: creating, voting, finalising, and executing. | -| [Token transfer API](../contracts/docs/api-token-transfer.md) | The `token_transfer` contract surface, including the memo field used to correlate a transfer with a chat message. | -| [Proposal lifecycle](../contracts/docs/concepts-proposal-lifecycle.md) | Every proposal status, the transitions between them, and what triggers each one. | -| [Token transfer flow](../contracts/docs/concepts-token-transfer-flow.md) | The end-to-end flow of an in-chat payment through the contract. | -| [Token transfer storage](../contracts/docs/contracts-token-transfer-storage.md) | Storage keys and value types used by `token_transfer`. | -| [WASM size and resource budget](../contracts/docs/concepts-resource-budget.md) | The 100 KB per-contract CI gate, current sizes and headroom, and the levers available when a contract approaches the limit. | +| Document | What it gives you | +| ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Contracts README](../contracts/README.md) | Workspace layout, toolchain, and how to build and test the contracts. | +| [Contract testing guide](../contracts/docs/testing.md) | Soroban test scaffolding, auth mocking vs. asserting real auth, testing expiry via the virtual ledger clock, and the proposals → group_treasury cross-contract test setup. | +| [Deployment and invocation](../contracts/docs/api-deployment-invocation.md) | Deploying each contract, initialising it, and invoking it from the CLI, including required environment variables. | +| [Contract events reference](../contracts/docs/contracts-events.md) | Every published event across all three contracts, its topic and data shape, the state change it signals, and whether the backend listener consumes it. | +| [Proposals API](../contracts/docs/api-proposals.md) | The `proposals` contract surface: creating, voting, finalising, and executing. | +| [Token transfer API](../contracts/docs/api-token-transfer.md) | The `token_transfer` contract surface, including the memo field used to correlate a transfer with a chat message. | +| [Proposal lifecycle](../contracts/docs/concepts-proposal-lifecycle.md) | Every proposal status, the transitions between them, and what triggers each one. | +| [Token transfer flow](../contracts/docs/concepts-token-transfer-flow.md) | The end-to-end flow of an in-chat payment through the contract. | +| [Token transfer storage](../contracts/docs/contracts-token-transfer-storage.md) | Storage keys and value types used by `token_transfer`. | +| [WASM size and resource budget](../contracts/docs/concepts-resource-budget.md) | The 100 KB per-contract CI gate, current sizes and headroom, and the levers available when a contract approaches the limit. | ### Operator Running and monitoring a deployment. -| Document | What it gives you | -| --- | --- | -| [Runbook](runbook.md) | Operational procedures: health checks, restarts, and incident response steps. | -| [Observability](observability.md) | Metrics, logs, and traces exposed by the services, and how to reach them. | +| Document | What it gives you | +| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| [Runbook](runbook.md) | Operational procedures: health checks, restarts, and incident response steps. | +| [Observability](observability.md) | Metrics, logs, and traces exposed by the services, and how to reach them. | | [Deployment and invocation](../contracts/docs/api-deployment-invocation.md) | Contract deployment steps and the environment variables the backend needs to watch the chain. | -| [Storage and push jobs](../apps/backend/docs/concepts-storage-push-jobs.md) | The background jobs that run on a schedule and the storage they clean up. | +| [Storage and push jobs](../apps/backend/docs/concepts-storage-push-jobs.md) | The background jobs that run on a schedule and the storage they clean up. | ### Security reviewer Threat model, hardening, and the crypto protocol documents. -| Document | What it gives you | -| --- | --- | -| [Security policy](../SECURITY.md) | The private disclosure channel, response windows, scope, and the extra care an on-chain finding needs. | -| [Threat model](threat-model.md) | Assets, adversaries, trust boundaries, and the mitigations claimed for each threat. | -| [Security fixes summary](../SECURITY_FIXES_SUMMARY.md) | A log of security issues found and the fixes applied for each. | -| [Audit logging](security/audit-logging.md) | What is audit-logged, in what format, and what is deliberately excluded. | -| [Rate limits](security/rate-limits.md) | Every rate limit in the system, its scope, and its threshold. | -| [TLS and pinning](security/tls-and-pinning.md) | Transport security requirements and certificate pinning behaviour. | -| [Backend security hardening](../apps/backend/docs/security-hardening.md) | Backend hardening measures and the threats each one closes. | -| [Signal integration](signal-integration.md) | How the Signal protocol is integrated and which guarantees it provides. | -| [Group epoch sync](group-epoch-sync.md) | How MLS group epochs stay synchronised across devices and what happens when they diverge. | -| [E2EE architecture](../apps/web/docs/concepts-e2ee-architecture.md) | The client-side key model — the basis for any claim that the server cannot read messages. | +| Document | What it gives you | +| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | +| [Security policy](../SECURITY.md) | The private disclosure channel, response windows, scope, and the extra care an on-chain finding needs. | +| [Threat model](threat-model.md) | Assets, adversaries, trust boundaries, and the mitigations claimed for each threat. | +| [Security fixes summary](../SECURITY_FIXES_SUMMARY.md) | A log of security issues found and the fixes applied for each. | +| [Audit logging](security/audit-logging.md) | What is audit-logged, in what format, and what is deliberately excluded. | +| [Rate limits](security/rate-limits.md) | Every rate limit in the system, its scope, and its threshold. | +| [TLS and pinning](security/tls-and-pinning.md) | Transport security requirements and certificate pinning behaviour. | +| [Backend security hardening](../apps/backend/docs/security-hardening.md) | Backend hardening measures and the threats each one closes. | +| [Signal integration](signal-integration.md) | How the Signal protocol is integrated and which guarantees it provides. | +| [Group epoch sync](group-epoch-sync.md) | How MLS group epochs stay synchronised across devices and what happens when they diverge. | +| [E2EE architecture](../apps/web/docs/concepts-e2ee-architecture.md) | The client-side key model — the basis for any claim that the server cannot read messages. | ### AI / data developer The FastAPI service in `apps/ai_agent`. -| Document | What it gives you | -| --- | --- | -| [AI agent README](../apps/ai_agent/README.md) | Running the service locally with `uv`, and its environment variables. | -| [Chat API](../apps/ai_agent/docs/api-chat.md) | The assistant chat endpoint: request, response, and system prompt behaviour. | -| [Index and search API](../apps/ai_agent/docs/api-index-search.md) | Indexing documents into the vector store and querying them. | -| [Proposals summarise API](../apps/ai_agent/docs/api-proposals-summarise.md) | Summarising a governance proposal into a short digest. | -| [Transfers analyse API](../apps/ai_agent/docs/api-transfers-analyse.md) | Risk-scoring a transfer and the flagging threshold. | -| [RAG search architecture](../apps/ai_agent/docs/concepts-rag-search-architecture.md) | Retrieval-augmented search design: chunking, embedding, and retrieval. | -| [Transfer risk analysis](../apps/ai_agent/docs/concepts-transfer-risk-analysis.md) | The heuristics behind transfer risk scoring. | -| [Pydantic models](../apps/ai_agent/docs/contracts-pydantic-models.md) | Request and response model definitions for the service. | -| [Weaviate schema](../apps/ai_agent/docs/contracts-weaviate-schema.md) | The vector store collection schema and its properties. | +| Document | What it gives you | +| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | +| [AI agent README](../apps/ai_agent/README.md) | Running the service locally with `uv`, and its environment variables. | +| [Chat API](../apps/ai_agent/docs/api-chat.md) | The assistant chat endpoint: request, response, and system prompt behaviour. | +| [Index and search API](../apps/ai_agent/docs/api-index-search.md) | Indexing documents into the vector store and querying them. | +| [Proposals summarise API](../apps/ai_agent/docs/api-proposals-summarise.md) | Summarising a governance proposal into a short digest. | +| [Transfers analyse API](../apps/ai_agent/docs/api-transfers-analyse.md) | Risk-scoring a transfer and the flagging threshold. | +| [RAG search architecture](../apps/ai_agent/docs/concepts-rag-search-architecture.md) | Retrieval-augmented search design: chunking, embedding, and retrieval. | +| [Transfer risk analysis](../apps/ai_agent/docs/concepts-transfer-risk-analysis.md) | The heuristics behind transfer risk scoring. | +| [Pydantic models](../apps/ai_agent/docs/contracts-pydantic-models.md) | Request and response model definitions for the service. | +| [Weaviate schema](../apps/ai_agent/docs/contracts-weaviate-schema.md) | The vector store collection schema and its properties. | --- @@ -200,12 +204,12 @@ The FastAPI service in `apps/ai_agent`. Documents about the repository itself rather than about the product. -| Document | What it gives you | -| --- | --- | -| [Security policy](../SECURITY.md) | How and where to report a vulnerability privately, and why never in a public issue or pull request. | -| [Testing strategy and conventions](testing.md) | Cross-app testing philosophy, runners, and the conventions a contributor must follow. | -| [Pull request template](../.github/pull_request_template.md) | The checklist every pull request is opened against. | -| [PR notes](../pr.md) | Scratch notes for an in-flight pull request; not a reference document. | +| Document | What it gives you | +| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | +| [Security policy](../SECURITY.md) | How and where to report a vulnerability privately, and why never in a public issue or pull request. | +| [Testing strategy and conventions](testing.md) | Cross-app testing philosophy, runners, and the conventions a contributor must follow. | +| [Pull request template](../.github/pull_request_template.md) | The checklist every pull request is opened against. | +| [PR notes](../pr.md) | Scratch notes for an in-flight pull request; not a reference document. | --- @@ -214,7 +218,7 @@ Documents about the repository itself rather than about the product. This file is the only entry point into the documentation, which means a document missing from it is a document nobody will find. -- **Adding a document:** add a row to the section matching the *reader* who needs it, not +- **Adding a document:** add a row to the section matching the _reader_ who needs it, not the directory it lives in. If two roles need it, list it under both — duplication across role sections is intentional, since each section is meant to be read on its own. - **Moving or deleting a document:** update or remove its row in the same commit. A broken