diff --git a/apps/web/docs/components-messaging.md b/apps/web/docs/components-messaging.md new file mode 100644 index 0000000..35ddcab --- /dev/null +++ b/apps/web/docs/components-messaging.md @@ -0,0 +1,204 @@ +# Messaging components + +Source: `apps/web/src/components/messaging/` + +- `MessageThread.tsx` +- `InboundMessageRow.tsx` +- `EncryptedThumbnail.tsx` +- `UnavailableMessagePlaceholder.tsx` + +## MessageThread + +`MessageThread` is the scrollable message list: infinite-scroll-to-load-older, scroll-position +preservation across prepends, and a live typing indicator. + +**Props** (`MessageThreadProps`): + +| Prop | Type | Purpose | +|---|---|---| +| `messages` | `ChatMessage[]` | Messages to render, oldest-first (see `useMessageHistory`'s `ChatMessage`). | +| `loadingOlder` | `boolean` | Shows a `Spinner` at the top while an older page is being fetched. | +| `hasReachedStart` | `boolean` | Shows "No more messages" (or the empty state) once the server has no earlier page. | +| `onLoadOlder` | `() => void` | Called when the user scrolls within `triggerDistance` of the top. | +| `triggerDistance?` | `number` (default `120`) | Pixel threshold from the top that re-arms `onLoadOlder`. | +| `renderMessage?` | `(message: ChatMessage) => React.ReactNode` | Row renderer override; falls back to the internal `DefaultMessageRow`. | +| `socket?` | `Socket \| null` | Socket.IO client used to listen for `typing_start` / `typing_stop` / `new_message`. | +| `currentUserId?` | `string` | Suppresses the typing indicator for the local user. | +| `conversationId?` | `string` | Scopes typing events to the conversation currently open. | + +Behavior notes: + +- A `useLayoutEffect` diffs `messages[0].id` against the previous render and adds the resulting + `scrollHeight` delta to `scrollTop`, so prepending older messages doesn't visually jump the view. +- Typing users are tracked in a `Set`, each with its own 3s auto-hide timer (cleared early + by `typing_stop` or by any `new_message` for the conversation). +- `messages.map` uses `message.id` as the React key — `useMessageHistory` dedupes by id specifically + so this key is always unique (see that hook's comments). +- The built-in `DefaultMessageRow` (used only when `renderMessage` is not supplied) reads + `message.content || message.ciphertext || ''` directly — it does **not** call into the + `InboundMessage`/decrypt pipeline described below. It is a plain, non-E2EE-aware fallback row. + +**Where it's rendered:** no current call site imports `MessageThread` outside its own file — it is +not yet wired into `apps/web/src/app/app/conversations/[id]/page.tsx`, which currently renders its +own inline message list (see below) rather than using this component. + +## InboundMessageRow + +Renders a single E2EE-decrypted `InboundMessage` as a chat bubble, switching on `message.status`. + +**Props** (`InboundMessageRowProps`): + +| Prop | Type | Purpose | +|---|---|---| +| `message` | `InboundMessage` (from `@/lib/crypto/types`) | The row's data — see "Message row states" below. | +| `isSelf` | `boolean` | Right-aligns the bubble and swaps to the accent-colored "self" style. | +| `senderName?` | `string` | Shown above the bubble for messages from others (not shown when `isSelf`). | + +Render branches, in order: + +1. `status === 'decrypted' && plaintext` → the bubble with `message.plaintext`. +2. `status === 'unavailable' && unavailableReason` → ``. +3. otherwise (i.e. `status === 'pending'`, or `'unavailable'` with no reason set) → an italic + "Decrypting…" placeholder. + +**Where it's rendered:** like `MessageThread`, `InboundMessageRow` has no current importer outside +its own file. The live conversation page (`apps/web/src/app/app/conversations/[id]/page.tsx`) +implements an equivalent branch inline (see "Message row states" below) rather than using this +component — `InboundMessageRow` / `MessageThread` / `useInboundPipeline` appear to be the +newer/parallel E2EE-message-list implementation. + +## EncryptedThumbnail + +Renders an inline decrypted preview for an image/video attachment's thumbnail. + +**Props** (`EncryptedThumbnailProps`): + +| Prop | Type | Purpose | +|---|---|---| +| `thumbnail` | `FileMessagePayload['thumbnail']` | The `{ fileId, fileKey, iv, mimeType }` reference from a decrypted file message. | +| `authToken` | `string` | JWT sent when fetching the presigned download URL. | +| `apiBaseUrl` | `string` | Backend base URL for the download request. | +| `alt?` | `string` (default `'File thumbnail'`) | `` alt text. | +| `className?` | `string` | Overrides the default `` classes. | + +**Decrypt-to-object-URL flow** (`useEffect` on `[thumbnail, authToken, apiBaseUrl]`): + +1. Calls `decryptThumbnailToObjectUrl(thumbnail, authToken, apiBaseUrl)` from `@/lib/thumbnail`, + which: + - downloads + decrypts the thumbnail ciphertext via `downloadAndDecryptFile(...)` (AES-GCM, + using `thumbnail.fileKey`/`thumbnail.iv`), producing a plaintext `Blob`; + - calls `URL.createObjectURL(plainBlob)` and returns that URL (or `null` on any decrypt/download + error, which it also `console.warn`s). +2. On resolution, `setObjectUrl(url)` swaps the skeleton for the real ``. +3. On rejection, `setError(true)` renders a small "⚠️" placeholder box instead. + +**Cleanup obligation:** the effect's cleanup function revokes the *current* `objectUrl` state +(`URL.revokeObjectURL(objectUrl)`) and sets a local `revoked` flag so an in-flight decrypt that +resolves after unmount/prop-change is ignored (`if (!revoked) setObjectUrl(url)`). Because +`objectUrl` is deliberately left out of the effect's dependency array (documented inline with an +`eslint-disable-next-line react-hooks/exhaustive-deps`), the revoke only happens once per +`[thumbnail, authToken, apiBaseUrl]` change/unmount rather than firing on every `objectUrl` update +— but it does mean the object URL created for state derived on the *previous* run of the effect is +the one revoked when the effect re-runs or the component unmounts, preventing the created blob URL +from leaking. + +Loading/empty states: renders `null` if `thumbnail` is falsy; an animated skeleton `div` while +`objectUrl` is not yet set and no error has occurred; and the `` once `objectUrl` resolves. + +**Where it's rendered:** `apps/web/src/app/app/conversations/[id]/page.tsx`, inside the message +list, for any message where `message.filePayload` is set and `message.contentType` is `'image'` or +`'video'` — paired with a "Download {fileName}" button that calls `handleFileDownload(message)`. + +## UnavailableMessagePlaceholder + +A small, reusable "this message can't be shown as text" bubble, used for both legitimately +unavailable messages and decrypt failures. + +**Props** (`UnavailableMessagePlaceholderProps`): + +| Prop | Type | Purpose | +|---|---|---| +| `reason` | `UnavailableReason` (from `@/lib/crypto/types`) | Selects the copy shown; `'pre-link' \| 'undecryptable' \| 'verification-failed'`. | + +Copy table (`REASON_COPY`): + +| `reason` | Copy shown | +|---|---| +| `'pre-link'` | "Waiting for secure session — message from before this device was linked." | +| `'undecryptable'` | "Unable to decrypt this message." | +| `'verification-failed'` | "Message could not be verified." | + +**The distinction it must preserve:** "no envelope for this device" is not the same failure as +"envelope exists but decryption/verification failed", and the component keeps them apart entirely +through the `reason` value it's handed — it does no inference of its own. The producing logic is +`apps/web/src/lib/crypto/processEnvelope.ts` (`processInboundEnvelope`): + +- **Legitimately unavailable (`'pre-link'`):** when `envelope.senderDeviceId` is `null` — + meaning this device was linked *after* the message was sent, so no per-device envelope was ever + created for it — `processInboundEnvelope` returns `{ ...base, status: 'unavailable', + unavailableReason: 'pre-link' }` *before* attempting any decryption at all. The same + `unavailableReason: 'pre-link'` is also set directly by `useInboundPipeline.ts`'s `ingestMeta` + when the backend flags a `new_message` payload's metadata as `meta.unavailable` — again short- + circuiting before a decrypt attempt. +- **Decryption/verification actually failed (envelope exists):** when an envelope *is* present and + `fetchSenderDevicePublicKey(...)` + `decryptAndVerifyEnvelope(...)` throws, the `catch` block sets + `status: 'unavailable'` with `unavailableReason: unavailableReasonFromError(err)`, which maps: + - `err instanceof VerificationFailedError` → `'verification-failed'` (signature check failed — + thrown by `decryptAndVerifyEnvelope` in `apps/web/src/lib/crypto/decrypt.ts`), + - `err instanceof DecryptError` (or anything else, as the catch-all) → `'undecryptable'` + (decrypt itself failed, e.g. corrupted/mismatched key material), + - note `PreLinkError` is also handled here (mapped to `'pre-link'`) since + `decryptAndVerifyEnvelope` throws it when no session exists for the sender device — the same + outward reason as the "no envelope at all" case above, but reached via a thrown error during an + actual decrypt attempt rather than skipping decryption up front. + +So the three `UnavailableReason` values are not synonyms for "can't read this message": `'pre-link'` +means "no session/envelope for this device — expected, not corruption"; `'undecryptable'` and +`'verification-failed'` both mean "an envelope existed and decrypting/verifying it failed." +`InboundMessageRow` and the conversation page both render this via `UnavailableMessagePlaceholder` +without collapsing the distinction — each `reason` gets its own copy. + +Note the inline usage in `apps/web/src/app/app/conversations/[id]/page.tsx` currently hardcodes +`` for any `message.unavailable` message, +rather than threading through the real reason from the backend/pipeline — so on that particular +page the pre-link vs. decrypt-failure distinction is not yet surfaced to the user, even though the +type (`UnavailableReason`) and the `processEnvelope.ts`/`useInboundPipeline.ts` pipeline do compute +it correctly for `InboundMessageRow` consumers. + +## Message row states + +Two parallel data models exist for a "message" in this codebase: + +1. **`ChatMessage`** (`apps/web/src/hooks/useMessageHistory.ts`) — the plain shape used by + `MessageThread`'s `DefaultMessageRow`. Fields: `id`, `conversationId`, `senderId`, `content`, + `createdAt`, plus optional E2EE-ish fields `ciphertext?`, `contentType?`, `sequenceNumber?`. There + is no `status` field — `DefaultMessageRow` just prints `content || ciphertext || ''` verbatim (see + the `#185` comment in `MessageThread.tsx` noting the decryption shim currently passes ciphertext + through as-is). + +2. **`InboundMessage`** (`apps/web/src/lib/crypto/types.ts`) — the richer, per-device-decrypted shape + produced by `processInboundEnvelope` / consumed by `InboundMessageRow`. Its `status` field + (`InboundMessageStatus = 'pending' | 'decrypted' | 'unavailable'`) is what actually drives + row rendering: + +| State | `status` | Other fields set | Handled by | +|---|---|---|---| +| Decrypted | `'decrypted'` | `plaintext` | `InboundMessageRow`'s first branch — plain bubble with `message.plaintext`. | +| Pending (ciphertext/meta not yet processed) | `'pending'` | — | `InboundMessageRow`'s fallback branch — "Decrypting…" italic placeholder. | +| Unavailable / pre-link (device linked after send, no envelope) | `'unavailable'` | `unavailableReason: 'pre-link'` | `UnavailableMessagePlaceholder` via `InboundMessageRow`'s second branch. | +| Unavailable / decrypt failed | `'unavailable'` | `unavailableReason: 'undecryptable'` | Same as above, different copy. | +| Unavailable / verification failed | `'unavailable'` | `unavailableReason: 'verification-failed'` | Same as above, different copy. | +| File / image / video | n/a here — handled outside `InboundMessage`/`ChatMessage` entirely | `message.filePayload`, `message.contentType` (`'image' \| 'video' \| 'file'`) | The conversation page (`apps/web/src/app/app/conversations/[id]/page.tsx`) branches directly: image/video → `EncryptedThumbnail` + a download button; other files → a file-only download row (not shown here). | + +**Not present in this codebase:** there is no "deleted tombstone" or "system event" message state — +no `tombstone`, `deleted`, or `system event` type/field was found anywhere under +`apps/web/src/components/messaging/`, `useMessageHistory.ts`, `useInboundPipeline.ts`, or +`processEnvelope.ts`. If those states are added later, they'd need a new `InboundMessageStatus` (or +a new `ChatMessage`/message-row discriminant) and a corresponding `InboundMessageRow` branch — none +currently exists to document. + +The conversation page's own inline render (not using `MessageThread`/`InboundMessageRow`) is the +current production code path; it branches on, in order: a parsed `transfer` payload → `TransferCard`; +`message.unavailable` → `UnavailableMessagePlaceholder` (reason hardcoded to `'undecryptable'`, see +caveat above); `message.filePayload` with `contentType` `'image'`/`'video'` → `EncryptedThumbnail`; +`message.filePayload` with `contentType === 'file'` → a file-download-only row. diff --git a/docs/ci-cd.md b/docs/ci-cd.md new file mode 100644 index 0000000..df84a70 --- /dev/null +++ b/docs/ci-cd.md @@ -0,0 +1,359 @@ +# CI/CD Pipeline Reference + +This document describes every GitHub Actions workflow in `.github/workflows/`: what +triggers each one, what it runs, and what a failure means for a contributor. + +## Contents + +- [Path-filtered app workflows](#path-filtered-app-workflows) + - [backend-ci](#backend-ci) + - [frontend-ci](#frontend-ci) + - [contracts-ci](#contracts-ci) + - [ai-agent-ci](#ai-agent-ci) +- [security-ci](#security-ci) +- [loadtest-nightly](#loadtest-nightly) +- [pr](#pr) +- [Repo-hygiene automation](#repo-hygiene-automation) + - [guard-main-branch](#guard-main-branch) + - [close-linked-issues](#close-linked-issues) + +## Path filters, in general + +Four workflows (`backend-ci`, `frontend-ci`, `contracts-ci`, `ai-agent-ci`) restrict +themselves to `push`/`pull_request` events whose changed files match a `paths:` filter +for one app directory, plus the workflow's own YAML file. GitHub Actions evaluates the +filter before deciding whether to even queue the workflow, so a PR that only touches, +say, `apps/web/**` will show a run for `frontend-ci` but **no run at all** for +`backend-ci`, `contracts-ci`, or `ai-agent-ci` — those checks are silently skipped, not +run-and-passed. Conversely, a PR that touches multiple app directories (e.g. a shared +type used by both frontend and backend) triggers every matching workflow. + +Practical implications: + +- If your PR is supposed to gate on a workflow and you don't see it appear in the + checks list, the most likely explanation is that your diff doesn't touch that app's + path filter — not that the workflow is broken. +- Editing a workflow file itself (e.g. `.github/workflows/backend-ci.yml`) also + triggers that workflow, so changes to the pipeline are self-testing. +- `security-ci` and `pr` have **no path filter** — they run on every push/PR + regardless of which files changed (see their sections below). + +--- + +## Path-filtered app workflows + +### backend-ci + +File: `.github/workflows/backend-ci.yml` + +| | | +|---|---| +| Trigger | `push` and `pull_request` where the diff touches `apps/backend/**` or the workflow file itself | +| Services | Postgres 16 (`localhost:5432`, db `clicked`), Redis 7 (`localhost:6379`) | +| Jobs | Single job `check` ("Format · Lint · Test") | + +Steps, in order: + +1. Checkout, set up Node 20 and pnpm. +2. `pnpm install --frozen-lockfile` (repo root). +3. Start a MinIO container (`minio/minio:RELEASE.2025-04-22T22-12-26Z`) manually via + `docker run`, wait for `/minio/health/ready`, then create the `clicked` bucket with + the AWS SDK (MinIO isn't a `services:` container — it's started by hand so the + bucket-creation script can run against it). +4. `pnpm db:migrate` against the Postgres service. +5. `pnpm format:check` +6. `pnpm lint` +7. `pnpm test` + +**Failure signal**: a failing step points at exactly one of format, lint, migration, or +test — the step names in the Actions log say which. A migration failure often means a +new migration file doesn't apply cleanly against a fresh database, not a code bug. + +### frontend-ci + +File: `.github/workflows/frontend-ci.yml` + +| | | +|---|---| +| Trigger | `push` / `pull_request` touching `apps/web/**` or the workflow file | +| Jobs | Single job `check` ("Lint · Build") | + +Steps: checkout, Node 20 + pnpm, `pnpm install --frozen-lockfile`, then +`pnpm --filter web lint`, `pnpm --filter web test`, `pnpm --filter web build`, in that +order. No services are needed — this workflow is pure static analysis, unit tests, and +a production build of the `web` package. + +**Failure signal**: lint failures are style/type issues; a `test` failure is a broken +unit test; a `build` failure usually means a type error or bundler error that only +surfaces at build time (dead code, unresolved import, etc.). + +### contracts-ci + +File: `.github/workflows/contracts-ci.yml` + +| | | +|---|---| +| Trigger | `push` / `pull_request` touching `contracts/**` or the workflow file, **plus** a weekly `schedule` (`0 8 * * 1` — every Monday 08:00 UTC) | +| Toolchain | `dtolnay/rust-toolchain@stable`, pinned by `contracts/rust-toolchain.toml` (`channel = "stable"`, target `wasm32-unknown-unknown`, components `clippy`, `rustfmt`) | +| Jobs | `test-and-build` (matrix), `clippy`, `audit` | + +**`test-and-build`** runs as a matrix over three Soroban contract packages — +`token_transfer`, `group_treasury`, `proposals` — each with `fail-fast: false` so one +package's failure doesn't cancel the others. Per package it: + +1. Installs the Rust stable toolchain + `wasm32-unknown-unknown` target. +2. Caches `~/.cargo/registry`, `~/.cargo/git/db`, and `contracts/target`, keyed by + `hashFiles('contracts/Cargo.lock')` (per-package cache key). +3. `cargo test -p `. +4. `cargo build -p --target wasm32-unknown-unknown --release`. +5. Installs `gh` (GitHub CLI) and runs the **WASM size gate** (see below). + +**WASM size gate + PR comment** (`Report WASM binary sizes` step, `if: always()` so it +runs even after a test/build failure): it scans every `*.wasm` file under +`contracts/target/wasm32-unknown-unknown/release`, computes byte and KB size for each, +and builds a markdown table. `THRESHOLD_BYTES=102400` (100 KB) — any WASM binary over +that size emits a `::error` annotation and sets `FAILED=1`, so the job exits non-zero +at the end of the step even if `cargo build` itself succeeded. The size table is always +appended to `$GITHUB_STEP_SUMMARY`. On `pull_request` events specifically, the step +also posts/updates a PR comment: it searches existing PR comments for one whose body +starts with the marker `` via `gh api ... --jq`, and either +`PATCH`es that comment in place or creates a new one — so repeated pushes to the same +PR update a single comment rather than spamming new ones on every run. + +**`clippy`** job: installs the same pinned toolchain (with `clippy` component), reuses +a workspace-wide cache key, and runs +`cargo clippy --workspace --target wasm32-unknown-unknown -- -D warnings -A dead_code -A clippy::too-many-arguments` +— all warnings are denied except `dead_code` and `too-many-arguments`, which are +explicitly allowed. + +**`audit`** job: installs `cargo-audit` and runs `cargo audit` against the workspace +for known security advisories in Rust dependencies. + +**Failure signal**: +- `test-and-build` failing on `cargo test` = a contract logic/unit-test regression in + that specific package (check the matrix leg name). +- `test-and-build` failing only on the size-gate step with a `cargo build` that + otherwise succeeded = the built WASM crossed the 100 KB ceiling; check the PR + comment / job summary table for which contract and by how much. +- `clippy` failing = a new clippy warning (lint), not a runtime bug. +- `audit` failing = a `RUSTSEC` advisory affecting a dependency version currently + pinned in `contracts/Cargo.lock`. + +**Toolchain-pinning caveat**: `contracts/rust-toolchain.toml` pins `channel = "stable"` +— not an exact version. `dtolnay/rust-toolchain@stable` therefore always installs +whatever the *current* stable Rust release is, which drifts over time. The scheduled +Monday run means this can go red **with zero code changes**, purely because a new +stable Rust release shipped a new warning, a clippy lint got stricter, or standard +library/codegen changes shifted a release WASM binary's size across the 100 KB +threshold. If `contracts-ci` fails on `main`/`dev` without a corresponding contracts +change, check the Rust release notes for the week before assuming a code regression. + +### ai-agent-ci + +File: `.github/workflows/ai-agent-ci.yml` + +| | | +|---|---| +| Trigger | `push` / `pull_request` touching `apps/ai_agent/**` or the workflow file | +| Jobs | `lint`, `typecheck`, `test` (independent, all run `working-directory: apps/ai_agent`) | + +All three jobs use `astral-sh/setup-uv@v5` (cache keyed on +`apps/ai_agent/uv.lock`) and `uv sync --group dev`. + +- **`lint`**: `uv run ruff check .` then `uv run ruff format --check .`. +- **`typecheck`**: `uv run mypy main.py`. +- **`test`**: `uv run pytest --cov=main --cov-report=xml --cov-report=term-missing`, + uploads coverage to Codecov (`continue-on-error: true` — a Codecov upload failure + does not fail the job), then appends a markdown coverage table to + `$GITHUB_STEP_SUMMARY` (falls back to a "Coverage data not available." note if + `coverage report` fails, `if: always()` so this runs even after a test failure). + +**Failure signal**: `lint`/`typecheck` failures are style/type issues in the Python +agent code; `test` failures are pytest assertion failures — check the coverage +summary/XML for which module regressed. + +--- + +## security-ci + +File: `.github/workflows/security-ci.yml` + +| | | +|---|---| +| Trigger | `push` to `main`, and **every** `pull_request` (no path filter, no branch filter on the PR side) | +| Jobs | `regression`, `dependency-audit` (independent) | + +Unlike the four workflows above, `security-ci` has no `paths:` filter, so it runs on +every PR regardless of which files changed. + +- **`regression`** ("Ciphertext-only guard + secret-field scan"): installs Node 20 + + pnpm, `pnpm install --frozen-lockfile`, then runs + `pnpm test -- security.regression.test.ts` in `apps/backend`, i.e. specifically + `apps/backend/src/__tests__/security.regression.test.ts`. This is the test suite that + guards against private keys / session-state fields leaking outside of ciphertext. +- **`dependency-audit`** ("Crypto dependency CVE audit"): runs + `node scripts/audit-crypto-deps.mjs`, which shells out to `pnpm audit --json` and + filters advisories down to a fixed allowlist of crypto-relevant packages: + `ioredis`, `jsonwebtoken`, `web-push`, `@stellar/stellar-sdk`, `drizzle-orm`, + `socket.io`, `@socket.io/redis-adapter`, `redis`, `jose`. Advisories against + transitive dependencies outside that list do not fail the job. + +**Failure signal**: `regression` failing means a change altered behavior the +ciphertext/secret-field guard test depends on — treat this as a potential plaintext +leak of keys or session state until proven otherwise, not a flaky test to retry. +`dependency-audit` failing means a new CVE was published against one of the nine +crypto-relevant packages above (or one of their dependents) — check the job output for +the specific advisory and affected version range. + +--- + +## loadtest-nightly + +File: `.github/workflows/loadtest-nightly.yml` (workflow name: `Nightly Load Test`) + +| | | +|---|---| +| Trigger | `schedule` — `0 3 * * *` (03:00 UTC nightly) — and `workflow_dispatch` (manual) | +| Services | Postgres 15, Redis 7 | +| Timeout | 20 minutes | + +This is a soak test, not a per-PR gate — it is deliberately not wired to `push` or +`pull_request`, so it never blocks a merge. Use `workflow_dispatch` to run it manually +before merging changes to the load-test scripts or gateway fan-out logic. + +Steps: + +1. Boot Postgres + Redis, `pnpm install --frozen-lockfile`, run `pnpm db:migrate`. +2. Start **two** backend instances (`pnpm dev`) on ports 3001 and 3002, both pointed at + the same `REDIS_URL` — this exercises Socket.IO fan-out and presence churn across + node boundaries via `@socket.io/redis-adapter`, which single-node CI can't catch. +3. Wait for both `/health` endpoints. +4. Seed a 200-device fixture: `npx tsx scripts/loadtest/seed.ts --devices 200 > fixture.json`. +5. Run the soak test: + `npx tsx scripts/loadtest/run.ts --fixture fixture.json --nodes http://localhost:3001,http://localhost:3002 --baseline scripts/loadtest/baseline.json --out loadtest-result.json`. +6. Always upload `loadtest-result.json` as a workflow artifact (`if: always()`), so a + failed run's data is still retrievable. + +**Baseline-regression comparison**: `scripts/loadtest/baseline.json` is a **static, +committed file** in the repo (not generated or updated by any workflow — no job in +this repo writes back to it). Its current contents: + +```json +{ + "deviceCount": 200, + "nodeCount": 2, + "latencyMs": { "p50": 80, "p95": 400, "p99": 900 }, + "peakRssMb": 350 +} +``` + +`scripts/loadtest/run.ts` reads this file when `--baseline` is passed and compares the +current run's summary against it with a fixed `REGRESSION_TOLERANCE = 1.25` (25%): +the run fails if `summary.latencyMs.p95 > baseline.latencyMs.p95 * 1.25` or +`summary.peakRssMb > baseline.peakRssMb * 1.25`. If the baseline file is missing or +unparsable, the script logs `no usable baseline at , skipping regression check` +and does not fail the run on that account. Because nothing updates this file +automatically, a deliberate, accepted performance change (e.g. a heavier feature that +legitimately raises p95 latency or memory) will keep failing every night until someone +manually edits `scripts/loadtest/baseline.json` to the new numbers. + +**Failure signal**: a failure is either a threshold breach inside `run.ts` (latency, +memory, or error-rate thresholds) or the 25%-worse-than-baseline regression check. +Since this only runs nightly/on-demand, a red run does not block any PR — but it +signals a real regression introduced sometime in the prior day's merges, and the +uploaded `loadtest-result.json` artifact has the concrete numbers to start from. + +--- + +## pr + +File: `.github/workflows/pr.yml` (workflow name: `PR Check`) + +| | | +|---|---| +| Trigger | `pull_request`, types `opened`, `synchronize`, `reopened` (no path filter) | +| Jobs | Single job `build` | + +Steps: checkout, Node 18.x, `npm i -g pnpm && pnpm install` (root-level, no +`--frozen-lockfile`), then `pnpm run lint`. + +This is a lightweight, repo-wide lint gate that runs on every PR regardless of which +paths changed — distinct from the per-app `lint` steps inside `backend-ci` / +`frontend-ci`, which only run when their respective app directories are touched. Note +it uses `npm` (not `pnpm`) to *install* pnpm itself, and does not use `--frozen-lockfile`, +so it will still install even if the lockfile is stale (unlike the other workflows). + +**Failure signal**: a failing `pnpm run lint` here is a whole-repo lint violation — +check which workspace package the lint error is reported against. + +--- + +## Repo-hygiene automation + +These two workflows don't test code — they enforce the repo's branching policy and +issue-closing behavior. Both use `pull_request_target` (runs with write-level +permissions and repo-owner secrets even for PRs opened from forks) and +`actions/github-script@v7` to call the GitHub API directly. + +### guard-main-branch + +File: `.github/workflows/guard-main-branch.yml` + +| | | +|---|---| +| Trigger | `pull_request_target`, types `opened`, `reopened`, `edited`, `ready_for_review`, only when `branches: [main]` | +| Permissions | `pull-requests: write` | + +This is the actual mechanism behind the "only `dev` accepts contributor PRs, `main` is +maintainer-only" rule described elsewhere in the repo docs. On every qualifying event +it: + +1. Re-checks `pr.base.ref === 'main'` (relevant because `edited` fires even when the PR + is edited to no longer target `main`) — if the base branch isn't `main` anymore, it + no-ops. +2. Determines whether the PR author is allowed to target `main`: the repo owner + (`context.repo.owner`) is always allowed; otherwise it looks up the author's + collaborator permission level via `getCollaboratorPermissionLevel` and allows + `admin` or `maintain`. Anyone else — including a normal `write`-level collaborator — + is treated as disallowed. A non-collaborator (typical fork contributor) is caught + by the `try`/`catch` and also treated as disallowed. +3. If disallowed: posts a comment explaining that PRs must target `dev`, then closes + the PR (`state: 'closed'`) via the API — it does not merge, retarget, or delete + anything, just closes. + +**Failure signal**: there's no pass/fail check here in the traditional CI sense — the +"signal" is the PR itself being auto-closed with an explanatory comment. If your PR +against `main` disappears/closes immediately, this workflow did it; retarget to `dev` +and reopen (or open a fresh PR against `dev`). + +### close-linked-issues + +File: `.github/workflows/close-linked-issues.yml` (workflow name: `Close Linked Issues +on Dev Merge`) + +| | | +|---|---| +| Trigger | `pull_request_target`, type `closed`, only when `branches: [dev]` | +| Permissions | `issues: write` | +| Guard | job-level `if: github.event.pull_request.merged == true` (skips PRs that were closed without merging) | + +GitHub's native "Closes #N" auto-close behavior only fires when a PR merges into the +repository's **default branch**. Since this repo's contributor workflow merges PRs into +`dev` rather than `main`, that native behavior never fires — this workflow re-implements +it for `dev` merges: + +1. Concatenates the merged PR's title + body. +2. Regex-matches closing keywords: `close(s|d)`, `fix(es|ed)`, `resolve(s|d)` followed + by `#` (case-insensitive, optional colon), de-duplicating issue numbers. +3. For each matched issue number: skips it if the number actually refers to a PR, or if + the issue is already closed; otherwise posts a + `Closed by #, merged into \`dev\`.` comment and closes the issue with + `state_reason: 'completed'`. +4. Any per-issue API error is caught and logged as a `core.warning` without failing the + whole job (so one bad issue number doesn't block the others). + +**Failure signal**: this workflow has no build/test output to fail in the usual sense. +If an issue you expected to auto-close after a `dev` merge is still open, check that +the PR title/body actually used one of the recognized keyword forms (`closes #123`, +`fixes #123`, `resolves #123`, etc.) — free-text like "related to #123" or "see #123" +is intentionally not matched. diff --git a/docs/development-setup.md b/docs/development-setup.md new file mode 100644 index 0000000..1556f50 --- /dev/null +++ b/docs/development-setup.md @@ -0,0 +1,219 @@ +# Local Development Setup + +This guide walks through setting up **clicked** — the Next.js web app, the Express/Socket.IO +backend, the Python AI agent, and the Soroban smart contracts — from a clean clone to a running +app on your machine. + +## 1. Prerequisites + +Install the exact versions below. Where the repo pins a version (via config files, not just a +README), that pin is called out. + +| Tool | Version | Where it's pinned | +|---|---|---| +| Node.js | **20.x** | `.github/workflows/backend-ci.yml`, `frontend-ci.yml`, `security-ci.yml` all set up Node 20; the root `package.json` targets pnpm 10 which requires modern Node. (One legacy workflow, `pr.yml`, still uses Node 18 — treat 20 as the source of truth for local dev.) | +| pnpm | **10.28.1** | Root `package.json` → `"packageManager": "pnpm@10.28.1+sha512..."`. Run `corepack enable` so the pinned version is used automatically. | +| Rust (stable) + `wasm32-unknown-unknown` target | stable channel, with `wasm32-unknown-unknown`, `clippy`, `rustfmt` | `contracts/rust-toolchain.toml`. `rustup` will auto-install the right toolchain/target the first time you build inside `contracts/`. | +| uv (Python package manager) | any recent uv; manages **Python 3.12** | `apps/ai_agent/pyproject.toml` (`requires-python = ">=3.12"`) and `apps/ai_agent/.python-version` (`3.12`). | +| Docker + Docker Compose | recent Docker Desktop / Engine with Compose v2 | Used to run `infra/docker-compose.yml`. | +| Stellar CLI | optional | Only needed if you're deploying/invoking Soroban contracts against a live network (see `contracts/docs/api-deployment-invocation.md`). Not required to build, test, or run the web/backend apps. | + +## 2. Clone and install JS/TS dependencies + +```bash +git clone https://github.com/codebestia/clicked.git +cd clicked +corepack enable # ensures the pinned pnpm 10.28.1 is used +pnpm install +``` + +This installs dependencies for every workspace declared in `pnpm-workspace.yaml`: +`apps/*` (`ai_agent`, `backend`, `tests`, `web`) and `contracts`. + +## 3. Configure environment variables + +There is one root-level env file, used primarily by the backend: + +```bash +cp .env.example .env +``` + +Fill in at minimum, to get a working local backend: + +- `JWT_SECRET` — any non-empty string for local dev. +- `DATABASE_URL` — e.g. `postgres://postgres:password@localhost:5432/clicked` (matches the + `postgres` service below). +- `REDIS_URL` — e.g. `redis://localhost:6379`. +- `OBJECT_STORE_*` — the defaults already in `.env.example` (`OBJECT_STORE_ENDPOINT=http://localhost:9000`, + bucket `clicked`, access key `clicked`, secret key `clickedsecret`) match the `minio`/`minio-init` + services below out of the box — you generally don't need to change these for local dev. +- `RPC_URL`, `TOKEN_TRANSFER_CONTRACT_ID`, `GROUP_TREASURY_CONTRACT_ID`, `PROPOSALS_CONTRACT_ID` — + only needed if you're exercising the blockchain/contract-linked features; leave blank otherwise. +- `OPENAI_API_KEY` — needed to run the AI agent for real; the AI agent's own test suite stubs it + out (see §6). + +Everything else in `.env.example` (TLS/pinning, rate limits, push/VAPID, XMTP, prekeys) has sane +defaults or is optional for local dev — see the inline comments in `.env.example` and +`docs/security/` for what each does. + +## 4. Start infrastructure with Docker Compose + +```bash +docker compose -f infra/docker-compose.yml up -d +``` + +This brings up four services (all with healthchecks so dependents can gate on readiness): + +- **`postgres`** — `postgres:15-alpine`, exposed on `localhost:5432`, user `postgres` / password + `password` / database `clicked`. Data persists in the `postgres_data` volume. +- **`redis`** — `redis:7-alpine`, exposed on `localhost:6379`. Data persists in `redis_data`. +- **`minio`** — S3-compatible object storage (`minio/minio`), exposed on `9000` (S3 API) and + `9001` (web console). Root credentials are `clicked` / `clickedsecret`. This is what + `OBJECT_STORE_*` in `.env` points at locally; swap those env vars to target real AWS S3 or + Cloudflare R2 in production — the backend uses the same S3 client path either way. +- **`minio-init`** — a one-shot `minio/mc` container that waits for `minio` to be healthy, then + creates the `clicked` bucket (`mc mb --ignore-existing`) and locks it down to no anonymous + access (`mc anonymous set none`). It's idempotent and exits after running — expect it to show as + "Exited (0)" in `docker ps`, not "running". + +You can confirm everything is healthy with `docker compose -f infra/docker-compose.yml ps`. + +## 5. Run database migrations + +The backend uses Drizzle ORM (`apps/backend/drizzle.config.ts`, dialect `postgresql`). With +Postgres up and `DATABASE_URL` set in `.env` (loaded via `dotenv`), run: + +```bash +pnpm --filter backend db:migrate +``` + +(This is also exposed as `make migrate`.) Other Drizzle commands available under +`apps/backend`: `db:generate` (generate migrations from schema changes), `db:push` (push schema +directly, no migration files), `db:studio` (Drizzle Studio UI). + +## 6. Bring up the apps + +### Everything at once + +```bash +pnpm dev +``` + +This runs `turbo run dev`, which fans out to every workspace's own `dev` script (currently `web` +and `backend`; `dev` is marked `persistent`/uncached in `turbo.json`). + +`make dev` does the same thing, but also starts Docker Compose first: + +```bash +make dev # = docker compose -f infra/docker-compose.yml up -d && pnpm dev +``` + +### Individually + +```bash +# Frontend (Next.js) — http://localhost:3000 +pnpm --filter web dev +# or: scripts/start-web.sh — a thin wrapper that cd's into apps/web and runs `pnpm run dev` +bash scripts/start-web.sh + +# Backend (Express + Socket.IO), tsx watch mode +pnpm --filter backend dev + +# AI agent (FastAPI), from apps/ai_agent +cd apps/ai_agent +uv run fastapi dev main.py +``` + +## 7. Smart contracts (Soroban / Rust) + +The `contracts` workspace (`contracts/Cargo.toml`) is a separate Cargo workspace, not part of the +pnpm workspace. `contracts/rust-toolchain.toml` pins `channel = "stable"` with the +`wasm32-unknown-unknown` target and the `clippy`/`rustfmt` components — `rustup` will install +these automatically the first time you build there. + +```bash +cd contracts +cargo build --target wasm32-unknown-unknown --release +cargo test +``` + +To build and deploy the individual contracts (token transfer, group treasury, proposals) against a +configured network, see `contracts/scripts/deploy_*.sh` and `contracts/docs/api-deployment-invocation.md`, +or run all of them via: + +```bash +make deploy-contracts +``` + +## 8. Running tests + +| Command | What it runs | +|---|---| +| `pnpm test` | Whatever `test` scripts Turbo finds across JS/TS workspaces. | +| `pnpm --filter backend test` | Backend Vitest suite (`vitest run`). | +| `pnpm --filter web test` | Web Vitest suite. | +| `make test` | `pnpm --filter backend test` **and** `cd contracts && cargo test`. | +| `cd apps/ai_agent && uv run pytest` | AI agent's pytest suite (config lives in `pyproject.toml`: `testpaths = ["tests"]`, coverage on by default). | + +### What needs Docker, and what doesn't + +**The backend unit test suite does not require Docker, Postgres, Redis, or MinIO to be running.** +`apps/backend/src/__tests__/setup.ts` only sets fake env vars (`DATABASE_URL=postgres://localhost/test`, +etc.) — it never opens a real connection. Individual test files back that up with `vi.mock(...)` +for `../db/index.js`, `../lib/redis.js`, S3 clients, and friends (see e.g. +`apps/backend/src/__tests__/devices.revoke.test.ts`). CI (`.github/workflows/backend-ci.yml`) does +spin up real Postgres/Redis/MinIO service containers and runs migrations before testing, but that's +so `pnpm lint`/`format:check`/build-adjacent steps and any integration-style tests have something to +talk to — the unit tests themselves are written to be mockable and don't require it. + +Similarly, the AI agent's tests (`apps/ai_agent/tests/conftest.py`) auto-patch `OPENAI_API_KEY` +and provide `mocker.patch(...)` fixtures for the OpenAI and Weaviate clients, so `uv run pytest` +runs without a real OpenAI key or a running Weaviate instance. + +**You do need Docker Compose running for:** + +- Actually using the app end-to-end (`pnpm dev` / `make dev`) — the backend will fail to boot + without a reachable Postgres and Redis. +- Running `db:migrate` / `db:push` / `db:studio` against a real database. +- Exercising file upload / object storage flows manually. +- Any ad-hoc integration test that intentionally hits a live service instead of a mock. + +## 9. Troubleshooting + +- **Backend fails to start with a Postgres/Redis connection error.** Confirm + `docker compose -f infra/docker-compose.yml ps` shows `postgres` and `redis` as healthy, and that + `DATABASE_URL`/`REDIS_URL` in `.env` match the compose file's exposed ports/credentials + (`postgres:password@localhost:5432/clicked`, `localhost:6379`). +- **`db:migrate` fails / tables missing.** Make sure Postgres is up and healthy *before* migrating + — `minio-init` and the migration step both depend on their upstream service's healthcheck for a + reason. Re-run `pnpm --filter backend db:migrate` after `docker compose ... up -d` reports the + `postgres` container healthy. +- **File uploads / S3 calls fail locally.** Check that `minio-init` actually completed (it exits + after creating the bucket — `docker compose -f infra/docker-compose.yml ps` should show it + `Exited (0)`, not stuck restarting) and that the `OBJECT_STORE_*` vars in `.env` match the MinIO + root credentials (`clicked` / `clickedsecret`) and endpoint (`http://localhost:9000`). +- **`pnpm install` picks the wrong pnpm version / lockfile mismatch.** Run `corepack enable` so the + `packageManager` pin in `package.json` (`pnpm@10.28.1`) is honored; avoid a globally-installed + pnpm of a different major version. +- **Rust build fails looking for the wasm target.** `rustup` should auto-install it from + `contracts/rust-toolchain.toml` on first `cargo build`/`cargo test` inside `contracts/`. If it + doesn't, run `rustup component add rust-std --target wasm32-unknown-unknown` explicitly (this is + also called out for Windows in `contracts/docs/api-deployment-invocation.md`). +- **AI agent 500s with an OpenAI auth error.** You need a real `OPENAI_API_KEY` in your environment + to use the agent live; the test suite doesn't need one since `conftest.py` stubs it and the + client. + +There is no dedicated `docs/troubleshooting.md` in this repo yet — for backend-specific runbooks +and deeper operational detail, see `docs/runbook.md` and `docs/observability.md`. + +## 10. Further reading + +- [`docs/runbook.md`](./runbook.md) — operational runbook. +- [`docs/observability.md`](./observability.md) — logging/metrics. +- [`docs/signal-integration.md`](./signal-integration.md), [`docs/group-epoch-sync.md`](./group-epoch-sync.md), [`docs/threat-model.md`](./threat-model.md) — protocol/security design docs. +- [`docs/security/`](./security) — TLS/pinning and rate-limit policy referenced from `.env.example`. +- [`apps/backend/docs/api-devices.md`](../apps/backend/docs/api-devices.md) and + [`apps/backend/docs/e2ee-onboarding.md`](../apps/backend/docs/e2ee-onboarding.md) — backend API + docs linked from the root `README.md`. +- [`contracts/docs/api-deployment-invocation.md`](../contracts/docs/api-deployment-invocation.md) — + full contract build/deploy/invoke walkthrough. diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..7375613 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,216 @@ +# Glossary + +This glossary documents the domain vocabulary used across the codebase. The +project mixes terms from three sources: + +- **Messaging / E2EE** — standard cryptographic-messaging protocol terms (Signal + double ratchet, MLS, NaCl sealed boxes). +- **Stellar / Soroban** — standard blockchain-platform terms from the Stellar + network and its Soroban smart-contract layer. +- **Project-specific coinages** — terms invented by this codebase that won't be + found in any external spec. + +Each entry is tagged **[standard]** or **[project-specific]** and, where useful, +points at the module that implements it. + +--- + +## 1. Messaging / E2EE terms **[standard, unless noted]** + +- **Envelope** — A single per-device encrypted payload: one message, sealed + separately for one recipient device. Persisted as one row per + `(message, recipient device)` pair in the `message_envelopes` table + (`apps/backend/src/db/schema.ts`), built client-side in `buildEnvelopes` + (`apps/web/src/lib/crypto.ts`) and consumed in + `apps/web/src/lib/crypto/processEnvelope.ts`. General term from + secure-messaging protocol design (Signal, MLS, etc.), not Stellar-specific. + +- **Prekey** — A pre-published public key (signed prekey / one-time prekey) + that lets a sender start an encrypted session with a device that is + currently offline, per the Signal/X3DH key-agreement pattern. Validated and + stored in `apps/backend/src/lib/keys.ts`; low-prekey alerting lives in + `apps/backend/src/services/prekeyLowSignal.ts`. + +- **Ratchet (Double Ratchet)** — The Signal Double Ratchet algorithm: a + per-session symmetric-key ratchet combined with a DH ratchet, giving + forward secrecy and post-compromise security for one-to-one sessions. + Implemented in `apps/web/src/lib/crypto/doubleRatchet.ts` (`ratchetEncrypt` + / `ratchetDecrypt`), with the persisted session layer in + `apps/web/src/lib/crypto/ratchetSession.ts` / `signalSession.ts`. + +- **Safety number** — A human-verifiable fingerprint computed from both + parties' identity keys, used to detect key/identity changes (aka "safety + number changed" in Signal). Surfaced in + `apps/web/src/app/app/conversations/[id]/page.tsx` + (`loadSafetyNumber`, `hasChangedSafetyNumber`) and the `safety_number_changed` + socket event. + +- **Epoch (MLS epoch)** — The MLS group-state version number: every commit + (member add/remove, key rotation) advances the group to a new epoch, and a + device can only decrypt messages encrypted in epochs it was a member for. + Tracked as `mlsGroups.currentEpoch`, `mlsCommits.epoch`, + `mlsWelcomes.epoch`, and the join/leave interval columns + `mlsGroupMembers.joinedAtEpoch` / `removedAtEpoch` + (`apps/backend/src/db/schema.ts`). See the disambiguation section below — + this is distinct from `groupControlEvents.sequence`. + +- **Sealed box** — A NaCl/libsodium "anonymous" sealed-box construction + (ephemeral ECDH + HKDF + AEAD) used for the project's original, pre-MLS + encryption scheme. Implemented as `sealedBoxEncrypt` in + `apps/web/src/lib/crypto.ts` and recorded as the `sealed_box` value of the + `e2ee_protocol` enum in `apps/backend/src/db/schema.ts` + (`messageEnvelopes.protocol`), alongside `signal` and `mls`. See + `apps/backend/src/services/e2eeProtocol.ts` for how protocol selection + across devices is negotiated. + +- **Fan-out** — The server-side act of taking one outbound message and + delivering/copying it out to every recipient device (one envelope per + device, socket push, offline queueing). Implemented in + `apps/backend/src/services/fanout.ts`, + `apps/backend/src/lib/messageFanout.ts`, and + `apps/backend/src/services/deviceDelivery.ts`. General distributed-messaging + term, not specific to any single protocol. + +--- + +## 2. Stellar / Soroban terms **[standard]** + +- **Ledger** — Stellar's unit of consensus: the blockchain's equivalent of a + block, each with a sequence number. The backend's on-chain event listener + tracks the `ledger` a transaction/event landed in ( + `apps/backend/src/services/stellarListener.ts`, field `ledger: number`) to + detect chain reorgs/gaps and resume from the right point. + +- **XDR** — Stellar/Soroban's binary wire encoding ("External Data + Representation") for transactions, contract values (`ScVal`), and results. + Used throughout `apps/web/src/lib/soroban.ts` (`xdr.ScVal.scvSymbol(...)`, + `signedTxXdr`) when building and signing Soroban invocations client-side. + +- **SAC (Stellar Asset Contract)** — Soroban's built-in contract wrapper that + exposes a classic Stellar asset (an `(issuer, code)` pair, e.g. + `USDC:GA5Z...`) as a SEP-41-compatible token contract with a deterministic + address. The project's `token_transfer` and `group_treasury` contracts are + written against the generic SEP-41 interface so they work with any SAC or + custom SEP-41 token (`contracts/contracts/token_transfer/src/token_interface.rs`, + documented in `contracts/docs/concepts-token-transfer-flow.md`). + +- **Passphrase (network passphrase)** — The string that identifies which + Stellar network (e.g. testnet vs. public) a transaction is signed for; + included in transaction signing to prevent cross-network replay. Read from + `NEXT_PUBLIC_NETWORK_PASSPHRASE` in `apps/web/src/lib/soroban.ts`. + +- **Freighter** — The browser-extension Stellar wallet used for user + authentication and transaction signing in this app ("Sign in with your + Freighter wallet" — `apps/web/src/components/landing/HowItWorks.tsx`). + Wrapped in `apps/web/src/lib/freighter.ts` + (`requestAccess`/`signMessage` from `@stellar/freighter-api`) and consumed + by `WalletContext.tsx` / `AuthContext.tsx`. + +--- + +## 3. Project-specific coinages **[project-specific — not standard terms]** + +- **Device set mismatch** — The error code (`device_set_mismatch`, issue + #133) returned when a client encrypts a message for a stale set of + recipient devices — i.e. it omitted an envelope for a device the server + knows about (a sibling device that came online, or wasn't yet known to the + sender). The server responds with the missing device IDs so the client can + re-encrypt and retry exactly once. Implemented client-side in + `apps/web/src/lib/crypto.ts` (search `device_set_mismatch`) and documented + in `apps/web/docs/concepts-message-pipeline.md`. + +- **Sibling device** — Any other device belonging to the *same* user as the + sender/recipient (as opposed to a different user's device). Because this + is a multi-device E2EE design, a user's own sibling devices are each + independent encryption targets requiring their own session/envelope, exactly + like another person's device. See `apps/web/src/lib/signalClient.ts` (its + header comment coins the term) and `fetchSiblingDeviceIds` referenced from + `apps/web/docs/concepts-message-pipeline.md`. + +- **Resume cursor** — A client-persisted "last seen event ID" used to resume + the realtime socket stream after a reconnect without re-fetching or missing + events, analogous to a Kafka offset/cursor but specific to this project's + socket protocol. Read/written via `getResumeCursor` / `setResumeCursor` in + `apps/web/src/lib/socket.ts` and `apps/web/src/hooks/useSocket.ts`, sent as + `lastEventId` on the `resume` socket event; server-side counterpart in + `apps/backend/src/services/resumeStream.ts`. + +- **Group control event** — A row in the `group_control_events` table + representing one membership-affecting action in a conversation + (`member_added`, `member_removed`, `member_left`, `commit`), each carrying + the group's post-event `epoch` and a gap-free per-conversation `sequence` + number. This is the project's own audit/catch-up log built on top of MLS + group state — the MLS commit/welcome material itself is carried opaquely + in the event's `payload` column. Defined in + `apps/backend/src/db/schema.ts` and produced by + `apps/backend/src/services/groupControl.ts`. + +--- + +## 4. Disambiguation: confusable pairs + +### `ciphertext` (message body) vs. envelope `ciphertext` (per-device payload) + +Both are literal column names but on different tables and mean different +things: + +- **`messages.ciphertext`** (`apps/backend/src/db/schema.ts`) is a single + column on the message row itself. It holds the opaque, E2EE-encrypted + message body for protocols that carry one ciphertext per message (or is + `NULL` when the message instead has per-device rows in + `message_envelopes`, or when it's a `system` message, which carries + `systemPayload` instead and must have `ciphertext IS NULL`, enforced by a + CHECK constraint). +- **`messageEnvelopes.ciphertext`** is a separate table, one row *per + recipient device*, holding that device's independently sealed copy of the + same plaintext (see "Envelope" above). A single logical message can have + zero-to-many envelope rows, each with its own ciphertext, versus at most one + `messages.ciphertext`. + +In short: `messages.ciphertext` is "the (optional) single encrypted blob on +the message," `message_envelopes.ciphertext` is "the per-device encrypted +copy," and the two are mutually complementary depending on which E2EE +protocol produced the message (see `protocol` / `mlsEpoch` columns and +`apps/backend/src/lib/ciphertextInvariant.ts`, which enforces the invariant +between them). + +### `deviceId` vs `senderDeviceId` + +- **`deviceId`** is the generic column name used wherever a table references + "some device" without an implied role — e.g. + `pushSubscriptions.deviceId`, `mlsGroupMembers.deviceId`, + `mlsCommits`/`mlsWelcomes`/`mlsKeyPackages`/`deviceKeyHistory` — all `deviceId`, + all foreign keys into the single canonical `devices` table. +- **`senderDeviceId`** is specifically the column on `messages` + (`apps/backend/src/db/schema.ts`) identifying *which one device, of + potentially several belonging to the sending user*, actually sent/encrypted + this particular message — needed because of multi-device support (a user's + sibling devices each have independent identity keys and sessions, so the + system must know exactly which device's key produced the message). + +So `senderDeviceId` is a role-qualified `deviceId` used only where the +sending device specifically (as opposed to some other role, e.g. recipient) +needs to be recorded; `messageEnvelopes.recipientDeviceId` is the +symmetric recipient-side equivalent. + +### `epoch` (MLS epoch) vs `sequence` (group control log sequence number) + +These live on the same `group_control_events` row but track different axes: + +- **`epoch`** is the *MLS group cryptographic state version* — it only + changes when a commit actually changes the group's secrets (member + add/remove, key rotation). It determines what a device can decrypt (see + "Epoch" above). +- **`sequence`** is a plain, strictly-increasing, gap-free *log position* + within one conversation's `group_control_events` (enforced by the unique + index `group_control_conversation_sequence_idx` on + `(conversationId, sequence)`), used purely for ordered catch-up/replay of + the control log — it has no cryptographic meaning and increments on every + control event, whether or not that event bumps the epoch. + +In other words: `sequence` numbers *events*, `epoch` numbers *group crypto +states* — an event's `sequence` always advances by exactly one per event, +while its `epoch` only advances when that event actually rotates group +secrets (comment in schema.ts: "join and leave can never be assigned the +same sequence number," clarifying that `sequence` is unconditionally unique +per event, unlike `epoch`).