fix: production hardening and runtime documentation

.agent/context/20260910T-part1-3-production-hardening.md

@@ -0,0 +1,93 @@
+# Production hardening parts 1-3
+
+## Goal
+
+Implement the three requested audit follow-ups on a short-lived branch, with
+one commit and one fresh Gate A review prompt after each part:
+
+1. Complete worker deployment configuration and add production API rate limiting.
+2. Harden the Privy fallback boundary and PostgreSQL integration-test cleanup.
+3. Reconcile stale operational and implementation documentation.
+
+## Branch and base
+
+- Branch: `fix/production-hardening-ops`
+- Base: current clean `develop` at the start of this session.
+- Scope: testnet-safe operational hardening; no mainnet activation, secrets, or
+  external payment effects.
+
+## Acceptance criteria
+
+- Part 1: API runtime defaults to a durable shared rate limiter; limits are
+  configurable and tested; worker deployment instructions include every
+  required runtime variable and preserve secret-store boundaries.
+- Part 2: Privy denied-relay fallback behavior is covered by regression tests;
+  ambiguous external effects remain fail-closed; PostgreSQL integration
+  cleanup tolerates setup failure without masking the original error.
+- Part 3: stale statuses, variable names, and architecture descriptions are
+  corrected without changing product scope or sponsor qualification claims.
+
+## Applicable safety cases
+
+The changes touch API access, worker runtime, Privy settlement boundaries, and
+PostgreSQL test infrastructure. The relevant matrix cases are Privy denial,
+ambiguous submission/no blind retry, service restart, and durable state and
+external-effect assertions. No live settlement is authorized for this work.
+
+## Initial validation
+
+The repository was clean on `develop`. Prior audit checks passed lint,
+typecheck, unit tests, build, format, generated artifacts, fixtures, browser
+tests, and markdown lint. PostgreSQL integration was not runnable locally
+because no container runtime was available; it must remain explicitly reported
+as unverified until CI or a container runtime executes it.
+
+## Part 1 evidence
+
+- `pnpm.cmd --filter @oneshot/api test`: PASS (50 tests).
+- `pnpm.cmd --filter @oneshot/storage-postgres test`: PASS (8 tests).
+- `pnpm.cmd typecheck`: PASS.
+- `pnpm.cmd lint`: PASS.
+- `pnpm.cmd format:check`: PASS.
+- `pnpm.cmd test`: PASS (65 files, 974 tests).
+- PostgreSQL integration remains unrun locally because no container runtime is
+  available; the new limiter SQL is covered by focused unit behavior and the
+  migration is included in the integration migration sequence.
+
+Part 1 implements a PostgreSQL-backed fixed-window limiter that fails closed
+when the limiter store is unavailable. It keys admission by request IP and
+route; correlation IDs remain diagnostic only. Worker deployment documentation
+now lists the complete non-secret runtime contract and sends only the three
+secret values to Secret Manager.
+
+## Part 2 evidence
+
+- `pnpm.cmd --filter @oneshot/privy-adapter test`: PASS (9 files, 130 tests).
+- `pnpm.cmd --filter @oneshot/storage-postgres test`: PASS (2 files, 8 tests).
+- `pnpm.cmd --filter @oneshot/api test`: PASS (6 files, 50 tests).
+- `pnpm.cmd --filter @oneshot/worker test`: PASS (5 files, 39 tests).
+- `pnpm.cmd typecheck`: PASS.
+- `pnpm.cmd lint`: PASS.
+- `pnpm.cmd format:check`: PASS.
+- `pnpm.cmd test`: PASS (66 files, 977 tests).
+- The Privy regression suite now covers relay-boundary fallback, generic 401
+  rejection without fallback, and policy denial without raw broadcast.
+- All PostgreSQL Testcontainers suites guard cleanup when setup fails. Actual
+  container-backed integration remains unrun locally because no runtime is
+  available.
+
+## Part 3 evidence
+
+- `pnpm.cmd format:check`: PASS.
+- `npx.cmd --yes markdownlint-cli2@0.18.1 "**/*.md" "#node_modules"`: PASS
+  (142 Markdown files, 0 errors).
+- A targeted stale-term scan over the reconciled operational and implementation
+  docs found no remaining references to the retired `ONESHOT_PROFILE`,
+  `ONESHOT_NETWORK`, or `ONESHOT_CONTRACT_VERSION` variables, old P5 test
+  count, pre-evidence Privy/Arc gap wording, or Graphile Worker as the
+  executable production scheduler.
+- The reconciled docs now match the executable production scheduler
+  (`RestartRunner`), current API rate-limit settings, current P5 implementation
+  status, current Privy/Arc qualification evidence, and the still-unverified
+  Graph claim. Historical context records are not treated as current runtime
+  documentation.

.env.example

@@ -7,6 +7,9 @@ PORT=3000
 # Worker deployments require at least 2 (one outbox claim + one ledger action).
 DB_POOL_MAX=10
 ONESHOT_SUBMISSIONS_DISABLED=false
+# Shared PostgreSQL-backed API admission control.
+ONESHOT_API_RATE_LIMIT_MAX_REQUESTS=60
+ONESHOT_API_RATE_LIMIT_WINDOW_MS=60000
 
 # Production worker effect boundary. Public identifiers are placeholders;
 # secrets must be injected by the deployment secret store, never committed.

apps/api/src/app.ts

@@ -125,7 +125,11 @@ export function buildApi(dependencies: ApiDependencies) {
     }
     if (
       request.method === 'POST' &&
-      !(await rateLimiter.allow({ correlationId, route: request.url }))
+      !(await rateLimiter.allow({
+        correlationId,
+        key: request.ip || 'unknown-client',
+        route: request.url.split('?')[0] ?? request.url,
+      }))
     ) {
       sendError(reply, 429, 'RATE_LIMITED', 'Request rate limit exceeded', correlationId);
       return reply;

apps/api/src/config.ts

@@ -14,6 +14,10 @@ export interface ApiRuntimeConfig {
   readonly serviceBearerToken: string;
   readonly database: PoolConfig;
   readonly submissionsDisabled: boolean;
+  readonly rateLimit: {
+    readonly maxRequests: number;
+    readonly windowMs: number;
+  };
   readonly privyAuth?: PrivyAuthRuntimeConfig;
 }
 
@@ -133,6 +137,10 @@ export function loadApiRuntimeConfig(
     serviceBearerToken: required(environment, 'SERVICE_BEARER_TOKEN', 16),
     database: databaseConfig(environment),
     submissionsDisabled: environment.ONESHOT_SUBMISSIONS_DISABLED === 'true',
+    rateLimit: {
+      maxRequests: integer(environment, 'ONESHOT_API_RATE_LIMIT_MAX_REQUESTS', 60, 1, 10_000),
+      windowMs: integer(environment, 'ONESHOT_API_RATE_LIMIT_WINDOW_MS', 60_000, 1_000, 3_600_000),
+    },
     ...(privyAuth ? { privyAuth } : {}),
   };
 }

apps/api/src/rate-limit.ts

@@ -1,12 +1,79 @@
+import type { Pool } from 'pg';
+
 export interface RateLimitInput {
   readonly correlationId: string;
+  readonly key: string;
   readonly route: string;
 }
 
 export interface RateLimiter {
   allow(input: RateLimitInput): Promise<boolean>;
 }
 
+export interface PostgresRateLimiterOptions {
+  readonly maxRequests: number;
+  readonly windowMs: number;
+  readonly now?: () => number;
+  readonly cleanupIntervalMs?: number;
+}
+
+/** Shared fixed-window admission control for horizontally scaled API nodes. */
+export class PostgresRateLimiter implements RateLimiter {
+  readonly #pool: Pool;
+  readonly #maxRequests: number;
+  readonly #windowMs: number;
+  readonly #now: () => number;
+  readonly #cleanupIntervalMs: number;
+  #nextCleanupAt = 0;
+
+  constructor(pool: Pool, options: PostgresRateLimiterOptions) {
+    if (!Number.isSafeInteger(options.maxRequests) || options.maxRequests < 1) {
+      throw new Error('Rate limiter maxRequests must be a positive integer');
+    }
+    if (!Number.isSafeInteger(options.windowMs) || options.windowMs < 1_000) {
+      throw new Error('Rate limiter windowMs must be at least 1000 milliseconds');
+    }
+    this.#pool = pool;
+    this.#maxRequests = options.maxRequests;
+    this.#windowMs = options.windowMs;
+    this.#now = options.now ?? Date.now;
+    this.#cleanupIntervalMs = options.cleanupIntervalMs ?? Math.max(options.windowMs, 60_000);
+  }
+
+  async allow(input: RateLimitInput): Promise<boolean> {
+    const now = this.#now();
+    const bucketStart = new Date(Math.floor(now / this.#windowMs) * this.#windowMs);
+    try {
+      const result = await this.#pool.query<{ request_count: number }>(
+        `INSERT INTO api_rate_limit_buckets
+           (bucket_start, client_key, route, request_count)
+         VALUES ($1, $2, $3, 1)
+         ON CONFLICT (bucket_start, client_key, route)
+         DO UPDATE SET request_count = LEAST(
+           api_rate_limit_buckets.request_count + 1,
+           $4 + 1
+         )
+         RETURNING request_count`,
+        [bucketStart, input.key, input.route, this.#maxRequests],
+      );
+      const requestCount = Number(result.rows[0]?.request_count ?? this.#maxRequests + 1);
+
+      if (now >= this.#nextCleanupAt) {
+        this.#nextCleanupAt = now + this.#cleanupIntervalMs;
+        void this.#pool
+          .query('DELETE FROM api_rate_limit_buckets WHERE bucket_start < $1', [
+            new Date(now - this.#windowMs * 2),
+          ])
+          .catch(() => undefined);
+      }
+      return requestCount <= this.#maxRequests;
+    } catch {
+      // Admission control fails closed if its durable store is unavailable.
+      return false;
+    }
+  }
+}
+
 export const allowAllRateLimiter: RateLimiter = {
   async allow() {
     return true;

apps/api/src/runtime.ts

@@ -9,6 +9,7 @@ import {
 } from './auth.js';
 import { loadApiRuntimeConfig, type ApiRuntimeConfig } from './config.js';
 import { createPrivyAccessTokenAuthenticator, isJwtCredential } from './privy-auth.js';
+import { PostgresRateLimiter } from './rate-limit.js';
 
 export interface ApiRuntime {
   readonly address: string;
@@ -46,6 +47,7 @@ export async function startApiRuntime(config: ApiRuntimeConfig): Promise<ApiRunt
     const app = buildApi({
       ledger,
       authenticator: buildApiAuthenticator(config),
+      rateLimiter: new PostgresRateLimiter(pool, config.rateLimit),
       config: {
         submissionsDisabled: config.submissionsDisabled,
         chainId: '5042002',

apps/api/test/api.integration.test.ts

@@ -15,8 +15,8 @@ const request = {
 };
 
 describePostgres('durable HTTP API', () => {
-  let container: StartedPostgreSqlContainer;
-  let pool: Pool;
+  let container!: StartedPostgreSqlContainer;
+  let pool!: Pool;
   let attempts = 0;
 
   beforeAll(async () => {
@@ -26,8 +26,8 @@ describePostgres('durable HTTP API', () => {
   });
 
   afterAll(async () => {
-    await pool.end();
-    await container.stop();
+    if (typeof pool !== 'undefined') await pool.end();
+    if (typeof container !== 'undefined') await container.stop();
   });
 
   const ledger = () =>
@@ -82,6 +82,7 @@ describePostgres('durable HTTP API', () => {
       serviceBearerToken: 'integration-token',
       database: { connectionString: container.getConnectionUri() },
       submissionsDisabled: false,
+      rateLimit: { maxRequests: 60, windowMs: 60_000 },
     });
     try {
       const response = await fetch(`${runtime.address}/health/ready`);

apps/api/test/config.test.ts

@@ -22,6 +22,7 @@ describe('API runtime configuration', () => {
     });
 
     expect(config.port).toBe(8080);
+    expect(config.rateLimit).toEqual({ maxRequests: 60, windowMs: 60_000 });
     expect(config.database).toEqual({
       connectionString: 'postgresql://oneshot:secret@localhost:5432/oneshot',
       max: 10,
@@ -146,6 +147,18 @@ describe('API runtime configuration', () => {
     ).toThrow('at least 16 characters');
   });
 
+  it('loads and validates the shared API rate-limit settings', () => {
+    const config = loadApiRuntimeConfig({
+      ...base,
+      ONESHOT_API_RATE_LIMIT_MAX_REQUESTS: '12',
+      ONESHOT_API_RATE_LIMIT_WINDOW_MS: '5000',
+    });
+    expect(config.rateLimit).toEqual({ maxRequests: 12, windowMs: 5000 });
+    expect(() =>
+      loadApiRuntimeConfig({ ...base, ONESHOT_API_RATE_LIMIT_MAX_REQUESTS: '0' }),
+    ).toThrow('ONESHOT_API_RATE_LIMIT_MAX_REQUESTS');
+  });
+
   it('rejects an allowlist entry that is not a Privy DID', () => {
     expect(() =>
       loadApiRuntimeConfig({

apps/api/test/rate-limit.test.ts

@@ -0,0 +1,45 @@
+import { describe, expect, it, vi } from 'vitest';
+import type { Pool } from 'pg';
+import { PostgresRateLimiter } from '../src/rate-limit.js';
+
+describe('PostgresRateLimiter', () => {
+  it('shares a fixed window and rejects requests after the configured limit', async () => {
+    let requestCount = 0;
+    const pool = {
+      query: vi.fn(async (sql: string) => {
+        if (sql.startsWith('INSERT')) {
+          requestCount = Math.min(requestCount + 1, 3);
+          return { rows: [{ request_count: requestCount }] };
+        }
+        return { rows: [] };
+      }),
+    } as unknown as Pool;
+    const limiter = new PostgresRateLimiter(pool, {
+      maxRequests: 2,
+      windowMs: 60_000,
+      now: () => 1_735_689_600_000,
+    });
+    const input = { correlationId: 'cid', key: 'client', route: '/v1/intents' };
+
+    await expect(limiter.allow(input)).resolves.toBe(true);
+    await expect(limiter.allow(input)).resolves.toBe(true);
+    await expect(limiter.allow(input)).resolves.toBe(false);
+    expect(pool.query).toHaveBeenCalledWith(
+      expect.stringContaining('ON CONFLICT (bucket_start, client_key, route)'),
+      [new Date(1_735_689_600_000), 'client', '/v1/intents', 2],
+    );
+  });
+
+  it('fails closed when PostgreSQL is unavailable', async () => {
+    const pool = {
+      query: vi.fn(async () => {
+        throw new Error('database unavailable');
+      }),
+    } as unknown as Pool;
+    const limiter = new PostgresRateLimiter(pool, { maxRequests: 60, windowMs: 60_000 });
+
+    await expect(
+      limiter.allow({ correlationId: 'cid', key: 'client', route: '/v1/intents' }),
+    ).resolves.toBe(false);
+  });
+});

apps/api/test/runtime-auth.test.ts

@@ -24,6 +24,7 @@ function config(privyEnabled: boolean): ApiRuntimeConfig {
     serviceBearerToken: 'service-token-1234',
     database: { connectionString: 'postgresql://localhost/oneshot' },
     submissionsDisabled: false,
+    rateLimit: { maxRequests: 60, windowMs: 60_000 },
     ...(privyEnabled
       ? { privyAuth: { appId: APP_ID, verificationKey, allowedSubjects: [OPERATOR] } }
       : {}),

apps/worker/README.md

@@ -17,10 +17,13 @@ Atomic at-most-once execution worker for OneShot Business Intents.
 
 ## Architecture and Dispatch
 
-The worker supports dual execution modes:
+The worker exports two scheduling interfaces:
 
-1. **Graphile Worker TaskList (`createTaskList`)**: Exposes standard typed job handlers conforming to Graphile Worker `TaskList` specification for production multi-worker runner pools.
-2. **Transactional Outbox Poller (`drainOutboxJobs`)**: Embedded transactional worker engine using PostgreSQL `FOR UPDATE SKIP LOCKED` for atomic job delivery without external message broker dependencies.
+1. **Graphile Worker TaskList (`createTaskList`)**: Exposes standard typed job handlers conforming to Graphile Worker `TaskList` for host integrations and tests.
+2. **Transactional Outbox Poller (`drainOutboxJobs`)**: The scheduler used by the executable `RestartRunner`, with PostgreSQL `FOR UPDATE SKIP LOCKED` for atomic job delivery without an external message broker.
+
+The production process does not start a Graphile Worker runner; `createTaskList`
+is an integration seam, while `RestartRunner` is the current runtime scheduler.
 
 ## Production process
 

apps/worker/test/restart-recovery.integration.test.ts

@@ -21,8 +21,8 @@ const sampleRequest = {
 };
 
 describePostgres('Startup recovery and restart safety (A04.1, A04.2)', () => {
-  let container: StartedPostgreSqlContainer;
-  let pool: Pool;
+  let container!: StartedPostgreSqlContainer;
+  let pool!: Pool;
   let attemptCounter = 0;
 
   beforeAll(async () => {
@@ -39,8 +39,8 @@ describePostgres('Startup recovery and restart safety (A04.1, A04.2)', () => {
   });
 
   afterAll(async () => {
-    await pool.end();
-    await container.stop();
+    if (typeof pool !== 'undefined') await pool.end();
+    if (typeof container !== 'undefined') await container.stop();
   });
 
   const newLedger = () =>