feat(ws/config): implement WS filtering, replay, metrics wiring, and dry-run flag (#257 #258 #259 #260)

docs/runbooks/on-call.md

@@ -39,8 +39,8 @@ read endpoints but does **not** take down the intent relay or WebSocket feed.
 | `GET /health` | `200 { status: "ok" }` |
 | `GET /api/v1/chain/health` | `200` with Soroban `status: "healthy"` |
 | Sweeper log (every 30 s) | Debug line: `sweep complete: expired=N duration=Xms` |
-| `MetricsRegistry.sweeper.sweepDurationMs` p99 | < 50 ms under normal load |
-| `MetricsRegistry.sweeper.expiredTotal` | Monotonically increasing; spikes expected near intent `deadline` clusters |
+| `vortex_sweeper_sweep_duration_ms` p99 | < 50 ms under normal load |
+| `vortex_sweeper_expired_total` | Monotonically increasing; spikes expected near intent `deadline` clusters |
 | WS subscriber count | Stable or slowly growing; sudden drops indicate client-side churn |
 | Node.js heap | Steady-state < 200 MB; no sustained upward trend between GC cycles |
 
@@ -142,8 +142,11 @@ A sweep that has been delayed or killed will simply be absent.
 2. Compares each intent's `deadline` (Unix timestamp) against `Date.now()`.
 3. Calls `IntentsService.update()` and `IntentsGateway.broadcast()` for each
    expired intent.
-4. Records `sweepDurationMs` and increments `expiredTotal` in
-   `MetricsRegistry.sweeper`.
+4. Records `vortex_sweeper_sweep_duration_ms` and increments
+   `vortex_sweeper_expired_total` via `MetricsService.recordSweep()` (Prometheus,
+   exposed on `GET /metrics`). The retired `MetricsRegistry` from
+   `src/common/metrics.ts` has been removed (issue #259) — use the
+   Prometheus metric names above for alerting and dashboards.
 
 Because the store is in-memory and the loop is synchronous, the sweep should
 complete in **single-digit milliseconds** for < 10 000 open intents.
@@ -182,8 +185,9 @@ or restart the service (the sweeper fires on the next 30-second tick after
 3. **Check metrics** (if a metrics endpoint is wired up):
    ```bash
    curl -s http://localhost:4000/metrics | grep sweeper
-   # sweeper_sweep_duration_ms_count
-   # sweeper_expired_total
+   # vortex_sweeper_sweep_duration_ms_count
+   # vortex_sweeper_sweep_duration_ms_sum
+   # vortex_sweeper_expired_total
    ```
 
 4. **Inspect process health**:

docs/runbooks/onchain-cutover.md

@@ -19,7 +19,7 @@ should be reviewed/updated as each lands:
 | On-chain intent registration (issue #22) | Replaces in-memory `create()` with a real Soroban tx | Open |
 | Solver-registry wiring (issue #23) | `accept()` calls the solver-registry contract | Open |
 | On-chain fill settlement (issue #24) | `fill()` submits + confirms a settlement tx | Open |
-| Dry-run mode (issue #35) | Config flag to simulate on-chain writes without submitting | Open |
+| Dry-run mode (issue #35) | Config flag to simulate on-chain writes without submitting | **Done** (issue #260) |
 | Intent audit trail (issue #62) | Append-only log of every state transition, independent of the state store | Open |
 
 Treat the checklist below as the gate for actually running this procedure:
@@ -86,32 +86,45 @@ immediately before flipping traffic:
 
 ## Dry-run flag and the cutover
 
-The dry-run flag (issue #35) is the primary safety mechanism this runbook
-leans on. It's a config-level switch (default **on** outside production,
-per that issue's requirements) that makes every on-chain-write code path
-build and simulate a Soroban transaction, log what *would* be submitted,
-and return without broadcasting it.
+The dry-run flag (`ONCHAIN_DRY_RUN`, issue #260 / #35) is the primary safety
+mechanism this runbook leans on. It's a config-level switch (default **true**
+outside production, per that issue's requirements) that makes every on-chain-write
+code path (`StellarTxService.invokeContract`, `SolverRegistryService.slashSolver`)
+build and simulate a Soroban transaction, log what *would* be submitted, and
+return without broadcasting it.
+
+**Runtime-toggleable limitation:** The flag is loaded from environment config at
+process start. Changing it requires a process restart — there is no hot-reload
+HTTP endpoint for this iteration. This is an intentional simplification: the
+staged rollout procedure below is designed around restart windows (not hot flips),
+and the cost of a restart in staging is negligible compared to the risk of a
+silent live-mode activation. A live-toggle mechanism is a separate future concern.
+
+**Production requirement:** `ONCHAIN_DRY_RUN` must be explicitly set in any
+`NODE_ENV=production` environment — the process refuses to start without it
+(validated by `src/config/env.validation.ts`). This prevents a misconfigured
+production deploy from silently defaulting to either mode.
 
 How it factors into cutover staging:
 
 1. **Stage 1 — dry-run in target environment.** Deploy the on-chain code
-   paths with the dry-run flag forced on, traffic unchanged (reads/writes
+   paths with `ONCHAIN_DRY_RUN=true` forced on, traffic unchanged (reads/writes
    still served from the in-memory store). This validates that transaction
    construction, contract ID wiring, and the signing key all work, with
    zero funds-moving risk. This is pre-check #2 above.
-2. **Stage 2 — shadow writes.** Flip dry-run off for a canary slice (or a
+2. **Stage 2 — shadow writes.** Flip `ONCHAIN_DRY_RUN=false` for a canary slice (or a
    single non-critical path, e.g. solver-registry reads before slashing
    writes) while the in-memory store remains authoritative for reads. Watch
    for transaction failures, unexpected fees, or confirmation-latency
    surprises.
 3. **Stage 3 — cutover.** Flip the in-memory store from authoritative to
    cache (or remove it, per how #22/#24 implement this) for the full
-   read/write path. Dry-run stays off. This is the point of no return for
-   this procedure — from here, rollback means the explicit procedure below,
-   not just re-flipping a flag.
+   read/write path. `ONCHAIN_DRY_RUN=false` stays set. This is the point of
+   no return for this procedure — from here, rollback means the explicit
+   procedure below, not just re-flipping a flag.
 
-Keep the dry-run flag itself deployed (not ripped out) after cutover — it's
-the fastest lever if a related on-chain code path needs to be redeployed or
+Keep `ONCHAIN_DRY_RUN` deployed (not ripped out) after cutover — it's the
+fastest lever if a related on-chain code path needs to be redeployed or
 patched later without another full staged rollout.
 
 ## Rollback plan

src/config/configuration.ts

@@ -49,6 +49,19 @@ export interface AppConfig {
     feePercentile: FeePercentile;
   };
   onchainIntentsEnabled: boolean;
+  /**
+   * Dry-run flag for on-chain write paths (issue #260).
+   *
+   * When true every write path (invokeContract, slashSolver) simulates and
+   * logs but never broadcasts a transaction.  Defaults to true outside
+   * production; must be explicitly set in production (validated by
+   * envValidationSchema — see src/config/env.validation.ts).
+   *
+   * Note: this flag takes effect on the next process restart; there is no
+   * hot-reload mechanism for this iteration.  See
+   * docs/runbooks/onchain-cutover.md for the staged rollout procedure.
+   */
+  onchainDryRun: boolean;
   corsOrigin: string;
   /** Maximum concurrent WebSocket connections (0 = unlimited). */
   wsMaxConnections: number;
@@ -69,6 +82,11 @@ export default (): AppConfig => ({
     feePercentile: (process.env.SOROBAN_FEE_PERCENTILE ?? "p50") as FeePercentile,
   },
   onchainIntentsEnabled: (process.env.ONCHAIN_INTENTS_ENABLED ?? "false") === "true",
+  // Default to dry-run (true) outside production; in production the value must
+  // be explicitly set (validated by envValidationSchema).
+  onchainDryRun: process.env.ONCHAIN_DRY_RUN !== undefined
+    ? process.env.ONCHAIN_DRY_RUN === "true"
+    : process.env.NODE_ENV !== "production",
   corsOrigin: process.env.CORS_ORIGIN ?? "*",
   wsMaxConnections: parseInt(process.env.WS_MAX_CONNECTIONS ?? "1000", 10),
 });

src/config/env.validation.spec.ts

@@ -50,8 +50,67 @@ describe("envValidationSchema — SOROBAN_SIGNING_KEY", () => {
     const { error, value } = envValidationSchema.validate({
       NODE_ENV: "production",
       SOROBAN_SIGNING_KEY: VALID_KEY,
+      // ONCHAIN_DRY_RUN is required in production (issue #260) — include it here
+      // so this test stays focused on SOROBAN_SIGNING_KEY validation only.
+      ONCHAIN_DRY_RUN: true,
     });
     expect(error).toBeUndefined();
     expect(value.SOROBAN_SIGNING_KEY).toBe(VALID_KEY);
   });
 });
+
+describe("envValidationSchema — ONCHAIN_DRY_RUN (#260)", () => {
+  it("defaults to true outside production when unset", () => {
+    const { error, value } = envValidationSchema.validate(BASE_ENV);
+    expect(error).toBeUndefined();
+    expect(value.ONCHAIN_DRY_RUN).toBe(true);
+  });
+
+  it("accepts true outside production", () => {
+    const { error, value } = envValidationSchema.validate({
+      ...BASE_ENV,
+      ONCHAIN_DRY_RUN: true,
+    });
+    expect(error).toBeUndefined();
+    expect(value.ONCHAIN_DRY_RUN).toBe(true);
+  });
+
+  it("accepts false outside production (explicit opt-out)", () => {
+    const { error, value } = envValidationSchema.validate({
+      ...BASE_ENV,
+      ONCHAIN_DRY_RUN: false,
+    });
+    expect(error).toBeUndefined();
+    expect(value.ONCHAIN_DRY_RUN).toBe(false);
+  });
+
+  it("is required in production — missing value fails validation", () => {
+    const { error } = envValidationSchema.validate({
+      NODE_ENV: "production",
+      SOROBAN_SIGNING_KEY: VALID_KEY,
+      // ONCHAIN_DRY_RUN deliberately omitted
+    });
+    expect(error).toBeDefined();
+    expect(error?.message).toContain("ONCHAIN_DRY_RUN");
+  });
+
+  it("accepts true in production (keep simulate-only after cutover)", () => {
+    const { error, value } = envValidationSchema.validate({
+      NODE_ENV: "production",
+      SOROBAN_SIGNING_KEY: VALID_KEY,
+      ONCHAIN_DRY_RUN: true,
+    });
+    expect(error).toBeUndefined();
+    expect(value.ONCHAIN_DRY_RUN).toBe(true);
+  });
+
+  it("accepts false in production (live on-chain writes enabled)", () => {
+    const { error, value } = envValidationSchema.validate({
+      NODE_ENV: "production",
+      SOROBAN_SIGNING_KEY: VALID_KEY,
+      ONCHAIN_DRY_RUN: false,
+    });
+    expect(error).toBeUndefined();
+    expect(value.ONCHAIN_DRY_RUN).toBe(false);
+  });
+});

src/config/env.validation.ts

@@ -60,4 +60,36 @@ export const envValidationSchema = Joi.object({
       // as a documentation hint and config validation guard only.
       "debug",
     ),
+
+  // ── On-chain write safety flag (issue #35 / issue #260) ──────────────────
+  // When true, every on-chain-write code path (invokeContract, slashSolver)
+  // builds and simulates the transaction, logs what it *would* submit, and
+  // returns without broadcasting — safe by construction.
+  //
+  // Default behaviour:
+  //   - Outside production: defaults to true (simulate-only, fail closed
+  //     toward safety — no real funds moved without an explicit opt-out).
+  //   - In production: *required* to be explicitly set.  Omitting it in a
+  //     production deploy fails validation so the operator must consciously
+  //     decide between dry-run and live mode before traffic reaches
+  //     on-chain write paths.  This matches the fail-closed pattern used
+  //     for SOROBAN_SIGNING_KEY.
+  //
+  // Limitations: the flag is config-driven and takes effect on the next
+  // process start; there is no HTTP endpoint to flip it at runtime without
+  // a restart.  This limitation is documented in onchain-cutover.md and is
+  // intentional for this iteration — a hot-reload mechanism is a separate
+  // concern.  Set ONCHAIN_DRY_RUN=false only after completing the dry-run
+  // soak described in docs/runbooks/onchain-cutover.md.
+  ONCHAIN_DRY_RUN: Joi.boolean()
+    .when("NODE_ENV", {
+      is: "production",
+      then: Joi.required().messages({
+        "any.required":
+          "ONCHAIN_DRY_RUN must be explicitly set in production. " +
+          "Set to true to remain in simulate-only mode, or false to enable live on-chain writes. " +
+          "See docs/runbooks/onchain-cutover.md for the staged rollout procedure.",
+      }),
+      otherwise: Joi.boolean().default(true),
+    }),
 });