Release: develop -> main

.github/workflows/ci.yaml

@@ -308,6 +308,20 @@ jobs:
       RUSTC_WRAPPER: sccache
       # See `node-tests` env block above for the 50-GiB rationale.
       SCCACHE_CACHE_SIZE: "50G"
+      # Activate the workspace's `coverage_nightly` cfg gate so the
+      # `#[cfg_attr(coverage_nightly, coverage(off))]` annotations
+      # (14× repo-wide, plus the platform-detection helpers in
+      # `node/src/r2_probe.rs`) actually take effect under
+      # `cargo llvm-cov`. cargo-llvm-cov does NOT auto-set this cfg —
+      # without it every `coverage(off)` in the workspace is inert and
+      # llvm-cov counts the excluded fns / lines as uncovered, which
+      # silently broke the 100%-line + 100%-function gate the moment
+      # the first annotation landed in the `node` crate. Set only on
+      # the coverage job: the `lint-and-build` job runs stable 1.81.0
+      # and would reject `feature(coverage_attribute)`, and
+      # `node-tests` doesn't need the cfg (test execution is
+      # orthogonal to the gate).
+      RUSTFLAGS: "--cfg coverage_nightly"
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -359,6 +373,31 @@ jobs:
             --test-threads 1 \
             -E 'not binary(api_remote)'
 
+      # On gate failure, re-format the existing llvm-cov data (no
+      # re-run, no new test execution — `report` reads the on-disk
+      # profraw / profdata produced by the previous step) and emit
+      # the per-file "Uncovered Lines" block plus a json digest of
+      # files below 100% line / function. `--show-missing-lines` on
+      # the gate step sometimes elides this section depending on the
+      # llvm-cov build (observed empirically across this repo's
+      # llvm-cov upgrades), so this step makes the detail
+      # deterministic: whenever the gate fails, the operator sees
+      # which file/line/function is below 100% without having to
+      # reproduce locally.
+      - name: Show missing coverage on gate failure
+        if: failure()
+        run: |
+          echo "--- llvm-cov report: --show-missing-lines (text) ---"
+          cargo llvm-cov report --release --show-missing-lines \
+            --ignore-filename-regex 'main\.rs|lib\.rs|publisher\.rs|runtime\.rs|scanner_runtime\.rs|scanner_ws\.rs|_tests\.rs$|bin/.*\.rs$' || true
+          echo "--- llvm-cov report: per-file json (filter < 100%) ---"
+          cargo llvm-cov report --release --json \
+            --ignore-filename-regex 'main\.rs|lib\.rs|publisher\.rs|runtime\.rs|scanner_runtime\.rs|scanner_ws\.rs|_tests\.rs$|bin/.*\.rs$' \
+            | jq -r '.data[0].files[]
+                | select(.summary.lines.percent < 100 or .summary.functions.percent < 100)
+                | {filename, lines: .summary.lines, functions: .summary.functions}' \
+          || echo "(jq not available or json parse failed)"
+
       - name: sccache stats (post-build)
         if: always()
         run: sccache --show-stats

CONTRIBUTING.md

@@ -35,19 +35,34 @@ Bitcoin / Esplora signals on the node's hot path are subscribed to,
 never polled. The scanner consumes block events from the
 mempool.space-compatible WebSocket stream (`scanner_ws.rs`,
 `ESPLORA_WS_URL`, default `wss://mutinynet.com/api/v1/ws`); the
-publisher waits for `track-tx` events between commit and reveal
-broadcasts instead of sleeping a fixed propagation interval. The
-previous 30-s tip-poll gated `/api/mint` and `/api/send` visibility
-by up to a full block-time + poll-interval (issue #84); event-driven
-ingestion brings that down to the WS round-trip.
+publisher broadcasts the commit and reveal transactions back-to-back
+via REST and never sleeps or polls between them. The previous 30-s
+tip-poll gated `/api/mint` and `/api/send` visibility by up to a full
+block-time + poll-interval (issue #84); event-driven ingestion brings
+that down to the WS round-trip.
+
+Historical note: issue #84 originally replaced a fixed 5 s
+`PROPAGATION_WAIT_SECS` sleep with a `{"action":"track-tx","data":"<commit_txid>"}`
+WS wait + REST safety-net. After the deployment moved to a
+self-hosted `mempool/backend:v3.3.1`, empirical measurement showed
+that backend version emits zero frames for `track-tx`, so the WS
+wait always timed out and the REST fallback always confirmed the
+tx as already on-chain. 16/16 fallbacks in 72 h DEV `request_log`,
+0 not-found, 0 errors. The wait was pure latency tax (~30 s/mint
+and ~30 s/send+commit) for an in-cluster scenario where bitcoind's
+local-mempool accept already orders the two POSTs correctly. The
+publisher now runs `client.broadcast(commit) → client.broadcast(reveal)`
+sequentially; race-freedom follows from the topology (node, electrs,
+bitcoind share the Docker `bitcoin` network), not from a WS
+subscription.
 
 Where it applies:
 
 - `node/src/scanner.rs` — pure inscription parsing, no polling.
 - `node/src/scanner_runtime.rs` — block-walk loop, drains the WS-fed channel.
 - `node/src/scanner_ws.rs` — WS subscriber + reconnect-with-backoff.
 - `node/src/scanner_ws_parse.rs` — pure WS frame parsers.
-- `node/src/publisher.rs` — `track-tx` wait between commit and reveal.
+- `node/src/publisher.rs` — direct sequential commit→reveal broadcast.
 
 Where it does NOT apply: integration tests
 (`node/tests/api_remote.rs`), health-readiness probes, and any
@@ -71,13 +86,16 @@ fails the build with a pointer to issue #84. The token is a plain
 comment marker — not an `#[allow(...)]` attribute, which would have
 been mistakable for a real lint suppression — and is the documented
 per-line opt-out for genuinely justified exceptions (today: the
-WS-reconnect backoff in `scanner_ws`, the inner `track-tx`
-reconnect-with-backoff in `scanner_ws`, and the bounded HTTP-retry
+WS-reconnect backoff in `scanner_ws` and the bounded HTTP-retry
 sleep in `scanner_runtime`). The same line must carry a comment
 explaining WHY this particular sleep is not a chain-tip poll. New
 uses require either changing the design or extending this section
 with the rationale.
 
+The publisher's previous per-broadcast `track-tx` reconnect-with-
+backoff inside `scanner_ws.rs` is no longer in the file — it was
+removed alongside the WS wait itself (see historical note above).
+
 ### Project invariants (non-negotiable)
 
 The five constraints below are decided and apply across every PR on
@@ -267,6 +285,13 @@ changing it, drop the local database (`docker rm -f zkcoins-pg`) and
 re-run `sqlx migrate run` against a fresh instance — there is no
 `down` migration in the MVP, the migration set is forward-only.
 
+R2-probe results land in `r2_probe_runs` (+ `r2_probe_hosts` /
+`r2_probe_warm_calls`) added by migration `0013_r2_probe_results.sql`.
+The `r2_probe_runs_summary` view drives `GET
+/api/admin/r2-probe/history`; the `probe_r2` binary writes via
+`--persist` when `DATABASE_URL` is set. See `node/src/r2_probe.rs`
+for the persistence module and the schema rationale.
+
 ## Setup
 
 After cloning, enable the repo's pre-push hook. The hook runs `cargo
@@ -458,8 +483,10 @@ The node continuously scans the Bitcoin blockchain:
 The publisher (`publisher.rs`) creates Taproot Inscriptions:
 - Commit/reveal pattern (two transactions)
 - Data split into 520-byte chunks (max push size)
-- Broadcasts via Esplora API, then waits for the WS `track-tx` event
-  between commit and reveal instead of sleeping a fixed interval
+- Broadcasts via Esplora REST: commit and reveal POSTs run back to
+  back with no inter-tx wait. Sequencing is provided by bitcoind's
+  local-mempool accept (node, electrs, bitcoind share the Docker
+  `bitcoin` network), not by a WS `track-tx` subscription.
 
 ### Plonky2 State-Transition Circuit
 

MIGRATION_RESEARCH.md

@@ -1314,6 +1314,41 @@ is a build-time signal, not a runtime-readiness signal.
 deploy-dev post-curl-retry fires on every DEV deploy. A regression
 that brings back the silent-panic shape fails one or both gates.
 
+### 7.24 Self-hosted `mempool/backend:v3.3.1` does not implement the `track-tx` WS action — **codified**
+
+**Discovered:** DEV operations (May 2026), via combined evidence:
+DEV `request_log` showed `/api/mint` p50 ≈ 40 s and
+`/api/send` p50 = 11 s + `/api/commit` p50 = 30.7 s — both with the
+30 s shape characteristic of the publisher's `track-tx` safety-net;
+16/16 REST fallbacks in the 72 h sample succeeded with 0 not-found
+and 0 errors; a direct WS probe against
+`mempool-api-mutinynet:8999/api/v1/ws` with
+`{"action":"track-tx","data":"<txid>"}` returned 0 frames in 15 s,
+while `{"action":"want","data":["blocks"]}` answered immediately.
+
+**Root cause:** the self-hosted `mempool/backend:v3.3.1` version
+does not emit any frame in response to `track-tx` (the action is
+recognised but no event arrives). Issue #84's original design
+assumed a public mutinynet endpoint where the action does fire;
+self-hosting flipped that assumption silently.
+
+**Fix:** drop the entire WS subscribe + safety-net + REST fallback
+path from `publisher::broadcast_inscription_txs`. The publisher now
+runs `client.broadcast(commit) → client.broadcast(reveal)` back to
+back; race-freedom comes from the deployment topology (node +
+electrs + bitcoind in the shared Docker `bitcoin` network,
+`bitcoind::sendrawtransaction` returns only after local-mempool
+accept), not from a WS subscription. Expected effect:
+`/api/mint` p50 ~40 s → ~11 s, `/api/send + /api/commit`
+~42 s → ~13 s. See the perf PR for the removed code.
+
+**Generalisation for future migrations:** when porting an
+event-driven path that was designed against a public upstream onto
+a self-hosted reimplementation of the same protocol, smoke-test
+each WS action against the self-hosted endpoint before assuming
+parity. Empty-frame-budget probe is cheap; silent latency tax is
+expensive.
+
 ---
 
 ## 8. Local Artifacts

README.md

@@ -254,7 +254,7 @@ Per-module coverage (CI-gated):
 | `publisher.rs`      | excluded          | Bitcoin commit/reveal broadcasting — needs live signet/regtest node                |
 | `main.rs`           | excluded          | Runtime bootstrap                                                                  |
 | `*_runtime.rs`      | excluded          | Background-loop wrappers; covered indirectly via integration tests against handlers |
-| `scanner_ws.rs`     | excluded          | WS subscriber + reconnect loop; pure helpers (`parse_ws_frame`, `frame_signals_tx_seen`) are unit-tested, the I/O loop is covered indirectly via the publisher's `track-tx` round-trip |
+| `scanner_ws.rs`     | excluded          | WS subscriber + reconnect loop; the pure helper `parse_ws_frame` is unit-tested, the I/O loop is covered by in-process WS-server tests |
 
 `publisher.rs`, `main.rs`, the `*_runtime.rs` wrappers, and `scanner_ws.rs` are excluded by design — they require a live Bitcoin node, a funded publisher key, a bound TCP socket, or an upstream WebSocket peer, none of which fit in a unit test. The exclusion list is encoded in the CI gate's `--ignore-filename-regex`; everything else is held at 100% lines + 100% functions. CI runs the MVP build, the all-features build, `cargo nextest run -p node -p shared --release --all-features --test-threads 1 -E 'not binary(api_remote)'` on the self-hosted M3 Ultra runner pool, and the `Coverage Gate (100% lines + functions)` job.
 

ROADMAP.md

@@ -374,7 +374,7 @@ Each stage carries the 100 % line coverage gate before commit.
   - DEV/PRD parity: PR [#73](https://github.com/zk-coins/node/pull/73) dropped the DEV-only Cargo features (`address-list`, `faucet`, `usernames`, `lnurl`) and removed the `DEV_SKIP_BROADCAST_FAILURE` env-gate so the two environments run the identical MVP-only binary. A follow-up refactor removed the `faucet` Cargo feature outright — mint is permanent MVP and ships unconditionally in every build — and a further refactor removed the `usernames` Cargo feature so usernames are permanent MVP and ship unconditionally too (PR [#76](https://github.com/zk-coins/node/pull/76)).
 **Remaining:**
   1. e2e roundtrip on signet from `dev.zkcoins.app`: create account → mint → send → recipient receives. Success criterion: one happy-path + one failure-path per route. Tracked via a follow-up GitHub issue.
-  2. Real performance measurement on the M3 Ultra. R2 budget: warm proof ≤ 5 s, ideally ≤ 1 s; cold-start ≤ 30 s including circuit-data load; peak mem < 64 GB during proving. Plonky2 currently runs CPU-only on Apple Silicon (no Metal backend); that's the operative baseline.
+  2. Real performance measurement on the M3 Ultra. R2 budget: warm proof ≤ 5 s, ideally ≤ 1 s; cold-start ≤ 30 s including circuit-data load; peak mem < 64 GB during proving. Plonky2 currently runs CPU-only on Apple Silicon (no Metal backend); that's the operative baseline. Tool: the `probe_r2` binary (`node/src/bin/probe_r2.rs`) drives the measurement; `--persist` writes every run into `r2_probe_runs` (migration 0013) and the trend is readable via the `r2_probe_runs_summary` view and `GET /api/admin/r2-probe/history` on the live node.
   3. If budget is missed: redesign per R2 (reduce `MAX_IN_COINS`, drop in-coin recursion, or switch to folding). **NOT** add external hardware or move to a cloud prover — the closed-environment + single-host constraint is non-negotiable.
 **Test plan:** the authoritative coverage gate runs in CI on the self-hosted M3 Ultra runner pool (`.github/workflows/ci.yaml`, jobs `Node + Shared Tests` and `Coverage Gate`, gated behind the `ci:full` label per PR [#48](https://github.com/zk-coins/node/pull/48)); the pre-push hook only enforces fmt + clippy + `cargo check`. Step 9 verifies integration, not unit coverage. e2e success criterion: every endpoint round-trips under realistic conditions (one happy-path traversal per route plus at least one failure path per route).
 **Risk:** Medium. First real exposure of the cyclic-recursive prover to production hardware under realistic load. If the budget holds, MVP is done.