A heartbeat that cannot be green without evidence

[Apolloccrypt]

relay.js is 6488 lines and 68 routes and, until this branch, no unit test loaded
it. The coverage table did not list the file: it was only ever started as a black
box by three suites that then killed it with SIGKILL, which is also why it could
never contribute a coverage profile. Five new suites boot the real relay.js and
drive it over HTTP.

Nothing here touches the network or production. The relay under test binds
127.0.0.1, gets no Mollie key, and every path that would default to /data is
redirected into a scratch dir.

What is covered

suite	tests	what it pins
route-auth-gate	15	auth before validation, the public envelope routes, admin and internal tokens
route-envelope-lifecycle	13	create, view, sign with real ML-DSA-65, receipt verified offline
route-transfer-burn	14	upload, download, burn-on-read, the link flow, per-tier ceilings
route-billing-entitlements	11 (1 todo)	entitlements across a restart, paid_until, tier to quota
route-rate-limits	4	the 50/hour create budget per key and fleet-wide, the per-IP limiters
total	57	56 pass, 1 todo (see finding 1), 6.3 s

Highlights, rather than a list of asserts:

• Auth before validation. A missing or wrong key returns 401 for a body that
  is absent, malformed, empty, or wrong-shaped. It can never turn into a 400,
  which is the shape the ParaID bypass had. A de-activated key answers byte for
  byte like an unknown one, so the 401 is not a key-existence oracle. An
  unconfigured ADMIN_TOKEN fails closed.
• The signature is really verified. Four forgeries are refused with 400
  bad_signature: a signature over another document, over another party index,
  from a key other than the one presented, and pure noise. The slot stays open
  and the good signature still lands afterwards.
• The receipt is verified offline. The test re-derives every signed message
  from the receipt's own fields, verifies each party signature and the notary
  signature with the raw engine, checks the notary key hash, and shows that
  changing one character of the document hash breaks the notary signature. It
  never asks the relay whether the receipt is good.
• Burn-on-read. First read returns the bytes and says X-Paramant-Burned: true, the second is 404, and that 404 is identical to the 404 for a hash that
  never existed. A refused read by another tenant does not consume the view.
• Restart. The billing suite restarts the relay on the same users.json and
  asks the same question again. That round trip is what billing: collect a second period, and make paid_until survive a restart #315 actually broke.

Coverage of relay.js

Measured with c8 over the suites exactly as CI runs them, --all --include relay.js, so the file appears even when nothing loads it.

	% lines	% branch	% funcs
before (CI suite as on main)	0.00 (1-6493 uncovered)	0.00	0.00
after (the five new suites)	41.42	58.21	43.88

The before figure is not an artefact of the tool: inbound-hash-verify,
deep-health-gate and billing-stance-boot do boot relay.js, but they SIGKILL
the child, and a killed process writes no V8 profile. The harness stops a relay
with SIGTERM (relay.js already handles it, zeroizes its blobs and exits 0), which
is what turned the profile on.

Sabotage, per suite

One line changed, the suite run, the change reverted. Each is a real change to
production code, not to a fixture.

#	sabotage	suite	result
S1	relay.js:4214 auth gate condition replaced by false (the gate lets everything through)	route-auth-gate	2 red: "POST /v2/envelopes without a key is 401", "create is NOT in that public list". Only 2 because the create handler has a second !keyData check behind the gate, which is defence in depth working as intended.
S2	relay.js:4529 views_remaining ... - 1 becomes - 0 (nothing ever burns)	route-transfer-burn	5 red, including "BURN ON READ", "the 404 after a burn is the same 404", both per-tier max_views tests
S3	relay.js:5657 sigVerify returns true unconditionally	route-envelope-lifecycle	1 red: "a signature over the WRONG message is refused, so the relay really verifies"
S4	relay.js:1643 _mutateUsersJson becomes a no-op (this is the #315 mechanism: entitlements live only in memory)	route-billing-entitlements	3 red: "an admin product-plan change is on disk", "leaves the other product's paid period alone", "the users.json write is atomic"
S5	relay.js:1404 and 1427 create budget 50 becomes 500	route-rate-limits	2 red: the per-process cap and the fleet-wide cap

git status was clean after each revert.

CI

• The route suites run in relay-crypto-tests, the job that installs the relay
  deps and builds @paramant/core. relay.js requires redis and the engine at
  load, so they cannot run in the no-install unit job, and they are excluded from
  its glob by name.
• That job gains a redis service (redis:7.4.8-alpine, the same tag
  docker-compose runs, on 6399). The envelope store is redis-only
  (relay.js:5650): without a server every /v2/envelopes route answers 503 and
  the ParaSign lifecycle cannot be tested. It is also what makes the fleet-wide
  half of the create limiter measurable, two relay processes against one budget.
• The step carries the same silent-suite gate as the unit job, with an empty
  expected set. Every suite prints its closing line through summary() in
  test/_requires.js, so a suite that runs and asserts nothing is caught here
  too. In this job it has redis and the engine, so a skip means broken, not
  under-equipped, and the gate fails rather than keeping a list.
• parasign-signs-quota moves into that step too. It needs redis and nothing
  else, and it ran in one place only: the no-install unit job, where it has no
  redis and skipped on every single run. Its five checks on the monthly signs cap
  (the metering that once sat behind a dead && false) had therefore never run
  on a pull request. With a redis service they do, and the unit job's
  expected-silent list is now empty: every suite that runs there asserts
  something.
• Both silence gates gained || true on the grep that collects the silent
  suites. GitHub runs a run: block under bash -e and grep exits 1 when it
  matches nothing, so an empty result, which is the good case, failed the step.
  That is what turned the first fully green route run into a red job; the suites
  themselves were 56 pass / 1 todo on that same run.

Findings

Found while writing the tests. Not fixed here: this branch is tests only.

1. An expired paid tier keeps granting on every account-level read

lib/entitlements.js:328-344 mergeAccountRecord() copies plan_parasign and
plan_parasend off the api-key records onto the merged account record, but not
their paid_until_* fields. Its own doc comment explains that those fields can
only live on the key records, because the accounts summary never carries the
per-product plans. So the merged record reaches getEntitlements with a paid
tier and no period, and entitlements.js:137 then correctly reads "no recorded
period" as "never expired".

getEntitlements(keyRec).parasign.tier                             -> 'free'
getEntitlements(mergeAccountRecord(acct, [keyRec])).parasign.tier -> 'business'

(same record, paid_until_parasign 20 days in the past)

Reached through entitlementRecordOf at:

• relay.js:5316 GET /v2/admin/entitlements/:account_id
• relay.js:5990 the signs-quota gate on POST /v2/envelopes/:id/sign
• relay.js:1804 the plan a newly minted psk_ key inherits

This is the same class as #315, one step further along: the date is written and
persisted correctly, and then dropped on the way to the gate that reads it. The
fix is two lines in mergeAccountRecord, but it changes billing behaviour, so it
belongs in its own PR with a decision about accounts that are already lapsed.

Encoded in the branch as a todo test (route-billing-entitlements, "#315: a
period that HAS passed stops granting"), which states the correct rule and fails
today without failing the run, plus a passing test named FINDING: that pins the
divergence between the two read paths as evidence. Both go away with the fix.

2. A ParaSend download answers with a 19 KB response header

GET /v2/outbound/:hash returns X-Paramant-Receipt, a base64 delivery receipt
carrying a full ML-DSA-65 signature and the inclusion proof. Measured on this
branch: 18551 bytes for that one header, 19560 bytes of response headers in
total (relay.js:4563-4590).

• Node's own default maxHeaderSize is 16384, so a Node client using fetch()
  or a default http.request cannot download a blob: it throws
  UND_ERR_HEADERS_OVERFLOW. This is how the finding surfaced, the test client
  hit it on the first download.
• No nginx config in this repo sets proxy_buffer_size (nginx-selfhost.conf,
  deploy/nginx-paramant-live.conf, deploy/nginx-paramant-public.conf,
  deploy/nginx-selfhost.conf all lack it). The nginx default is 4k/8k, which
  answers 502 on an upstream header this size. Whether production is configured
  differently was not checked, that is a production read.

The obvious shape of a fix is to move the receipt into the body or behind a
second endpoint, or return only a receipt id in the header. Not touched here.

3. Two smaller inconsistencies

• relay.js:4386 takes the TTL ceiling with a 'community' fallback for a key
  with no plan, relay.js:4391 takes the max_views ceiling for the same key with
  a 'pro' fallback. Two different defaults, one line apart, and the more
  permissive one is on the views.
• relay.js:1607 OUTBOUND_RATE = { free: 50, pro: 500, enterprise: Infinity }
  has no community and no business key, so both fall to ?? free = 50/hour.
  A Business account pays for the highest ParaSign volume and gets the free
  outbound rate.

4. A testability gap, not a bug

The only code path that writes paid_until is the Mollie webhook, and
lib/mollie.js:12 hard-codes api.mollie.com with no override (deliberately, as
an anti-SSRF measure). So the write half cannot be driven offline and this branch
pins the read-back and the admin write instead. If that path is ever to be tested
end to end it needs an injection seam that does not widen the SSRF surface.

How to run it

cd relay && npm install
docker run -d --rm -p 6399:6379 redis:7.4.8-alpine
REDIS_URL=redis://127.0.0.1:6399 node --test test/route-*.test.js

What the review changed

Four of the findings were comments promising behaviour the code did not have.
That is the same failure as a skipping canary, in a smaller frame, so they are
worth listing.

finding	what the comment said	what the code did	fix
dry run	lib.mjs:26 and run.mjs:51: a dry run can never stand in for proof	4/4 GREEN, exit 0	summary.ok false and exit 2 however many steps passed. steps_ok records separately that the wiring is sound, so the two are never confused
cleanup	parasign.mjs: "cleanup first, so a failure anywhere below still tries to retract"	the void ran only on the success path	try/finally around the receipt walk. The runs that left an envelope behind were exactly the ones that mattered
the issue	the workflow comment said "reopened"	it searched state: open, so every red episode after a green one filed a new issue	searches state: all by title, skips pull requests, reopens and comments
/health	recorded as "is alive"	only the status code	the body must be JSON, must say it is healthy, and must carry a mode or a version. nginx, a cache, a maintenance page and a parked domain all answer 200

Nothing watched the watcher

scripts/heartbeat/ had no test coverage, and it showed: a swapped ML-DSA-65
argument order lived in it while the whole suite stayed green. That is the same
bug the old canary carried.

tests/heartbeat-lib.test.mjs (11 tests) pins:

• the ML-DSA-65 argument order, including that the wrong order throws rather
  than returning false, which is precisely why it was invisible
• the recipe-v4 sign message: stable, mixes in the signer public key, and binds
  party index and document hash
• runStep fails a step that recorded no proof, and writes the evidence either
  way
• requireSecret fails by name, and treats a whitespace-only value as unset
  (an unset repository secret expands to an empty string)
• a dry run exits non-zero and never reports itself as green

Reintroducing the swapped order fails two of its tests. Verified by doing it.

The self-check moved off module load into assertMlDsaWrapper(), called at the
top of each step. A throw during import happens before run.mjs has a step to
attribute it to, so no summary.json and no annotation would be written and the
alarm would degrade into a stack trace. It now arrives as a named red step, with
the noble RangeError caught and rewritten.

The suite needs @noble/post-quantum, so the root integration job installs the
root dependencies. Making it skip when the package is absent was the obvious
alternative and it is exactly the bug being fixed.

Docs pointing at deleted files

docs/site-claims.md row 8, tests/README.md, docs/frontend-loading-contract.md
and deploy/DEPLOY-3.1.md no longer send a reader to
tests/transfer-canary.test.mjs or tests/parasign-canary.test.mjs. The
DEPLOY entry for #316 keeps its history and records that #338 replaced it, and
why.

Rebased on main through #327.

The SLA page, and the order this has to happen in

Section 5 of /sla was made false by this branch, so it is rewritten here
rather than left for a follow-up:

sentence	why it became false
"the product-heartbeat workflow ... runs hourly"	that file no longer has a schedule
"the four sector relays ... are not part of that hourly run"	surface.mjs POSTs to all four and goes red on status 0

What it says now is what is true, and it is less than before: /status is
the only thing running today. The hourly check is built and public, and the
paragraph says it runs hourly once enabled and is switched off until its
credentials are in place.

Why "once enabled", and the gate

That clause is not hedging, it is the third finding. PARAMANT_CANARY_KEY and
PARASIGN_CANARY_KEY do not exist and can only be created from the machine that
holds the keys. Merged with a live schedule, this alarm would go red at :17
every hour and comment on the same issue twenty-four times a day, for a reason
everybody already knows. An alarm that cries hourly for a known reason is one
people learn to ignore, which is how a monitor dies a second time.

So the job carries:

if: github.event_name == 'workflow_dispatch' || vars.HEARTBEAT_ENABLED == 'true'

workflow_dispatch ignores the gate, so a run can always be started by hand,
including before the variable exists, to see exactly which secret it still
wants. docs/heartbeat.md has the commands in order:

gh secret set PARAMANT_CANARY_KEY    # a relay API key (pgp_)
gh secret set PARASIGN_CANARY_KEY    # a ParaSign bearer key (psk_test_)
gh workflow run heartbeat.yml        # try it by hand and watch it
gh variable set HEARTBEAT_ENABLED --body true   # only once that run is green

Step 3 matters more than it looks: the first real run is the first time anything
will have checked ParaSign against production, so it is also the first chance to
find that ParaSign has been broken for the eleven days nobody signed anything.
Worth watching, not worth discovering at three in the morning.

The test got stricter, not looser

• the workflow name is matched exactly, and product-heartbeat workflow is now
  forbidden on the page
• the gate is pinned in both directions: while it is there the page must say
  the check is switched off, and once it goes the page must stop saying so. So
  setting HEARTBEAT_ENABLED fails this test until section 5 is rewritten. The
  follow-up cannot be forgotten, because the switch itself trips it
• the sector clause lost the exemption it had in the previous round. It is
  pinned to surface.mjs reaching all four hosts for the ParaID deny check, and
  the page must carry both halves: what the run does there, and that uptime is
  not measured
• "a run that fails opens an issue in the repository" is tied to the
  if: failure() step that calls issues.create, plus issues: write

Each was checked by breaking it:

broken	test says
gate removed	the gate is gone, so the page must stop saying the check is switched off
if: failure() changed to always()	the workflow must have a step that runs only on failure
old sector sentence restored	the page must say both what the run does at those hosts and what it does not measure