Skip to content

/api/rpc forwards to the upstream RPC without a timeout — one hung upstream request stalls its initiator AND every deduplicated waiter until the platform kills the invocation #2522

Description

@Bayyan16
Weakness class CWE-754 (Improper Check for Unusual or Exceptional Conditions)
Affected route POST /api/rpc — the app's core RPC proxy (600 req/min/IP budget, middleware.ts:33)
Affected file app/app/api/rpc/route.ts — unbounded fetch lines 340-347; in-flight registration lines 348-350; waiter await lines 335-338; cleanup finally lines 377-379
Assessed at 7ce465b — HEAD of playground as of 2026-08-16 (file untouched by GH#2516/#2518/#2519; line numbers identical at the previously audited 8d3e280)
Related issues none — no existing issue covers the missing upstream timeout

TL;DR

The RPC proxy's one outbound call has no AbortSignal — it is the only unbounded fetch in the codebase (oracle/publishers, oracle-keeper/register, and lib/api-proxy.ts all carry 5-8 s timeouts). If the upstream RPC accepts the connection but never answers (the standard failure mode of a rate-limited or degraded provider), the request hangs until the serverless platform kills the invocation. Worse, the hung promise is registered in the dedup map, so every concurrent identical read request coalesces onto it and hangs too — one stall fans out into as many hung responses as there are overlapping callers, and the designed failure path (per-item -32603 errors, already implemented) never engages because the promise never settles.

1. What the route does

/api/rpc proxies JSON-RPC calls to Helius (embedding the paid API key server-side), with a small TTL cache and in-flight deduplication: concurrent identical read requests share one upstream fetch (inflightRequests, keyed by method+params). The trading UI drives this path constantly — getAccountInfo, getMultipleAccounts, getProgramAccounts poll at 1-1.5 s TTL from every open tab.

2. The affected code

// app/app/api/rpc/route.ts:340-350
const fetchPromise = (async () => {
  const response = await fetch(getRpcUrl(networkOverride), {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(req),
  });                                // ← no signal: AbortSignal.timeout(...) is absent
  return await response.json();
})();

// Register in-flight for dedup
if (!isMutating) {
  inflightRequests.set(cacheKey, fetchPromise);   // waiters await this exact promise
}

Three compounding problems when the upstream black-holes:

  1. The initiator hangs until the platform deadline (Vercel maxDuration) kills the invocation. The client receives an opaque network error instead of a fast, retryable JSON-RPC error.
  2. Waiters hang transitively. Concurrent identical requests await the stored promise at lines 335-338 — one stall becomes N stalled responses, proportional to live traffic.
  3. The poisoned entry blocks replacement. The finally cleanup (inflightRequests.delete, lines 377-379) runs only after the promise settles; until then a fresh request for the same key joins the waiters instead of issuing its own fetch.

The designed failure path — the "BUG 14" catch at lines 363-376 that converts rejections into per-item -32603 errors — works only if the promise rejects. A hang produces no rejection, so the route's own error handling is bypassed entirely.

3. Why every other fetch in the codebase is bounded

Call site Timeout
app/app/api/oracle/publishers/route.ts (Pythnet + bridge fetches) 5-8 s
app/app/api/oracle-keeper/register/route.ts (keeper forward) 8 s
app/lib/api-proxy.ts (backend proxy) 8 s
app/app/api/rpc/route.ts:341 (the hottest outbound path) none

This is an omission, not a design choice — and it sits on the route with the highest request budget (600/min/IP).

4. Impact

During any upstream brownout that manifests as a hang rather than a clean error (common with rate-limited providers):

  • hung serverless invocations accumulate in proportion to read traffic, each holding resources for up to the platform maximum;
  • users see the app freeze on data loads rather than degrade gracefully;
  • the client-visible symptom (network timeout) gives no signal that the proxy itself is healthy, complicating incident diagnosis;
  • invocation-time costs scale with the stall duration.

5. Proof of concept

Deterministic reproduction, no live deployment needed — point the proxy at a socket that accepts connections and never answers:

// tests/api/rpc-upstream-stall.poc.test.ts
import http from "node:http";
import { setTimeout as sleep } from "node:timers/promises";

it("a black-holed upstream stalls the initiating call AND all dedup waiters", async () => {
  // 1) A server that accepts connections and never responds.
  const blackhole = http.createServer(() => { /* never responds */ });
  await new Promise<void>((r) => blackhole.listen(0, "127.0.0.1", r));
  const port = (blackhole.address() as { port: number }).port;
  process.env.NEXT_PUBLIC_HELIUS_RPC_URL = `http://127.0.0.1:${port}`;  // consumed by getRpcUrl()

  // 2) One initiating request + three concurrent identical ones.
  //    Identical method+params ⇒ identical cacheKey ⇒ the extra calls become
  //    dedup waiters on the first's promise (route.ts:335-338).
  const call = () => fetch("http://localhost:3000/api/rpc", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Origin: "http://localhost:3000",          // passes isAllowedOrigin in non-production
    },
    body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "getEpochInfo", params: [] }),
  });

  const outcome = await Promise.race([
    Promise.allSettled([call(), call(), call(), call()]).then(() => "settled"),
    sleep(10_000).then(() => "STILL PENDING AFTER 10s"),
  ]);

  expect(outcome).toBe("settled");   // ← FAILS today: "STILL PENDING AFTER 10s"
  // Expected after the fix: all four settle quickly with the per-item shape
  // {"error":{"code":-32603,"message":"Upstream RPC request failed"}}.

  blackhole.close();
});

Live-environment variant: temporarily block egress to *.helius-rpc.com on a dev deployment (or observe during a genuine provider brownout) and watch /api/rpc requests for popular read methods pile up together instead of failing independently — the dedup map makes their fates identical.

6. Recommended fix

Bound the fetch like every other outbound call in the app — one added line:

const response = await fetch(getRpcUrl(networkOverride), {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(req),
  signal: AbortSignal.timeout(20_000),
});

The timeout composes directly with the existing machinery: on expiry the fetch rejects (an AbortError), the "BUG 14" catch at lines 363-376 converts it into a per-item -32603 error for the initiator, and the finally deletes the in-flight entry so the next request issues a fresh fetch. Waiters currently bypass that catch (see the companion issue on the dedup-waiter await placement) — fixing both together makes every caller, initiator or waiter, fail fast with the same well-formed error.

7. References

  • app/app/api/oracle/publishers/route.ts, app/app/api/oracle-keeper/register/route.ts, app/lib/api-proxy.ts — in-repo precedent for AbortSignal.timeout on all outbound fetches
  • app/app/api/rpc/route.ts:335-338, 348-350 — the dedup registration that amplifies one stall into many hung responses
  • app/app/api/rpc/route.ts:363-376, 377-379 — the rejection handler and cleanup the timeout feeds into
  • app/middleware.ts:33 — the 600 req/min/IP budget that makes this the highest-volume outbound path

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions