feat: RPC timeout budgets for policy deploy + circuit breaker for verification-service - #381
Conversation
…ification-service Closes Vellar-Wallet#327 Closes Vellar-Wallet#326
|
@onuibeblessing2019-hash Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
|
@onuibeblessing2019-hash is attempting to deploy a commit to the david's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @onuibeblessing2019-hash — thanks for the contribution! We do not accept pull requests to the You don't need to reopen anything — your work is preserved and this PR now targets |
…and-circuit-breaker # Conflicts: # packages/service-kit/src/index.ts # packages/service-kit/src/metrics.ts # services/api-gateway/README.md # services/api-gateway/src/server.ts
|
Merged Two real, pre-existing bugs surfaced during that merge, worth flagging separately from #326/#327's own scope:
|
Closes #327
Closes #326
#327 — timeout budgets for policy deployment RPC calls
services/policy-service/src/deploy.ts's deploy path (getAccount,simulateTransaction,prepareTransaction,sendTransaction, thegetTransactionpolling loop) had no explicit timeout on any individual RPC call — only the polling loop had an overall deadline.Real finding while implementing this:
rpc.Server's constructor accepts atimeoutoption in@stellar/stellar-sdk's TypeScript types, but as of SDK 16.0.1 it has no effect —RpcServer's constructor callscreateHttpClient(opts.headers), which only ever receivesheaders, neveropts.timeout. Confirmed by readingrpc/axios.js/http-client/fetch-client.js, and experimentally: a test against a deliberately-hanging local server using the SDK's owntimeoutoption hung indefinitely. Implemented a real, self-contained timeout (withTimeoutError, aPromise.raceagainst this module's own timer) instead of relying on the non-functional SDK option — documented prominently in the code so it isn't "fixed" back to the broken approach later.DEPLOY_RPC_TIMEOUT_MS(default 10s, bounds each individual RPC call) andDEPLOY_POLL_TIMEOUT_MS(default 60s, bounds the overall polling loop) — documented inpolicy-service/README.mdper the issue's requirement.deploy_rpc_timeout(a single call stalled) vs. the existingdeploy_timeout(poll loop exceeded its overall budget) vs. every other existing failure code — a timeout is never confused with a real RPC/contract failure.deploy.test.ts, using a real local HTTP server that never responds (the only reliable way to exercise a genuine timeout end-to-end) — confirms the budget is actually enforced, is configurable, and is distinct from a real connection-refused error.#326 — circuit breaker for api-gateway → verification-service
No existing circuit-breaker dependency in this repo, so added a small, dependency-free state machine (
packages/service-kit/src/circuit-breaker.ts) — closed → open (afterfailureThresholdconsecutive failures) → half-open (one trial call aftercooldownMs) → closed or back to open, matching the issue's exact requirement.api-gateway's/verification/*proxy via@fastify/http-proxy'spreHandler(fast-fail 503 while open, no network attempt) andreplyOptions.onResponse/onErrorhooks (record success/failure) — only a genuine connection-level failure counts against the breaker; a normal upstream 4xx/5xx (the verification service correctly reporting a domain outcome) does not.VERIFICATION_CB_FAILURE_THRESHOLD(default 5),VERIFICATION_CB_COOLDOWN_MS(default 30s) — documented inapi-gateway/README.md.vela_circuit_breaker_state_changes_total{breaker="verification-service"}per the issue's "add a metric tracking circuit breaker state changes" requirement, plus a log line on every transition.circuit-breaker.test.ts(closed/open/half-open transitions, the metric hook, the lower-levelbeforeCall/recordOutcomepair the gateway integration actually uses since@fastify/http-proxycan't wrap a promise this code controls) + 2 new integration tests inapi-gateway/server.test.tsexercising the full cycle through a real gateway instance against a real (unlistened-port) connection failure, confirming: opens after 2 consecutive failures, fast-fails with 503 +retryAfterMs, recovers via a half-open trial once the real upstream comes back, and — separately — confirms a real reachable upstream returning 404 five times in a row never trips a breaker configured to open on just 1 failure.Test plan
packages/service-kit: 71/71 tests pass (circuit-breaker.test.tsnew, everything else unaffected).services/policy-service: 50/50 tests pass (deploy.test.tsnew).services/api-gateway: 15/15 tests pass (2 new circuit-breaker integration tests).npx tsc --noEmitclean in all 3 touched packages.