Skip to content

fix(edge): serve an Openship page for upstream-down 502/504 instead of OpenResty's stock one - #557

Open
AbdullahM07 wants to merge 1 commit into
oblien:mainfrom
AbdullahM07:fix/edge-upstream-down-page
Open

fix(edge): serve an Openship page for upstream-down 502/504 instead of OpenResty's stock one#557
AbdullahM07 wants to merge 1 commit into
oblien:mainfrom
AbdullahM07:fix/edge-upstream-down-page

Conversation

@AbdullahM07

Copy link
Copy Markdown
Member

Closes #556.

What was broken

git grep error_page over the repo returned zero hits: the edge defined no page for any
5xx, so an app that was restarting, crashed, or still booting produced OpenResty's stock
502 Bad Gateway. Behind Openship Cloud's shared edge — which forwards to the box by IP
and relays whatever the box answers (edge-target.ts) — that stock page is what a visitor
to the operator's own domain reads, carrying a third party's branding and links.

What this does

Serves an Openship-branded page for upstream-down errors from the box's own edge, mirroring
what #431 did for unrouted hosts.

  • packages/adapters/src/infra/edge-upstream-down.ts — the page, built exactly like
    edge-not-found.ts: inline in the config (it has to arrive on every install path at once,
    and a shared include would be a hard -t failure that freezes every later reload on the
    box), the same five tokenizer invariants, and no links or reflected values since it is
    served to strangers. Removing the third-party link is the fix — re-pointing it at the
    operator's own domain would only link to the site that is currently down.
  • Wired at server scope in every vhost that proxies to an upstream. Not host redirects,
    and not static routes unless a compiled vercel.json gave them proxy locations — which
    is why the gate tests for an upstream rather than for !staticRoot.
  • 502 and 504 only, never 503. blockStatus (default 403) and rateLimit.status
    (default 429) are operator-overridable and limit_req_status is 429, so a 503 arm would
    brand an operator's deliberate block as an outage.
  • VHOST_GENERATION 1 → 2. Without the bump this reaches no existing box:
    registerRoute is the sole writer and reapplyStoredRoutes replays only vhosts stamped
    below the current generation.
  • No = before the named location, which is load-bearing rather than stylistic — see
    below.

nginx semantics, verified rather than assumed

error_page … @named preserves or clobbers the status code depending on =, so I probed it
in openresty/openresty:1.27.1.1-alpine with dead and hanging upstreams before settling the
design:

Case Syntax Trigger Status seen Body seen
A error_page 502 504 @loc; nginx-generated 502 502 preserved our handler
B error_page 502 504 = @loc; nginx-generated 502 503 — handler's code clobbers it our handler
C no = real 504 read-timeout 504 preserved our handler
D no = upstream's own 502 502 the app's own body

So the form without = lets one shared ~2 KB body cover both codes without collapsing 504
into 502. Case D is the one most worth having: with proxy_intercept_errors off (nginx's
default, and set nowhere in this repo) an app that answers 502 itself keeps its own body, so
this page never hijacks a real response from a running app. That default is inherited rather
than written in any vhost, so the e2e below is what fails if a future
proxy_intercept_errors on; ever starts masking every app's real error responses.

How it was verified

  • apps/api/test/e2e/edge-upstream-down-page.e2e.test.ts — boots real OpenResty with
    vhosts from the real NginxProvider and checks: a refused upstream (502 + our page,
    and explicitly not the stock body), a read timeout (504 + our page), an app's own 502
    (its body passes through untouched), a healthy app (200, untouched), a .css URI on a
    down app (text/html, per the empty types { } map), and a static vhost (never serves
    it). openresty -t runs inside the same container first, making this the pre-merge
    equivalent of the Dockerfile's gate.
  • Three of those six cases fail without this change, returning
    <title>502 Bad Gateway</title> and <title>504 Gateway Time-out</title> — the reported
    bug itself. Confirmed by temporarily disabling the wiring and re-running. The other three
    pass either way by design: they guard against regressions rather than proving the feature.
  • edge-upstream-down.test.ts (16) — the tokenizer invariants plus the stripComments +
    extractBlocks round trip that Unrecognized hostnames return raw TLS / OpenResty errors instead of a friendly “Service Not Found” page #431's fix originally broke.
  • nginx.test.ts (+7) — handler in both blocks of a proxy vhost, absent from static and
    host-redirect vhosts, present on a static vhost that has proxy locations, error_page at
    server scope, no 503, no =.
  • Full @repo/adapters suite: 2879 pass. tsc --noEmit clean in @repo/adapters and
    @repo/api.

bun format was deliberately not run across the repo: nginx.ts, nginx.test.ts and
index.ts already carry Prettier drift on main, and --write would have reformatted lines
this change never touches. The two new files pass prettier --check as written.

Known limitation

This covers every 502/504 the box generates — an app that is down, on a custom domain or
a free *.opsh.io one. It does not cover the case where the box itself is unreachable
from Openship Cloud's edge: that response is generated entirely by the Cloud edge and no
change in this repository can reach it. #556 records the same boundary.

The page carries an openship-edge-upstream-down marker immediately after the doctype, so
curl -s https://host | head -c 200 tells the two cases apart in support.

The edge defined no `error_page` for any 5xx, so an app that was restarting,
crashed, or still booting produced OpenResty's stock `502 Bad Gateway`. Behind
Openship Cloud's shared edge — which forwards to the box by IP and relays
whatever the box answers — that page is what a visitor to the operator's OWN
domain reads, branded for a third party. Closes oblien#556.

Adds the page as `edge-upstream-down.ts`, mirroring the unrouted-host page
(oblien#431): inline in the config for the same reason (it has to arrive on every
install path at once, and a missing `include` would be a hard `-t` failure that
freezes every later reload), the same five tokenizer invariants, and no links or
reflected values since it is served to strangers. Removing the third-party link
IS the fix — re-pointing it at the operator's own domain would only link to the
site that is currently down.

Wired at server scope in every vhost that proxies to an upstream: not host
redirects, and not static routes unless a compiled vercel.json gave them proxy
locations. Intercepts 502 and 504 only — `blockStatus` (403) and
`rateLimit.status` (429) are operator-overridable, so a 503 arm would brand a
deliberate block as an outage.

`VHOST_GENERATION` 1 -> 2, without which this reaches no existing box:
`registerRoute` is the sole writer, and `reapplyStoredRoutes` replays only
vhosts stamped below the current generation.

No `=` before the named location, so an intercepted 504 stays a 504 and one body
serves both codes. Verified against openresty/openresty:1.27.1.1-alpine rather
than assumed: with `=`, the named location's own return code replaces the
original and a timeout stops being reportable as one.

Verified: a new e2e boots real OpenResty with vhosts from the real
NginxProvider and checks a refused upstream (502 + our page), a read timeout
(504 + our page), an app's own 502 (its body passes through untouched, since
`proxy_intercept_errors` is off), a healthy app, a `.css` URI on a down app, and
a static vhost. Three of its six cases fail without this change, returning the
stock pages. Full adapters suite: 2879 pass.
@AbdullahM07
AbdullahM07 force-pushed the fix/edge-upstream-down-page branch from 4743cdd to 394c06b Compare August 16, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upstream-down hostnames return a raw OpenResty 502 instead of a friendly “Application unavailable” page

1 participant