Skip to content

Document runtime roles and health checks #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cursor wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Document runtime roles and health checks #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 46 additions & 7 deletions docs/01-deployment-infrastructure/OBSERVABILITY_MANUAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Once configured, you will have visibility into:
- **LiveView**: Mount times, handle_event durations
- **Business KPIs**: Active rides, pending rides, users, marketplace listings, bookings, study sessions, payment processing
- **Distributed Traces**: Full request waterfall from HTTP through controllers, plugs, Ecto queries
- **Health Check**: App status, database connectivity, all 12 background worker statuses
- **Health Check**: App status, `RELEASE_ROLE`, database connectivity, BEAM stats, and worker status when the node runs background processes

---

Expand Down Expand Up @@ -90,7 +90,7 @@ Once configured, you will have visibility into:
2. **Traces** go directly from your app to Grafana Cloud Tempo via OTLP HTTP (no extra infrastructure)
3. **Dashboards** are auto-uploaded to Grafana Cloud by PromEx when the app starts (no extra infrastructure)
4. **Logs** are structured JSON written to stdout, visible in Render's log viewer
5. **Health check** at `/health` returns app status, BEAM stats, and worker health
5. **Health check** at `/health` returns app status, `RELEASE_ROLE`, BEAM stats, and role-specific worker health

---

Expand Down Expand Up @@ -297,7 +297,7 @@ Open in your browser:
https://studentjunxion.co.za/health
```

You should see JSON with app status, BEAM stats, database status, and worker health.
You should see JSON with app status, `release_role`, BEAM stats, and database status. Worker status appears only on `RELEASE_ROLE=all` and `RELEASE_ROLE=worker` nodes.

---

Expand Down Expand Up @@ -657,10 +657,43 @@ rate(studentplug_business_ride_requests_total[5m]) * 60

`GET /health` is always available without authentication.

**Response when healthy (HTTP 200):**
The response shape depends on `RELEASE_ROLE`:

| Role | Health checks | `workers` key |
|------|---------------|---------------|
| unset / `all` | Database, BEAM stats, background GenServers | Present |
| `web` | Database, BEAM stats | Omitted |
| `worker` | Database, BEAM stats, background GenServers | Present |

HTTP status is `503` only when the database check fails. If the database is reachable but a worker GenServer is stopped, the JSON `status` is `"degraded"` and the response remains HTTP 200 so operators can distinguish a node-level problem from total outage.

**Web response when healthy (HTTP 200):**
```json
{
"status": "ok",
"release_role": "web",
"version": "0.1.1",
"database": "ok",
"beam": {
"uptime_seconds": 86400,
"process_count": 1234,
"process_limit": 262144,
"memory_total_mb": 256.5,
"memory_processes_mb": 128.3,
"memory_ets_mb": 45.2,
"atom_count": 12345,
"schedulers_online": 4,
"otp_release": "25"
},
"timestamp": "2026-03-13T12:00:00Z"
}
```

**Worker or single-service response when healthy (HTTP 200):**
```json
{
"status": "ok",
"release_role": "worker",
"version": "0.1.1",
"database": "ok",
"beam": {
Expand All @@ -683,18 +716,24 @@ rate(studentplug_business_ride_requests_total[5m]) * 60
"SafetyMonitoringJob": "running",
"ScheduledRidesJob": "running",
"DriverPayoutJob": "running",
"PartnerPayoutJob": "running",
"ReserveReleaseJob": "running",
"QualityControlJob": "running",
"SharedRideJob": "running",
"RideExpirationJob": "running",
"DailyViewsEmailJob": "running"
"JobExpirationJob": "running",
"LocationStaleJob": "running",
"PayoutJob": "running",
"DailyViewsEmailJob": "running",
"StudyDigestJob": "running"
},
"timestamp": "2026-03-13T12:00:00Z"
}
```

**Status values:**
- `ok` -- database reachable, all workers running
- `degraded` -- database reachable, but some workers stopped
- `ok` -- database reachable, and all expected workers running for `all` / `worker` roles
- `degraded` -- database reachable, but some workers stopped on an `all` / `worker` node
- `unhealthy` -- database unreachable (returns HTTP 503)

Use this for:
Expand Down
72 changes: 41 additions & 31 deletions docs/01-deployment-infrastructure/SCALING_POWERHOUSE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,16 +87,16 @@ Elixir on the BEAM gives you capabilities other stacks need Kafka + Redis + Side
| **Render workspace** | **Professional (Pro)** — **not** Hobby | Already paid; enables autoscale/preview when you add services |
| **Team members** | **2** on workspace | Billed per seat on Professional plan |
| **Build minutes** | **Overage charges** apply | Frequent deploys (multiple per day) consume included minutes; extra billed per Render rates |
| Redis | **None** | Cannot scale web horizontally safely |
| `RELEASE_ROLE` | **Implemented** (`all` default; `web` / `worker` for split deploy) | Safe to add worker service when budget approved |
| Oban | **Single `default: 10` queue** | Critical jobs compete with analytics digests |
| Hammer | **ETS backend** | Per-node only |
| PubSub | **Default (local)** | Breaks multi-node channels |
| Redis | **No production service yet**; code switches on `REDIS_URL` | Cannot scale web horizontally safely until Redis is provisioned and set on every node |
| `RELEASE_ROLE` | **Implemented** (`unset`/`all` for single service; `web` / `worker` for split deploy) | Safe to add worker service when budget approved; do not run duplicate `all` services |
| Oban | **Segmented queues** (`critical`, `notifications`, `default`, `analytics`, `cleanup`) | Still competes with HTTP until `RELEASE_ROLE=worker` services process jobs off-web |
| Hammer | **ETS by default; Redis when `REDIS_URL` is set** | Per-node only until Redis is live on every web node |
| PubSub | **Local by default; Redis adapter when `REDIS_URL` is set** | Multi-node channels require Redis before web autoscale |
| `DNS_CLUSTER_QUERY` | Likely unset | Clustering inactive |
| OpenTelemetry | **Disabled** (`otel_enabled: false`) | Blind to p95/p99 under load |
| Grafana | Was configured; **broken after rebrand** | No dashboards/alerts |
| `UserSocket.id/1` | Returns `nil` | No Presence; no per-socket tracking |
| Static assets | `gzip: false` in endpoint | Slower transfers |
| Static assets | Production endpoint config has `gzip: true` | Keep enabled for marketing/app asset delivery |

**Root cause of crashes:** not “too many users” yet—**too many subsystems on one 512MB node** (`Studentplug.Application` starts Endpoint, Oban, AccountCleanup, AdminNotificationJob, 7× ride jobs, 3× delivery jobs, marketplace/study jobs, PromEx, etc.).

Expand Down Expand Up @@ -270,16 +270,21 @@ What you pay **today** before powerhouse service upgrades (web/worker/redis/Post

**Goal:** Web nodes never run Oban or cron GenServers; worker nodes never open HTTP port.

**Implement in** `studentplug/lib/studentplug/application.ex`:
**Implemented in** `studentplug/lib/studentplug/release_role.ex` and wired from `studentplug/lib/studentplug/application.ex`:

| `RELEASE_ROLE` | Start | Do not start |
|----------------|-------|--------------|
| `web` (default) | `StudentplugWeb.Endpoint`, `Repo`, `Phoenix.PubSub`, `Finch`, `Telemetry`, `PromEx`, `ReleaseMigrator` (web only if needed) | `Oban`, `AccountCleanup`, `AdminNotificationJob`, all `Rides.Jobs.*`, all `Delivery.Jobs.*`, `MarketingDataSync`, `StudyDigestJob`, `DailyViewsEmailJob`, etc. |
| `worker` | `Repo`, `Oban`, all background GenServers, `Finch`, `Telemetry` | `StudentplugWeb.Endpoint` |
| unset / `all` | Shared children + web children + worker children | Nothing — current single Render service behaviour |
| `web` | `StudentplugWeb.Endpoint`, `Repo`, `Phoenix.PubSub`, `Finch`, `Telemetry`, `PromEx`, `ReleaseMigrator`, `MetricsPusher` | `Oban`, `AccountCleanup`, `AdminNotificationJob`, all `Rides.Jobs.*`, all `Delivery.Jobs.*`, `MarketingDataSync`, `StudyDigestJob`, `DailyViewsEmailJob`, etc. |
| `worker` | `Repo`, `Oban`, all background GenServers, `Finch`, `Telemetry`, `Phoenix.PubSub` | `StudentplugWeb.Endpoint`, `PromEx`, `ReleaseMigrator`, `MetricsPusher` |

**Env vars**

```bash
# Current single-service production default
RELEASE_ROLE=all # or leave unset
PHX_SERVER=true

# Web service (Render)
RELEASE_ROLE=web
PHX_SERVER=true
Expand All @@ -293,31 +298,34 @@ POOL_SIZE=20

**Why this matters for Elixir:** the BEAM can supervise thousands of processes—but **not** when one VM is memory-starved. Isolating HTTP from batch work is OTP 101.

**Critical:** Never run **two full stacks** (both with Oban + all GenServers). You will **duplicate cron** (double emails, double payouts, corrupted state).
**Critical:** Never run **two full stacks** (`RELEASE_ROLE=all` / unset on multiple services). You will **duplicate cron** (double emails, double payouts, corrupted state).

---

### Step 8.2 — Redis: PubSub + Hammer + cache

**Goal:** All web nodes see the same PubSub messages; rate limits apply cluster-wide; hot caches are consistent.

**Add dependencies** (in `mix.exs`):
**Dependencies present** (in `mix.exs`):

- `phoenix_pubsub_redis` (or official Redis PubSub adapter for your Phoenix version)
- `hammer` Redis backend (e.g. `hammer_backend_redis` or Hammer 7 Redis integration)
- `phoenix_pubsub_redis`
- `hammer_backend_redis`
- `redix` for cache access

**Configure PubSub** (replace local adapter):
**Configure PubSub and Hammer** by setting `REDIS_URL` at runtime. With no `REDIS_URL`, the app keeps local PubSub and ETS Hammer for dev / single-box production. With `REDIS_URL`, `runtime.exs` configures both:

```elixir
# config/runtime.exs (prod) — illustrative
# config/runtime.exs
config :studentplug, Studentplug.PubSub,
adapter: Phoenix.PubSub.Redis,
url: System.get_env("REDIS_URL"),
node_name: System.get_env("RENDER_INSTANCE_ID") || "local"

config :hammer,
backend: {Hammer.Backend.Redis, [redis_url: System.get_env("REDIS_URL")]}
```

**Configure Hammer** — switch from `Hammer.Backend.ETS` in `config/config.exs` to Redis backend using `REDIS_URL`.
Render must use the **internal** Redis URL and the same value on every `web` and `worker` service. Check startup logs for `Redis configured (PubSub + Hammer)` after deploy.

**Migrate ETS caches** (priority order):

Expand All @@ -341,17 +349,12 @@ REDIS_URL=redis://... # Render internal Key Value URL

**Goal:** Critical user paths (ride accept, payment, booking) never wait behind weekly digest emails.

**Replace** in `config/config.exs`:
**Current `config/config.exs`:**

```elixir
# Today (insufficient):
queues: [default: 10]

# Powerhouse (example starting point):
queues: [
critical: 20,
notifications: 15,
realtime: 10,
critical: 10,
notifications: 10,
default: 10,
analytics: 5,
cleanup: 3
Expand All @@ -364,16 +367,17 @@ queues: [
|-------|----------|----------------------|
| `critical` | Ride state transitions, payment capture, booking confirm | Seconds |
| `notifications` | Email, FCM push fan-out | Minutes |
| `realtime` | Location batch persist, status sync helpers | Seconds |
| `default` | General domain work | Minutes |
| `analytics` | Marketing rollups, grocery intelligence | Hours |
| `cleanup` | Expiry, pruning | Hours |

Only Oban workers belong on these queues. Timer-based GenServers still run under `RELEASE_ROLE=worker` / `all`; migrate them to Oban where retries, uniqueness, and queue back-pressure matter.

**Migrate GenServer cron jobs to Oban workers** where feasible:

| Current GenServer (in `application.ex`) | Target |
|----------------------------------------|--------|
| `SurgePricingJob` | Oban worker, `realtime` or `critical` |
| `SurgePricingJob` | Oban worker, `critical` or `default` |
| `SafetyMonitoringJob` | Oban worker, `critical` |
| `ScheduledRidesJob` | Oban worker, `critical` |
| `RideExpirationJob` | Oban worker, `cleanup` |
Expand Down Expand Up @@ -418,7 +422,7 @@ config :studentplug, Studentplug.Repo,

**Finch:** ensure pooled HTTP to SendGrid, Mapbox, Cloudinary with reasonable pool sizes on workers.

**Endpoint:** enable `gzip: true` for static in production.
**Endpoint:** production config sets `gzip: true`; keep it enabled for static asset delivery.

---

Expand All @@ -437,8 +441,10 @@ Redis PubSub is **sufficient** for most horizontal scale; DNS cluster helps if y
### Phase 1 completion checklist

- [x] `RELEASE_ROLE` implemented and tested locally
- [ ] Redis PubSub live; channel test passes across two local nodes
- [ ] Hammer Redis live
- [x] Redis PubSub runtime wiring implemented; live only when `REDIS_URL` is set
- [x] Hammer Redis runtime wiring implemented; live only when `REDIS_URL` is set
- [ ] Redis service provisioned and `REDIS_URL` set on all Render web/worker services
- [ ] Cross-node channel test passes across two local or staging nodes
- [x] Oban queues segmented; critical jobs tagged (Oban workers only; GenServer cron unchanged)
- [ ] At least high-risk GenServers migrated to Oban workers
- [x] `POOL_SIZE` documented per role (runtime.exs + ReleaseRole moduledoc)
Expand Down Expand Up @@ -513,7 +519,7 @@ mix deps.get --only prod && mix compile && mix assets.deploy && MIX_ENV=prod mix
_build/prod/rel/studentplug/bin/server
```

8. Health check path: `/health` (returns DB + worker status on web—adjust after split so web health does not require GenServers).
8. Health check path: `/health`. In `RELEASE_ROLE=web`, `/health` checks DB + BEAM stats and omits worker GenServer status so Render does not fail web instances because workers live elsewhere.

---

Expand Down Expand Up @@ -629,7 +635,11 @@ Web + worker → Notifications → email/Slack: deploy failed, instance restarte

### Step 10.5 — `/health` endpoint

Already at `HealthController` — extend after worker split so web health checks DB + Endpoint only; worker health checks DB + Oban heartbeat.
Already at `HealthController`.

- `RELEASE_ROLE=web`: database + BEAM stats; worker status is skipped.
- `RELEASE_ROLE=worker` or `all`: database + BEAM stats + background GenServer status.
- HTTP status is `503` only when the database check fails. Stopped workers make the JSON `status` `"degraded"` while the response remains HTTP 200.

---

Expand Down
4 changes: 4 additions & 0 deletions docs/06-admin-system-operations/POST_DEPLOY_SMOKE_CHECKLIST.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@
---

- [ ] **1.** `GET /health` returns OK (open `https://studentjunxion.co.za/health`)
- [ ] Confirm `release_role` is expected:
- Single service today: `all` (or unset in Render, reported as `all`)
- Split deploy: web service reports `web`; worker service health/logs report `worker`
- [ ] If more than one service reports `all`, stop: duplicate Oban/cron workers may run.
- [ ] **2.** Login page loads on web (`https://studentjunxion.co.za`)
- [ ] **3.** Mobile app opens to home on test device (release or prod API)
- [ ] **4.** One critical journey today (rotate daily):
Expand Down