diff --git a/docs/01-deployment-infrastructure/OBSERVABILITY_MANUAL.md b/docs/01-deployment-infrastructure/OBSERVABILITY_MANUAL.md index 7dce70cb..c0172f99 100644 --- a/docs/01-deployment-infrastructure/OBSERVABILITY_MANUAL.md +++ b/docs/01-deployment-infrastructure/OBSERVABILITY_MANUAL.md @@ -52,7 +52,7 @@ Once configured, you will have visibility into: - **LiveView**: Mount times, handle_event durations - **Business KPIs**: Active rides, pending rides, users, marketplace listings, bookings, study sessions, payment processing - **Distributed Traces**: Full request waterfall from HTTP through controllers, plugs, Ecto queries -- **Health Check**: App status, database connectivity, all 12 background worker statuses +- **Health Check**: App status, `RELEASE_ROLE`, database connectivity, BEAM stats, and worker status when the node runs background processes --- @@ -90,7 +90,7 @@ Once configured, you will have visibility into: 2. **Traces** go directly from your app to Grafana Cloud Tempo via OTLP HTTP (no extra infrastructure) 3. **Dashboards** are auto-uploaded to Grafana Cloud by PromEx when the app starts (no extra infrastructure) 4. **Logs** are structured JSON written to stdout, visible in Render's log viewer -5. **Health check** at `/health` returns app status, BEAM stats, and worker health +5. **Health check** at `/health` returns app status, `RELEASE_ROLE`, BEAM stats, and role-specific worker health --- @@ -297,7 +297,7 @@ Open in your browser: https://studentjunxion.co.za/health ``` -You should see JSON with app status, BEAM stats, database status, and worker health. +You should see JSON with app status, `release_role`, BEAM stats, and database status. Worker status appears only on `RELEASE_ROLE=all` and `RELEASE_ROLE=worker` nodes. --- @@ -657,10 +657,43 @@ rate(studentplug_business_ride_requests_total[5m]) * 60 `GET /health` is always available without authentication. -**Response when healthy (HTTP 200):** +The response shape depends on `RELEASE_ROLE`: + +| Role | Health checks | `workers` key | +|------|---------------|---------------| +| unset / `all` | Database, BEAM stats, background GenServers | Present | +| `web` | Database, BEAM stats | Omitted | +| `worker` | Database, BEAM stats, background GenServers | Present | + +HTTP status is `503` only when the database check fails. If the database is reachable but a worker GenServer is stopped, the JSON `status` is `"degraded"` and the response remains HTTP 200 so operators can distinguish a node-level problem from total outage. + +**Web response when healthy (HTTP 200):** +```json +{ + "status": "ok", + "release_role": "web", + "version": "0.1.1", + "database": "ok", + "beam": { + "uptime_seconds": 86400, + "process_count": 1234, + "process_limit": 262144, + "memory_total_mb": 256.5, + "memory_processes_mb": 128.3, + "memory_ets_mb": 45.2, + "atom_count": 12345, + "schedulers_online": 4, + "otp_release": "25" + }, + "timestamp": "2026-03-13T12:00:00Z" +} +``` + +**Worker or single-service response when healthy (HTTP 200):** ```json { "status": "ok", + "release_role": "worker", "version": "0.1.1", "database": "ok", "beam": { @@ -683,18 +716,24 @@ rate(studentplug_business_ride_requests_total[5m]) * 60 "SafetyMonitoringJob": "running", "ScheduledRidesJob": "running", "DriverPayoutJob": "running", + "PartnerPayoutJob": "running", + "ReserveReleaseJob": "running", "QualityControlJob": "running", "SharedRideJob": "running", "RideExpirationJob": "running", - "DailyViewsEmailJob": "running" + "JobExpirationJob": "running", + "LocationStaleJob": "running", + "PayoutJob": "running", + "DailyViewsEmailJob": "running", + "StudyDigestJob": "running" }, "timestamp": "2026-03-13T12:00:00Z" } ``` **Status values:** -- `ok` -- database reachable, all workers running -- `degraded` -- database reachable, but some workers stopped +- `ok` -- database reachable, and all expected workers running for `all` / `worker` roles +- `degraded` -- database reachable, but some workers stopped on an `all` / `worker` node - `unhealthy` -- database unreachable (returns HTTP 503) Use this for: diff --git a/docs/01-deployment-infrastructure/SCALING_POWERHOUSE.md b/docs/01-deployment-infrastructure/SCALING_POWERHOUSE.md index 7e1dc90e..0d9b36af 100644 --- a/docs/01-deployment-infrastructure/SCALING_POWERHOUSE.md +++ b/docs/01-deployment-infrastructure/SCALING_POWERHOUSE.md @@ -87,16 +87,16 @@ Elixir on the BEAM gives you capabilities other stacks need Kafka + Redis + Side | **Render workspace** | **Professional (Pro)** — **not** Hobby | Already paid; enables autoscale/preview when you add services | | **Team members** | **2** on workspace | Billed per seat on Professional plan | | **Build minutes** | **Overage charges** apply | Frequent deploys (multiple per day) consume included minutes; extra billed per Render rates | -| Redis | **None** | Cannot scale web horizontally safely | -| `RELEASE_ROLE` | **Implemented** (`all` default; `web` / `worker` for split deploy) | Safe to add worker service when budget approved | -| Oban | **Single `default: 10` queue** | Critical jobs compete with analytics digests | -| Hammer | **ETS backend** | Per-node only | -| PubSub | **Default (local)** | Breaks multi-node channels | +| Redis | **No production service yet**; code switches on `REDIS_URL` | Cannot scale web horizontally safely until Redis is provisioned and set on every node | +| `RELEASE_ROLE` | **Implemented** (`unset`/`all` for single service; `web` / `worker` for split deploy) | Safe to add worker service when budget approved; do not run duplicate `all` services | +| Oban | **Segmented queues** (`critical`, `notifications`, `default`, `analytics`, `cleanup`) | Still competes with HTTP until `RELEASE_ROLE=worker` services process jobs off-web | +| Hammer | **ETS by default; Redis when `REDIS_URL` is set** | Per-node only until Redis is live on every web node | +| PubSub | **Local by default; Redis adapter when `REDIS_URL` is set** | Multi-node channels require Redis before web autoscale | | `DNS_CLUSTER_QUERY` | Likely unset | Clustering inactive | | OpenTelemetry | **Disabled** (`otel_enabled: false`) | Blind to p95/p99 under load | | Grafana | Was configured; **broken after rebrand** | No dashboards/alerts | | `UserSocket.id/1` | Returns `nil` | No Presence; no per-socket tracking | -| Static assets | `gzip: false` in endpoint | Slower transfers | +| Static assets | Production endpoint config has `gzip: true` | Keep enabled for marketing/app asset delivery | **Root cause of crashes:** not “too many users” yet—**too many subsystems on one 512MB node** (`Studentplug.Application` starts Endpoint, Oban, AccountCleanup, AdminNotificationJob, 7× ride jobs, 3× delivery jobs, marketplace/study jobs, PromEx, etc.). @@ -270,16 +270,21 @@ What you pay **today** before powerhouse service upgrades (web/worker/redis/Post **Goal:** Web nodes never run Oban or cron GenServers; worker nodes never open HTTP port. -**Implement in** `studentplug/lib/studentplug/application.ex`: +**Implemented in** `studentplug/lib/studentplug/release_role.ex` and wired from `studentplug/lib/studentplug/application.ex`: | `RELEASE_ROLE` | Start | Do not start | |----------------|-------|--------------| -| `web` (default) | `StudentplugWeb.Endpoint`, `Repo`, `Phoenix.PubSub`, `Finch`, `Telemetry`, `PromEx`, `ReleaseMigrator` (web only if needed) | `Oban`, `AccountCleanup`, `AdminNotificationJob`, all `Rides.Jobs.*`, all `Delivery.Jobs.*`, `MarketingDataSync`, `StudyDigestJob`, `DailyViewsEmailJob`, etc. | -| `worker` | `Repo`, `Oban`, all background GenServers, `Finch`, `Telemetry` | `StudentplugWeb.Endpoint` | +| unset / `all` | Shared children + web children + worker children | Nothing — current single Render service behaviour | +| `web` | `StudentplugWeb.Endpoint`, `Repo`, `Phoenix.PubSub`, `Finch`, `Telemetry`, `PromEx`, `ReleaseMigrator`, `MetricsPusher` | `Oban`, `AccountCleanup`, `AdminNotificationJob`, all `Rides.Jobs.*`, all `Delivery.Jobs.*`, `MarketingDataSync`, `StudyDigestJob`, `DailyViewsEmailJob`, etc. | +| `worker` | `Repo`, `Oban`, all background GenServers, `Finch`, `Telemetry`, `Phoenix.PubSub` | `StudentplugWeb.Endpoint`, `PromEx`, `ReleaseMigrator`, `MetricsPusher` | **Env vars** ```bash +# Current single-service production default +RELEASE_ROLE=all # or leave unset +PHX_SERVER=true + # Web service (Render) RELEASE_ROLE=web PHX_SERVER=true @@ -293,7 +298,7 @@ POOL_SIZE=20 **Why this matters for Elixir:** the BEAM can supervise thousands of processes—but **not** when one VM is memory-starved. Isolating HTTP from batch work is OTP 101. -**Critical:** Never run **two full stacks** (both with Oban + all GenServers). You will **duplicate cron** (double emails, double payouts, corrupted state). +**Critical:** Never run **two full stacks** (`RELEASE_ROLE=all` / unset on multiple services). You will **duplicate cron** (double emails, double payouts, corrupted state). --- @@ -301,23 +306,26 @@ POOL_SIZE=20 **Goal:** All web nodes see the same PubSub messages; rate limits apply cluster-wide; hot caches are consistent. -**Add dependencies** (in `mix.exs`): +**Dependencies present** (in `mix.exs`): -- `phoenix_pubsub_redis` (or official Redis PubSub adapter for your Phoenix version) -- `hammer` Redis backend (e.g. `hammer_backend_redis` or Hammer 7 Redis integration) +- `phoenix_pubsub_redis` +- `hammer_backend_redis` - `redix` for cache access -**Configure PubSub** (replace local adapter): +**Configure PubSub and Hammer** by setting `REDIS_URL` at runtime. With no `REDIS_URL`, the app keeps local PubSub and ETS Hammer for dev / single-box production. With `REDIS_URL`, `runtime.exs` configures both: ```elixir -# config/runtime.exs (prod) — illustrative +# config/runtime.exs config :studentplug, Studentplug.PubSub, adapter: Phoenix.PubSub.Redis, url: System.get_env("REDIS_URL"), node_name: System.get_env("RENDER_INSTANCE_ID") || "local" + +config :hammer, + backend: {Hammer.Backend.Redis, [redis_url: System.get_env("REDIS_URL")]} ``` -**Configure Hammer** — switch from `Hammer.Backend.ETS` in `config/config.exs` to Redis backend using `REDIS_URL`. +Render must use the **internal** Redis URL and the same value on every `web` and `worker` service. Check startup logs for `Redis configured (PubSub + Hammer)` after deploy. **Migrate ETS caches** (priority order): @@ -341,17 +349,12 @@ REDIS_URL=redis://... # Render internal Key Value URL **Goal:** Critical user paths (ride accept, payment, booking) never wait behind weekly digest emails. -**Replace** in `config/config.exs`: +**Current `config/config.exs`:** ```elixir -# Today (insufficient): -queues: [default: 10] - -# Powerhouse (example starting point): queues: [ - critical: 20, - notifications: 15, - realtime: 10, + critical: 10, + notifications: 10, default: 10, analytics: 5, cleanup: 3 @@ -364,16 +367,17 @@ queues: [ |-------|----------|----------------------| | `critical` | Ride state transitions, payment capture, booking confirm | Seconds | | `notifications` | Email, FCM push fan-out | Minutes | -| `realtime` | Location batch persist, status sync helpers | Seconds | | `default` | General domain work | Minutes | | `analytics` | Marketing rollups, grocery intelligence | Hours | | `cleanup` | Expiry, pruning | Hours | +Only Oban workers belong on these queues. Timer-based GenServers still run under `RELEASE_ROLE=worker` / `all`; migrate them to Oban where retries, uniqueness, and queue back-pressure matter. + **Migrate GenServer cron jobs to Oban workers** where feasible: | Current GenServer (in `application.ex`) | Target | |----------------------------------------|--------| -| `SurgePricingJob` | Oban worker, `realtime` or `critical` | +| `SurgePricingJob` | Oban worker, `critical` or `default` | | `SafetyMonitoringJob` | Oban worker, `critical` | | `ScheduledRidesJob` | Oban worker, `critical` | | `RideExpirationJob` | Oban worker, `cleanup` | @@ -418,7 +422,7 @@ config :studentplug, Studentplug.Repo, **Finch:** ensure pooled HTTP to SendGrid, Mapbox, Cloudinary with reasonable pool sizes on workers. -**Endpoint:** enable `gzip: true` for static in production. +**Endpoint:** production config sets `gzip: true`; keep it enabled for static asset delivery. --- @@ -437,8 +441,10 @@ Redis PubSub is **sufficient** for most horizontal scale; DNS cluster helps if y ### Phase 1 completion checklist - [x] `RELEASE_ROLE` implemented and tested locally -- [ ] Redis PubSub live; channel test passes across two local nodes -- [ ] Hammer Redis live +- [x] Redis PubSub runtime wiring implemented; live only when `REDIS_URL` is set +- [x] Hammer Redis runtime wiring implemented; live only when `REDIS_URL` is set +- [ ] Redis service provisioned and `REDIS_URL` set on all Render web/worker services +- [ ] Cross-node channel test passes across two local or staging nodes - [x] Oban queues segmented; critical jobs tagged (Oban workers only; GenServer cron unchanged) - [ ] At least high-risk GenServers migrated to Oban workers - [x] `POOL_SIZE` documented per role (runtime.exs + ReleaseRole moduledoc) @@ -513,7 +519,7 @@ mix deps.get --only prod && mix compile && mix assets.deploy && MIX_ENV=prod mix _build/prod/rel/studentplug/bin/server ``` -8. Health check path: `/health` (returns DB + worker status on web—adjust after split so web health does not require GenServers). +8. Health check path: `/health`. In `RELEASE_ROLE=web`, `/health` checks DB + BEAM stats and omits worker GenServer status so Render does not fail web instances because workers live elsewhere. --- @@ -629,7 +635,11 @@ Web + worker → Notifications → email/Slack: deploy failed, instance restarte ### Step 10.5 — `/health` endpoint -Already at `HealthController` — extend after worker split so web health checks DB + Endpoint only; worker health checks DB + Oban heartbeat. +Already at `HealthController`. + +- `RELEASE_ROLE=web`: database + BEAM stats; worker status is skipped. +- `RELEASE_ROLE=worker` or `all`: database + BEAM stats + background GenServer status. +- HTTP status is `503` only when the database check fails. Stopped workers make the JSON `status` `"degraded"` while the response remains HTTP 200. --- diff --git a/docs/06-admin-system-operations/POST_DEPLOY_SMOKE_CHECKLIST.md b/docs/06-admin-system-operations/POST_DEPLOY_SMOKE_CHECKLIST.md index cf17ebe6..3dc70151 100644 --- a/docs/06-admin-system-operations/POST_DEPLOY_SMOKE_CHECKLIST.md +++ b/docs/06-admin-system-operations/POST_DEPLOY_SMOKE_CHECKLIST.md @@ -11,6 +11,10 @@ --- - [ ] **1.** `GET /health` returns OK (open `https://studentjunxion.co.za/health`) + - [ ] Confirm `release_role` is expected: + - Single service today: `all` (or unset in Render, reported as `all`) + - Split deploy: web service reports `web`; worker service health/logs report `worker` + - [ ] If more than one service reports `all`, stop: duplicate Oban/cron workers may run. - [ ] **2.** Login page loads on web (`https://studentjunxion.co.za`) - [ ] **3.** Mobile app opens to home on test device (release or prod API) - [ ] **4.** One critical journey today (rotate daily):