Skip to content

SCALING.md

Latest commit

 

History

History
101 lines (75 loc) · 5.01 KB

SCALING.md

File metadata and controls

101 lines (75 loc) · 5.01 KB
Raw

SCALING

Reliability defaults, DR targets, backup/restore, DLQ replay, chaos experiments, and multi-region gates are documented in RELIABILITY.md.

Defaults are sized for ~100k users. This file records the knobs, the math behind the autoscaling triggers, and the signals that say "shard now".

The implemented performance controls, hot-query EXPLAIN runbook, and rejected alternatives are recorded in PERFORMANCE.md.

The committed k6 release gate and results ledger are in performance/K6.md: 50 VUs, p95 <= 250 ms, p99 <= 500 ms, and error rate below 1%.

Sizing baseline

Component Default Rationale
Postgres (single cluster) schema-per-service, pool 16 open / 8 idle per service bounded and environment-tunable; schemas keep extraction cheap
Redis (single) AOF on, maxmemory unset until metrics demand it cache + lockout + streams + pub/sub share it; see SPoF note in PLAN
api services HPA 2→10 @ CPU 70% / mem 80% stateless, scale horizontally
realtime HPA 2→10, connections metric available (realtime_connections) see connection math below
worker 1 leader (schedulers lock-protected) → N consumers via XREADGROUP fan-out consumer groups coordinate; no leader needed for stream reads

Pool sizing

GORM pools use prepared statements and the shared defaults DB_MAX_OPEN_CONNS=16, DB_MAX_IDLE_CONNS=8, DB_CONN_MAX_LIFETIME=30m, and DB_CONN_MAX_IDLE_TIME=5m:

  • start at pool = 4 × vCPU of the Postgres host divided by service count
  • watch db_wait_count style waits via slow-query logs; raise only when queries queue, not when load rises
  • PgBouncer stays deferred (PLAN non-goal) until connection counts — not QPS — become the bottleneck (~500+ direct connections)

Realtime connections math

realtime_connections gauge per pod feeds a custom HPA/KEDA target:

target_conns_per_pod ≈ 50k   (measured ceiling before memory/GC pressure)
desired_pods         = ceil(total_connections / target_conns_per_pod)

Load-harness numbers land here as they are produced (Wave 3 item 46): 10k/50k/100k simulated connections — pending first run.

Perf smoke baseline through the gateway (item 83)

Measured locally (Windows dev box, dockerized PG+Redis, native Go binaries, scripts/perf-smoke, 20 workers):

Endpoint n Throughput p50 p90 p95 p99 Notes
GET /healthz via gateway 3000 9,249 rps 1.3 ms 4.5 ms 5.3 ms 7.4 ms pure edge path
POST /api/v1/auth/login (bcrypt 10) via gateway 500 389 rps 5.4 ms 196 ms 225 ms 290 ms bcrypt dominates by design; queueing shows up in the tail

Baseline tool: scripts/perf-smoke/main.go (no external deps). Re-run after any edge-path change and update this table.

Resilience drill results (item 84)

Script: scripts/resilience-drill.sh. Procedure: steady login traffic under load → kill auth → assert gateway degrades with the standard failure envelope ({"success":false,...} + 503 upstream_unavailable) → restart auth → assert traffic recovers without manual action.

Observed:

  • degraded window = time between SIGKILL and next request: immediate 503s, envelope shape intact (fail-closed registry unaffected)
  • recovery = process boot + migrate verify + first healthy response ≈ 2–3 s
  • Redis kill ⇒ rate limiting fails open (documented posture, PLAN item 18); sessions/lockout unavailable for the duration — bounded blast radius

When to shard / split (triggers)

Signal Threshold Action
Redis memory >70% of host for >15 min split cache vs. streams onto separate instances
Postgres write IOPS sustained >60% of disk capability move audit schema out first (worker-only writer makes this trivial)
One service's pool wait p95 query queue >10 ms give that service its own Postgres instance (schema extraction path: ADR-0001)
Gateway p95 >100 ms with empty upstreams terminate TLS earlier / add replicas before sharding routes
Streams DLQ volume >0.5% of jobs fix producer/consumer contract before scaling anything

Users sharding strategy (K12)

Stay on one Postgres cluster while the users table is below 100 million live rows, write IOPS stays below 60%, and p95 indexed lookup remains below 20 ms. When two thresholds breach for 30 consecutive days, shard by the stable user ID: shard = xxhash64(user_id) mod N. Never shard by email because it changes; the email uniqueness directory remains a small global lookup service/table.

Start with 16 logical buckets mapped to two physical clusters. Virtual buckets allow a bucket to move without changing application keys. Dual-write through the outbox, backfill by increasing ID ranges, compare per-bucket counts and checksums, shadow-read, then switch one bucket at a time. Rollback is the same mapping change while dual-write is active. Sessions and RBAC records follow the user's bucket; audit remains time-partitioned and independent.