diff --git a/docs/loadtest-runbook.md b/docs/loadtest-runbook.md new file mode 100644 index 0000000..e675c33 --- /dev/null +++ b/docs/loadtest-runbook.md @@ -0,0 +1,110 @@ +# 1000-player load run — runbook + +Everything below is staged and ready. On "go" it runs top to bottom. + +Target: `https://gajendra.fsn.frappe.cloud` (private FC bench). +Generator: 1–2 DoppioBoxes on `test-01` (Hetzner FSN, same region as the target, +so measured latency is server time rather than network). + +## 0. Prerequisites (one-time, before "go") + +- [ ] `join_session` rate limit lifted on the target — **and a reminder set to restore it** +- [ ] global `frappe.conf.rate_limit` checked: `grep -i rate_limit sites//site_config.json sites/common_site_config.json` + (site-wide, not per-IP — if set, a 1000-wide salvo trips it and every bot gets a bare 429) +- [ ] a quiz with **8+ questions** seeded, so the ramp has enough salvos to show a trend +- [ ] devbox slugs agreed and provisioned +- [ ] host API key + secret issued (optional — without it you drive the host screen) + +## 1. Bootstrap the generator box + +```bash +git clone https://github.com/bwhtech/quizzly && cd quizzly +npm i --no-save undici socket.io-client +ulimit -n 8192 # each bot holds a websocket + an http connection +``` + +## 2. Pre-flight + +```bash +QZ_ORIGIN=https://gajendra.fsn.frappe.cloud QZ_PIN= \ + scripts/loadtest_preflight.sh 1000 +``` + +Checks clock skew (a snapshot-cloned microVM fails every TLS handshake with +"certificate is not yet valid", which reads exactly like a dead target), node +version, `ulimit -n`, deps, then runs a 2-bot smoke and cleans it up. +**Do not ramp until this is clean.** + +## 3. Ramp + +Self-hosting — the generator creates each session, seats the bots, starts the +game and plays to the podium, no clicking: + +```bash +QZ_QUIZ="General Knowledge" QZ_API_KEY=... QZ_API_SECRET=... \ + scripts/loadtest_ramp.sh +``` + +Manual host — you open a fresh lobby per stage, the script asks for the pin: + +```bash +scripts/loadtest_ramp.sh +``` + +Stages default to `100 250 500 1000` (`QZ_STAGES` to change), 60s cooldown +between them, one JSON report per stage in `./loadtest-reports/`. The ramp +**stops at the first degraded stage** rather than piling failure on failure. + +Splitting across two boxes: run 500 on each with `QZ_STAGES=500` against the +same pin. If two-box 500+500 beats one-box 1000, the generator was the +bottleneck, not the server. + +## 4. Watch, server-side + +The client numbers say *when* it broke; these say *what* broke. + +- gunicorn worker saturation and request queue depth +- the socket.io node process — CPU and RSS during the podium burst +- MariaDB slow log, and lock waits on `QZ Answer` inserts +- RQ: whether the shared ticker stays on schedule or drifts + +## 5. Pass / fail, decided before the run + +| Measure | Threshold at 1000 | Why | +|---|---|---| +| players seated / sockets live | 100% | anything less and the rest is unreadable | +| question delivery skew p99 | **< 500 ms** | scoring is `(1 - (response_ms/window_ms)/2) * 1000`; on a 20s window 500ms costs ~12 of 1000 points, 2s costs ~50 — 500ms is the edge of fair | +| `submit_answer` p99 | < 1000 ms | beyond this the countdown on screen is lying | +| dropped answers | 0 | excludes legitimate "Already answered" | +| podium delivered | 100%, p99 < 2s | see the prediction below | +| HTTP 429 | 0 | any means a limiter is still in the path | + +## 6. The prediction to confirm or kill + +`engine.py:403` broadcasts the podium carrying the **full leaderboard** to the +whole room. At 1000 players that is ~80KB × 1000 sockets ≈ **80MB pushed from a +single node process in one burst**, immediately followed by 1000 `get_state` +calls that each return the same leaderboard again over HTTP. + +My bet is the first failure is here, not in the submit salvo the existing +`scripts/loadtest.py` measures. The generator reports podium payload size, +fan-out total and delivery skew specifically to settle this. If confirmed, the +fix is small: broadcast top-N only, and let players fetch their own placement. + +## 7. Abort + +Stop if the target starts serving 5xx to real traffic, or FC's proxy begins +rate-limiting site-wide. `ctrl-c` on the ramp; sessions in flight can be ended +from the host screen. + +## 8. Cleanup + +```bash +# dry run first +echo 'exec(open("apps/quizzly/scripts/loadtest_cleanup.py").read())' | bench --site console +# then +QZ_CLEANUP_APPLY=1 bash -c 'echo "exec(open(\"apps/quizzly/scripts/loadtest_cleanup.py\").read())" | bench --site console' +``` + +Deletes bot participants and their answers, and drops only sessions where +*every* player was synthetic. Then: **restore the `join_session` rate limit.** diff --git a/quizzly/api.py b/quizzly/api.py index cc6f69d..dc2ca36 100644 --- a/quizzly/api.py +++ b/quizzly/api.py @@ -64,6 +64,9 @@ def get_host_state(session: str | None = None) -> dict: session_doc = get_host_session(session) if session else get_live_host_session() if not session_doc or session_doc.status == "Cancelled": return {} + # the host screen is the most frequent caller, so revive a dead ticker here + # too: a game whose loop was killed resumes within a poll, not a scheduler minute + engine.ensure_ticker_running() if engine.is_abandoned(session_doc): # the host reloaded into a game whose worker is gone: settle it and show the podium engine.finish_session(session_doc) diff --git a/quizzly/engine.py b/quizzly/engine.py index 7403671..af0e7d0 100644 --- a/quizzly/engine.py +++ b/quizzly/engine.py @@ -37,16 +37,36 @@ def enqueue_game_loop(session_doc) -> None: clear_control(session_doc.name) get_ready(session_doc, questions[0], 0, len(questions)) frappe.cache.sadd(ACTIVE_SESSIONS_KEY, session_doc.name) + # after_commit so the ticker only starts once the get_ready state and the + # active-set membership it reads are actually visible + enqueue_ticker(after_commit=True) + + +def enqueue_ticker(after_commit: bool = False) -> None: frappe.enqueue( "quizzly.engine.run_ticker", queue="long", timeout=TICKER_TIMEOUT, job_id="qz_ticker", deduplicate=True, - enqueue_after_commit=True, + enqueue_after_commit=after_commit, ) +def ensure_ticker_running() -> None: + """Bring the shared ticker back if it died with games still live. + + Every game is driven by this one loop. If its worker is lost mid-game — an OOM + under a large answer flood, a deploy or worker restart — nothing advances the + games and they freeze exactly where they stood, with no exception to log. A + deduplicated re-enqueue is a no-op while the loop is alive, and otherwise starts + a fresh one that resumes every game from the Redis state it left behind. Driven + from the scheduler and the host poll so recovery never needs a human. + """ + if active_sessions(): + enqueue_ticker() + + def run_ticker() -> None: """One shared self-looping job. Advances every active session on time or host command.""" # process-local: the ticker is a single deduplicated job, so per-session throttle diff --git a/quizzly/hooks.py b/quizzly/hooks.py index 1257187..e9e44bd 100644 --- a/quizzly/hooks.py +++ b/quizzly/hooks.py @@ -29,3 +29,14 @@ export_python_type_annotations = True require_type_annotated_api_methods = True + +# The live-game loop is one shared background job; if its worker dies mid-game +# nothing else advances the games. This re-enqueues it (deduplicated, so a live +# loop is untouched) so a killed ticker recovers without a human. +scheduler_events = { + "cron": { + "* * * * *": [ + "quizzly.engine.ensure_ticker_running", + ] + } +} diff --git a/quizzly/tests/test_engine.py b/quizzly/tests/test_engine.py index c564f0c..47bd3f1 100644 --- a/quizzly/tests/test_engine.py +++ b/quizzly/tests/test_engine.py @@ -521,3 +521,27 @@ def test_question_without_explanation_skips_the_screen(self): with patch("frappe.publish_realtime"), patch("frappe.db.commit"): engine.close_question(self.session_doc, question, 0, len(self.questions)) self.assertEqual(engine.get_state(self.session)["phase"], "stats") + + +class TestTickerRecovery(IntegrationTestCase): + """The shared ticker can be killed mid-game (OOM under a large answer flood, a + worker restart). Nothing else advances a game, so it must be revivable.""" + + def setUp(self): + frappe.cache.delete_value(engine.ACTIVE_SESSIONS_KEY) + + def tearDown(self): + frappe.cache.delete_value(engine.ACTIVE_SESSIONS_KEY) + + def test_ensure_ticker_running_revives_when_a_game_is_live(self): + frappe.cache.sadd(engine.ACTIVE_SESSIONS_KEY, "some-session") + with patch("frappe.enqueue") as enqueue: + engine.ensure_ticker_running() + enqueue.assert_called_once() + self.assertEqual(enqueue.call_args.kwargs["job_id"], "qz_ticker") + self.assertTrue(enqueue.call_args.kwargs["deduplicate"]) + + def test_ensure_ticker_running_is_a_noop_with_no_live_games(self): + with patch("frappe.enqueue") as enqueue: + engine.ensure_ticker_running() + enqueue.assert_not_called() diff --git a/scripts/loadtest_cleanup.py b/scripts/loadtest_cleanup.py new file mode 100644 index 0000000..71d3938 --- /dev/null +++ b/scripts/loadtest_cleanup.py @@ -0,0 +1,64 @@ +# Removes what a load run leaves behind: bot participants, their answers, and any +# session that ends up with no real players. Run it on the bench host after a ramp. +# +# IPython's autoindent mangles indented blocks pasted into `bench console`, so this +# has to be exec'd from the file rather than piped in as source: +# echo 'exec(open("apps/quizzly/scripts/loadtest_cleanup.py").read())' \ +# | bench --site SITE console +# +# env: +# QZ_CLEANUP_PREFIXES comma-separated nickname prefixes (default bot,probe,preflight,tmp) +# QZ_CLEANUP_APPLY 1 to actually delete; anything else only reports + +import os + +import frappe + +prefixes = os.environ.get("QZ_CLEANUP_PREFIXES", "bot,probe,preflight,tmp").split(",") +apply_changes = os.environ.get("QZ_CLEANUP_APPLY") == "1" + +participants = [] +for prefix in prefixes: + participants += frappe.get_all( + "QZ Participant", + filters={"nickname": ["like", f"{prefix}%"]}, + fields=["name", "session", "nickname"], + ) + +if not participants: + print("nothing to clean") +else: + bot_sessions = {row.session for row in participants} + bot_names = {row.name for row in participants} + + # a session is only disposable when every player in it was synthetic + real_counts = frappe.get_all( + "QZ Participant", + filters={"session": ["in", list(bot_sessions)]}, + fields=["session", "count(name) as total"], + group_by="session", + ) + total_by_session = {row.session: row.total for row in real_counts} + bots_by_session = {} + for row in participants: + bots_by_session[row.session] = bots_by_session.get(row.session, 0) + 1 + + disposable = [s for s in bot_sessions if bots_by_session[s] == total_by_session.get(s)] + mixed = sorted(bot_sessions - set(disposable)) + + answers = frappe.db.count("QZ Answer", {"participant": ["in", list(bot_names)]}) + print(f"bot participants: {len(bot_names)} answers: {answers}") + print(f"sessions fully synthetic: {len(disposable)} sessions with real players too: {len(mixed)}") + for session in mixed: + print(f" keeping session {session} ({bots_by_session[session]}/{total_by_session[session]} synthetic)") + + if not apply_changes: + print("\ndry run — set QZ_CLEANUP_APPLY=1 to delete") + else: + frappe.db.delete("QZ Answer", {"participant": ["in", list(bot_names)]}) + frappe.db.delete("QZ Participant", {"name": ["in", list(bot_names)]}) + for session in disposable: + frappe.cache.srem("qz:active_sessions", session) + frappe.delete_doc("QZ Session", session, force=True, ignore_permissions=True) + frappe.db.commit() + print(f"\ndeleted {len(bot_names)} participants, {answers} answers, {len(disposable)} sessions") diff --git a/scripts/loadtest_live.mjs b/scripts/loadtest_live.mjs new file mode 100644 index 0000000..0bb9498 --- /dev/null +++ b/scripts/loadtest_live.mjs @@ -0,0 +1,336 @@ +#!/usr/bin/env node +// Live-fire load generator: N guest players against a running session, over the +// same three transports a real phone uses — HTTP join, a socket.io room +// subscription, and one submit_answer per question. +// +// QZ_PIN=010805 QZ_PLAYERS=5 node scripts/loadtest_live.mjs +// +// Unlike scripts/loadtest.py this needs no bench access and holds a real socket +// per player, so it measures the two things that HTTP-only driving cannot: how +// far apart a question lands across every device, and the end-of-game fan-out. +// +// env: +// QZ_ORIGIN target site (default https://gajendra.fsn.frappe.cloud) +// QZ_PIN game pin (required unless QZ_QUIZ + api key are set) +// QZ_QUIZ quiz name; with QZ_API_KEY/SECRET the run hosts itself: +// create session -> seat bots -> start -> play to the podium +// QZ_PLAYERS bot count (default 5) +// QZ_NAME_PREFIX nickname prefix (default bot) +// QZ_JOIN_RATE joins per minute, 0=unthrottled (default 0) +// QZ_ANSWER_DELAY ms to wait before submitting; 0 = worst-case herd (default 0) +// QZ_TIMEOUT overall budget seconds (default 900) +// QZ_LEAVE 1 = leave_session on exit, only usable while still in the +// lobby and capped by its own 10/60s per-IP limit (default 0) +// QZ_API_KEY host api key (only for self-hosted runs) +// QZ_API_SECRET host api secret (only for self-hosted runs) +// QZ_OUT write the raw report here as JSON (optional) + +import { Agent, request } from "undici"; +import { io } from "socket.io-client"; + +const ORIGIN = process.env.QZ_ORIGIN || "https://gajendra.fsn.frappe.cloud"; +let PIN = process.env.QZ_PIN; +const QUIZ = process.env.QZ_QUIZ; +const API_KEY = process.env.QZ_API_KEY; +const API_SECRET = process.env.QZ_API_SECRET; +const HOSTED = Boolean(QUIZ && API_KEY && API_SECRET); +const PLAYERS = Number(process.env.QZ_PLAYERS || 5); +const NAME_PREFIX = process.env.QZ_NAME_PREFIX || "bot"; +const JOIN_RATE = Number(process.env.QZ_JOIN_RATE || 0); +const ANSWER_DELAY = Number(process.env.QZ_ANSWER_DELAY || 0); +const TIMEOUT_MS = Number(process.env.QZ_TIMEOUT || 900) * 1000; +const LEAVE = process.env.QZ_LEAVE === "1"; +const OUT = process.env.QZ_OUT; + +if (!PIN && !HOSTED) { + console.error("set QZ_PIN, or QZ_QUIZ + QZ_API_KEY + QZ_API_SECRET to host the run"); + process.exit(2); +} + +const SITE = new URL(ORIGIN).hostname; +const API = `${ORIGIN}/api/method/quizzly.api`; +let ROOM_EVENT; + +// One pool wide enough that a 1000-player salvo is never queued client-side; a +// queued request would be charged to the server as latency it did not spend. +const pool = new Agent({ connections: PLAYERS + 16, pipelining: 1 }); + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +async function callApi(method, params, httpMethod = "POST", asHost = false) { + const started = Date.now(); + const isPost = httpMethod === "POST"; + const url = isPost + ? `${API}.${method}` + : `${API}.${method}?${new URLSearchParams(params)}`; + try { + const response = await request(url, { + method: httpMethod, + dispatcher: pool, + headers: { + "content-type": "application/json", + accept: "application/json", + ...(asHost + ? { authorization: `token ${API_KEY}:${API_SECRET}` } + : { cookie: "sid=Guest" }), + }, + body: isPost ? JSON.stringify(params) : undefined, + headersTimeout: 60000, + bodyTimeout: 60000, + }); + const text = await response.body.text(); + const ms = Date.now() - started; + if (response.statusCode !== 200) { + return { ms, error: `http_${response.statusCode}`, detail: serverMessage(text) }; + } + return { ms, data: JSON.parse(text).message }; + } catch (error) { + return { ms: Date.now() - started, error: error.code || error.name }; + } +} + +// frappe hides thrown messages inside a JSON-encoded _server_messages list +function serverMessage(text) { + try { + const messages = JSON.parse(JSON.parse(text)._server_messages || "[]"); + return messages.map((m) => JSON.parse(m).message).join("; ").slice(0, 120); + } catch { + return text.slice(0, 120); + } +} + +function percentile(sorted, p) { + if (!sorted.length) return 0; + const index = Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1); + return sorted[index]; +} + +function summarise(values) { + const sorted = [...values].sort((a, b) => a - b); + return { + n: sorted.length, + p50: Math.round(percentile(sorted, 50)), + p95: Math.round(percentile(sorted, 95)), + p99: Math.round(percentile(sorted, 99)), + max: Math.round(sorted.at(-1) || 0), + }; +} + +function tally(items) { + const counts = {}; + for (const item of items) counts[item] = (counts[item] || 0) + 1; + return counts; +} + +async function joinAll() { + const players = []; + const failures = []; + const latencies = []; + const gap = JOIN_RATE > 0 ? 60000 / JOIN_RATE : 0; + + const attempts = Array.from({ length: PLAYERS }, (_, index) => async () => { + if (gap) await sleep(index * gap); + const nickname = `${NAME_PREFIX}${String(index).padStart(4, "0")}`; + const result = await callApi("join_session", { pin: PIN, nickname }); + latencies.push(result.ms); + if (result.error) { + failures.push(`${result.error}${result.detail ? `: ${result.detail}` : ""}`); + return; + } + players.push({ index, nickname, token: result.data.participant_token, events: [], submits: [] }); + }); + + await Promise.all(attempts.map((run) => run())); + return { players, failures, latencies }; +} + +function connect(player, onEvent) { + return new Promise((resolve) => { + // websocket-only: the real client starts on polling and upgrades, but a + // generator that does the same doubles the handshake cost per player and + // charges it to the server. Note this when reading connect timings. + const socket = io(`${ORIGIN}/${SITE}`, { + transports: ["websocket"], + extraHeaders: { Origin: ORIGIN, Cookie: "sid=Guest" }, + reconnection: true, + }); + const started = Date.now(); + player.socket = socket; + + socket.on("connect", () => { + socket.emit("qz_join", PIN); + if (player.connectMs === undefined) { + player.connectMs = Date.now() - started; + resolve(true); + } + }); + socket.on(ROOM_EVENT, (message) => onEvent(player, message)); + socket.on("connect_error", (error) => { + if (player.connectMs === undefined) { + player.connectError = error.message; + player.connectMs = Date.now() - started; + resolve(false); + } + }); + setTimeout(() => { + if (player.connectMs === undefined) { + player.connectError = "timeout"; + resolve(false); + } + }, 30000); + }); +} + +async function main() { + let session; + if (HOSTED) { + console.log(`==> creating session for quiz ${QUIZ}`); + const created = await callApi("create_session", { quiz: QUIZ }, "POST", true); + if (created.error) { + console.error(` could not create session: ${created.error} ${created.detail || ""}`); + return 2; + } + session = created.data.session; + PIN = created.data.game_pin; + // without auto-advance the ticker waits on a host click that never comes + await callApi("set_auto_advance", { session, enabled: 1 }, "POST", true); + console.log(` session=${session} pin=${PIN}`); + } + ROOM_EVENT = `qz_session_${PIN}`; + + console.log(`target=${ORIGIN} pin=${PIN} players=${PLAYERS} answer_delay=${ANSWER_DELAY}ms\n`); + + console.log("==> joining"); + const { players, failures, latencies } = await joinAll(); + console.log(` seated ${players.length}/${PLAYERS} join latency ${JSON.stringify(summarise(latencies))}`); + if (failures.length) console.log(` join failures: ${JSON.stringify(tally(failures))}`); + if (!players.length) { + console.error("nobody got in, aborting"); + return 2; + } + + const questions = new Map(); // question_row -> {arrivals: [], submits: []} + let podium = null; + let finished; + const done = new Promise((resolve) => { + finished = resolve; + }); + + function onEvent(player, message) { + const at = Date.now(); + player.events.push({ at, type: message.type }); + + if (message.type === "question") { + const row = message.question_row; + if (!questions.has(row)) { + questions.set(row, { index: message.q_index, arrivals: [], submits: [], errors: [] }); + } + const bucket = questions.get(row); + bucket.arrivals.push(at); + if (!player.answered?.has(row)) { + (player.answered ??= new Set()).add(row); + submit(player, row, bucket); + } + } + + if (message.type === "podium" && !podium) { + podium = { at, bytes: JSON.stringify(message).length, arrivals: [] }; + } + if (message.type === "podium") podium.arrivals.push(at); + } + + async function submit(player, row, bucket) { + if (ANSWER_DELAY) await sleep(Math.random() * ANSWER_DELAY); + const option = String(1 + (player.index % 4)); + const result = await callApi("submit_answer", { + pin: PIN, + token: player.token, + question_row: row, + selected_option: option, + }); + if (result.error) bucket.errors.push(`${result.error}${result.detail ? `: ${result.detail}` : ""}`); + else bucket.submits.push(result.ms); + } + + console.log("==> opening sockets"); + const connectStarted = Date.now(); + const connected = await Promise.all(players.map((player) => connect(player, onEvent))); + const live = connected.filter(Boolean).length; + console.log( + ` ${live}/${players.length} sockets live in ${Date.now() - connectStarted}ms ` + + `connect ${JSON.stringify(summarise(players.filter((p) => !p.connectError).map((p) => p.connectMs)))}` + ); + const connectErrors = tally(players.filter((p) => p.connectError).map((p) => p.connectError)); + if (Object.keys(connectErrors).length) console.log(` connect errors: ${JSON.stringify(connectErrors)}`); + + if (HOSTED) { + console.log("\n==> starting the game"); + const started = await callApi("start_session", { session }, "POST", true); + if (started.error) console.error(` start failed: ${started.error} ${started.detail || ""}`); + } else { + console.log("\n==> waiting for the host to run the game (ctrl-c to stop)\n"); + } + const watchdog = setInterval(() => { + if (podium && podium.arrivals.length >= live) finished(); + }, 500); + const timeout = setTimeout(finished, TIMEOUT_MS); + await done; + clearInterval(watchdog); + clearTimeout(timeout); + + const report = { origin: ORIGIN, pin: PIN, session, players: players.length, sockets: live, questions: [] }; + + console.log("=== per question ==="); + for (const [row, bucket] of questions) { + const first = Math.min(...bucket.arrivals); + const skew = bucket.arrivals.map((at) => at - first); + const line = { + q: bucket.index, + delivered: `${bucket.arrivals.length}/${live}`, + skew_ms: summarise(skew), + submit_ms: summarise(bucket.submits), + ok: bucket.submits.length, + errors: tally(bucket.errors), + }; + report.questions.push({ row, ...line }); + console.log( + ` Q${line.q} delivered ${line.delivered} ` + + `skew p50=${line.skew_ms.p50} p95=${line.skew_ms.p95} max=${line.skew_ms.max}ms ` + + `submit p50=${line.submit_ms.p50} p95=${line.submit_ms.p95} max=${line.submit_ms.max}ms ` + + `ok=${line.ok} errors=${JSON.stringify(line.errors)}` + ); + } + + if (podium) { + const first = Math.min(...podium.arrivals); + const skew = summarise(podium.arrivals.map((at) => at - first)); + report.podium = { bytes: podium.bytes, delivered: podium.arrivals.length, skew_ms: skew }; + console.log( + `\n=== podium ===\n payload ${(podium.bytes / 1024).toFixed(1)}KB x ${podium.arrivals.length} sockets ` + + `= ${((podium.bytes * podium.arrivals.length) / 1048576).toFixed(1)}MB fan-out ` + + `skew p50=${skew.p50} p95=${skew.p95} max=${skew.max}ms` + ); + } else { + console.log("\n=== podium ===\n never arrived"); + } + + if (OUT) { + await (await import("node:fs/promises")).writeFile(OUT, JSON.stringify(report, null, 2)); + console.log(`\nreport written to ${OUT}`); + } + + for (const player of players) player.socket?.close(); + + // leave_session is capped 10/60s per IP, so this only clears a smoke run. A + // full-size run leaves its rows behind for the bench-side cleanup snippet. + if (LEAVE) { + for (const player of players) { + await callApi("leave_session", { pin: PIN, token: player.token }); + } + console.log(`\nleft ${players.length} players`); + } + return 0; +} + +process.exitCode = await main(); +process.exit(process.exitCode); diff --git a/scripts/loadtest_preflight.sh b/scripts/loadtest_preflight.sh new file mode 100755 index 0000000..22376d1 --- /dev/null +++ b/scripts/loadtest_preflight.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Pre-flight for a live load run. Run this on the load-generating box before the +# ramp: it catches the failures that otherwise look like the target breaking. +# +# QZ_ORIGIN=https://site QZ_PIN=123456 scripts/loadtest_preflight.sh [players] +set -uo pipefail + +ORIGIN="${QZ_ORIGIN:-https://gajendra.fsn.frappe.cloud}" +PLAYERS="${1:-1000}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +fails=0 + +check() { + if [ "$1" = "ok" ]; then printf ' \033[32mok\033[0m %s\n' "$2" + else printf ' \033[31mFAIL\033[0m %s\n' "$2"; fails=$((fails + 1)); fi +} + +echo "pre-flight: origin=$ORIGIN players=$PLAYERS" + +# A snapshot-cloned microVM comes up with a skewed clock, and every TLS handshake +# then fails with "certificate is not yet valid" — which reads as a dead target. +remote_date="$(curl -sI "$ORIGIN" | awk 'BEGIN{IGNORECASE=1} /^date:/{sub(/^[Dd]ate: /,""); print}' | tr -d '\r')" +if [ -n "$remote_date" ]; then + skew=$(( $(date -u +%s) - $(date -u -d "$remote_date" +%s 2>/dev/null || date -u -jf "%a, %d %b %Y %T %Z" "$remote_date" +%s) )) + [ "${skew#-}" -lt 30 ] && check ok "clock skew ${skew}s" || check fail "clock skew ${skew}s — sync ntp before running" +else + check fail "no Date header from $ORIGIN (target unreachable or TLS failed)" +fi + +node -e 'process.exit(Number(process.versions.node.split(".")[0]) >= 20 ? 0 : 1)' 2>/dev/null \ + && check ok "node $(node -v)" || check fail "node >= 20 required (have $(node -v 2>/dev/null || echo none))" + +# every bot holds a websocket and an http connection; the default 1024 is not enough +limit="$(ulimit -n)" +[ "$limit" = "unlimited" ] || [ "$limit" -ge $((PLAYERS * 3)) ] \ + && check ok "ulimit -n $limit" \ + || check fail "ulimit -n $limit, need >= $((PLAYERS * 3)) — run: ulimit -n $((PLAYERS * 4))" + +node -e 'import("undici");import("socket.io-client")' 2>/dev/null \ + && check ok "deps installed" || check fail "run: npm i --no-save undici socket.io-client" + +if [ -n "${QZ_PIN:-}" ]; then + out="$(QZ_PLAYERS=2 QZ_TIMEOUT=8 QZ_LEAVE=1 QZ_NAME_PREFIX=preflight \ + node "$SCRIPT_DIR/loadtest_live.mjs" 2>&1)" + echo "$out" | grep -q "sockets live" \ + && check ok "2-bot smoke: $(echo "$out" | grep 'seated' | xargs)" \ + || { check fail "2-bot smoke failed"; echo "$out" | sed 's/^/ /'; } +else + echo " skip QZ_PIN not set, skipping the smoke run" +fi + +echo +[ "$fails" -eq 0 ] && echo "pre-flight clean" || echo "$fails check(s) failed — fix before ramping" +exit "$fails" diff --git a/scripts/loadtest_ramp.sh b/scripts/loadtest_ramp.sh new file mode 100755 index 0000000..c7cd131 --- /dev/null +++ b/scripts/loadtest_ramp.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Ramp a live quiz through increasing player counts, one full game per stage, +# writing a JSON report per stage. Stops at the first stage that degrades, so a +# broken run does not burn the whole ramp. +# +# Self-hosting (preferred, no host clicking): +# QZ_QUIZ="General Knowledge" QZ_API_KEY=... QZ_API_SECRET=... scripts/loadtest_ramp.sh +# +# Manual host (you drive the host screen, script prompts for each stage's pin): +# scripts/loadtest_ramp.sh +set -uo pipefail + +ORIGIN="${QZ_ORIGIN:-https://gajendra.fsn.frappe.cloud}" +STAGES="${QZ_STAGES:-100 250 500 1000}" +OUT_DIR="${QZ_OUT_DIR:-./loadtest-reports}" +COOLDOWN="${QZ_COOLDOWN:-60}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +mkdir -p "$OUT_DIR" +stamp="$(date -u +%Y%m%dT%H%M%SZ)" +echo "ramp $stamp origin=$ORIGIN stages=[$STAGES] reports=$OUT_DIR" + +for players in $STAGES; do + echo + echo "════ stage: $players players ════" + + # the generator only creates its own session in hosted mode; otherwise the + # host makes one and we need its pin before any bot can join + if [ -z "${QZ_QUIZ:-}" ]; then + read -rp " open a fresh lobby on the host screen, then enter its pin: " QZ_PIN + export QZ_PIN + fi + + report="$OUT_DIR/${stamp}-${players}.json" + QZ_PLAYERS="$players" QZ_OUT="$report" QZ_NAME_PREFIX="bot" \ + node "$SCRIPT_DIR/loadtest_live.mjs" + status=$? + + if [ "$status" -ne 0 ]; then + echo " stage $players exited $status — stopping the ramp here" + exit "$status" + fi + + # a stage that lost players or dropped answers makes every larger stage + # unreadable, so the ramp stops rather than piling failure on failure + if node -e ' + const r = require(process.argv[1]); + const lost = r.sockets < r.players; + const dropped = r.questions.some((q) => Object.keys(q.errors).length); + const undelivered = r.questions.some((q) => { + const [got, want] = q.delivered.split("/").map(Number); + return got < want; + }); + process.exit(lost || dropped || undelivered ? 1 : 0); + ' "$(cd "$(dirname "$report")" && pwd)/$(basename "$report")"; then + echo " stage $players clean" + else + echo " stage $players degraded — see $report. Stopping." + exit 1 + fi + + echo " cooling down ${COOLDOWN}s" + sleep "$COOLDOWN" +done + +echo +echo "ramp complete, reports in $OUT_DIR"