fix: revive the shared game ticker if its worker dies mid-game

docs/loadtest-runbook.md

@@ -0,0 +1,110 @@
+# 1000-player load run — runbook
+
+Everything below is staged and ready. On "go" it runs top to bottom.
+
+Target: `https://gajendra.fsn.frappe.cloud` (private FC bench).
+Generator: 1–2 DoppioBoxes on `test-01` (Hetzner FSN, same region as the target,
+so measured latency is server time rather than network).
+
+## 0. Prerequisites (one-time, before "go")
+
+- [ ] `join_session` rate limit lifted on the target — **and a reminder set to restore it**
+- [ ] global `frappe.conf.rate_limit` checked: `grep -i rate_limit sites/<site>/site_config.json sites/common_site_config.json`
+      (site-wide, not per-IP — if set, a 1000-wide salvo trips it and every bot gets a bare 429)
+- [ ] a quiz with **8+ questions** seeded, so the ramp has enough salvos to show a trend
+- [ ] devbox slugs agreed and provisioned
+- [ ] host API key + secret issued (optional — without it you drive the host screen)
+
+## 1. Bootstrap the generator box
+
+```bash
+git clone https://github.com/bwhtech/quizzly && cd quizzly
+npm i --no-save undici socket.io-client
+ulimit -n 8192          # each bot holds a websocket + an http connection
+```
+
+## 2. Pre-flight
+
+```bash
+QZ_ORIGIN=https://gajendra.fsn.frappe.cloud QZ_PIN=<lobby pin> \
+  scripts/loadtest_preflight.sh 1000
+```
+
+Checks clock skew (a snapshot-cloned microVM fails every TLS handshake with
+"certificate is not yet valid", which reads exactly like a dead target), node
+version, `ulimit -n`, deps, then runs a 2-bot smoke and cleans it up.
+**Do not ramp until this is clean.**
+
+## 3. Ramp
+
+Self-hosting — the generator creates each session, seats the bots, starts the
+game and plays to the podium, no clicking:
+
+```bash
+QZ_QUIZ="General Knowledge" QZ_API_KEY=... QZ_API_SECRET=... \
+  scripts/loadtest_ramp.sh
+```
+
+Manual host — you open a fresh lobby per stage, the script asks for the pin:
+
+```bash
+scripts/loadtest_ramp.sh
+```
+
+Stages default to `100 250 500 1000` (`QZ_STAGES` to change), 60s cooldown
+between them, one JSON report per stage in `./loadtest-reports/`. The ramp
+**stops at the first degraded stage** rather than piling failure on failure.
+
+Splitting across two boxes: run 500 on each with `QZ_STAGES=500` against the
+same pin. If two-box 500+500 beats one-box 1000, the generator was the
+bottleneck, not the server.
+
+## 4. Watch, server-side
+
+The client numbers say *when* it broke; these say *what* broke.
+
+- gunicorn worker saturation and request queue depth
+- the socket.io node process — CPU and RSS during the podium burst
+- MariaDB slow log, and lock waits on `QZ Answer` inserts
+- RQ: whether the shared ticker stays on schedule or drifts
+
+## 5. Pass / fail, decided before the run
+
+| Measure | Threshold at 1000 | Why |
+|---|---|---|
+| players seated / sockets live | 100% | anything less and the rest is unreadable |
+| question delivery skew p99 | **< 500 ms** | scoring is `(1 - (response_ms/window_ms)/2) * 1000`; on a 20s window 500ms costs ~12 of 1000 points, 2s costs ~50 — 500ms is the edge of fair |
+| `submit_answer` p99 | < 1000 ms | beyond this the countdown on screen is lying |
+| dropped answers | 0 | excludes legitimate "Already answered" |
+| podium delivered | 100%, p99 < 2s | see the prediction below |
+| HTTP 429 | 0 | any means a limiter is still in the path |
+
+## 6. The prediction to confirm or kill
+
+`engine.py:403` broadcasts the podium carrying the **full leaderboard** to the
+whole room. At 1000 players that is ~80KB × 1000 sockets ≈ **80MB pushed from a
+single node process in one burst**, immediately followed by 1000 `get_state`
+calls that each return the same leaderboard again over HTTP.
+
+My bet is the first failure is here, not in the submit salvo the existing
+`scripts/loadtest.py` measures. The generator reports podium payload size,
+fan-out total and delivery skew specifically to settle this. If confirmed, the
+fix is small: broadcast top-N only, and let players fetch their own placement.
+
+## 7. Abort
+
+Stop if the target starts serving 5xx to real traffic, or FC's proxy begins
+rate-limiting site-wide. `ctrl-c` on the ramp; sessions in flight can be ended
+from the host screen.
+
+## 8. Cleanup
+
+```bash
+# dry run first
+echo 'exec(open("apps/quizzly/scripts/loadtest_cleanup.py").read())' | bench --site <site> console
+# then
+QZ_CLEANUP_APPLY=1 bash -c 'echo "exec(open(\"apps/quizzly/scripts/loadtest_cleanup.py\").read())" | bench --site <site> console'
+```
+
+Deletes bot participants and their answers, and drops only sessions where
+*every* player was synthetic. Then: **restore the `join_session` rate limit.**

quizzly/api.py

@@ -64,6 +64,9 @@ def get_host_state(session: str | None = None) -> dict:
 	session_doc = get_host_session(session) if session else get_live_host_session()
 	if not session_doc or session_doc.status == "Cancelled":
 		return {}
+	# the host screen is the most frequent caller, so revive a dead ticker here
+	# too: a game whose loop was killed resumes within a poll, not a scheduler minute
+	engine.ensure_ticker_running()
 	if engine.is_abandoned(session_doc):
 		# the host reloaded into a game whose worker is gone: settle it and show the podium
 		engine.finish_session(session_doc)

quizzly/engine.py

@@ -37,16 +37,36 @@ def enqueue_game_loop(session_doc) -> None:
 	clear_control(session_doc.name)
 	get_ready(session_doc, questions[0], 0, len(questions))
 	frappe.cache.sadd(ACTIVE_SESSIONS_KEY, session_doc.name)
+	# after_commit so the ticker only starts once the get_ready state and the
+	# active-set membership it reads are actually visible
+	enqueue_ticker(after_commit=True)
+
+
+def enqueue_ticker(after_commit: bool = False) -> None:
 	frappe.enqueue(
 		"quizzly.engine.run_ticker",
 		queue="long",
 		timeout=TICKER_TIMEOUT,
 		job_id="qz_ticker",
 		deduplicate=True,
-		enqueue_after_commit=True,
+		enqueue_after_commit=after_commit,
 	)
 
 
+def ensure_ticker_running() -> None:
+	"""Bring the shared ticker back if it died with games still live.
+
+	Every game is driven by this one loop. If its worker is lost mid-game — an OOM
+	under a large answer flood, a deploy or worker restart — nothing advances the
+	games and they freeze exactly where they stood, with no exception to log. A
+	deduplicated re-enqueue is a no-op while the loop is alive, and otherwise starts
+	a fresh one that resumes every game from the Redis state it left behind. Driven
+	from the scheduler and the host poll so recovery never needs a human.
+	"""
+	if active_sessions():
+		enqueue_ticker()
+
+
 def run_ticker() -> None:
 	"""One shared self-looping job. Advances every active session on time or host command."""
 	# process-local: the ticker is a single deduplicated job, so per-session throttle

quizzly/hooks.py

@@ -29,3 +29,14 @@
 
 export_python_type_annotations = True
 require_type_annotated_api_methods = True
+
+# The live-game loop is one shared background job; if its worker dies mid-game
+# nothing else advances the games. This re-enqueues it (deduplicated, so a live
+# loop is untouched) so a killed ticker recovers without a human.
+scheduler_events = {
+	"cron": {
+		"* * * * *": [
+			"quizzly.engine.ensure_ticker_running",
+		]
+	}
+}

quizzly/tests/test_engine.py

@@ -521,3 +521,27 @@ def test_question_without_explanation_skips_the_screen(self):
 		with patch("frappe.publish_realtime"), patch("frappe.db.commit"):
 			engine.close_question(self.session_doc, question, 0, len(self.questions))
 		self.assertEqual(engine.get_state(self.session)["phase"], "stats")
+
+
+class TestTickerRecovery(IntegrationTestCase):
+	"""The shared ticker can be killed mid-game (OOM under a large answer flood, a
+	worker restart). Nothing else advances a game, so it must be revivable."""
+
+	def setUp(self):
+		frappe.cache.delete_value(engine.ACTIVE_SESSIONS_KEY)
+
+	def tearDown(self):
+		frappe.cache.delete_value(engine.ACTIVE_SESSIONS_KEY)
+
+	def test_ensure_ticker_running_revives_when_a_game_is_live(self):
+		frappe.cache.sadd(engine.ACTIVE_SESSIONS_KEY, "some-session")
+		with patch("frappe.enqueue") as enqueue:
+			engine.ensure_ticker_running()
+		enqueue.assert_called_once()
+		self.assertEqual(enqueue.call_args.kwargs["job_id"], "qz_ticker")
+		self.assertTrue(enqueue.call_args.kwargs["deduplicate"])
+
+	def test_ensure_ticker_running_is_a_noop_with_no_live_games(self):
+		with patch("frappe.enqueue") as enqueue:
+			engine.ensure_ticker_running()
+		enqueue.assert_not_called()

scripts/loadtest_cleanup.py

@@ -0,0 +1,64 @@
+# Removes what a load run leaves behind: bot participants, their answers, and any
+# session that ends up with no real players. Run it on the bench host after a ramp.
+#
+# IPython's autoindent mangles indented blocks pasted into `bench console`, so this
+# has to be exec'd from the file rather than piped in as source:
+#   echo 'exec(open("apps/quizzly/scripts/loadtest_cleanup.py").read())' \
+#     | bench --site SITE console
+#
+# env:
+#   QZ_CLEANUP_PREFIXES  comma-separated nickname prefixes (default bot,probe,preflight,tmp)
+#   QZ_CLEANUP_APPLY     1 to actually delete; anything else only reports
+
+import os
+
+import frappe
+
+prefixes = os.environ.get("QZ_CLEANUP_PREFIXES", "bot,probe,preflight,tmp").split(",")
+apply_changes = os.environ.get("QZ_CLEANUP_APPLY") == "1"
+
+participants = []
+for prefix in prefixes:
+	participants += frappe.get_all(
+		"QZ Participant",
+		filters={"nickname": ["like", f"{prefix}%"]},
+		fields=["name", "session", "nickname"],
+	)
+
+if not participants:
+	print("nothing to clean")
+else:
+	bot_sessions = {row.session for row in participants}
+	bot_names = {row.name for row in participants}
+
+	# a session is only disposable when every player in it was synthetic
+	real_counts = frappe.get_all(
+		"QZ Participant",
+		filters={"session": ["in", list(bot_sessions)]},
+		fields=["session", "count(name) as total"],
+		group_by="session",
+	)
+	total_by_session = {row.session: row.total for row in real_counts}
+	bots_by_session = {}
+	for row in participants:
+		bots_by_session[row.session] = bots_by_session.get(row.session, 0) + 1
+
+	disposable = [s for s in bot_sessions if bots_by_session[s] == total_by_session.get(s)]
+	mixed = sorted(bot_sessions - set(disposable))
+
+	answers = frappe.db.count("QZ Answer", {"participant": ["in", list(bot_names)]})
+	print(f"bot participants: {len(bot_names)}  answers: {answers}")
+	print(f"sessions fully synthetic: {len(disposable)}  sessions with real players too: {len(mixed)}")
+	for session in mixed:
+		print(f"  keeping session {session} ({bots_by_session[session]}/{total_by_session[session]} synthetic)")
+
+	if not apply_changes:
+		print("\ndry run — set QZ_CLEANUP_APPLY=1 to delete")
+	else:
+		frappe.db.delete("QZ Answer", {"participant": ["in", list(bot_names)]})
+		frappe.db.delete("QZ Participant", {"name": ["in", list(bot_names)]})
+		for session in disposable:
+			frappe.cache.srem("qz:active_sessions", session)
+			frappe.delete_doc("QZ Session", session, force=True, ignore_permissions=True)
+		frappe.db.commit()
+		print(f"\ndeleted {len(bot_names)} participants, {answers} answers, {len(disposable)} sessions")