fix(fly): size zygote pool + slots to real RAM to stop concurrency OOM#17
Open
teovillanueva wants to merge 1 commit into
Open
fix(fly): size zygote pool + slots to real RAM to stop concurrency OOM#17teovillanueva wants to merge 1 commit into
teovillanueva wants to merge 1 commit into
Conversation
The zygote tier funnels every concurrent Python job into ONE shared pool container, but its memory cap (ZYGOTE_POOL_MEMORY_MB=1024) only fit ~2 jobs at edalef's memoryMb=512, while the worker still advertised 24 flat slots. Under exam-time concurrency the 1 GB pool cgroup OOM'd — sometimes killing the privileged pool *parent* (observed: python UID:0 killed → "zygote: read STARTED: EOF" → Docker-tier fallback), collapsing every in-flight job sharing that pool. Camino A (config-only, no VM resize), deployed + verified in prod: - ZYGOTE_POOL_MEMORY_MB 1024 -> 2560 (uses the idle RAM on the 4 GB VM; holds 5 x 512 MB jobs with ~1.3 GB left for dockerd/OS). - WORKER_MAX_SANDBOXES 24 -> 5: the slot semaphore is acquired before the pool child is forked, so a count matching usableRAM/512MB makes it a real admission gate — the worker refuses the 6th job instead of overcommitting the pool. - Autoscaler divisor /24 -> /5 so the fleet scales out at real per-node capacity (a single node's pool OOM'd long before 24 inflight). Verified: python smoke job state=done; live pool mem.limit=2684354560 (2560 MiB). Follow-ups (not in this change): memory-weighted admission in the worker (decouple slot count from RAM) is the durable fix; pre-create more pool machines + raise the autoscaler ceiling for larger exams. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problema
edalef reportó ejecuciones que "se clavan" bajo concurrencia (exámenes). Root cause: el tier zygote mete todos los jobs de Python concurrentes en un solo contenedor-pool compartido por
(lenguaje,versión), pero su cap de memoria (ZYGOTE_POOL_MEMORY_MB=1024) solo entraba ~2 jobs con elmemoryMb=512que usa edalef para data-science — mientras el worker seguía anunciando 24 slots planos.Bajo carga, el cgroup del pool de 1 GB hacía OOM. A veces el OOM-killer mataba al parent privilegiado del pool (observado en prod:
python UID:0killed →zygote: read STARTED: EOF→ fallback a Docker tier), tumbando todos los jobs in-flight que compartían ese pool.Cambio (Camino A — solo config, sin resize de VM)
Ya desplegado y verificado en prod:
ZYGOTE_POOL_MEMORY_MB(worker)WORKER_MAX_SANDBOXES(worker)FAS_STARTED_MACHINE_COUNTdivisor (autoscaler)/24/5usableRAM/512MBlo convierte en gate de admisión real: el worker rechaza el 6º job en vez de sobrecargar el pool.Verificación
state: "done"✅mem.limit = 2684354560= 2560 MiB ✅Follow-ups (fuera de este PR)
provision-pool.sh grow N+ subir el ceiling del autoscaler para exámenes más grandes.🤖 Generated with Claude Code