Repository navigation
ci: agent test jobs run under systemd limits so a wedge fails fast and leaves evidence - #117
Merged
Merged
Conversation
…he runner The agent-rest / agent-code-mode wedge leaves the test step in_progress past nextest's global-timeout, the step timeout and the job timeout, all of which need the runner to act. Memory exhaustion is ruled out by measurement under the runner's limits (4 CPUs, 16 GB): run peak ~0.3 GB / 80 tasks, compile anon peak ~3.2 GB. The test step now runs nextest via `sudo systemd-run --wait` with RuntimeMaxSec=11min, MemoryMax=8G, MemorySwapMax=0 and TasksMax=4096, so an overrun fails loudly with a systemd result and the always() steps still run; the unit's cgroup kill replaces the pkill sweep. A ci-resources sampler unit records memory, swap, PSI, pids, disk, top RSS and D-state processes every 5s, uploaded with nextest.log as an artifact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
test (agent-rest)/test (agent-code-mode)CI jobs keep wedging at a flat 30:00 with no log. It happened on 3 of the last 4 runs on #115/#116.sudo systemd-run --waitwith RuntimeMaxSec=11min, MemoryMax=8G, MemorySwapMax=0 and TasksMax=4096.systemd-runreturns, so the job fails fast and thealways()steps still run.ci-resourcessampler: memory, swap, pressure stats, load, pids, disk, top memory users and D-state processes every 5 s. It's uploaded with nextest.log, so the next wedge leaves evidence.Validated locally under the emulated limits: 3/3 runs of each suite. A forced tiny MemoryMax fails loudly in 1 s, and a short RuntimeMaxSec kills at 8 s.
To confirm on this PR's first run: that hosted runners allow
sudo systemd-run(runner image systemd 255).🤖 Generated with Claude Code
https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk