Skip to content

Swift CodeQL dies with a runner OOM on ~20% of runs #570

Description

@OrellBuehler

Analyze (swift) intermittently fails with:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The tell is the runtime. A run that fits in memory finishes the whole job in 43-55 min; a run that doesn't gets stuck in the Build iOS app step and the runner is killed at ~90-100 min. The step never completes, so the logs 404 — read the check-run annotations instead (gh api repos/{owner}/{repo}/check-runs/<job-id>/annotations).

Rate

3 of the last 18 runs of the job in codeql-mobile.yml, then 3 of 3 attempts on the split-out codeql-swift.yml in #558 (2026-08-31/09-01).

What was ruled out on #558

  • Memory budget. ci: cut Swift CodeQL build time and stop the runner OOM #557 cut ram 6144 → 4096, which turned constant deaths into occasional ones. Going further to 3072 did not help — the build still died at 92 min — so the extractor's own budget is not the remaining factor. Reverted.
  • Cold TRAP cache under the new workflow file: the logs say swift does not support TRAP caching (missing option group), so there is no cache to be cold.
  • The workflow split itself. The parsed swift job in codeql-swift.yml is byte-identical to the one in codeql-mobile.yml, workflow-level keys included; only name: and the concurrency group differ.
  • A bad runner. Every failure was a different runner, interleaved in time with successes on other branches.

Where the time goes (measured on job 99276431786, see the comments in codeql-swift.yml)

25m41s  emit-module for Bissbilanz    (one frontend call, 145 files)
 7m14s  emit-module for BissbilanzWidgets
  ~4m   queries and everything else

The untraced build of the same scheme in mobile-ios.yml takes ~2 min, so the extractor is the entire cost.

Things worth trying

  • Serialize the traced build (-jobs 1, -parallelizeTargets NO) so only one swift-frontend + extractor pair is resident at a time. Risk: a successful run may then exceed the 150-min timeout.
  • Cut the number of traced targets — Widgets and Watch recompile Shared/ and PhoneShared/, so ~230 file-compiles cover 166 unique files.
  • Accept the flake and retry the job automatically instead, now that ci: path-scope swift codeql and group dependabot bumps #558 has made it fire only for mobile/iosApp/** changes, pushes to main, and the weekly schedule.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions