Skip to content

ci(sonar): stop a runner backlog from failing an unrelated PR - #325

Merged
phsb5321 merged 4 commits into
mainfrom
325-sonar-timeout
Aug 10, 2026
Merged

ci(sonar): stop a runner backlog from failing an unrelated PR#325
phsb5321 merged 4 commits into
mainfrom
325-sonar-timeout

Conversation

@phsb5321

@phsb5321 phsb5321 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Closes #319.

The failure

SonarQube standalone scan is a required check. When its 10-minute job timeout fires mid-coverage, the PR goes red for a reason that has nothing to do with the diff under review, and the only remedy is a manual re-run once the runner queue drains.

Observed 2026-08-01 — a 13-run backlog hit the two self-hosted runners:

run outcome why
02:21 cancelled job ran 02:23:00 → 02:33:23, killed mid Generate Go coverage report
02:12 cancelled superseded sha, cancelled by hand — not a timeout
01:54 cancelled main, same shape
01:37 and back to 07-28 success

Nothing about the code changed across that boundary. 10m23s is the whole story.

The fix

All three candidates from the issue, because they address different halves of the problem.

1. concurrency group per ref. sonar.yml had no concurrency: block at all, so a superseded push left its scan running and competing for a runner with the scan that replaced it. cancel-in-progress is ${{ github.event_name == 'pull_request' }} rather than a flat true: a push to main is not superseded in the same sense, and cancelling it would silently drop the baseline scan that PR scans are compared against.

2. Drop -race from the coverage run. Test (go test -race) (ci.yml:144, also a required check) already gates races on every PR. Running the detector a second time buys no coverage signal and roughly doubles this step's wall time on a contended runner. -covermode=atomic stays — it does not require -race.

3. timeout-minutes: 1020. Headroom for a genuinely contended run, now that (1) and (2) have removed most of the contention. The step still cannot hang indefinitely on a real failure: the || true on the coverage command swallows non-zero exits, not wall time.

Verification

  • actionlint — clean.
  • zizmor --persona=auditor .github/workflows/sonar.ymlNo findings to report. Good job! (2 ignored).
  • Required-check names confirmed against the live ruleset (gh api repos/yolo-labz/wa/rulesets): both SonarQube standalone scan and Test (go test -race) are required, which is what makes (2) safe and (1) worth doing.

The real proof is this PR's own scan finishing inside the new budget — and that only exercises the uncontended path. The contended path cannot be reproduced on demand; if a backlog recurs and the job still times out at 20 minutes, the next step is splitting coverage out of the scan job rather than raising the number again.

Escalation

[pending] Pedro — this edits a file under .github/workflows/, which is escalate-class per CLAUDE.md rule 33. Not self-merging.

Revert path: git revert a10578f.

@phsb5321
phsb5321 requested a review from a team as a code owner August 1, 2026 05:08
`SonarQube standalone scan` is a required check, so when the 10-minute
job timeout killed it mid-coverage the PR went red for a reason that had
nothing to do with the diff — and the only remedy was a manual re-run
once the queue drained.

Observed 2026-08-01: a 13-run backlog hit the two self-hosted runners and
the job ran 02:23:00 -> 02:33:23 before the limit cancelled it. Runs back
to 07-28 were green; nothing about the code changed.

All three fixes from #319, because they address different halves of the
problem:

- concurrency group per ref, cancel-in-progress on pull_request only.
  Superseded pushes no longer leave scans competing for runners. Pushes
  to main are not superseded in the same sense, so they are never
  cancelled.
- drop `-race` from the coverage run. `Test (go test -race)` in ci.yml is
  also a required check and already gates races on every PR; running the
  detector twice buys no coverage signal and roughly doubles this step.
  `-covermode=atomic` stays and does not need the race detector.
- raise `timeout-minutes` 10 -> 20 as headroom for a genuinely contended
  run. The step cannot hang indefinitely on a real failure: `|| true`
  swallows non-zero exits, not wall time.

Closes #319

Signed-off-by: Pedro H S Balbino <30302237+phsb5321@users.noreply.github.com>
@phsb5321
phsb5321 force-pushed the 325-sonar-timeout branch from a10578f to 933159e Compare August 1, 2026 06:00
@phsb5321

phsb5321 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto ef97bb2 (current main) — merge-ready, all checks green.

[pending] Pedro — edits .github/workflows/, which is escalate-class. Not self-merging. Revert path once landed: git revert <squash-sha>.

phsb5321 added a commit that referenced this pull request Aug 10, 2026
…2) (#326)

* ci(bench): reproduce idle-RSS + hot-path benchmarks in CI (roadmap 2.2)

Roadmap 2.2's "benchmark artifact reproducible in CI" metric was the
last unmet repo-local roadmap item: nothing in .github/workflows/ ran
the bench harnesses and the bench/README reference numbers (10/06/2026,
commit 2f80ed3) were unguarded — a silent drift risk for the README's
RSS/market claims.

Additive-only bench.yml (new workflow, no existing file modified, no
secrets, no write surface, deliberately not a required check):
- triggers: push to main + pull_request, concurrency group per ref with
  PR-only cancel (shape from #325)
- builds wad from the working tree via ./bench/idle-rss.sh (unpaired
  boot) and runs the four hot-path microbenchmarks with -count=1
  -benchmem -vet=off
- compares results against committed thresholds (10/06 baseline x 2x
  RSS / 3x ns/op, derived + cited in the workflow header), fails on
  regression, uploads raw output as the bench-results artifact
- comparison is sed/awk only — no jq/benchstat, no new dependencies

bench/README.md documents the CI job, the threshold policy and how to
read the artifact; the 10/06 table is relabeled as the baseline it is.

.zizmor.yml: bench.yml joins the documented self-hosted-runner
exception list — every workflow on the [self-hosted, dokku] pool is
listed there, and the required "workflow static analysis" check would
otherwise stay red on the new workflow (zizmor enforce mode).

Verified locally: harnesses + compare step pass on this host
(32 MiB RSS; 338/3822/2912/4750 ns/op vs thresholds 1000/10000/8000/
13000); actionlint 1.7.12 clean; zizmor 1.24.1 --persona=auditor: no
findings.

Signed-off-by: Pedro H S Balbino <30302237+phsb5321@users.noreply.github.com>

* ci: re-trigger checks after GitHub Actions incident (2026-08-06)

The initial run of this PR's checks landed inside a GitHub Actions
incident (status page: critical, 14:12-21:10 BRT). Three workflow runs
(Reproducibility, CodeQL, quality-gates) were left in a zombie state —
run-level 'queued' with zero jobs, refusing both cancel and rerun
(HTTP 403 'already running'). All other checks passed on the original
commit. This empty commit moves the head SHA so every workflow
re-triggers fresh; no code or configuration changed.

Signed-off-by: Pedro H S Balbino <30302237+phsb5321@users.noreply.github.com>

* ci(bench): round thresholds up to the documented factor product

314×3.2 = 1004.8 → 1005 ns/op (was 1000, factor 3.185× below 3.2)
4.1 µs×3.2 = 13120 ns/op (was 13000, factor 3.171× below 3.2)

qa (w8:p3) issuecomment-5207658880 G5: committed thresholds sat below
their own documented slack product. Header comments and README table
updated in lockstep.

Signed-off-by: Pedro H S Balbino <30302237+phsb5321@users.noreply.github.com>

---------

Signed-off-by: Pedro H S Balbino <30302237+phsb5321@users.noreply.github.com>
@phsb5321
phsb5321 merged commit 1db658b into main Aug 10, 2026
18 checks passed
@phsb5321
phsb5321 deleted the 325-sonar-timeout branch August 10, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sonar.yml: 10-min timeout cancels the coverage step under runner contention

1 participant