ci: split benchmarks switchboard release flow - #140
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
More reviews will be available in 18 minutes and 58 seconds. Learn how PR review limits work. To continue reviewing without waiting, enable usage-based billing in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
26aeb62 to
dca2d98
Compare
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| - name: Checkout repository | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| ref: ${{ steps.input.outputs.deploy_ref }} | ||
| fetch-depth: 0 | ||
| persist-credentials: false | ||
|
|
||
| - name: Resolve image tags | ||
| id: context | ||
| run: | | ||
| set -euo pipefail | ||
| deploy_sha="$(git rev-parse HEAD)" | ||
| runner_image="us-east1-docker.pkg.dev/coval-benchmarks-prod/runner/coval-bench-runner:${deploy_sha}" | ||
| api_image="us-east1-docker.pkg.dev/coval-benchmarks-prod/runner/coval-bench-api:${deploy_sha}" | ||
|
|
||
| git fetch origin production-runner:refs/remotes/origin/production-runner 2>/dev/null || true | ||
| skip=false | ||
| if git rev-parse --verify origin/production-runner >/dev/null 2>&1 && [ "$deploy_sha" = "$(git rev-parse origin/production-runner)" ]; then | ||
| skip=true | ||
| fi |
There was a problem hiding this comment.
The context job checks out with persist-credentials: false, then immediately runs git fetch origin production-runner ... 2>/dev/null || true. On a private repository the fetch needs git credentials — which persist-credentials: false deliberately removes — so it silently fails every time. skip therefore stays false on every run and the early-exit optimization is permanently bypassed.
Compare web-release.yml lines 70-88: the equivalent job there uses token: ${{ steps.app_token.outputs.token }} (without persist-credentials: false), so its skip check works as intended. The runner-release context job needs the same treatment — either generate an app token before checkout, or drop persist-credentials: false here.
| name: Methodology marker check | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| if: github.event_name == 'pull_request' | ||
| # Advisory only: never let this job block a PR, even on an unexpected error. | ||
| continue-on-error: true |
There was a problem hiding this comment.
The new if: github.event_name == 'pull_request' guard is appropriate to prevent this advisory job from running in release pipelines, but it also prevents it from running when ci.yml is invoked from switchboard.yml via workflow_call. In that context github.event_name is "workflow_call", not "pull_request", so the check is always skipped for every PR that goes through Switchboard. The check is continue-on-error: true so this isn't blocking, but the behavioral change may be unintentional — the check ran on all PRs before this PR.
dca2d98 to
4b07d3b
Compare
4b07d3b to
736aecf
Compare
736aecf to
7fb02e5
Compare
7fb02e5 to
ebb0eea
Compare
|
Addressed the reviewbot idempotency finding in the latest push: the runner-release context checkout now keeps credentials so the production-runner fetch can work on the private repo and no-op releases can skip correctly. Also fixed the fresh-runner pnpm setup issue by declaring packageManager and setting up pnpm before node cache restore. |
ebb0eea to
cdc250a
Compare
|
One more rollout fix after the latest CI pass: removed the shared concurrency block from the reusable runner/API CI workflow, because the direct PR CI and Switchboard-called adapter were canceling each other during the transition. Same rollout-race shape as the benchmark-infra Terraform plan fix. |
1 participant
Summary
runner-release.ymlrepository_dispatchweb-release.ymlfor Vercel web production promotion and theproduction-webmarker branch--max-warnings 0packageManagerand set up pnpm before node cache restore on fresh GitHub runnersproduction-runneridempotency fetch authenticated so no-op releases can skip correctlyValidation
actionlintpnpm codegen,pnpm lint,pnpm test(32 passed),pnpm buildpnpm install --frozen-lockfileuv sync --frozen --dev,uv run ruff check .,uv run ruff format --check .,uv run mypy --strict src,uv run pytest -q --disable-socket --allow-unix-socket --allow-hosts=127.0.0.1,::1(582 passed, 1 skipped)Rollout note
Merge before the ci-cd-switchboard config PR that starts dispatching
switchboard_benchmarks_web_releaseandswitchboard_benchmarks_runner_release. Until that config is on Switchboard main, the new gate conservatively falls back to running both local adapters if planning cannot run; once switchboard#150 lands, the normal selected-check path takes over.Greptile Summary
This PR splits the monolithic push-to-main release pipeline into two independently triggered release workflows (
runner-release.ymlandweb-release.yml), each driven byrepository_dispatchfrom a Switchboard config, and introduces a newswitchboard.ymlorchestrator that selectively runs only the relevant CI adapters (ci.ymlorweb-ci.yml) based on changed files. Two frontend lint warnings are also fixed to satisfy--max-warnings 0.switchboard.ymlplans which CI adapters to invoke per PR using changed-file selectors, falls back conservatively to running both if Switchboard cannot plan, and enforces a hard gate job that fails the PR if any selected adapter fails.runner-release.ymladds idempotency (skip ifproduction-runneralready points to the target SHA), Docker image builds, infra dispatch + run-watching, and a marker fast-forward — replacing the oldrelease.yml.web-release.ymladds Vercel production promotion with an identical idempotency gate via aproduction-webmarker branch, andweb-ci.ymladds local OpenAPI schema generation for type-safe codegen in CI.Confidence Score: 5/5
Safe to merge — all workflow logic is well-guarded, fallback paths are explicit, and no existing release or CI gate is removed without a replacement.
The new workflows are carefully structured: release idempotency is gated on marker-branch comparisons, Switchboard failures fall back to running the full suite, and the gate job correctly enforces selected-adapter results. The only new issue found is a benign race window between the background HTTP server start and codegen in web-ci.yml — unlikely to fire in practice and non-blocking to merge.
.github/workflows/runner-release.yml — the context job idempotency skip path is worth verifying once merged with a live dispatch to confirm the skip logic behaves as expected.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant GH as GitHub Event participant SW as switchboard.yml participant RCI as ci.yml participant WCI as web-ci.yml participant SG as switchboard_gate participant RR as runner-release.yml participant WR as web-release.yml participant INFRA as benchmark-infra participant VERCEL as Vercel GH->>SW: pull_request / merge_group SW->>SW: Resolve changed files SW->>SW: Plan Switchboard checks SW->>SW: Resolve selected adapters alt "run_runner_ci == true" SW->>RCI: workflow_call(ref) RCI->>RCI: uv sync / ruff / mypy / pytest end alt "run_web_ci == true" SW->>WCI: workflow_call(ref) WCI->>WCI: pnpm codegen / lint / test / build end SW->>SG: check adapter results GH->>RR: repository_dispatch RR->>RR: context (resolve ref / skip check) RR->>RR: build_and_push (Docker to AR) RR->>INFRA: dispatch image-bump INFRA-->>RR: watch run until complete RR->>RR: fast-forward production-runner GH->>WR: repository_dispatch WR->>WR: context (resolve ref / skip check) WR->>WR: pnpm codegen / vercel build WR->>VERCEL: deploy --prebuilt --prod VERCEL-->>WR: deployment_url WR->>VERCEL: promote deployment WR->>WR: fast-forward production-web%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant GH as GitHub Event participant SW as switchboard.yml participant RCI as ci.yml participant WCI as web-ci.yml participant SG as switchboard_gate participant RR as runner-release.yml participant WR as web-release.yml participant INFRA as benchmark-infra participant VERCEL as Vercel GH->>SW: pull_request / merge_group SW->>SW: Resolve changed files SW->>SW: Plan Switchboard checks SW->>SW: Resolve selected adapters alt "run_runner_ci == true" SW->>RCI: workflow_call(ref) RCI->>RCI: uv sync / ruff / mypy / pytest end alt "run_web_ci == true" SW->>WCI: workflow_call(ref) WCI->>WCI: pnpm codegen / lint / test / build end SW->>SG: check adapter results GH->>RR: repository_dispatch RR->>RR: context (resolve ref / skip check) RR->>RR: build_and_push (Docker to AR) RR->>INFRA: dispatch image-bump INFRA-->>RR: watch run until complete RR->>RR: fast-forward production-runner GH->>WR: repository_dispatch WR->>WR: context (resolve ref / skip check) WR->>WR: pnpm codegen / vercel build WR->>VERCEL: deploy --prebuilt --prod VERCEL-->>WR: deployment_url WR->>VERCEL: promote deployment WR->>WR: fast-forward production-webReviews (2): Last reviewed commit: "ci: split benchmarks switchboard release..." | Re-trigger Greptile