Skip to content

ci(swe-bench): fix invalid job-level secrets gate#145

Merged
principle-lgtm merged 1 commit into
developfrom
fix/swe-bench-workflow
May 18, 2026
Merged

ci(swe-bench): fix invalid job-level secrets gate#145
principle-lgtm merged 1 commit into
developfrom
fix/swe-bench-workflow

Conversation

@principle-lgtm
Copy link
Copy Markdown
Contributor

Summary

Fixes #143 — `swe-bench.yml` was failing in 0s on every push to every branch because of a job-level `if:` referencing the `secrets` context:

```yaml

Gate on API key availability

if: ${{ secrets.ANTHROPIC_API_KEY != '' }}
```

The `secrets` context isn't available in job-level `if:` expressions — only inside step `run:`/`uses:`/`with:` blocks. So GitHub Actions rejected the file on every event (not just `workflow_dispatch`), surfacing as a noisy failed run that masked real CI failures from readers scanning the workflows list.

Fix

  • Drop the job-level `if:`.
  • Add a first-step "Require ANTHROPIC_API_KEY secret" guard that exits 1 with a clear `::error::` message if the secret is empty at runtime. Manual triggers without the secret configured now fail fast and loud at the gate instead of mid-evaluation with a cryptic Rust panic.
  • Also quote `$GITHUB_STEP_SUMMARY` in the final `report` step (`SC2086` shellcheck warning, called out in CI: swe-bench.yml fails to parse due to job-level if: referencing secrets #143).

Verification

  • `actionlint` no longer flags either issue on this file. (Two remaining actionlint findings on `swe-bench.yml` are about `actions/cache@v3` and `actions/upload-artifact@v3` being too old — out of scope here; PR ci: bump GitHub Actions to Node 24-compatible versions #142 bumps those.)
  • `python3 -c "import yaml; yaml.safe_load(...)"` parses cleanly.
  • The workflow is `workflow_dispatch` only, so true end-to-end verification is a manual trigger after merge. The next push to `develop` should no longer show a `swe-bench.yml` failed-run entry in the actions list.

Closes #143

🤖 Generated with Claude Code

The `if: ${{ secrets.ANTHROPIC_API_KEY != '' }}` at job level was
invalid — the `secrets` context isn't available in job-level `if:`
expressions, only inside step `run:`/`uses:`/`with:` blocks. GitHub
Actions therefore rejected the file on every push event, surfacing a
0-second "workflow file issue" failure even though the workflow is
`workflow_dispatch` only.

Fix:
- Drop the job-level `if:`.
- Add a first-step guard that exits 1 with a clear ::error:: message
  if ANTHROPIC_API_KEY is empty at runtime, so manual triggers without
  the secret configured fail loud at the gate instead of mid-evaluation.
- Quote `$GITHUB_STEP_SUMMARY` in the report step (SC2086).

Closes #143
@principle-lgtm principle-lgtm merged commit e0a00b2 into develop May 18, 2026
7 checks passed
@principle-lgtm principle-lgtm deleted the fix/swe-bench-workflow branch May 18, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant