Skip to content

ci: skip queue-verified Dev and Dependencies jobs on pushes to main - #25443

Merged
alamb merged 1 commit into
apache:mainfrom
comphead:ci/skip-post-merge-dev-dependencies
Sep 18, 2026
Merged

alamb merged 1 commit into
apache:mainfrom
comphead:ci/skip-post-merge-dev-dependencies

Conversation

@comphead

@comphead comphead commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The inflated test count on the main commits page is duplicated checks. Every commit is checked twice at the same SHA, once by the merge queue and once by the push that follows.

The merge_group event is separate from the pull_request and push events.

— Events that trigger workflows

Neither event knows the other ran, so a workflow listing both runs twice. At 3ab72e6 the two Dev runs share a head_sha and differ only in event and ref: 35262885559 (merge_group, gh-readonly-queue/main/pr-25383-98b26c5f9…) and 35263735243 (push, main). Dependencies is the same pair. GET /commits/3ab72e61…/check-runs returns each of these seven names twice:

Check License Header                              Circular Dependency Check
Check Markdown Links                              Detect Unused Dependencies
Spell Check with Typos
Use prettier to check formatting of documents
Validate required_status_checks in .asf.yaml

The existing branches-ignore: 'gh-readonly-queue/**' does not prevent this. It suppresses a third run, the push event on the queue's own temporary branch, not the push to main.

Scale: from September 9 to 17, 2026 each workflow ran 156 times for pushes to main, and all 156 SHAs already had a merge_group run of the same workflow at the same SHA. Over the seven full days September 10 to 16 there were 120 such commits, at 1.81 (Dev) plus 4.69 (Dependencies) mean runner-minutes each, so about 780 runner-minutes per week. All seven jobs report runner_group: "GitHub Actions" and none use Runs-On, so every minute saved comes off the shared GitHub-hosted quota #25148 is about.

Seven other names are also doubled at that SHA. Those are the rust.yml jobs #25365 deliberately kept on main for caches, coverage, and the non-required FFI check. This PR leaves them alone.

What changes are included in this PR?

main is added to the existing branches-ignore list in .github/workflows/dev.yml and .github/workflows/dependencies.yml.

Event Dev / Dependencies jobs
Push to main 0
Merge group, pull request, push to any other branch 7
workflow_dispatch (Dependencies) 2

Unlike rust.yml, neither workflow has a job worth keeping post-merge: none uses actions/cache or rust-cache, none uploads an artifact or coverage, and all seven are required checks the merge queue already satisfies. With nothing to preserve, excluding the event at the trigger is simpler and cheaper than a per-job if:, since no run is created at all.

Keeping branches-ignore rather than a positive branches: list is deliberate: the merge queue is configured only for main, so release branches have no pre-merge run to inherit and must keep post-push coverage.

Every commit reaches upstream `main` through the merge queue, which runs
`dev.yml` and `dependencies.yml` at the same SHA. The push that follows
runs the same seven jobs again at that SHA.

Add `main` to the existing `branches-ignore` list in both workflows.
Release branches and every other branch keep their post-push runs, and
the merge queue, pull requests, and `workflow_dispatch` are unaffected.
@github-actions github-actions Bot added the development-process Related to development process of DataFusion label Sep 17, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.33%. Comparing base (edc936f) to head (6076818).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25443      +/-   ##
==========================================
- Coverage   82.33%   82.33%   -0.01%     
==========================================
  Files        1137     1137              
  Lines      432498   432498              
  Branches   432498   432498              
==========================================
- Hits       356116   356115       -1     
- Misses      54843    54844       +1     
  Partials    21539    21539              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb

alamb commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Let's give it a try!

@alamb
alamb added this pull request to the merge queue Sep 18, 2026
@alamb

alamb commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Thanks @comphead

Merged via the queue into apache:main with commit 3a647e4 Sep 18, 2026
43 checks passed
@comphead

Copy link
Copy Markdown
Contributor Author

Less duplicates, but some still exist, would need a followup

@alamb

alamb commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Though it looks good to me that it is now explicitly skipping the push checks (as they already ran in the merge_group) 👍

Thanks for this @comphead

Screenshot 2026-09-18 at 3 23 21 PM

@comphead

Copy link
Copy Markdown
Contributor Author

it still runs linux tests twice, I'm brainstorming how to dedup it gently, prob would require some CI restructure.

haohuaijin pushed a commit to haohuaijin/arrow-datafusion that referenced this pull request Sep 19, 2026
…pache#25443)

## Which issue does this PR close?

- Part of apache#25148. This PR does not close the issue.
- Follow-up to apache#25365, which did the same for `rust.yml`. Answers
apache#25365 (comment).

## Rationale for this change

The inflated test count on the `main` commits page is duplicated checks.
Every commit is checked twice at the same SHA, once by the merge queue
and once by the push that follows.

> The `merge_group` event is separate from the `pull_request` and `push`
events.
>
> — [Events that trigger
workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#merge_group)

Neither event knows the other ran, so a workflow listing both runs
twice. At 3ab72e6 the two `Dev` runs share a `head_sha` and differ only
in event and ref:
[35262885559](https://github.com/apache/datafusion/actions/runs/35262885559)
(`merge_group`, `gh-readonly-queue/main/pr-25383-98b26c5f9…`) and
[35263735243](https://github.com/apache/datafusion/actions/runs/35263735243)
(`push`, `main`). `Dependencies` is the same pair. `GET
/commits/3ab72e61…/check-runs` returns each of these seven names twice:

```
Check License Header                              Circular Dependency Check
Check Markdown Links                              Detect Unused Dependencies
Spell Check with Typos
Use prettier to check formatting of documents
Validate required_status_checks in .asf.yaml
```

The existing `branches-ignore: 'gh-readonly-queue/**'` does not prevent
this. It suppresses a *third* run, the push event on the queue's own
temporary branch, not the push to `main`.

Scale: from September 9 to 17, 2026 each workflow ran 156 times for
pushes to `main`, and all 156 SHAs already had a `merge_group` run of
the same workflow at the same SHA. Over the seven full days September 10
to 16 there were 120 such commits, at 1.81 (`Dev`) plus 4.69
(`Dependencies`) mean runner-minutes each, so about 780 runner-minutes
per week. All seven jobs report `runner_group: "GitHub Actions"` and
none use Runs-On, so every minute saved comes off the shared
GitHub-hosted quota apache#25148 is about.

Seven other names are also doubled at that SHA. Those are the `rust.yml`
jobs apache#25365 deliberately kept on `main` for caches, coverage, and the
non-required FFI check. This PR leaves them alone.

## What changes are included in this PR?

`main` is added to the existing `branches-ignore` list in
`.github/workflows/dev.yml` and `.github/workflows/dependencies.yml`.

| Event | Dev / Dependencies jobs |
| --- | --: |
| Push to `main` | 0 |
| Merge group, pull request, push to any other branch | 7 |
| `workflow_dispatch` (`Dependencies`) | 2 |

Unlike `rust.yml`, neither workflow has a job worth keeping post-merge:
none uses `actions/cache` or `rust-cache`, none uploads an artifact or
coverage, and all seven are required checks the merge queue already
satisfies. With nothing to preserve, excluding the event at the trigger
is simpler and cheaper than a per-job `if:`, since no run is created at
all.

Keeping `branches-ignore` rather than a positive `branches:` list is
deliberate: the merge queue is configured only for `main`, so release
branches have no pre-merge run to inherit and must keep post-push
coverage.
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Oct 11, 2026
…pache#25443)

## Which issue does this PR close?

- Part of apache#25148. This PR does not close the issue.
- Follow-up to apache#25365, which did the same for `rust.yml`. Answers
apache#25365 (comment).

## Rationale for this change

The inflated test count on the `main` commits page is duplicated checks.
Every commit is checked twice at the same SHA, once by the merge queue
and once by the push that follows.

> The `merge_group` event is separate from the `pull_request` and `push`
events.
>
> — [Events that trigger
workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#merge_group)

Neither event knows the other ran, so a workflow listing both runs
twice. At 3ab72e6 the two `Dev` runs share a `head_sha` and differ only
in event and ref:
[35262885559](https://github.com/apache/datafusion/actions/runs/35262885559)
(`merge_group`, `gh-readonly-queue/main/pr-25383-98b26c5f9…`) and
[35263735243](https://github.com/apache/datafusion/actions/runs/35263735243)
(`push`, `main`). `Dependencies` is the same pair. `GET
/commits/3ab72e61…/check-runs` returns each of these seven names twice:

```
Check License Header                              Circular Dependency Check
Check Markdown Links                              Detect Unused Dependencies
Spell Check with Typos
Use prettier to check formatting of documents
Validate required_status_checks in .asf.yaml
```

The existing `branches-ignore: 'gh-readonly-queue/**'` does not prevent
this. It suppresses a *third* run, the push event on the queue's own
temporary branch, not the push to `main`.

Scale: from September 9 to 17, 2026 each workflow ran 156 times for
pushes to `main`, and all 156 SHAs already had a `merge_group` run of
the same workflow at the same SHA. Over the seven full days September 10
to 16 there were 120 such commits, at 1.81 (`Dev`) plus 4.69
(`Dependencies`) mean runner-minutes each, so about 780 runner-minutes
per week. All seven jobs report `runner_group: "GitHub Actions"` and
none use Runs-On, so every minute saved comes off the shared
GitHub-hosted quota apache#25148 is about.

Seven other names are also doubled at that SHA. Those are the `rust.yml`
jobs apache#25365 deliberately kept on `main` for caches, coverage, and the
non-required FFI check. This PR leaves them alone.

## What changes are included in this PR?

`main` is added to the existing `branches-ignore` list in
`.github/workflows/dev.yml` and `.github/workflows/dependencies.yml`.

| Event | Dev / Dependencies jobs |
| --- | --: |
| Push to `main` | 0 |
| Merge group, pull request, push to any other branch | 7 |
| `workflow_dispatch` (`Dependencies`) | 2 |

Unlike `rust.yml`, neither workflow has a job worth keeping post-merge:
none uses `actions/cache` or `rust-cache`, none uploads an artifact or
coverage, and all seven are required checks the merge queue already
satisfies. With nothing to preserve, excluding the event at the trigger
is simpler and cheaper than a per-job `if:`, since no run is created at
all.

Keeping `branches-ignore` rather than a positive `branches:` list is
deliberate: the merge queue is configured only for `main`, so release
branches have no pre-merge run to inherit and must keep post-push
coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants