Skip to content

Canonical CI: grouped-tests.yml + root test/test_groups.toml#1484

Open
ChrisRackauckas-Claude wants to merge 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:grouped-tests-ci
Open

Canonical CI: grouped-tests.yml + root test/test_groups.toml#1484
ChrisRackauckas-Claude wants to merge 2 commits into
SciML:masterfrom
ChrisRackauckas-Claude:grouped-tests-ci

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Summary

Converts the root test workflow (.github/workflows/Test.yml, name: "Tests") from a hand-maintained version × group matrix into the canonical grouped-tests.yml@v1 thin caller, with the matrix declared once in a root test/test_groups.toml.

What changed

  • .github/workflows/Test.yml — the matrix test job (formerly tests.yml@v1 with a 3-version × 5-group matrix) becomes the thin caller:
    jobs:
      tests:
        uses: "SciML/.github/.github/workflows/grouped-tests.yml@v1"
        with:
          coverage: false
        secrets: "inherit"
    Filename, name: "Tests", on: triggers, and concurrency: are preserved verbatim. coverage: false is kept (the old job disabled coverage). group-env-name is omitted because runtests.jl reads the canonical GROUP var; check-bounds, coverage-directories, apt-packages are all left at their defaults (the old job used the defaults). Linux-only: no os field (default ubuntu-latest).
  • test/test_groups.toml (new) — [Core] on ["lts","1","pre"], [QA] on ["lts","1"].
  • Canonical Core/QA refactor — this repo had no Aqua/JET; its only QA-style check was the ExplicitImports import-hygiene test running inline in the Misc group. That check is moved into an isolated QA group:
    • test/qa/Project.toml[deps] = Catalyst (via [sources] path = "../..") + ExplicitImports + Test; [compat] julia = "1.10", ExplicitImports = "1.15".
    • test/qa/qa.jl — the moved ExplicitImports checks.
    • runtests.jl — when GROUP == "QA", activate test/qa, Pkg.develop the repo root, instantiate, and include("qa/qa.jl"). The functional suite (formerly Modeling/Simulation/Hybrid/Misc/Spatial) now runs under Core (and the local default All); the fine-grained group names still work for local runs.
  • Project.tomlExplicitImports dropped from [extras], [targets].test, and [compat] (it now lives only in the QA env). julia compat floor is already "1.10" (LTS), unchanged.

Extension tests remain owned by the separate TestExtensions.yml workflow (untouched), as before; the root matrix does not run the Extensions group. All other workflow files are untouched.

Matrix match

compute_affected_sublibraries.jl . --root-matrix (from SciML/.github@v1) emits exactly the intended canonical matrix:

group versions runner
Core lts, 1, pre ubuntu-latest
QA lts, 1 ubuntu-latest

= 5 cells, all ubuntu-latest. This is a deliberate consolidation of the old 15-cell matrix (5 functional groups × 3 versions): the 5 functional groups collapse into a single Core group (running the union of all functional tests) on the same 3 versions, and the inline ExplicitImports check becomes a dedicated QA group on [lts, 1]. Same test coverage, canonical Core/QA shape. Emitted matrix verified to match the authored test_groups.toml exactly (5/5).

QA validation

Catalyst is a heavy package (ModelingToolkitBase / Symbolics / SymbolicUtils precompile, > ~20 min cold), so the QA group was not run locally — CI validates it. The QA structure was statically verified: test_groups.toml, test/qa/Project.toml, and Project.toml all parse as TOML; Test.yml parses as YAML; runtests.jl and qa/qa.jl parse as Julia; the [sources] path ../.. resolves to the Catalyst Project.toml. No tests were skipped, broken, or silenced.

Ignore until reviewed by @ChrisRackauckas.

🤖 Generated with Claude Code

Convert the root test workflow (Test.yml) to the canonical
grouped-tests.yml@v1 thin caller, declaring the group x version matrix
once in test/test_groups.toml instead of a hand-maintained YAML matrix.

- Test.yml: replace the version x group matrix job (tests.yml@v1) with
  the grouped-tests.yml@v1 thin caller. Preserve filename, name: "Tests",
  on: triggers, and concurrency: verbatim. Keep coverage: false (the old
  job disabled coverage); all other inputs are defaults.
- test/test_groups.toml: [Core] on [lts, 1, pre]; [QA] on [lts, 1].
  Linux-only (no os field -> ubuntu-latest default).
- Refactor to canonical Core/QA. The functional suite (formerly the
  Modeling/Simulation/Hybrid/Misc/Spatial groups) now runs under GROUP
  "Core" (and the local-default "All"); the fine-grained names still work
  for local runs. The ExplicitImports import-hygiene check moves out of
  the inline Misc group into an isolated QA group (test/qa/Project.toml
  with [sources] path to the repo root + test/qa/qa.jl) gated on
  GROUP == "QA".
- Project.toml: drop ExplicitImports from [extras]/[targets].test and its
  [compat] entry (now lives only in the QA env). julia compat floor is
  already "1.10" (LTS).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@isaacsas

isaacsas commented Jun 8, 2026

Copy link
Copy Markdown
Member

Why is this eliminating the current group structure? That was intentional as otherwise CI times become gigantic.

@ChrisRackauckas

Copy link
Copy Markdown
Member

Nothing is eliminated.

@ChrisRackauckas

Copy link
Copy Markdown
Member

The matrix is exactly the same.

@isaacsas

isaacsas commented Jun 8, 2026

Copy link
Copy Markdown
Member

The PR says it explicitly now runs all tests in one group except for QA though, and most of the prior groups are not present in the group CI file.

@ChrisRackauckas

Copy link
Copy Markdown
Member

Okay that's an error. They aren't supposed to be in the CI file, but they aren't reflected in the test_groups. I'll get that fixed.

@ChrisRackauckas-Claude ChrisRackauckas-Claude marked this pull request as ready for review June 9, 2026 21:52
The grouped-tests-ci conversion collapsed Catalyst's six parallel
functional test groups into a single [Core] entry and dropped the
Extensions group entirely. runtests.jl dispatches on Modeling,
Simulation, Hybrid, Misc, Spatial, and Extensions (the last gated on
`GROUP == "All" || GROUP == "Extensions"`, which RUN_ALL/Core never
covers), so with only [Core]+[QA] the Extensions suite never ran and
the other five no longer ran as separate CI jobs.

List each named functional group as its own entry, matching the old
Test.yml group matrix (Modeling/Simulation/Hybrid/Misc/Spatial on
lts/1/pre) plus Extensions (lts/1/pre, from TestExtensions.yml), and
keep [QA] on lts/1.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread test/test_groups.toml
Comment on lines +1 to +17
[Modeling]
versions = ["lts", "1", "pre"]

[Simulation]
versions = ["lts", "1", "pre"]

[Hybrid]
versions = ["lts", "1", "pre"]

[Misc]
versions = ["lts", "1", "pre"]

[Spatial]
versions = ["lts", "1", "pre"]

[Extensions]
versions = ["lts", "1", "pre"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sis good now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants