Improve CI workflows

[ZiaCheemaGit]

To ensure we are guaranteeing everything works on our support matrix. We support Python 3.9-3.14, so we need to test all of them. We support Linux and MacOS, at least one test for each. We support GCC and Clang on Linux so we at least one test each.

But you are right we are probably testing some thing unnecessarily. For example, we don't need to run the C++-only tests on every combination of Python version, just every compiler and OS.

We could split the C++ and Python tests up so that we only run on tests where that matters.

First, lets get this in. We can discuss improvements on Friday.

Originally posted by @ktbarrett in #224 (comment)

[ZiaCheemaGit]

We should be running the limited set of integration tests (test_logic.py, test_range.py, etc.) always, for coverage. What I don't want is running the entire cocotb regression every time since it takes forever.
I think we can fold the integration_tests.yaml flow into the dev flow now that it's running on every push.

@ktbarrett

• But we are just running minimal tests which doesnt even contain heavy platform and dependency differences so why run same tests ten time.
• A benifit to merge both would be that, we will need to build only once. Current approach builds coconext two times. once for dev_tests and once for integration_tests. and if we combine then makefile can also be simplified. but then in future we will to rework.

Your call.

_Originally posted by @ZiaCheemaGit in #224 (comment)

[ZiaCheemaGit]

Problems to address

• CI builds coconext twice
• All tests dont need to run on full CI matrix
• Only report integration test failures and dont block PRs on failing integration. And also coverage drops if integration tests fails.

[ktbarrett]

I think we can set this up similar to how cocotb does it, but a bit better. Basically the flow is:

1. Multiple flows have different triggers (build-test-dev.yaml, build-test-release.yaml in cocotb)
2. All flows that need to run tests invoke the same common Github Action test flow (regression-test.yaml)
3. We programmatically derive the test matrix using info about which flow we are running (release vs dev vs whatever) (generate-envs.py in cocotb)
4. We have different test groups (pure Python, pure C++, nanobind, integration, etc.) each has it's own test matrix which are then joined together into a single test matrix for the test flow
5. We have the common Github Action test flow and we use the test groups derived by the Python script determine which steps in the flow to run/skip to set up our test environment
6. Final step is to run the Makefiles passing the test groups we want. Common actions (like compiling coconext and cocotb) will only happen once because all these flows only run common tasks once.

Triggers

We have multiple kinds of flow triggers:

• pull requests run dev test flow
• dev branch merges run truncated release flow (release smoke test)
• tags run the full release test flow and upload build artifacts (full release)

Support matrix

For C++ tests without a simulator our test matrix is limited to the following:

• platform (implies minimum support compiler on that platform, so Ubuntu 22.04 implies GCC 11 or clang 14 minimum)
  • Ubuntu 22.04
  • Ubuntu 24.04
  • Ubuntu 26.04 (should be available soon)
• C++ version:
  • C++20
  • C++23
  • C++26 (unlikely to see any platform with support for this any time soon)

For pure Python test without a simulator we only care about:

• Python version:
  • CPython 3.9 - 3.14

For integration tests without a simulator we need the combination of the two above as compiling nanobind code with newer C++ versions can change behavior or add new functionality. And also the Python version because we might bind features that only work on newer versions of Python.

For code that requires a simulator we also need to consider simulator and simulator version. Since we are in development I don't want to care about a ton of simulators or versions right now. Basically we should limit ourselves to FOSS simulators and the most recent release. So:

• Simulators
  • NVC
  • Verilator
  • Icarus
  • GHDL (Maybe? this simulator has a bunch of issues)

Test matrix per flow

Dev tests: Kit every support option at least once, but not an exhaustive cross product
Release smoke test: Like dev test but testing the build release objects
Release: Test the full cross-product of support

Test groups

These should map to kinds of tests we run in the Makefile. So some ideas are:

• cpp_tests
• python_tests
• integration_tests

Options

In addition to test groups there are other options we need to set:

• coverage: enables the codecov upload, sets Makefile variable and adds a Make rule for joining coverage and generating the report
• release: enables step for downloading build artifacts and sets Makefile variable so the tests install the wheel or sdist instead of installing the local package when installing coconext

Main Github Action flow

Basically what we have now works, just merge the release and dev steps.

Makefile

We need individual rules for each test group. Another for uv sync and installing the package. Another for building the wheel and sdist which will be used by the first part of the release flow.

[ZiaCheemaGit]

Future work must consider this comment