python profile: benchmarks run in the unit test session, and the docstring names a non-existent target

[simontaurus]

The benchmark scaffold in the python profile contradicts itself in two ways.

1. Benchmarks are collected by the unit test session

profiles/python/tests/benchmarks/test_benchmarks.py states benchmarks should never run with the unit tests:

"""Performance benchmarks; run with `make benchmark`, never with the unit tests."""

But profiles/python/pyproject.toml collects them anyway:

[tool.pytest.ini_options]
testpaths = ["tests"]

tests/benchmarks/ is under tests/, so make test runs the benchmarks with coverage enabled. That is what profiles/python/pytest.benchmark.ini exists to avoid:

# Benchmarks run in their own session: no coverage (it distorts timings) and
# only the tests under tests/benchmarks/.

Effect: benchmarks execute twice (once in make test, once in make bench), make test is slower, and the timings taken in the unit session are distorted by coverage instrumentation.

2. The docstring names a target that does not exist

The docstring says make benchmark. profiles/python/Makefile defines bench, not benchmark.

Suggested fix

Exclude the benchmark directory from the default session:

addopts = "--cov --cov-config=pyproject.toml --cov-report=term-missing --ignore=tests/benchmarks"

and correct the docstring to make bench.

Found while applying the python profile to OO-LD/awl-python.