Summary
mutation-cargo.yml's unmutated baseline build fails deterministically
for leynos/tei-rapporteur, a caller whose tei-py crate embeds a
Python interpreter via PyO3 for its tests. The baseline cargo test
run tries to execute the tei-py test binary, which is dynamically
linked against a uv-managed libpython3.13.so.1.0 that is absent from
the runtime linker's search path. No mutant is ever tested.
Observed
Two scheduled runs, both full (unfiltered) mutation sweeps, fail
identically:
Both runs show, byte-for-byte, the same failure in the "Run mutation
testing" step:
Found <N> mutants to test
FAILED Unmutated baseline in 51-53s build + 0s test
*** baseline
*** .../cargo test --no-run --verbose --package=tei-core@0.1.0 --package=tei-py@0.1.0 --package=tei-xml@0.1.0 --all-features
...
Running `.../target/debug/deps/tei_py-18a10b569daa82cc`
/home/runner/work/tei-rapporteur/tei-rapporteur/target/debug/deps/tei_py-18a10b569daa82cc: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory
error: test failed, to rerun pass `-p tei-py --lib`
Caused by:
process didn't exit successfully: `.../tei_py-18a10b569daa82cc` (exit status: 127)
*** result: Failure(127)
mutation_cargo_exit_code=4
mutation_cargo_outcome=baseline tests failing
##[error]Process completed with exit code 4.
Both runs fail on the exact same test binary hash
(tei_py-18a10b569daa82cc), which strongly suggests a deterministic
environment gap rather than a flake — every other crate's test suite
in the baseline (tei-core, tei-xml, dozens of test result: ok
lines) builds and passes; only tei-py's embedded-interpreter tests
fail, and always the same way.
Job summaries for both runs confirm zero mutants were ever scored:
0% - 0 hits, 0 misses, 0 errors. There is no partial survivor data
to recover from either run — the baseline failure aborts before any
mutant is generated or tested.
Root cause
mutation-cargo.yml installs a uv-managed Python (here, UV_PYTHON: 3.13, cpython-3.13.14-linux-x86_64-gnu) to run its own orchestration
script (workflow_scripts/mutation_run_cargo.py). Because uv's shims
land ahead of the system python3 on PATH, PyO3's build script
(pyo3-build-config, invoked while compiling tei-py) resolves that
uv-managed interpreter and links tei-py's test binaries against its
libpython3.13.so.1.0. That shared object lives under uv's private
Python install directory
($UV_PYTHON_INSTALL_DIR/~/.local/share/uv/python/...), which is
never added to LD_LIBRARY_PATH (or baked into the binary's RPATH),
so the dynamic linker cannot find it at test-run time.
leynos/tei-rapporteur's own ci.yml does not hit this: its
generate-coverage step (which runs cargo test/nextest) executes
before actions/setup-python/uv are installed in that job, so
PyO3 links against the runner image's system Python instead — a
library that is already on the default ld.so search path. The
mutation workflow's ordering (uv/Python set up before the Rust test
build) is the opposite, and exposes the gap.
Any caller crate that embeds a Python interpreter via PyO3 for its
tests (the auto-initialize/embedding pattern, as opposed to a pure
extension-module build) will hit this whenever the reusable workflow
provisions its own Python ahead of the cargo build.
Suggested fix
One of:
- Export
LD_LIBRARY_PATH (including the directory containing
libpython*.so* under uv's managed install) before invoking cargo mutants/cargo test in mutation-cargo.yml, so PyO3 test binaries
that link the uv-managed interpreter can find it at runtime; or
- Set
PYO3_PYTHON (or otherwise pin the interpreter PyO3's build
script resolves) to a system Python already on the default library
search path, so the workflow's own Python provisioning cannot shadow
it; or
- Document a
setup-commands-based workaround (the input already
exists) in docs/mutation-cargo-workflow.md for callers with
PyO3-embedding crates, until one of the above lands.
Impact
leynos/tei-rapporteur's mutation-testing workflow has never produced a
usable result: every scheduled/dispatched run fails the baseline
before a single mutant is scored, for a crate (tei-py) that is
central to the workspace's mutation scope (it appears in both shards
sampled here). This is a total loss of mutation-testing coverage for
that crate and anything gated on --test-workspace=true transitively
exercising it.
Summary
mutation-cargo.yml's unmutated baseline build fails deterministicallyfor leynos/tei-rapporteur, a caller whose
tei-pycrate embeds aPython interpreter via PyO3 for its tests. The baseline
cargo testrun tries to execute the
tei-pytest binary, which is dynamicallylinked against a uv-managed
libpython3.13.so.1.0that is absent fromthe runtime linker's search path. No mutant is ever tested.
Observed
Two scheduled runs, both full (unfiltered) mutation sweeps, fail
identically:
(job failed after ~1m26s of active compute; "Found 418 mutants to
test")
(job failed after ~1m25s of active compute — the run shows 106
minutes overall, but ~104 minutes of that is scheduling/queue delay
before the
mutantsjob starts; "Found 486 mutants to test")Both runs show, byte-for-byte, the same failure in the "Run mutation
testing" step:
Both runs fail on the exact same test binary hash
(
tei_py-18a10b569daa82cc), which strongly suggests a deterministicenvironment gap rather than a flake — every other crate's test suite
in the baseline (
tei-core,tei-xml, dozens oftest result: oklines) builds and passes; only
tei-py's embedded-interpreter testsfail, and always the same way.
Job summaries for both runs confirm zero mutants were ever scored:
0% - 0 hits, 0 misses, 0 errors. There is no partial survivor datato recover from either run — the baseline failure aborts before any
mutant is generated or tested.
Root cause
mutation-cargo.ymlinstalls a uv-managed Python (here,UV_PYTHON: 3.13,cpython-3.13.14-linux-x86_64-gnu) to run its own orchestrationscript (
workflow_scripts/mutation_run_cargo.py). Because uv's shimsland ahead of the system
python3onPATH, PyO3's build script(
pyo3-build-config, invoked while compilingtei-py) resolves thatuv-managed interpreter and links
tei-py's test binaries against itslibpython3.13.so.1.0. That shared object lives under uv's privatePython install directory
(
$UV_PYTHON_INSTALL_DIR/~/.local/share/uv/python/...), which isnever added to
LD_LIBRARY_PATH(or baked into the binary's RPATH),so the dynamic linker cannot find it at test-run time.
leynos/tei-rapporteur's own
ci.ymldoes not hit this: itsgenerate-coveragestep (which runscargo test/nextest) executesbefore
actions/setup-python/uvare installed in that job, soPyO3 links against the runner image's system Python instead — a
library that is already on the default
ld.sosearch path. Themutation workflow's ordering (uv/Python set up before the Rust test
build) is the opposite, and exposes the gap.
Any caller crate that embeds a Python interpreter via PyO3 for its
tests (the
auto-initialize/embedding pattern, as opposed to a pureextension-modulebuild) will hit this whenever the reusable workflowprovisions its own Python ahead of the cargo build.
Suggested fix
One of:
LD_LIBRARY_PATH(including the directory containinglibpython*.so*under uv's managed install) before invokingcargo mutants/cargo testinmutation-cargo.yml, so PyO3 test binariesthat link the uv-managed interpreter can find it at runtime; or
PYO3_PYTHON(or otherwise pin the interpreter PyO3's buildscript resolves) to a system Python already on the default library
search path, so the workflow's own Python provisioning cannot shadow
it; or
setup-commands-based workaround (the input alreadyexists) in
docs/mutation-cargo-workflow.mdfor callers withPyO3-embedding crates, until one of the above lands.
Impact
leynos/tei-rapporteur's mutation-testing workflow has never produced a
usable result: every scheduled/dispatched run fails the baseline
before a single mutant is scored, for a crate (
tei-py) that iscentral to the workspace's mutation scope (it appears in both shards
sampled here). This is a total loss of mutation-testing coverage for
that crate and anything gated on
--test-workspace=truetransitivelyexercising it.