You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Replace Riot with uv as the only test runner while preserving the existing compatibility matrix and locked dependencies.
Test environments are defined in tests/suitespec.py and use flat, descriptive locks under .uv/.
CI reuses build_base_venvs artifacts, so test jobs do not rebuild ddtrace and its native extensions.
Existing Riot lock contents were moved byte-for-byte; riotfile.py and 18 stale OpenTracing locks remain temporarily for follow-up cleanup.
Contributor docs cover the uv workflow and conflict resolution for PRs that still modify Riot.
Each commit is self-contained and can be shipped independently.
Testing
CI is the source of truth. Local validation covers lock parity, matrix expansion, runner behavior, lint, and representative suites.
Area
Jobs, main → uv
Slowest job, main → uv
Critical-path change
Total runner time, main → uv
Environment setup
7 → 7
3m 36s → 2m 41s
25.4% faster
18m 44s → 15m 04s
tracer
19 → 19
11m 54s → 9m 45s
18.0% faster
2h 13m 23s → 2h 01m 49s
aiohttp
5 → 13
4m 49s → 2m 07s
56.2% faster
20m 38s → 23m 55s
Flask
25 → 25
5m 49s → 3m 47s
34.8% faster
1h 19m 20s → 1h 11m 17s
requests
1 → 9
8m 14s → 3m 37s
56.0% faster
8m 14s → 19m 47s
Four suites combined
50 → 66
11m 54s → 9m 45s
18.0% faster
4h 01m 34s → 3h 56m 49s
All four suites have a shorter critical path, and their combined runner time is 2.0% lower. The additional aiohttp and requests shards trade runner concurrency for substantially shorter feedback time.
Source: latest complete focused branch test pipeline at b966e97c5c, compared with the latest successful origin/main test pipeline at 3bb9ecda22. The current head pipeline stopped at prechecks before suite jobs, so it is not a timing sample.
Validation requires exact environment coverage, byte-identical moved locks, successful representative local runs, and passing affected CI jobs.
Risks
The matrix is large, so parity checks and CI catch coverage or dependency drift. Cache and performance optimization remain deferred unless CI shows a material regression.
Additional Notes
This changes internal test infrastructure only and uses the changelog/no-changelog label.
DataDog/apm-reliability/dd-trace-py | ci_visibility/testing 2/18 — ❌ 1 test failed
❌ TestPytestBdd::test_pytest_bdd_with_missing_step_implementation[py3.9] from test_pytest_bdd.py
[Errno 2] No such file or directory: '/tmp/pytest-of-bits/pytest-0/popen-gw3/test_pytest_bdd_with_missing_step_implementation5/bdd_capture.json'
❌ UnittestSnapshotTestCase::test_unittest_will_force_run_multiple_unskippable_tests[py3.10] from test_unittest_snapshot.py
At request <Request GET /test/session/snapshot >:
At snapshot (token='tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_multiple_unskippable_tests'):
- Directory: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots
- CI mode: 1
- Trace File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_multiple_unskippable_tests.json
- Stats File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_multiple_unskippable_tests_tracestats.json
At compare of 1 expected trace(s) to 1 received trace(s):
At trace 'unittest.test_session' (6 spans):
Received fewer spans (1) than expected (6). Expected unmatched spans: 'unittest.test_module', 'unittest.test_suite', 'unittest.test', 'unittest.test', 'unittest.test'
❌ UnittestSnapshotTestCase::test_unittest_will_force_run_unskippable_tests[py3.10] from test_unittest_snapshot.py
At request <Request GET /test/session/snapshot >:
At snapshot (token='tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_unskippable_tests'):
- Directory: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots
- CI mode: 1
- Trace File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_unskippable_tests.json
- Stats File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_force_run_unskippable_tests_tracestats.json
At compare of 1 expected trace(s) to 1 received trace(s):
At trace 'unittest.test_session' (5 spans):
Received fewer spans (1) than expected (5). Expected unmatched spans: 'unittest.test_module', 'unittest.test_suite', 'unittest.test', 'unittest.test'
❌ UnittestSnapshotTestCase::test_unittest_will_include_custom_tests[py3.10] from test_unittest_snapshot.py
At request <Request GET /test/session/snapshot >:
At snapshot (token='tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_include_custom_tests'):
- Directory: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots
- CI mode: 1
- Trace File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_include_custom_tests.json
- Stats File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.unittest.test_unittest_snapshot.test_unittest_will_include_custom_tests_tracestats.json
At compare of 1 expected trace(s) to 1 received trace(s):
At trace 'unittest.test_session' (5 spans):
Received fewer spans (1) than expected (5). Expected unmatched spans: 'unittest.test_module', 'unittest.test_suite', 'unittest.test', 'unittest.test'
❌ test_does_not_trace_empty_poll_when_disabled[py3.9] from test_kafka.py
Traceback (most recent call last):
File "/tmp/pytest-of-bits/pytest-0/popen-gw2/test_does_not_trace_empty_poll6/ddtrace_subprocess_dir/test.py", line 10, in <module>
from tests.contrib.kafka.conftest import group_id
ModuleNotFoundError: No module named 'tests'
assert 1 == 0
❌ test_schematized_span_service_and_operation[None-None][py3.9] from test_kafka.py
assert 1 == 0
❌ test_schematized_span_service_and_operation[None-mysvc][py3.9] from test_kafka.py
❌ test_schematization[None-v0][py3.9] from test_wsgi.py
b'Traceback (most recent call last):
File "/tmp/pytest-of-bits/pytest-0/test_schematization_None_v0__p5/ddtrace_sub..._dir/test.py", line 4, in <module>
from tests.conftest import *
ModuleNotFoundError: No module named \'tests\'
'
assert 1 == 0
❌ test_schematization[mysvc-v0][py3.9] from test_wsgi.py
b'Traceback (most recent call last):
File "/tmp/pytest-of-bits/pytest-0/test_schematization_mysvc_v0__5/ddtrace_sub..._dir/test.py", line 4, in <module>
from tests.conftest import *
ModuleNotFoundError: No module named \'tests\'
'
assert 1 == 0
❌ test_schematization[mysvc-v1][py3.9] from test_wsgi.py
b'Traceback (most recent call last):
File "/tmp/pytest-of-bits/pytest-0/test_schematization_mysvc_v1__5/ddtrace_sub..._dir/test.py", line 4, in <module>
from tests.conftest import *
ModuleNotFoundError: No module named \'tests\'
'
assert 1 == 0
Found 0 performance improvements and 11 performance regressions! Performance is the same for 611 metrics, 10 unstable metrics.
Explanation
This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:
🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline
We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.
If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.
Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.
More details about the CI and significant changes
You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.
CIs of the difference of means are often centered around 0%, because often changes are not that big:
---------------------------------(------|---^--------)-------------------------------->
-0.6% 0% 0.3% +1.2%
| | |
lower bound of the CI --' | |
sample mean (center of the CI) -------------' |
upper bound of the CI ----------------------'
As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).
For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:
----------------------------------------|---------|---(---------^---------)---------->
0% 1% 1.3% 2.2% 3.1%
| | | |
significant impact threshold --------------' | | |
lower bound of CI --------------' | |
sample mean (center of the CI) --------------------------' |
upper bound of CI ----------------------------------'
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AI GeneratedLargely based on code generated by an AI or LLM. This label is the same across all dd-trace-* reposchangelog/no-changelogA changelog entry is not required for this PR.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace Riot with uv as the only test runner while preserving the existing compatibility matrix and locked dependencies.
tests/suitespec.pyand use flat, descriptive locks under.uv/.build_base_venvsartifacts, so test jobs do not rebuild ddtrace and its native extensions.riotfile.pyand 18 stale OpenTracing locks remain temporarily for follow-up cleanup.Each commit is self-contained and can be shipped independently.
Testing
CI is the source of truth. Local validation covers lock parity, matrix expansion, runner behavior, lint, and representative suites.
All four suites have a shorter critical path, and their combined runner time is 2.0% lower. The additional aiohttp and requests shards trade runner concurrency for substantially shorter feedback time.
Source: latest complete focused branch test pipeline at
b966e97c5c, compared with the latest successful origin/main test pipeline at3bb9ecda22. The current head pipeline stopped at prechecks before suite jobs, so it is not a timing sample.Validation requires exact environment coverage, byte-identical moved locks, successful representative local runs, and passing affected CI jobs.
Risks
The matrix is large, so parity checks and CI catch coverage or dependency drift. Cache and performance optimization remain deferred unless CI shows a material regression.
Additional Notes
This changes internal test infrastructure only and uses the
changelog/no-changeloglabel.