Skip to content

Conversation

@bm1549
Copy link
Contributor

@bm1549 bm1549 commented Feb 9, 2026

Motivation

Re-running parametric tests (e.g. when iterating on a test or debugging) was slow because each run could rebuild the parametric library image and, with multiple workers, start a new test agent and library container per test. This PR documents and implements options so parametric re-runs are faster: single-worker mode (+1) for shared agent and library container reuse, and --skip-parametric-build to skip rebuilding the image when only test code or docs change.

Runtime impact (parametric re-runs)

Measured on a single parametric test: tests/parametric/test_tracer.py::Test_Tracer::test_tracer_span_top_level_attributes) with TEST_LIBRARY=nodejs

On a secondary run, using +1 and --skip-parametric-build is about 3-7× faster in pytest time and ~2-5× faster in wall time, from skipping the parametric image build and reusing one agent and one library container.

Before After
Pytest time ~7.3s ~1.1-4.5s
Wall time ~10s ~2-6s

Changes

  • Docs: Added “Making parametric runs faster” in docs/scenarios/parametric.md; added .cursor/rules/parametric-testing.mdc and updated AGENTS.md to recommend +1 and --skip-parametric-build for re-runs.
  • run.sh: Documented +1 and that single-worker PARAMETRIC runs reuse one agent and one library container per session.
  • Framework: Added --skip-parametric-build (and SKIP_PARAMETRIC_BUILD) in conftest.py and utils/_context/_scenarios/parametric.py; in utils/docker_fixtures/ implemented session-scoped agent and library container reuse for parametric when using a single worker.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

…rary builds and reuse containers. Introduce new rules for parametric testing in documentation. Update related scripts and fixtures for improved performance during test runs.
@bm1549 bm1549 added the ai-generated The pull request includes a significant amount of AI-generated code label Feb 9, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

CODEOWNERS have been resolved as:

.cursor/rules/parametric-testing.mdc                                    @DataDog/system-tests-core
AGENTS.md                                                               @DataDog/system-tests-core
conftest.py                                                             @DataDog/system-tests-core
docs/scenarios/parametric.md                                            @DataDog/system-tests-core
run.sh                                                                  @DataDog/system-tests-core
tests/parametric/conftest.py                                            @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/_context/_scenarios/parametric.py                                 @DataDog/system-tests-core
utils/docker_fixtures/_core.py                                          @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

@datadog-datadog-prod-us1

This comment has been minimized.

@bm1549 bm1549 changed the title Speed up local parametric testing capabilities Speed up local parametric testing capabilities by 2-5x Feb 10, 2026
Copy link
Collaborator

@nccatoni nccatoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes could cause reliability issues. While it might be fine to speedup local development you should make sure that all parametric tests do indeed pass. You should also get a review from @cbeauchesne (OOO this week)


### Making parametric runs faster

- **Skip the build when the image already exists:** Use `--skip-parametric-build` (or set `SKIP_PARAMETRIC_BUILD=1`) when you are only changing test code. This avoids rebuilding the parametric library image on every run. When you change the Dockerfile or app code under `utils/build/docker/<lang>/parametric/`, run without this option so the image is rebuilt.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure that docker doesn't already cache the images in this case ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going based on empirical evidence, it saved an additional ~1-2 seconds per run since we avoid the layer cache validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated The pull request includes a significant amount of AI-generated code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants