-
Notifications
You must be signed in to change notification settings - Fork 14
Speed up local parametric testing capabilities by 2-5x #6256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…rary builds and reuse containers. Introduce new rules for parametric testing in documentation. Update related scripts and fixtures for improved performance during test runs.
|
|
This comment has been minimized.
This comment has been minimized.
nccatoni
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes could cause reliability issues. While it might be fine to speedup local development you should make sure that all parametric tests do indeed pass. You should also get a review from @cbeauchesne (OOO this week)
|
|
||
| ### Making parametric runs faster | ||
|
|
||
| - **Skip the build when the image already exists:** Use `--skip-parametric-build` (or set `SKIP_PARAMETRIC_BUILD=1`) when you are only changing test code. This avoids rebuilding the parametric library image on every run. When you change the Dockerfile or app code under `utils/build/docker/<lang>/parametric/`, run without this option so the image is rebuilt. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure that docker doesn't already cache the images in this case ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Going based on empirical evidence, it saved an additional ~1-2 seconds per run since we avoid the layer cache validation
Motivation
Re-running parametric tests (e.g. when iterating on a test or debugging) was slow because each run could rebuild the parametric library image and, with multiple workers, start a new test agent and library container per test. This PR documents and implements options so parametric re-runs are faster: single-worker mode (
+1) for shared agent and library container reuse, and--skip-parametric-buildto skip rebuilding the image when only test code or docs change.Runtime impact (parametric re-runs)
Measured on a single parametric test:
tests/parametric/test_tracer.py::Test_Tracer::test_tracer_span_top_level_attributes) with TEST_LIBRARY=nodejsOn a secondary run, using +1 and --skip-parametric-build is about 3-7× faster in pytest time and ~2-5× faster in wall time, from skipping the parametric image build and reusing one agent and one library container.
Changes
Workflow
🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist
tests/ormanifests/is modified ? I have the approval from R&P teambuild-XXX-imagelabel is present