Add tests for startup log behavior in tracer libraries #6241

bm1549 · 2026-02-06T23:32:19Z

Motivation

Add comprehensive tests for tracer startup log behavior across supported languages. These tests verify that tracers correctly emit startup configuration logs with the appropriate default behavior and when explicitly configured via DD_TRACE_STARTUP_LOGS, ensuring proper observability and configuration visibility.

Changes

Adds tests/parametric/test_startup_logs.py with four test cases:

test_startup_logs_default: Verifies default startup log behavior when DD_TRACE_STARTUP_LOGS is not set
test_startup_logs_enabled: Verifies that startup logs are emitted when DD_TRACE_STARTUP_LOGS=true
test_startup_logs_disabled: Verifies that startup logs are suppressed when DD_TRACE_STARTUP_LOGS=false
test_startup_logs_diagnostic_agent_unreachable: Verifies diagnostic messages appear when the agent is unreachable

Language-Specific Handling

The tests handle language-specific differences:

Node.js Writes on first tracer flush, rather than on startup
Node.js and Ruby: Write to stdout (require trace triggering for Node.js)
.NET: Writes to a file (dotnet-tracer-managed*) instead of stdout/stderr
Other languages: Output to stderr

Recent Updates

Refactored test implementation to reduce duplication and improve maintainability (addresses code review feedback)

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-02-06T23:32:47Z

CODEOWNERS have been resolved as:

tests/parametric/test_startup_logs.py                                   @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/rust.yml                                                      @DataDog/apm-rust

datadog-datadog-prod-us1 · 2026-02-06T23:43:23Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 98790a0 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19694ec8ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tests/parametric/test_startup_logs.py

nccatoni · 2026-02-10T14:50:03Z

tests/parametric/test_startup_logs.py

+            else:
+                try:
+                    # Node.js and Ruby log to stdout, other libraries only log stderr
+                    if context.library in ("nodejs", "ruby"):


Optional: Ideally a test should not have different execution paths for different languages. You should make multiple tests and use the manifests to activate the relevant one for each language. This let's the easy win process catch any evolution of the tracer behavior in the furture.

I'm going to keep the test as-is for now. I plan to update the other languages to fix them in the very short term via DataDog/dd-trace-rb#5342 and DataDog/dd-trace-js#7470 - one is merged and the other one is close to being merged

Once these are in, I can remove all of the language-conditional logic, with the exception of the .NET log file logic. However, I believe it is helpful to have in the same test to showcase the deviation of .NET, rather than appearing as two valid behaviors

tests/parametric/test_startup_logs.py

nccatoni · 2026-02-10T14:55:39Z

I noticed that you put the AI generated label on this PR, would you mind telling me how much of the code was generated, with what tool and how much prompting you did (one prompt or multiple, and whether you needed to give the llm a lot of information or not). I'm asking to evaluate the effectiveness of the AI rules in the repo

bm1549 · 2026-02-10T15:32:01Z

I noticed that you put the AI generated label on this PR, would you mind telling me how much of the code was generated, with what tool and how much prompting you did (one prompt or multiple, and whether you needed to give the llm a lot of information or not). I'm asking to evaluate the effectiveness of the AI rules in the repo

I started using Cursor, but moved to Claude Code afterwards since it tended to perform better in this repo. I'm a bit of a novice at them, so I'm just experimenting

Nearly all of the code was generated, but it took a ton of prompting to identify the different language-specific behaviors present and it couldn't figure out C++/PHP at all. After making the changes, it was a bit verbose/hard to follow so I prompted it to clean itself up

One piece that was a struggle was the runtime for format.sh and re-running parametric tests as it tried to fix the tests - this was the clear largest time spend in this task. As a result, I asked Claude to optimize both of them, resulting in #6256 and #6258, which I have as draft since I haven't fully wrapped my head around each line of code. If you're more comfortable in those parts of the codebase, I'd encourage you to check them out since it may take me some time to go through them and understand the potential consequences of the optimizations

Moved test skip declarations from inline decorators (@missing_feature, @incomplete_test_app) to manifest files for cpp, nodejs, php, and rust. Also added span trigger in test_startup_logs_diagnostic_agent_unreachable to ensure tracers attempt agent connection. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

p-datadog · 2026-02-10T17:17:08Z

Sorry meant to update DataDog/dd-trace-rb#5342 instead of this PR.

genesor

Looks ok overall, the readability could be improve to have most of the test follow a single path where possible.

genesor · 2026-02-10T22:23:41Z

tests/parametric/test_startup_logs.py

+                test_library.dd_flush()
+
+            # For .NET, startup logs are written to a file instead of stdout/stderr
+            if context.library == "dotnet":


Could we get this condition reworked (and the similar ones in the other tests) to an simpler if / else if / else

I don't think it's useful to have dotnet in its own if and other language in another if in the else condition.

genesor · 2026-02-10T22:28:35Z

tests/parametric/test_startup_logs.py

+                if match:
+                    logger.error(logs)
+                    pytest.fail(
+                        # f"Startup log found when DD_TRACE_STARTUP_LOGS=false. "


Shouldn't this be uncommented so we understand what's going on ?

genesor · 2026-02-10T22:29:09Z

tests/parametric/test_startup_logs.py

+                if logs is not None:
+                    startup_log_pattern = r"DATADOG (TRACER )?CONFIGURATION( - (CORE|TRACING|PROFILING|.*))?"
+                    if re.search(startup_log_pattern, logs, re.IGNORECASE):
+                        pytest.fail(
+                            "Startup log found in .NET log file when DD_TRACE_STARTUP_LOGS=false. "
+                            f"Content (first 1000 chars): {logs[:1000]}"
+                        )


This could be factorized with the same check below

nccatoni

Thank you for the changes and for the insides in your AI development workflow. Everything looks good to me now but you should also get an approval from someone familiar with the feature

Done

Addresses code review feedback by consolidating duplicated log retrieval logic and simplifying conditional structures. Changes include: - Extract STARTUP_LOG_PATTERN to module-level constant - Add unified _get_startup_logs() helper function - Replace nested if/else with clearer if/elif/else pattern - Improve error messages to include matched pattern information This reduces code duplication (~60 lines to ~30 shared lines) and makes the test logic easier to follow and maintain. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

bm1549 · 2026-02-12T01:55:44Z

Thanks for the review! I've addressed all the feedback:

✅ Line 50 (nested if/else): Replaced with if/elif/else pattern in new _get_startup_logs() helper
✅ Line 133 (duplication): Consolidated all log retrieval into single helper function
✅ Line 150 (error message): Uncommented and fixed typo (10000→1000)

Changes:

Added STARTUP_LOG_PATTERN module constant (eliminates 4 duplicate regex definitions)
Created _get_startup_logs() helper with clearer conditional structure
All 4 test methods now use the helper (net -16 lines)
Error messages now include matched pattern for easier debugging

bm1549 added the ai-generated The pull request includes a significant amount of AI-generated code label Feb 6, 2026

bm1549 force-pushed the brian.marks/startup-log branch from 4a29519 to 4fd5c1a Compare February 9, 2026 19:50

bm1549 mentioned this pull request Feb 9, 2026

fix(startup-log): emit immediately after init and send to stderr DataDog/dd-trace-js#7470

Merged

2 tasks

Add tests for startup log behavior in tracer libraries

19694ec

bm1549 force-pushed the brian.marks/startup-log branch from b90773c to 19694ec Compare February 10, 2026 00:18

bm1549 marked this pull request as ready for review February 10, 2026 00:18

bm1549 requested review from a team as code owners February 10, 2026 00:18

bm1549 requested review from BridgeAR and removed request for a team February 10, 2026 00:18

chatgpt-codex-connector bot reviewed Feb 10, 2026

View reviewed changes

tests/parametric/test_startup_logs.py Outdated Show resolved Hide resolved

nccatoni previously requested changes Feb 10, 2026

View reviewed changes

bm1549 mentioned this pull request Feb 10, 2026

Change default logger output from stdout to stderr DataDog/dd-trace-rb#5342

Merged

bm1549 requested review from a team as code owners February 10, 2026 16:19

bm1549 requested review from dubloom and removed request for a team February 10, 2026 16:19

Merge branch 'main' into brian.marks/startup-log

20e8b40

bm1549 requested a review from nccatoni February 10, 2026 18:08

genesor reviewed Feb 10, 2026

View reviewed changes

nccatoni reviewed Feb 11, 2026

View reviewed changes

Ruby dev now goes to stderr

98790a0

bm1549 mentioned this pull request Feb 12, 2026

feat(config): enable startup logs by default DataDog/dd-trace-js#7474

Draft

2 tasks

Add tests for startup log behavior in tracer libraries #6241

Are you sure you want to change the base?

Add tests for startup log behavior in tracer libraries #6241

Conversation

bm1549 commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Language-Specific Handling

Recent Updates

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented Feb 6, 2026 • edited by datadog-official bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

nccatoni Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

bm1549 Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nccatoni commented Feb 10, 2026

Uh oh!

bm1549 commented Feb 10, 2026

Uh oh!

p-datadog commented Feb 10, 2026

Uh oh!

genesor left a comment

Choose a reason for hiding this comment

Uh oh!

genesor Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

genesor Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

genesor Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

nccatoni left a comment

Choose a reason for hiding this comment

Uh oh!

bm1549 commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bm1549 commented Feb 6, 2026 •

edited

Loading

github-actions bot commented Feb 6, 2026 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 6, 2026 •

edited by datadog-official bot

Loading