feat(aws-lambda-microvm): refresh identity on run hook by litianningdatadog · Pull Request #19781 · DataDog/dd-trace-py

litianningdatadog · 2026-08-20T04:32:16Z

Description

AWS Lambda MicroVM instances created from the same image start with the same in-memory runtime id and Remote Config client id. The platform POST /aws/lambda-microvms/runtime/v1/run request is the earliest shared signal that a restored instance is becoming active.

This registers the web request listener only inside MicroVM images, matches the fixed /run hook, and refreshes identity once per process. The once-only guard handles stacked web layers observing the same platform request.

Testing

Added hook matcher coverage for exact match, ignored requests, listener registration outside MicroVMs, root-span ordering, and concurrent observations. Updated web event tests to use the real MicroVM hook path.

Risks

Medium. This is the PR that activates identity refresh from request dispatch, but only when AWS_LAMBDA_MICROVM_IMAGE_ARN is present and only for the fixed platform hook.

Additional Notes

Stacked on #19780.

cit-pr-commenter-54b7da · 2026-08-20T04:32:28Z

Codeowners resolved as

Resolved from the full PR diff against tianning.li/3-runtime-id-refresh-consumers using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/runtime/__init__.py                                    @DataDog/apm-sdk-capabilities-python
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml  @DataDog/apm-python
tests/contrib/asgi/test_microvm_identity_refresh.py                     @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/bottle/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/cherrypy/test_microvm_identity_refresh.py                 @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/django/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/falcon/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/flask/test_microvm_identity_refresh.py                    @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/http_server/test_microvm_identity_refresh.py              @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/molten/test_microvm_identity_refresh.py                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/pyramid/test_microvm_identity_refresh.py                  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/sanic/test_sanic.py                                       @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/tornado/test_microvm_identity_refresh.py                  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/tracer/runtime/test_runtime_id.py                                 @DataDog/apm-sdk-capabilities-python

cit-pr-commenter-54b7da · 2026-08-20T04:32:34Z

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

datadog-datadog-prod-us1-2 · 2026-08-20T04:32:45Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 10 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v126532182-233089d-musllinux_1_2_aarch64, 1] — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

View all 10 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 621c074 | Docs | View more details | Give us feedback!}

cit-pr-commenter-54b7da · 2026-08-20T04:32:49Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 254 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 254 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=132)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.internal.opentelemetry.span -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

pr-commenter · 2026-08-20T04:59:07Z

Benchmarks

Benchmark execution time: 2026-08-20 11:54:47

Comparing candidate commit 621c074 in PR branch tianning.li/4-microvm-run-hook-activation with baseline commit c9ede47 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 4 performance regressions! Performance is the same for 382 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.820µs; +3.004µs] or [+13.074%; +13.925%]

scenario:span-start

🟥 execution_time [+1.434ms; +1.614ms] or [+9.495%; +10.683%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+369.793ns; +416.833ns] or [+14.012%; +15.794%]

scenario:tracer-small

🟥 execution_time [+30.437µs; +33.258µs] or [+8.986%; +9.818%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-691.623ns; +771.989ns] or [-6.332%; +7.068%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-23.503ns; +42.737ns] or [-3.870%; +7.037%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-1839.113ns; +1422.673ns] or [-10.814%; +8.365%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1090.920ns; +1411.073ns] or [-8.402%; +10.868%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-333.773ns; +313.379ns] or [-9.095%; +8.539%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-289.760ns; +215.637ns] or [-9.904%; +7.371%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [-63.436ns; +88.110ns] or [-5.497%; +7.634%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-3920.982ns; +4168.575ns] or [-9.560%; +10.164%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-761.786ns; +791.162ns] or [-9.338%; +9.698%]

Copilot

Pull request overview

This PR activates “identity refresh” for AWS Lambda MicroVM restores by hooking the earliest shared platform signal (POST /aws/lambda-microvms/runtime/v1/run) into the existing core.WEB_REQUEST_STARTING event, ensuring runtime- and identity-bound components get refreshed once per process in MicroVM environments.

Changes:

Add MicroVM /run hook matching + once-per-process identity refresh registration to ddtrace.internal.runtime.
Expand runtime-id tests to cover exact hook matching, ordering (pre-root-span), and concurrency behavior.
Update web framework tests to use the real MicroVM hook path constant and add a release note.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ddtrace/internal/runtime/init.py	Registers a MicroVM-only `WEB_REQUEST_STARTING` listener to refresh identity once when the exact `/run` hook is observed.
tests/tracer/runtime/test_runtime_id.py	Adds subprocess tests for hook matching, ordering vs root span creation, concurrency safety, and non-MicroVM no-op behavior.
tests/contrib/asgi/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in ASGI middleware tests.
tests/contrib/bottle/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Bottle integration tests.
tests/contrib/cherrypy/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in CherryPy integration tests.
tests/contrib/django/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Django integration tests.
tests/contrib/falcon/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Falcon integration tests.
tests/contrib/flask/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions (including WSGI environ) in Flask integration tests.
tests/contrib/http_server/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in `http.server` integration tests.
tests/contrib/molten/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Molten integration tests.
tests/contrib/pyramid/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Pyramid integration tests (including tracing-disabled path).
tests/contrib/sanic/test_sanic.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Sanic integration tests.
tests/contrib/tornado/test_microvm_identity_refresh.py	Uses `MICROVM_RUN_HOOK_PATH` for request-start dispatch assertions in Tornado integration tests.
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml	Documents MicroVM-only stable identifier regeneration behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+import threading
 import typing as t
 import uuid
 import weakref

+from ddtrace.internal import core
 from ddtrace.internal import forksafe


AWS Lambda MicroVM instances restored from the same image start with the same in-memory runtime id and Remote Config client id. The platform's /run lifecycle request is the earliest common signal that a restored instance is becoming active. Register the web request listener only inside MicroVM images, match the fixed POST /run hook, and refresh identity once per process. Stacked web layers can observe the same request without rotating identity more than once.

litianningdatadog requested a lite review from Copilot August 20, 2026 05:23

Copilot started reviewing on behalf of litianningdatadog August 20, 2026 05:24 View session

litianningdatadog requested a review from brettlangdon August 20, 2026 05:26

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Comment thread ddtrace/internal/runtime/__init__.py

Comment on lines +1 to 7

import threading

import typing as t

import uuid

import weakref

from ddtrace.internal import core

from ddtrace.internal import forksafe

litianningdatadog force-pushed the tianning.li/3-runtime-id-refresh-consumers branch 2 times, most recently from c992c42 to d424d55 Compare August 20, 2026 11:22

litianningdatadog force-pushed the tianning.li/4-microvm-run-hook-activation branch from 03c4230 to 55f826c Compare August 20, 2026 11:24

litianningdatadog force-pushed the tianning.li/4-microvm-run-hook-activation branch from 55f826c to 621c074 Compare August 20, 2026 11:30

litianningdatadog added the aws-microvm Work related to AWS MicroVM onboarding label Aug 21, 2026

Navigation Menu

Uh oh!

feat(aws-lambda-microvm): refresh identity on run hook - #19781

feat(aws-lambda-microvm): refresh identity on run hook#19781
litianningdatadog wants to merge 1 commit into
tianning.li/3-runtime-id-refresh-consumersfrom
tianning.li/4-microvm-run-hook-activation

litianningdatadog commented Aug 20, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 20, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 20, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

litianningdatadog commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Risks

Additional Notes

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 20, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

pr-commenter Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationinject-ids_only

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

litianningdatadog commented Aug 20, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Aug 20, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

cit-pr-commenter-54b7da Bot commented Aug 20, 2026 •

edited

Loading

pr-commenter Bot commented Aug 20, 2026 •

edited

Loading