fix(llmobs): stop inlining base64 images into langchain span content - #19803
fix(llmobs): stop inlining base64 images into langchain span content#19803joizddog wants to merge 2 commits into
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 340de84 | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-08-20 23:26:25 Comparing candidate commit 1623095 in PR branch Found 0 performance improvements and 9 performance regressions! Performance is the same for 613 metrics, 10 unstable metrics.
|
langchain builds its own LLMObs message content with a bare str() and no branch for list-shaped multimodal content, so a data:image/...;base64 URL landed verbatim in span content. This hit both paths: input_messages is built before the is_workflow branch, and when demoted it is serialized into input.value. Past DD_LLMOBS_EVENT_SIZE_BYTES, _truncate_span_event blanks the span's whole input and output, not just the offending field. Route list content through the shared _extract_content_parts, which parses the OpenAI-shaped image_url blocks langchain passes. The llm path gets image_parts and audio_parts; the demoted path keeps a marker instead, since a workflow span has nowhere to put structured parts and the provider's child span carries the payload. Covers both input sites, streaming and not. Bare strings in the content list are normalized first, since langchain allows them and the extractor does not read them.
1623095 to
807ec2e
Compare
Codeowners resolved asResolved from the full PR diff against |
Dependency direction analysis
|
Circular import analysis
|
There was a problem hiding this comment.
Pull request overview
This PR updates the LangChain LLMObs integration to prevent inline base64 media (notably data:image/...;base64,...) from being stringified into span message text, instead capturing it as structured image_parts / audio_parts where applicable, and falling back to markers on demoted (“workflow”) spans.
Changes:
- Add LangChain-specific message-content extraction that routes list-shaped multimodal content through the shared
_extract_content_partshelper to avoid base64-in-text. - Ensure demoted/workflow spans emit markers rather than structured parts (to avoid re-serializing media back into
input_value). - Add targeted LangChain LLMObs tests covering inline image/audio capture, oversize degradation, streaming/non-streaming paths, and regression pins; add a release note.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
ddtrace/llmobs/_integrations/langchain.py |
Parse list-shaped multimodal message content via shared extractor, attach image_parts/audio_parts only when captured, and add workflow-marker behavior. |
tests/contrib/langchain/test_langchain_llmobs.py |
Adds coverage to ensure base64 payloads are not inlined into span content (including streaming and workflow/demoted behavior). |
releasenotes/notes/fix-langchain-inline-base64-images-54f331500a30449f.yaml |
Documents the behavior change and the size-guard/marker semantics for LangChain LLMObs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| render in LLM Observability. Images referenced by a remote URL are not fetched and keep their | ||
| existing text reference. A single inline image whose base64 payload exceeds 80% of the |
… note _capture_inline_image returns no marker for a non-data URL, so the shared extractor falls back to "[image]" and the URL string is not recorded. The note claimed the reference was preserved. Pin the behavior with a test, since the note now asserts it.
fix(llmobs): stop inlining base64 images into langchain span content
What does this PR do?
Stops the langchain integration writing raw base64 image payloads into span content, and captures
them as typed
image_parts/audio_partson the llm path instead.The defect
langchain builds its own LLMObs message content rather than delegating to the provider integration,
and did so with a bare
str()and no branch for list-shaped multimodal content(
_integrations/langchain.py, non-stream chat and both streamed call sites). Adata:image/...;base64,...URL therefore landed in span content verbatim.This happens on both paths, because
input_messagesis built before theis_workflowbranch;that branch only chooses which key the result lands under. When demoted, it is JSON-serialized into
input.value.Past
DD_LLMOBS_EVENT_SIZE_BYTES,_truncate_span_eventblanks the span's entire input andoutput, not just the image. So a single vision prompt can cost the whole record of the call.
The delegated path does not save you either: three cases produce no provider llm span at all
(proxy requests, raw-response streaming, and a provider registered-but-never-imported), and in those
the langchain span carrying the inlined base64 is the only record.
Precedent
No new design here. List content is routed through the shared
_extract_content_parts, the samehelper #19690 extended to return image parts, so the wire shape and the capture semantics are
identical to the merged OpenAI and Anthropic work (#19148, #19690), which in turn build on
#18809.
Changes
ddtrace/llmobs/_integrations/langchain.py(+73/-12):_extract_message_content(content, is_workflow)— list content goes through_extract_content_parts; a data URL becomes a structured part and never reaches the text. Blocksthe shared extractor does not recognise (langchain's own
image/audiostandard blocks) stillleave a
[image]/[audio]marker rather than the payload._build_message(content, role, is_workflow)— attaches media keys only when something wascaptured, so a text-only message keeps byte-identical shape to before.
(
_handle_stream_input_messagesnow takesis_workflow).Audio is fixed by the same change; langchain had never handled it either.
Size guarding (the question asked on #19148 and #19690)
This inherits the per-image budget from
_capture_inline_image, which #19690 added to_extract_content_parts— an over-budget image degrades to[image omitted: too large]before itcan push the event over the cap. There is no new guard here and no guard bypass.
The cumulative case is still open, exactly as on the merged PRs: N images that each fit the
per-image budget can together exceed the per-event limit. That is unchanged by this PR and remains
tracked under MLOB-6408 as a writer-side fix. Flagging it explicitly so it is not re-litigated here.
Why the demoted path gets markers, not typed parts
On a demoted (
workflow) span, typed messages have nowhere to land — those spans carryinput_value, notmessages— so structured parts would be serialized straight back into thevalue, which is the bug this PR fixes. Those keep a marker only; the provider's child llm span
carries the real payload.
Note this interacts with the pending non-LLM span-kind widening: once
workflow/task/tool/stepcan keep typed messages, this branch becomes eligible to emit real parts there instead ofmarkers. Deliberately left as a follow-up rather than pre-empting a change that has not landed.
Testing
tests/contrib/langchain/test_langchain_llmobs.py(+216). 8/8 of the new tests pass;lint fmt,style,typingandspellingclean; banned-terms clean.The existing
test_langchain.py:99-123already fed animage_urlblock throughChatOpenAI, butasserted only "no error" and used an
httpsURL, so it could not catch this. The new cases use adata:URL and assert the base64 is absent from span content.Note
conftest.pysetsintegrations_enabled=False, so the suite exercises the non-delegated path;the demoted path needs a fixture that patches a provider and is called out rather than silently
uncovered.
Known adjacent defects, deliberately not in scope
format_langchain_io) can still stringify content blocks.Both are pre-existing, independent of this change, and worth their own issues rather than widening
this diff.