Skip to content

LLM Analytics: Provider wrappers don't respect context distinct_id. #443

@ethanporcaro

Description

@ethanporcaro

Bug Description

I'm using PydanticAI for LLM agents in Python. I wrapped it in the AsyncOpenAI from the posthog Python library, and am getting the traces correctly.

However, I am setting the distinct ID for posthog at a context level, and it seems that is not respected by the wrapper. Looking into openai_Async.py, I see that the distinct ID can be provided through the OpenAI-compatible methods such as openai.responses.create, but PydanticAI doesn't care about passing any of those through. I think it defaults to the trace ID.

I should be able to call identify_context and have it stored.

Mininum reproducable sample

import asyncio
import os

import dotenv
import posthog
from posthog import identify_context
from posthog.ai.openai import AsyncOpenAI
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Required env.
# Or just hardcode it.
"""
POSTHOG_API_KEY=
OPENAI_API_KEY=
OPENAI_ENDPOINT=
AI_MODEL=
"""
dotenv.load_dotenv()

POSTHOG_API_KEY = os.getenv("POSTHOG_API_KEY")
posthog_client = posthog.Posthog(POSTHOG_API_KEY, host="https://us.i.posthog.com", debug=True)

client = AsyncOpenAI(
    api_key=os.getenv("OPENAI_API_KEY", ""),
    base_url=os.getenv("OPENAI_ENDPOINT", "http://localhost:1234/v1"),
    posthog_client=posthog_client,
)
model = OpenAIChatModel(
    os.getenv("AI_MODEL", "qwen/qwen3-vl-30b"),
    provider=OpenAIProvider(openai_client=client),
)
agent = Agent(
    model=model,
    system_prompt="Repeat the word you get ten times."
)


async def main():
    with posthog_client.new_context():
        identify_context("test_user")

        posthog_client.capture("test_event")  # Correctly uses distinct_id.
        print(await agent.run("posthog"))  # Does not log distinct_id.


if __name__ == "__main__":
    asyncio.run(main())

Additional context

I am using FastAPI and setting the context in the middleware. However, the result is the same as the independant sample above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions