Feature — MCP-based trace source for observability backends

[henrikrexed]

Problem or motivation

Currently, agentevals supports Jaeger JSON and OTLP trace files as inputs. Users must export traces to files before evaluating. This adds friction — especially for teams whose traces already live in an observability backend (Dynatrace, Grafana, Datadog, etc.).

Proposed solution

Add an MCP-based trace loader that pulls traces directly from any observability backend that exposes an MCP server.

Usage

# Pull traces from Dynatrace via MCP
agentevals run --source mcp \
  --mcp-url "http://localhost:3000" \
  --mcp-tool "execute_dql" \
  --query 'fetch spans | filter service.name == "my-agent" | sort start_time desc | limit 100' \
  --eval-set my-eval.json

# Pull from any MCP server
agentevals run --source mcp \
  --mcp-url "http://grafana-mcp:8080" \
  --mcp-tool "query_traces" \
  --query '...' \
  --eval-set my-eval.json

Why MCP?

MCP (Model Context Protocol) is becoming the standard for AI tool integration. Multiple observability vendors already have MCP servers:

• Dynatrace MCP — execute_dql tool
• Grafana MCP — trace queries
• More coming (Datadog, New Relic, etc.)

By supporting MCP as a trace source, agentevals becomes instantly compatible with any backend that has an MCP server — without writing vendor-specific integrations.

Implementation

New file: src/agentevals/loader/mcp.py

The loader would:

1. Connect to the MCP server via HTTP
2. Call the specified tool with the query
3. Parse the response as OTLP spans
4. Convert to agentevals internal trace format

Alternatives considered

No response

Additional context

Benefits

• No file export needed — evaluate directly from production traces
• Vendor-agnostic — works with any MCP-compatible backend
• CI/CD ready — agentevals run --source mcp ... in pipelines
• Real-time — evaluate the latest traces, not stale exports

Human confirmation

• I am a human (not a bot, agent, or AI) filing this issue.

[krisztianfekete]

Thanks @henrikrexed, this is very reasonable feature request. You can sorta do this already today, but it's a bit manual, so we will definitely improve on the UX here.

Just for context, we will be also packaging this for Kubernetes soon, where users will be able to point their OTel collectors to agentevals (after some filtering/sampling, until we figure out the "continuous evaluation" design), so that's another way to consume agentevals.

[krisztianfekete]

I'd expect most of the complexity of this feature is handling all the various format MCP servers will use to return data. E.g. IIRC Dynatrace's MCP server's output is not native OTLP, can you confirm that?

[henrikrexed]

@krisztianfekete Thanks for the update! A few thoughts on the OTel endpoint approach:

1. MCP support is essential
   Most observability vendors provide MCP servers, so relying only on native OTLP could be a significant limitation (e.g., Dynatrace's MCP output isn't native OTLP). If Agenteval becomes an OTel endpoint where users point collectors, you'd need to handle filtering/sampling until "continuous evaluation" is finalized.

2. Storage challenge for trace investigations
   Agent investigations can take several minutes with many tool calls, and you'll need to cache all spans involved. This is heavy lifting.

3. Efficiency metric
   The number of tool calls per investigation should be tracked — it's a useful efficiency metric for agent evaluation.

4. Alternative: signal2metrics connector
   Instead of storing all spans, the Opentelemetry collector has a ⁠ signal2metrics ⁠ connector that can generate metrics from spans. We could rely on those metrics for evaluation rather than caching everything, which would be much more scalable for longer-running agents.

Would love to hear your thoughts on this approach!

[krisztianfekete]

We can mostly ignore the native otlp support for this issue at hand and have a discussion on this topic on it's own GH issue, I just wanted to mention it as part of the larger context.

I agree MCP support is a must, but this is a bit more complex than stated here:

By supporting MCP as a trace source, agentevals becomes instantly compatible with any backend that has an MCP server — without writing vendor-specific integrations.

You also confirm that Dynatrace MCP server's output is in a proprietary format:

Most observability vendors provide MCP servers, so relying only on native OTLP could be a significant limitation (e.g., Dynatrace's MCP output isn't native OTLP).

So unfortunately, we will very likely need to write vendor/project-specific adapters for MCP integrations anyway.

[henrikrexed]

a few things to consider from our experience with the k8s AI agent benchmark:

1. MCP response format is not standardized

The issue assumes MCP tools return OTLP-formatted spans, but in practice each vendor's MCP server returns data in its own format. Dynatrace's ⁠ execute_dql ⁠ returns DQL result records, Grafana's returns Tempo-style JSON, etc. None of them return raw OTLP.

This means we can't just "parse the response as OTLP spans" — we need a vendor-agnostic parser interface where each backend provides its own parsing rules to normalize responses into agentevals' internal trace format. Something like:

⁠ python
class TraceParser(Protocol):
def parse(self, mcp_response: dict) -> list[Span]: ...

Implementations per vendor

class DynatraceParser(TraceParser): ...
class GrafanaParser(TraceParser): ...
 ⁠

2. Caching layer needed for K8s deployment

If agentevals runs as a Kubernetes workload (which makes sense for CI/CD pipelines), performance evaluators like ⁠ token_efficiency ⁠ and ⁠ time_efficiency ⁠ need the complete trace — all spans from root to leaf. Traces arrive span-by-span and can take seconds to complete.

This means agentevals would need a trace caching/assembly layer that collects spans by trace ID, waits for completion (or a timeout), then hands the full trace to evaluators. Without this, evaluators would see incomplete data and produce unreliable scores.

3. Consider multi-signal evaluation beyond traces

Traces alone don't tell the full story. For a complete agent evaluation, we should consider:
•⁠ ⁠Metrics — token usage counters, latency histograms, error rates over time (not just per-trace)
•⁠ ⁠Logs — reasoning chains, tool output content, error details that don't fit in span attributes
•⁠ ⁠Other dimensions like cost data, rate limit events, etc.

The MCP loader design should be extensible to pull not just traces but also metrics and logs from backends. This would enable richer evaluators that correlate signals — e.g., "did the agent's token usage spike correlate with degraded response quality?"

[henrikrexed]

⁠Interested in contributing this. I have been working on performance evaluators (token/tool/time efficiency) that would pair well with MCP-based trace loading — evaluate agent performance directly from production traces without file exports.
For the implementation, the MCP loader could use the official MCP Go/Python SDK as the client, keeping it lightweight. The key challenge will be normalizing responses across different backends (Dynatrace DQL returns a different span format than Grafana's trace queries), so a pluggable response parser would help.
Happy to take this on if it's available.

[krisztianfekete]

I am not fully sold on the UX proposed here, especially with all the many MCP-specific commands. I'll update this thread with a cleaner design, hopefully shortly. Regarding supporting MCP vendors, I'd also like to start with something OSS and official, probably Tempo first.

[henrikrexed]

⁠Agreed on starting simple. One thought , Tempo is OSS but still Grafana Labs' product. For a truly vendor-neutral starting point, Jaeger (CNCF graduated, standard query API) or OpenSearch might be better first targets.
Then each vendor can add their own — I'd naturally handle the Dynatrace DQL normalization layer.
Looking forward to your updated design.

[krisztianfekete]

I was not able to find any official Jaeger MCP servers, and OpenSearch might not be as popular for traces, hence my recommendations. Happy to be corrected wrong on the first point, though.

[henrikrexed]

opensearch: https://opensearch.org/blog/introducing-mcp-in-opensearch/ , jaeger for now it is not an official one: https://github.com/serkan-ozal/jaeger-mcp-server