Skip to content

load-testing: add MLflow tracing structure, validation + bottleneck examples - #251

Open
RamVegiraju wants to merge 1 commit into
databricks:mainfrom
RamVegiraju:feat/load-testing-mlflow-tracing
Open

load-testing: add MLflow tracing structure, validation + bottleneck examples#251
RamVegiraju wants to merge 1 commit into
databricks:mainfrom
RamVegiraju:feat/load-testing-mlflow-tracing

Conversation

@RamVegiraju

Copy link
Copy Markdown

Summary

  • Documents that the load-testing skill assumes the app is served by the MLflow AgentServer (the /invocations SSE contract the scripts hit, and the source of the per-request traces).
  • Adds Step 6 — Validate & Diagnose with MLflow Tracing: reliability from trace state, latency over successful traces, and bottleneck diagnosis via the span waterfall (CHAT_MODEL vs TOOL split, per-tool p95). Locust measures the system from the outside, MLflow traces from the inside — together they turn a QPS number into an explanation.
  • Adds two starter examples: examples/locustfile.py (streaming, TTFT, M2M OAuth, ramp shape) and examples/validate_with_mlflow.py (reliability + latency + span breakdown).
  • Adds a Resources section (Locust, MLflow tracing/search-traces/entities API, AgentServer, Databricks load-test docs) and an MLflow-tracing trigger to description.
  • Fixes a tracking gap: adds load-testing to the .gitignore source-skill allowlist so its example files are versioned and sync-skills.py can reproduce the template copies from a clean clone.

All changes are additive (existing wording untouched aside from the extended description) and synced to all templates via .scripts/sync-skills.py.

Test plan

  • Both example scripts run end-to-end against a real experiment; validate_with_mlflow.py correctly surfaced the bottleneck tool (get_stock_price p95 ≈ 897ms vs get_weather ≈ 200ms) and the model-vs-tool split.
  • APIs validated against installed OSS source + live data: Locust events.request.fire / LoadTestShape.tick; MLflow search_traces(locations=…), trace.data.spans, span_type/start_time_ns/end_time_ns.
  • examples/*.py compile in source and all synced template copies.
  • Doc claims validated against mlflow.org (span types CHAT_MODEL/TOOL/CHAIN, leaf-span caveat).

…xamples

Additive update to the load-testing skill (source of truth + synced template copies):
- Note that templates are served by the MLflow AgentServer (the /invocations SSE
  contract the scripts hit, and the source of the per-request traces).
- Step 6 "Validate & Diagnose with MLflow Tracing": reliability from trace state,
  latency over successful traces, and bottleneck diagnosis via the span waterfall
  (CHAT_MODEL vs TOOL split, per-tool p95).
- examples/locustfile.py (streaming, TTFT, M2M OAuth, ramp shape) and
  examples/validate_with_mlflow.py (reliability + latency + span breakdown).
- Resources: Locust, MLflow tracing/search-traces/entities API, AgentServer,
  Databricks load-test-agent-app docs.
- description gains an MLflow-tracing trigger.

Also add load-testing to the .gitignore source-skill allowlist
(!.claude/skills/load-testing/) so its example files are versioned and
sync-skills.py can reproduce the template copies from a clean clone.

Synced to all templates via .scripts/sync-skills.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant