feat(llama-index): add loongsuite-instrumentation-llama-index (#18)

[RichardoMrMu]

What

Adds a new instrumentation package loongsuite-instrumentation-llama-index, providing automatic OpenTelemetry instrumentation for LlamaIndex (llama-index-core).

Closes #18 (roadmap #36 — "Add instrumentation for llama-index", contribution welcome).

How

Rather than monkey-patching call sites, this attaches to LlamaIndex's native instrumentation dispatcher (llama_index.core.instrumentation). LlamaIndex already emits a span/event stream through a root Dispatcher, assigning each instrumented call a span id_ and a parent_span_id that reflects the logical call tree (e.g. query → retrieve/synthesize, chat → complete). The package registers a BaseSpanHandler + BaseEventHandler on that dispatcher and re-projects the stream onto OpenTelemetry spans following the ARMS gen-ai semantic conventions, consuming parent_span_id directly so the OTel trace preserves LlamaIndex's own parent/child structure.

Span-kind mapping (gen_ai.span.kind): LLM / EMBEDDING / RETRIEVER / RERANKER / TASK (synthesis) / CHAIN (query engine) / AGENT (chat engine, agent run). Classification is method-first so class names embedding a misleading keyword (e.g. RetrieverQueryEngine.query) are classified correctly (CHAIN, not RETRIEVER). LLM/embedding events fold request model, messages and provider token usage onto the span. Content capture can be disabled with OTEL_INSTRUMENTATION_LLAMA_INDEX_CAPTURE_CONTENT=false.

Testing

tests/test_instrumentor.py drives the real llama-index-core dispatcher with in-process MockLLM / MockEmbedding and asserts on spans exported to an InMemorySpanExporter (no network, no credentials). Each GREEN assertion is paired with a RED baseline:

• RED: an LLM chat with no instrumentor active produces zero spans.
• GREEN: chat produces an LLM span with correct gen_ai.* attributes; chat→complete share one trace_id with complete nested under chat (proving parent_span_id mapping); embedding produces an EMBEDDING span.
• RED-after-teardown: after uninstrument(), a subsequent chat produces zero spans.
• Plus classification unit tests and idempotent-uninstrument.

15 tests pass (pytest), on llama-index-core 0.14.25 + opentelemetry-sdk.

Notes

• Package layout, pyproject.toml, entry point, CHANGELOG and bootstrap-registry entry follow the existing loongsuite-instrumentation-* conventions (e.g. terminus2).
• CHANGELOG.md updated (repo requirement).

[Copilot]

Copilot review overview

🟡 Changes recommended

Structured prediction classification, active-span teardown, hierarchy verification, and CI registration need correction.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Structured prediction methods are misclassified as CHAIN · New
• Uninstrumenting strands active spans and context tokens · New
• New package missing from tox-loongsuite configuration · New
• Test does not assert the exact parent span ID · New

What changed in this PR

Adds automatic OpenTelemetry instrumentation for LlamaIndex via its native dispatcher.

Changes:

• Maps LlamaIndex spans and events to GenAI telemetry.
• Adds lifecycle, hierarchy, classification, and embedding tests.
• Registers packaging and bootstrap metadata.

File	Description
pyproject.toml	Adds Ruff exceptions.
loongsuite-distro/​.../​loongsuite_instrumentation_llama_index.py	Adds bootstrap registry metadata.
.../​tests/​test_instrumentor.py	Tests spans and lifecycle.
.../​tests/​conftest.py	Configures test tracing.
.../​tests/​__init__.py	Initializes tests.
.../​test-requirements.txt	Adds test dependencies.
.../​version.py	Defines package version.
.../​package.py	Defines instrument dependency.
.../​llama_index/​__init__.py	Implements instrumentation.
.../​README.md	Documents usage.
.../​pyproject.toml	Defines package metadata.
.../​LICENSE	Adds Apache license.
.../​CHANGELOG.md	Records initial release.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sipercai]

Thanks for adding LlamaIndex support! Could we use the shared GenAI util in this LoongSuite repository, following the Hermes instrumentation as a reference, so semantic conventions and content-capture controls can be managed consistently?

Also, span types should reflect the actual operation. A class name containing Agent should not cause internal methods such as setup, parse, and call_tool to all become AGENT spans. AGENT spans should represent actual agent invocations, while tool execution should be represented as TOOL spans.

[RichardoMrMu]

Thanks for the review, @sipercai — both points are addressed, and I pushed the changes plus tests.

1. Use the shared GenAI util for semconv + content-capture (following hermes)

• Span kinds now come from opentelemetry.util.genai.extended_semconv...GenAiSpanKindValues and the gen_ai.span.kind key from the shared GEN_AI_SPAN_KIND, so this package speaks the same vocabulary as the rest of loongsuite instead of local string literals.
• Content capture is now governed by the shared get_content_capturing_mode() / ContentCapturingMode (the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT switch). I removed the package-specific OTEL_INSTRUMENTATION_LLAMA_INDEX_CAPTURE_CONTENT env var. Two tests assert SPAN_ONLY captures messages and NO_CONTENT (the default) suppresses them while keeping the structural span.

2. Span types must reflect the real operation (not AGENT-by-classname)

• Fixed _classify so a class name containing Agent no longer forces internal methods to AGENT:
  • call_tool / acall_tool → TOOL (execute_tool)
  • agent-loop steps (setup_agent, init_run, take_step, finalize, handle_tool_call_results, ...) → CHAIN (step)
  • only a genuine agent invocation (run/arun, or chat/complete on an agent/chat-engine) stays AGENT
  • the blanket "agent" in class_name → AGENT fallback is gone.
• I introspected the workflow agents (FunctionAgent/ReActAgent) to ground the method list; verified against the old code that FunctionAgent.call_tool/take_step/setup_agent/handle_tool_call_results all classified as AGENT before, and are TOOL/CHAIN now.

Also picked up the Copilot findings in the same pass:

• astructured_predict / stream_structured_predict / astream_structured_predict now classify as LLM (were falling through to CHAIN).
• uninstrument() no longer strands open spans: the handler stops creating new spans and drains (ends) any still-open spans before it is detached from the dispatcher (the dispatcher only routes exit/drop to still-attached handlers). Added a test that opens a span, uninstruments, and asserts it was ended.
• The chat→complete nesting test now asserts the exact parent span id, not just "some exported span".
• Registered the package's test/lint envs in tox-loongsuite.ini so CI can select it.

All 28 unit tests pass locally and ruff check is clean. Happy to adjust the step/chain mapping if you'd prefer a dedicated step kind for the agent-loop internals.

[sipercai]

Thanks for the update! The span classification fixes look good. For shared-util integration, please follow Hermes and use this repository’s ExtendedTelemetryHandler (start/stop_invoke_agent, start/stop_execute_tool) from the LlamaIndex callbacks, rather than only reusing enums and config helpers.

Please also add the missing GenAI util dependency—clean installation currently fails on import—and delegate content emission to the handler so EVENT_ONLY works. Tests should verify that telemetry failures do not affect business execution.

[Copilot]

Copilot review overview

🟡 Changes recommended

Span shutdown and concurrent teardown can strand spans or contexts, and output telemetry and content-capture documentation are inaccurate.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 · 1

Open (6)

• Race between stopped check and span registration strands spans · New
• Live spans omitted from shutdown drain registry · New
• Cross-context cleanup leaves ended span attached · New
• Chat input incorrectly reported as generated output · New
• New package missing from tox-loongsuite configuration
• GenAI content capture documentation contradicts implementation · New

Resolved since last review (3)

• Uninstrumenting strands active spans and context tokens
• Structured prediction methods are misclassified as CHAIN
• Test does not assert the exact parent span ID