Skip to content

Detection: agent frameworks (pydantic-ai / LangChain / LlamaIndex) — resolve provider from the model string, treat .run()/.invoke() as metered #144

Description

@AndresL230

Part of #142.

Summary

The primary LLM path in modern Python apps is an agent abstraction, not a direct SDK call. In sapling, every AI feature goes through pydantic-ai: classifier_agent = Agent("google-gla:gemini-2.5-flash", ...) then classifier_agent.run(...) (e.g. routes/documents.py:698, all of agents/*). None are detected — the provider (Gemini) is hidden inside the model string, and agent.run() looks like a local method call.

Root cause

  • agent = Agent(...) binds the var to package pydantic_ai, which is not in PACKAGE_TO_PROVIDER and is not a registered provider → resolveProvider → dropped at core-scanner.ts:269. There is no concept of "the provider is named in a constructor argument string."
  • Irony: src/scanner/python-waste-detector.ts:18-30 already enumerates langchain*, llama_index*, google.generativeai for N+1 loop detection — that awareness never reaches endpoint detection.

Proposed fix

  • Recognize agent/LLM construction for the common frameworks: pydantic-ai Agent(model, ...); LangChain ChatOpenAI / ChatGoogleGenerativeAI / ChatAnthropic / init_chat_model(...); LlamaIndex LLM ctors.
  • Parse the model string → provider: "google-gla:" / "google-vertex:" → gemini/vertex; "openai:" / bare gpt-* → openai; "anthropic:" / claude-* → anthropic; etc. Bind the agent/LLM var to the resolved provider so the normal resolution path picks up the call.
  • Treat .run() / .run_sync() / .invoke() / .ainvoke() / .complete() / .chat() on a bound agent as a metered call. Note the tool-loop cost multiplier — orchestrators fire N model round-trips per call.
  • If the model string is dynamic/unreadable → emit an outbound LLM call of unknown provider (depends on #D), not nothing.

Evidence

Manual scan of sapling found ~26 Gemini sites, the majority via pydantic-ai agent.run() across agents/document.py, agents/chat_tutor.py, agents/quiz.py, routes/documents.py:698/718/721/725, routes/learn.py:499, routes/notes.py:204/223/252. Extension detected 0.

Acceptance criteria

  • Agent("google-gla:gemini-2.5-flash") + .run() → provider gemini.
  • LangChain ChatOpenAI(...).invoke(...) → provider openai.
  • A dynamic model string still emits an outbound (unknown) LLM call, not nothing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/detectiondocs/accuracy/detection.md — finding every call site, attributing correctlyenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions