Skip to content

Measurement: benchmark fixtures for modern call shapes + sapling/plate-gallery recall fixtures + re-baseline #148

Description

@AndresL230

Part of #142.

Summary

The detection fixes (#A–#E) need corpus coverage and a guarded baseline so recall gains are measured and precision changes don't regress silently. The current corpus (recost-dev/extension-benchmark) has no fixtures for SDK-client-via-handle, agent frameworks, non-literal URLs, Python AI SDKs, or route-def exclusion.

Proposed fix

  • Add minimal fixtures (in recost-dev/extension-benchmark) for each shape, with must_detect endpoints:
    • genai.Client().models.generate_content (gemini), including the with-scope variant (#A)
    • pydantic-ai Agent("google-gla:...").run() and LangChain ChatOpenAI().invoke() (#B)
    • httpx.AsyncClient()client.post(literal) and httpx.put(f"{BASE}/..") (#C)
    • a route-def file that must produce 0 external endpoints (precision fixture, #E)
    • an unknown-capture fixture: outbound call with env-var host → provider:"unknown" + non-empty capture (#D)
  • Add sapling and plate-gallery (or trimmed copies) as larger recall fixtures, with expected external call sites documented from the manual audit.
  • Re-baseline detection metrics; wire the precision fixtures into the CI gate so route-def / test noise can't creep back.

Notes

Acceptance criteria

  • Each #A–#E shape has a passing benchmark fixture.
  • Recall delta on sapling / plate-gallery is measured and recorded.
  • CI gate fails if route-def / test calls reappear as external endpoints.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/measurementdocs/accuracy/measurement.md — benchmark corpus and CI gates

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions