Skip to content

[corpus] Fixture: fastapi-route-defs (precision — route definitions must not be counted as endpoints) #14

Description

@AndresL230

Part of #10. Validates recost-dev/extension#147.

Goal

A precision fixture: a FastAPI app with several route definitions and exactly one real outbound call. expected.json lists only the outbound call, so every route-def the scanner emits as an endpoint is counted as a false positive — quantifying the route-def noise (the biggest driver of the current 0.41 detectionPrecision).

Branch

fixture/fastapi-route-defs

File to create

fastapi-route-defs/src/main.py

import httpx
from fastapi import FastAPI

app = FastAPI()


@app.get("/health")
def health():
    return {"ok": True}


@app.get("/users")
def list_users():
    return []


@app.get("/users/{user_id}")
def get_user(user_id: str):
    return {"id": user_id}


@app.post("/users")
def create_user(name: str):
    return {"created": name}


@app.delete("/users/{user_id}")
def delete_user(user_id: str):
    return {"deleted": user_id}


@app.patch("/users/{user_id}")
def update_user(user_id: str):
    return {"updated": user_id}


@app.post("/moderate")
def moderate(text: str):
    resp = httpx.post("https://api.openai.com/v1/moderations", json={"input": text})
    return {"flagged": resp.json()}

fastapi-route-defs/expected.json

{
  "schemaVersion": 1,
  "fixtureSlug": "fastapi-route-defs",
  "endpoints": [
    { "file": "src/main.py", "function": "moderate", "line": 39, "provider": "openai", "method": "POST", "must_detect": true, "notes": "The ONLY real outbound call — httpx.post with a literal OpenAI URL." }
  ],
  "findings": []
}

FIXTURE.md

Synthetic fixture authored for the ReCost benchmark (no upstream source). Scope: a FastAPI router whose @app.<verb>("/path") definitions must NOT be counted as external endpoints, alongside a single genuine outbound call that must. License: n/a (synthetic).

Current behaviour (the gap)

  • matchRouteDefinitionLine + isHighConfidenceUrl (core-scanner.ts:284/95) emit each @app.<verb>("/…") as an endpoint. Route-defs at lines 7, 12, 17, 22, 27, 32, 37 are therefore detected but absent from expected.json7 false positives.
  • The real call at line 39 is detected (literal URL, httpx.post) → 1 true positive.
  • Expected current per-fixture precision: ~1/8 = 0.13 (drags global detectionPrecision).

Flips green when

recost-dev/extension#147 lands (classify decorator route definitions as internal-route and keep them out of the external-endpoint list).

Acceptance criteria

  • Fixture added with src/main.py + expected.json + FIXTURE.md.
  • Runner output records the current ~1/8 precision for fastapi-route-defs (7 route-def false positives).
  • After #147, the 7 route-defs are no longer emitted; precision → 1.0, recall stays 1/1.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/measurementdocs/accuracy/measurement.md — benchmark corpus and CI gatesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions