Skip to content

[benchmark corpus] Resolve Python cross-function batch FN: relabel sibling helpers or accept as unrecoverable #134

Description

@AndresL230

Context

Follow-up to #117 (FN #1) and PR #132. #117 bundled two C1 false negatives; #132 recovered the unbatched_parallel FN but deferred the Python cross-function batch FN after attempting and reverting a fix.

The reverted attempt (a module-scope (provider, methodChain) batching pass in src/scanner/python-waste-detector.ts) proved this is not a detector gap — it's a corpus-labeling inconsistency. This issue tracks the corpus-side decision that #117's original "design sketch" did not anticipate.

⚠️ This issue belongs in recost-dev/extension-benchmark (where the fixtures live), filed here only because that repo is outside the current automation scope. Please move/mirror it there.

The problem

The corpus labels exactly one batch FN:

  • flask-mixed-providers/src/providers/anthropic_helper.py:11 — two calls to _client.messages.create(...) across two module-level functions (summarize, summarize_with_style).

But sibling helpers in the same fixture — openai_helper.py, cohere_helper.py — have structurally identical two-calls-across-two-functions shapes and are not labeled as batch findings. Any cross-function pass precise enough to fire on the Anthropic helper also fires on the siblings → 6 FPs, batch precision drops to 14.3%. No AST signal distinguishes the labeled case from the unlabeled-but-identical ones (would require cross-file call-graph signal the scanner doesn't currently expose).

Decision needed — pick one

Option A — Relabel (recover the FN). Add batch-finding labels to the structurally-identical sibling helpers (openai_helper.py, cohere_helper.py, and any others) so the cross-function pass becomes precise. Then re-implement the reverted (provider, methodChain) module-scope pass in python-waste-detector.ts.

Option B — Accept (drop the FN from the recall target). Formally mark the anthropic_helper.py:11 batch case as unrecoverable without cross-file call-graph signal in the corpus, removing it from the batch recall denominator. python-waste-detector.ts stays byte-identical to main.

Acceptance criteria

  • Decision recorded (A or B) with rationale, in extension-benchmark.
  • If A: sibling helpers labeled consistently; the reverted module-scope pass re-implemented; npm run benchmark shows batch at TP 1+ / FP 0 / FN 0; no regression to the 7 existing C1 tests; per-type precision stays 100%.
  • If B: corpus annotation marks the case unrecoverable; batch recall target adjusted; baseline.json regenerated so the row is no longer counted as an outstanding FN.

Do NOT

Re-add the (provider, methodChain) pass without first resolving the labeling — #132 already proved it regresses batch precision to 14.3% on the current corpus.

Links

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/findingsdocs/accuracy/findings.md — calibrated waste detector findingsenhancementNew feature or requestwave/4-recall-recoveryRecover the two C1 FNs (anthropic_helper.py batch + dalle.ts unbatched_parallel)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions