Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions benchmark/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"detectionRecall": 0.5147058823529411,
"providerAttributionAccuracy": 0.8275862068965517,
"findingPrecision": 1,
"findingRecall": 0.3333333333333333,
"findingRecall": 0.6666666666666666,
"findingMetricsByType": {
"batch": {
"truePositives": 0,
Expand All @@ -20,11 +20,11 @@
"recall": 1
},
"unbatched_parallel": {
"truePositives": 0,
"truePositives": 1,
"falsePositives": 0,
"falseNegatives": 1,
"falseNegatives": 0,
"precision": 1,
"recall": 0
"recall": 1
}
}
}
3 changes: 2 additions & 1 deletion dashboard/src/lib/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ export type SuggestionType =
| "redundancy"
| "n_plus_one"
| "rate_limit"
| "concurrency_control";
| "concurrency_control"
| "unbatched_parallel";

export type Severity = "high" | "medium" | "low";

Expand Down
2 changes: 2 additions & 0 deletions dashboard/src/pages/Suggestions.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ const severityConfig: Record<Severity, { color: string; bg: string; icon: Elemen
const typeIcons: Partial<Record<SuggestionType, ElementType>> = {
cache: Archive,
batch: Layers,
unbatched_parallel: Layers,
redundancy: RefreshCw,
n_plus_one: Layers,
rate_limit: Zap,
Expand All @@ -25,6 +26,7 @@ const typeIcons: Partial<Record<SuggestionType, ElementType>> = {
const typeLabels: Record<string, string> = {
cache: 'Cacheable',
batch: 'Batchable',
unbatched_parallel: 'Unbatched Parallel',
redundancy: 'Redundant Call',
n_plus_one: 'N+1 Query',
rate_limit: 'Rate Limit Risk',
Expand Down
2 changes: 0 additions & 2 deletions dashboard/src/styles/theme.css
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,6 @@
display: none;
}

<<<<<<< HEAD
=======
@keyframes pulseGlow {
0% {
fill: #444;
Expand Down
17 changes: 9 additions & 8 deletions docs/accuracy/findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,17 +38,18 @@ Each detector has a measured false-positive rate against the benchmark corpus (D
- [ ] FPR is re-measured on every benchmark CI run; regressions fail the build.
- [ ] False positives that remain are by-design (documented exceptions, e.g., "we choose to flag this conservatively because the cost of missing it is high").

### Calibration table (measured 2026-05-13 against corpus v1 — 7 fixtures, 3 expected findings; refreshed after C1 PR-3 merged)
### Calibration table (measured 2026-05-13 against corpus v1 — 7 fixtures, 3 expected findings; refreshed after C1 PR-3 merged; updated after Wave 4 / #117)

| Detector (scanner `type`) | TP | FP | FN | FPR | Precision | Severity (current) | Notes |
|---|---|---|---|---|---|---|---|
| `n_plus_one` | 1 | 0 | 0 | 0% | 100% | high | Only detector with a corpus TP. Sample size = 1. |
| `cache` | 0 | 0 | 0 | — | — | medium | C1 PR-2 dropped emissions from 7 to 0 — Python detector now suppresses generative endpoints + explicit write-shaped HTTP methods, AST detector buckets fetch/axios redundancy by URL. No emissions, no expected entries; row collapses to absent in `findingMetricsByType`. |
| `batch` | 0 | 1 | 1 | 100% | 0% | medium | C1 PR-3 dropped emissions from 9 to 1 by bucketing both TS and Python sequential-batching detectors by `(provider, enclosingFunction)` (calls in different functions can't be batched together) plus line-dedup for cross-file resolver expansion. The one remaining FP is `bedrock-raw-fetch/src/index.ts:5` — two sequential `await handleApi(...)` calls in `main()`, arguably a true positive that the corpus didn't label. The expected `batch` TP at `flask-mixed-providers/src/providers/anthropic_helper.py:11` is still missed (the detector no longer mis-emits at line 7, so finding-recall is unchanged at 33.33%). Sample size = 1, below per-type gate threshold. |
| `rate_limit` | 0 | 1 | 0 | 100% | 0% | low | One FP. No expected entries. |
| `concurrency_control` | — | — | — | — | — | low | Scanner emits nothing on the corpus; not in the table. See "Type-name mismatch" below. |

The corpus labels one fan-out finding as `unbatched_parallel`; the scanner emits `concurrency_control` for the same pattern. The matcher compares type strings exactly, so the expected `unbatched_parallel` shows up as a recall miss (FN = 1) and the scanner's `concurrency_control` (if it were ever emitted on this corpus) would show up as a separate row of FPs. As of 2026-05-13 the scanner emits zero `concurrency_control` findings on the corpus, so only the FN side appears. The label gap is tracked as a corpus follow-up — either rename the expected type to `concurrency_control` or have the scanner emit `unbatched_parallel` for this specific pattern.
| `n_plus_one` | 1 | 0 | 0 | 0% | 100% | high | Only detector with a corpus TP. Sample size = 1. |
| `cache` | 0 | 0 | 0 | — | — | medium | C1 PR-2 dropped emissions from 7 to 0 — Python detector now suppresses generative endpoints + explicit write-shaped HTTP methods, AST detector buckets fetch/axios redundancy by URL. No emissions, no expected entries; row collapses to absent in `findingMetricsByType`. |
| `batch` | 0 | 0 | 1 | — | 100% | medium | C1 PR-3 bucketed sequential-batching detectors by `(provider, enclosingFunction)` + line-dedup. The prior bedrock FP (`bedrock-raw-fetch/src/index.ts:5`) is gone — live benchmark shows `batch` clean at 0 FP. The expected TP at `flask-mixed-providers/src/providers/anthropic_helper.py:11` is still missed (FN = 1, recall 0%). Wave 4 / #117 attempted a cross-function `(provider, methodChain)` batching pass to recover this FN, but it fired equally on structurally-identical sibling helpers (`openai_helper.py`, `cohere_helper.py`) that are unlabeled in `expected.json`, producing 6 FPs and dropping `batch` precision to 14.3%. Because no AST signal distinguishes the labeled-TP case from the unlabeled-but-equivalent cases, this is a **corpus-labeling inconsistency**: either the sibling helpers should also be labeled as batch findings, or the FN should be accepted as unrecoverable without cross-file call-graph signal. The cross-function pass was reverted. Tracked as a corpus follow-up, not a detector fix. |
| `unbatched_parallel` | 1 | 0 | 0 | 0% | 100% | derived | Wave 4 / #117 recovered the DALL-E `Promise.all(Array.from({length:n}).map(() => images.generate()))` FN. A dedicated `unbatched_parallel` SuggestionType was added; `detectInlineParallel` (AST) and `detectInlineParallelFinding` (regex fallback) now emit it. The `BOUNDED_REPLICATION` guard was removed from the inline-parallel path because `inlineParallelCapable` is the precision control. Severity is derived from the score-based `deriveSeverity()` pipeline (not hardcoded). AST + regex paths both emit it. |
| `rate_limit` | 0 | 1 | 0 | 100% | 0% | low | One FP. No expected entries. |
| `concurrency_control` | — | — | — | — | — | low | Scanner emits nothing on the corpus; not in the table. |

**Wave 4 / #117 outcome (2026-05-29):** Two benchmark finding false-negatives were targeted. One shipped: the DALL-E `unbatched_parallel` FN is recovered — finding recall rose from 33.33% → 66.67%, precision held at 100%. One did not ship: the Python cross-function `batch` FN at `flask-mixed-providers/anthropic_helper.py:11` remains open. The attempted recovery pass produced 6 FPs on structurally-identical but unlabeled sibling helper files — a corpus-labeling inconsistency, not a detector gap. The pass was reverted; the `batch` row is now clean (0 FP, precision 100%) with recall 0%. Recommended corpus follow-up: either label `openai_helper.py` and `cohere_helper.py` as batch findings too, or formally accept this FN as unrecoverable without cross-file call-graph signal.

Acceptance criterion "no detector with FPR > 30%" passes for `n_plus_one` and `cache` after PR-2 and effectively passes for `batch` after PR-3 (sample size 1 below per-type gate threshold; remaining FP is borderline TP). Still fails for `rate_limit` (sample size 1). Sample sizes remain small (corpus v1 has 3 expected findings total), so the FPR numbers are diagnostic, not statistically robust. Wait until the corpus grows past N ≥ 10 expected findings per type before defending an "FPR < 30%" target as final.

Expand Down
Loading
Loading