Skip to content

A7: URL-path method fallback for raw fetch (closes #79) #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndresL230 merged 3 commits into from
May 13, 2026
Merged

A7: URL-path method fallback for raw fetch (closes #79) #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4920bbc
feat(fingerprints): add urlPathKey + lookupByUrlPath for raw-fetch ma…
AndresL230 May 13, 2026
fe16e41
fix(detection): A7 — URL-path method fallback for raw fetch (closes #79)
AndresL230 May 13, 2026
24ed60e
fix(a7): address CodeRabbit feedback — boundary matching, monthlyCost…
AndresL230 May 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@
"build:webview": "cd webview && npm run build",
"build:dashboard": "cd dashboard && npm run build && rm -rf ../dashboard-dist && cp -r dist ../dashboard-dist",
"test": "npm run test:scanner",
"test:scanner": "tsc -p tsconfig.scanner-tests.json && tsc -p tsconfig.benchmark.json && node dist-test/test/scanner-patterns.test.js && node dist-test/test/workspace-scanner.test.js && node dist-test/test/workspace-file-access.test.js && node dist-test/test/endpoint-classification.test.js && node dist-test/test/local-waste-detector.test.js && node dist-test/test/chat-providers.test.js && node dist-test/test/fingerprint-registry.test.js && node dist-test/test/pricing-sync.test.js && node dist-test/test/ast-parser-loader.test.js && node dist-test/test/ast-call-visitor.test.js && node dist-test/test/ast-import-resolver.test.js && node dist-test/test/ast-scanner.test.js && node dist-test/test/ast-python.test.js && node dist-test/test/ast-frequency-analyzer.test.js && node dist-test/test/ast-cache-detector.test.js && node dist-test/test/ast-batch-detector.test.js && node dist-test/test/ast-concurrency-detector.test.js && node dist-test/test/ast-cross-file-resolver.test.js && node dist-test/intelligence/__tests__/builder.test.js && node dist-test/intelligence/__tests__/clusters.test.js && node dist-test/intelligence/__tests__/compression.test.js && node dist-test/intelligence/__tests__/export.test.js && node dist-test/test/api-client.test.js && node dist-test/test/key-management.test.js && node dist-test/test/ast-parser-loader-fallback.test.js && node dist-test/intelligence/__tests__/cost-utils.test.js && node dist-test/test/intelligence-compression-async.test.js && node dist-test/test/webview-provider-dispatch.test.js && node dist-test/test/extension-activation.test.js && node dist-test/test/source-span.test.js && node dist-test/test/url-template.test.js && node dist-test/test/enclosing-function.test.js && node dist-test/test/endpoint-id.test.js && node dist-test/test/parity.test.js && node dist-test/test/a6-object-literal-fps.test.js && node dist-test/test/a2-const-fold.test.js && node dist-test/src/test/benchmark-schema.test.js && node dist-test/src/test/benchmark-metrics.test.js",
"test:scanner": "tsc -p tsconfig.scanner-tests.json && tsc -p tsconfig.benchmark.json && node dist-test/test/scanner-patterns.test.js && node dist-test/test/workspace-scanner.test.js && node dist-test/test/workspace-file-access.test.js && node dist-test/test/endpoint-classification.test.js && node dist-test/test/local-waste-detector.test.js && node dist-test/test/chat-providers.test.js && node dist-test/test/fingerprint-registry.test.js && node dist-test/test/pricing-sync.test.js && node dist-test/test/ast-parser-loader.test.js && node dist-test/test/ast-call-visitor.test.js && node dist-test/test/ast-import-resolver.test.js && node dist-test/test/ast-scanner.test.js && node dist-test/test/ast-python.test.js && node dist-test/test/ast-frequency-analyzer.test.js && node dist-test/test/ast-cache-detector.test.js && node dist-test/test/ast-batch-detector.test.js && node dist-test/test/ast-concurrency-detector.test.js && node dist-test/test/ast-cross-file-resolver.test.js && node dist-test/intelligence/__tests__/builder.test.js && node dist-test/intelligence/__tests__/clusters.test.js && node dist-test/intelligence/__tests__/compression.test.js && node dist-test/intelligence/__tests__/export.test.js && node dist-test/test/api-client.test.js && node dist-test/test/key-management.test.js && node dist-test/test/ast-parser-loader-fallback.test.js && node dist-test/intelligence/__tests__/cost-utils.test.js && node dist-test/test/intelligence-compression-async.test.js && node dist-test/test/webview-provider-dispatch.test.js && node dist-test/test/extension-activation.test.js && node dist-test/test/source-span.test.js && node dist-test/test/url-template.test.js && node dist-test/test/enclosing-function.test.js && node dist-test/test/endpoint-id.test.js && node dist-test/test/parity.test.js && node dist-test/test/a6-object-literal-fps.test.js && node dist-test/test/a2-const-fold.test.js && node dist-test/test/a7-url-path-fallback.test.js && node dist-test/src/test/benchmark-schema.test.js && node dist-test/src/test/benchmark-metrics.test.js",
"calibrate-detectors": "tsc -p tsconfig.scanner-tests.json && node dist-test/test/waste-calibration.js",
"watch:ext": "node esbuild.mjs --watch",
"watch:webview": "cd webview && npm run build -- --watch",
Expand Down
12 changes: 10 additions & 2 deletions src/intelligence/compression.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,13 +361,21 @@ function buildFileSummary(filePath: string, snapshot: RepoIntelligenceSnapshot):
const context = buildFileContext(snapshot, filePath);
const provider = context.providers[0] ?? null;
const callsPerDay = estimateCallsPerDay(context.apiCalls);
const methodSig = context.apiCalls[0]?.method ?? undefined;
// A7: In mixed-provider clusters, pair the chosen provider with one of its
// own calls so method/url align with the provider used for pricing.
const representativeCall = provider
? (context.apiCalls.find((call) => normalizeProviderId(call.provider) === provider) ?? context.apiCalls[0])
: context.apiCalls[0];
const methodSig = representativeCall?.method ?? undefined;
// A7: pass the matching call's URL so URL-path lookup can resolve pricing
// when there is no SDK method chain (raw fetch).
const url = representativeCall?.url ?? undefined;
return {
filePath,
description: ensureMaxSentences(getDescription(context), 2),
providers: context.providers,
topRisks: getTopRisks(context),
estimatedMonthlyCost: provider ? (estimateLocalMonthlyCost(provider, callsPerDay, methodSig) ?? null) : null,
estimatedMonthlyCost: provider ? (estimateLocalMonthlyCost(provider, callsPerDay, methodSig, url) ?? null) : null,
whyItMatters: ensureMaxSentences(getWhyItMatters(context), 1),
};
}
Expand Down
53 changes: 31 additions & 22 deletions src/intelligence/cost-utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { lookupMethod } from "../scanner/fingerprints/registry";
import { lookupMethod, lookupByUrlPath } from "../scanner/fingerprints/registry";

// Best-effort local cost estimation shared by the webview and intelligence
// layer. When a provider or pricing signal is missing, callers can fall back
Expand Down Expand Up @@ -44,32 +44,41 @@ const DEFAULT_PER_CALL_COST = 0.0001;
export function estimateLocalMonthlyCost(
provider: string,
callsPerDay: number,
methodSignature?: string
methodSignature?: string,
/**
* A7 (issue #79): URL for the call, used when `methodSignature` is undefined
* (e.g. raw `fetch()` with a known provider host but no SDK method chain).
*/
url?: string,
): number | null {
if (!provider || provider === "unknown") return null;
if (!Number.isFinite(callsPerDay) || callsPerDay < 0) return null;

if (methodSignature) {
const fingerprint = lookupMethod(provider, methodSignature);
if (fingerprint) {
if (fingerprint.costModel === "free") return 0;
if (fingerprint.costModel === "per_token") {
const inputTokens = 500;
const outputTokens = 200;
const inputCost = (inputTokens / 1_000_000) * (fingerprint.inputPricePer1M ?? 0);
const outputCost = (outputTokens / 1_000_000) * (fingerprint.outputPricePer1M ?? 0);
return Math.round((inputCost + outputCost) * callsPerDay * 30 * 100) / 100;
}
if (fingerprint.costModel === "per_transaction") {
const txValue = 50;
const fee = (fingerprint.fixedFee ?? 0) + txValue * (fingerprint.percentageFee ?? 0);
return Math.round(fee * callsPerDay * 30 * 100) / 100;
}
if (fingerprint.costModel === "per_request") {
return Math.round((fingerprint.fixedFee ?? fingerprint.perRequestCostUsd ?? DEFAULT_PER_CALL_COST) * callsPerDay * 30 * 100) / 100;
}
return null;
let fingerprint = methodSignature ? lookupMethod(provider, methodSignature) : null;

// A7: fall back to URL-path lookup when the SDK chain didn't resolve.
if (!fingerprint && url) {
fingerprint = lookupByUrlPath(provider, url);
}

if (fingerprint) {
if (fingerprint.costModel === "free") return 0;
if (fingerprint.costModel === "per_token") {
const inputTokens = 500;
const outputTokens = 200;
const inputCost = (inputTokens / 1_000_000) * (fingerprint.inputPricePer1M ?? 0);
const outputCost = (outputTokens / 1_000_000) * (fingerprint.outputPricePer1M ?? 0);
return Math.round((inputCost + outputCost) * callsPerDay * 30 * 100) / 100;
}
if (fingerprint.costModel === "per_transaction") {
const txValue = 50;
const fee = (fingerprint.fixedFee ?? 0) + txValue * (fingerprint.percentageFee ?? 0);
return Math.round(fee * callsPerDay * 30 * 100) / 100;
}
if (fingerprint.costModel === "per_request") {
return Math.round((fingerprint.fixedFee ?? fingerprint.perRequestCostUsd ?? DEFAULT_PER_CALL_COST) * callsPerDay * 30 * 100) / 100;
}
return null;
}

const perCall = LOCAL_PRICING[provider];
Expand Down
8 changes: 7 additions & 1 deletion src/scan-results.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ export function mergeRemoteAndLocalEndpoints(
crossFileOrigin: call.crossFileOrigin ?? null,
}],
callsPerDay,
monthlyCost: estimateLocalMonthlyCost(provider, callsPerDay, call.methodSignature) ?? 0,
monthlyCost: estimateLocalMonthlyCost(provider, callsPerDay, call.methodSignature, call.url) ?? 0,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
status: call.frequency === "per-request" ? "n_plus_one_risk" : "normal",
methodSignature: call.methodSignature,
costModel: call.costModel,
Expand Down Expand Up @@ -457,6 +457,12 @@ export function mergeRemoteAndLocalEndpoints(
synthetic.crossFileOrigins = synthetic.crossFileOrigins ?? [];
synthetic.crossFileOrigins.push(call.crossFileOrigin);
}
synthetic.monthlyCost = estimateLocalMonthlyCost(
synthetic.provider,
synthetic.callsPerDay,
synthetic.methodSignature,
synthetic.url,
) ?? 0;
}

return [...merged, ...syntheticByMethodUrl.values()];
Expand Down
11 changes: 10 additions & 1 deletion src/scanner/fingerprints/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ Valid `Language` values: `"javascript"`, `"typescript"`, `"python"`, `"go"`, `"j

| Field | Type | Required | Description |
|---|---|---|---|
| `pattern` | `string` | ✓ | SDK method chain without the variable prefix, e.g. `"chat.completions.create"` |
| `pattern` | `string` | one of | SDK method chain without the variable prefix, e.g. `"chat.completions.create"`. Use this OR `urlPathKey` |
| `urlPathKey` | `string` | one of | URL-path segment for raw-`fetch`-only providers, e.g. `"v1/text-to-speech"`. Use this OR `pattern`. See [URL-path fingerprints](#url-path-fingerprints-urlpathkey) below |
| `httpMethod` | `string` | ✓ | HTTP verb: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `SUBSCRIBE`, or `RPC` |
| `endpoint` | `string` | ✓ | Full URL of the API endpoint, e.g. `"https://api.sendgrid.com/v3/mail/send"` |
| `costModel` | `CostModel` | ✓ | One of `"per_token"`, `"per_transaction"`, `"per_request"`, `"free"` |
Expand All @@ -115,12 +116,20 @@ Valid `Language` values: `"javascript"`, `"typescript"`, `"python"`, `"go"`, `"j
| `cacheCapable` | `boolean` | — | `true` if responses can be cached |
| `description` | `string` | — | One-line human-readable description |

Either `pattern` or `urlPathKey` must be set on every method entry — the registry will reject a file where any method has neither.

**Cost model rules:**
- `per_token` — must have `inputPricePer1M`
- `per_transaction` — must have `fixedFee` or `percentageFee` (or both)
- `per_request` — no pricing fields required (usage-based tiers)
- `free` — no pricing fields

### URL-path fingerprints (`urlPathKey`)

For providers commonly called via raw `fetch(...)` (with no SDK method chain), add a method entry with `urlPathKey` instead of `pattern`. The matcher walks longest-key first and falls back to a `_default` entry if you add one. Path matching is segment-aware (matches on `/` boundaries), so `"v1/text-to-speech"` matches `/v1/text-to-speech/voice-abc/stream` but NOT `/api/v1`.

See `elevenlabs.json` for a worked example.

---

## Worked Example — Twilio
Expand Down
31 changes: 31 additions & 0 deletions src/scanner/fingerprints/elevenlabs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,37 @@
"costModel": "free",
"cacheCapable": true,
"description": "Conversational AI — list agents is free"
},
{
"urlPathKey": "v1/text-to-speech",
"httpMethod": "POST",
"endpoint": "https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
"costModel": "per_request",
"perRequestCostUsd": 0.0003,
"description": "Raw-fetch TTS path (A7)"
},
{
"urlPathKey": "v1/speech-to-text",
"httpMethod": "POST",
"endpoint": "https://api.elevenlabs.io/v1/speech-to-text",
"costModel": "per_request",
"perRequestCostUsd": 0.00006,
"description": "Raw-fetch STT path (A7)"
},
{
"urlPathKey": "v1/voices",
"httpMethod": "GET",
"endpoint": "https://api.elevenlabs.io/v1/voices",
"costModel": "free",
"description": "Raw-fetch list voices (A7)"
},
{
"urlPathKey": "_default",
"httpMethod": "POST",
"endpoint": "https://api.elevenlabs.io/v1",
"costModel": "per_request",
"perRequestCostUsd": 0.0001,
"description": "Unrecognized ElevenLabs path — conservative fallback (A7)"
}
]
}
74 changes: 72 additions & 2 deletions src/scanner/fingerprints/registry.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ const DEBUG_BUNDLE_LOGS = process.env.RECOST_DEBUG_SCAN === "1";
/** provider (lowercase) → pattern → MethodFingerprint */
const methodIndex = new Map<string, Map<string, MethodFingerprint>>();

/**
* A7 (issue #79): provider (lowercase) → list of URL-path methods, sorted by
* descending urlPathKey length so longest-match wins. The special key
* `"_default"` is the provider-wide fallback and is kept at the end.
*/
const urlPathIndex = new Map<string, MethodFingerprint[]>();

/** lowercase exact hostname → provider id */
const exactHostIndex = new Map<string, string>();

Expand All @@ -17,13 +24,32 @@ const regexHostIndex: Array<{ regex: RegExp; provider: string }> = [];
for (const fp of ALL_PROVIDERS) {
const key = fp.provider.toLowerCase();

// Method index
// Method index — SDK-chain entries only (those with `pattern`)
const methods = new Map<string, MethodFingerprint>();
const urlPathEntries: MethodFingerprint[] = [];
for (const m of fp.methods) {
methods.set(m.pattern, m);
if (m.pattern) {
methods.set(m.pattern, m);
}
if (m.urlPathKey) {
urlPathEntries.push(m);
}
}
methodIndex.set(key, methods);

if (urlPathEntries.length > 0) {
// Longest urlPathKey first so specific matches win over short prefixes.
// `_default` is always the longest-tail fallback.
urlPathEntries.sort((a, b) => {
const aIsDefault = a.urlPathKey === "_default";
const bIsDefault = b.urlPathKey === "_default";
if (aIsDefault && !bIsDefault) return 1;
if (!aIsDefault && bIsDefault) return -1;
return (b.urlPathKey?.length ?? 0) - (a.urlPathKey?.length ?? 0);
});
urlPathIndex.set(key, urlPathEntries);
}

// Host index (exact entries in ALL_PROVIDERS take priority)
for (const h of fp.hosts) {
const resolvedProvider = h.provider ?? fp.provider;
Expand Down Expand Up @@ -122,6 +148,50 @@ export function lookupHost(hostname: string): string | null {
return null;
}

/**
* Find a fingerprint method by matching the request URL's path against
* `urlPathKey` entries. Falls back to the `_default` entry if no specific
* match. Returns `null` if the provider is unknown, has no URL-path entries,
* or the URL is malformed.
*
* A7 (issue #79): raw-fetch calls have a provider attributed via host match
* (see `lookupHost`) but no SDK method chain. Match by URL path instead so
* the cost layer can produce a non-stub estimate.
*
* Matching is longest-key-first against the request path+query, so
* `"v1/text-to-speech"` wins over a hypothetical shorter `"v1/"` prefix.
*/
export function lookupByUrlPath(provider: string, url: string): MethodFingerprint | null {
if (!provider || !url) return null;

const entries = urlPathIndex.get(provider.toLowerCase());
if (!entries || entries.length === 0) return null;

let pathAndQuery: string;
try {
const parsed = new URL(url);
pathAndQuery = parsed.pathname + (parsed.search ?? "");
} catch {
return null;
}

let fallback: MethodFingerprint | null = null;
for (const entry of entries) {
if (entry.urlPathKey === "_default") {
fallback = entry;
continue;
}
if (entry.urlPathKey) {
const escaped = entry.urlPathKey.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const boundary = new RegExp(`(^|/)${escaped}(/|$|\\?)`);
if (boundary.test(pathAndQuery)) {
return entry;
}
}
}
return fallback;
}

/**
* Return all registered provider ids (in registration order).
*/
Expand Down
14 changes: 12 additions & 2 deletions src/scanner/fingerprints/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,18 @@ export type Language =
| "rust";

export interface MethodFingerprint {
/** SDK method chain pattern, e.g. "chat.completions.create" */
pattern: string;
/**
* SDK method chain pattern, e.g. "chat.completions.create".
* Either `pattern` or `urlPathKey` must be set on every entry.
*/
pattern?: string;
/**
* URL-path substring used by `lookupByUrlPath` when an API call has a known
* provider but no SDK method chain (e.g. raw `fetch(...)`). The matcher tries
* the longest `urlPathKey` first; the special value `"_default"` is a
* provider-wide fallback (A7, issue #79).
*/
urlPathKey?: string;
Comment on lines +12 to +23

@coderabbitai coderabbitai Bot May 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Encode the pattern/urlPathKey requirement in the type, not only in comments.

MethodFingerprint currently allows entries with neither field set, despite Line 14’s contract. Enforcing this at type level prevents invalid fingerprint records from compiling.

Proposed type-safe shape 
-export interface MethodFingerprint {
+type MethodFingerprintKey =
+  | { pattern: string; urlPathKey?: string }
+  | { pattern?: string; urlPathKey: string };
+
+export type MethodFingerprint = MethodFingerprintKey & {
   /**
    * SDK method chain pattern, e.g. "chat.completions.create".
    * Either `pattern` or `urlPathKey` must be set on every entry.
    */
-  pattern?: string;
+  pattern?: string;
   /**
    * URL-path substring used by `lookupByUrlPath` when an API call has a known
    * provider but no SDK method chain (e.g. raw `fetch(...)`). The matcher tries
    * the longest `urlPathKey` first; the special value `"_default"` is a
    * provider-wide fallback (A7, issue `#79`).
    */
   urlPathKey?: string;
   /** HTTP verb: GET | POST | PUT | PATCH | DELETE | SUBSCRIBE | RPC */
   httpMethod: string;
   /** Full URL or URL template for the mapped endpoint */
   endpoint: string;
   costModel: CostModel;
   ...
-}
+};
As per coding guidelines, "`src/**/*.ts`: Use TypeScript in strict mode for extension backend."
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/scanner/fingerprints/types.ts` around lines 12 - 23, The
MethodFingerprint type currently allows neither pattern nor urlPathKey to be
set; change its definition in src/scanner/fingerprints/types.ts to encode the
contract as a discriminated union so each entry must have at least one of the
two fields (e.g. a union of a shape with required pattern and optional
urlPathKey and a shape with required urlPathKey and optional pattern), preserve
any other existing optional properties on MethodFingerprint, and update any call
sites or constructors that create MethodFingerprint objects to satisfy the new
union (adjust tests/fixtures as needed).

@AndresL230 AndresL230 May 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping. The runtime validator in registry.ts already enforces pattern || urlPathKey at JSON load time, and MethodFingerprint instances are only constructed by JSON load (no in-code constructors). A discriminated union would force type-narrowing at every read of m.pattern (which happens many places) for a constraint already covered at the single entry point. The ergonomic cost exceeds the value of compile-time enforcement here.

@coderabbitai coderabbitai Bot May 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AndresL230, understood — that's a reasonable trade-off. Centralising the invariant at the single JSON-load entry point via the runtime validator avoids scattered type-narrowing across all m.pattern read sites, and the ergonomic cost of a discriminated union would be real given the number of call sites.

✏️ Learnings added 
Learnt from: AndresL230
Repo: recost-dev/extension PR: 104
File: src/scanner/fingerprints/types.ts:12-23
Timestamp: 2026-05-13T20:11:35.796Z
Learning: In `src/scanner/fingerprints/types.ts`, `MethodFingerprint` deliberately keeps `pattern` and `urlPathKey` as plain optional fields rather than a discriminated union. The `pattern || urlPathKey` invariant is enforced at the single entry point (JSON load) by the runtime validator in `src/scanner/fingerprints/registry.ts`. All `MethodFingerprint` instances are constructed only via JSON load with no in-code constructors, so compile-time union enforcement was judged to have too high an ergonomic cost (type-narrowing at every `m.pattern` read site) relative to the benefit.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

/** HTTP verb: GET | POST | PUT | PATCH | DELETE | SUBSCRIBE | RPC */
httpMethod: string;
/** Full URL or URL template for the mapped endpoint */
Expand Down
Loading
Loading