diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e785013..a16484a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,6 +39,8 @@ jobs: FLEETDIFF_BIN="$PWD/bin/fleetdiff" FLEETDIFF_SPEND_DOCKER=1 python3 -m unittest discover -s examples/litellm-spend -p 'test_*.py' -v - name: Offline capture accounting contract run: | + go run ./examples/inspect --variant stock > "$RUNNER_TEMP/inspect-synthetic-traces.json" + cmp examples/inspect/synthetic-traces.json "$RUNNER_TEMP/inspect-synthetic-traces.json" (cd testdata/inspect-contract/v1 && shasum -a 256 -c SHA256SUMS) go test ./internal/inspect -run '^TestInspectAccountingContract$' -count=1 sh examples/inspect.sh @@ -107,7 +109,7 @@ jobs: go test ./internal/compare -run '^$' -fuzz '^FuzzCanonicalSummary$' -fuzztime 10s -parallel 2 go test ./internal/compare -run '^$' -fuzz '^FuzzEmbeddedSketch$' -fuzztime 10s -parallel 2 go test ./internal/compare -run '^$' -fuzz '^FuzzScanSummary$' -fuzztime 10s -parallel 2 - go test ./internal/diagnose -run '^$' -fuzz '^FuzzDiagnose$' -fuzztime 10s -parallel 2 + go test ./internal/diagnose -run '^$' -fuzz '^FuzzDiagnose$' -fuzztime 10000x -parallel 2 -timeout 2m go test ./internal/inspect -run '^$' -fuzz '^FuzzTraceRecord$' -fuzztime 10s -parallel 2 - name: Two operators through the released collector if: runner.os == 'Linux' diff --git a/CHANGELOG.md b/CHANGELOG.md index 0043cf4..ac1c3fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,21 @@ Notable user-visible changes are recorded here, newest first. -## Unreleased +## [0.7.0] - 2026-10-11 + +See the [release notes](docs/releases/v0.7.0.md) for the SQL export default and +report upgrade. + +- Compare recorded cache-read and cache-write shares and uncached input on + LiteLLM exports, with keyed uncached-growth rankings and separate coverage. + Include version-pinned cache and reasoning columns in the SQL export by + default, preserving absent writer paths as NULL. +- Extend summary cache comparisons with independent read/write coverage and + uncached totals; identify unavailable model and contributor breakdowns. +- Ship a deterministic synthetic OTLP capture for binary-only inspection, + checked byte for byte against its generator in CI. +- Lead with significant recorded cache changes, show request coverage in words, + and direct cache-read-share drops to provider diagnostics and route checks. ## [0.6.1] - 2026-10-11 diff --git a/README.md b/README.md index c69557d..129bbc5 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ whether the usage records are complete. Use the [read-only SQL projection](examples/litellm-spend/export.sql) and your existing PostgreSQL access. The [export guide](examples/litellm-spend/README.md#export-request-level-rows) -covers connection settings and optional cache fields. Once your approved +covers connection settings and version-pinned cache fields. Once your approved read-only connection is named `litellm_readonly`, this example exports 16 UTC days: ```sh @@ -35,7 +35,7 @@ umask 077 export_dir="$HOME/litellm-export" mkdir "$export_dir" curl --fail --location --proto '=https' --tlsv1.2 \ - https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.6.0/examples/litellm-spend/export.sql \ + https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.7.0/examples/litellm-spend/export.sql \ --output "$export_dir/export.sql" psql 'service=litellm_readonly' -Xq -v ON_ERROR_STOP=1 \ -v start_utc=2026-09-23T00:00:00Z -v end_utc=2026-10-09T00:00:00Z \ @@ -48,13 +48,14 @@ The SQL writes CSV in one pass and excludes prompts and responses. Keep the export directory private. Existing request-level JSON and JSONL work too. + -### 2. Install v0.6.1 +### 2. Install v0.7.0 Run this in Bash with `curl`, `gh`, `shasum` and `tar` available: ```sh -bash -o pipefail -c 'curl --fail --location --proto "=https" --tlsv1.2 https://raw.githubusercontent.com/llm-measurement/fleetdiff/7773f212e4ae6e800f066b394e5507c0df8ada5f/scripts/install.sh | sh -s -- v0.6.1 ./fleetdiff-install' +bash -o pipefail -c 'curl --fail --location --proto "=https" --tlsv1.2 https://raw.githubusercontent.com/llm-measurement/fleetdiff/7773f212e4ae6e800f066b394e5507c0df8ada5f/scripts/install.sh | sh -s -- v0.7.0 ./fleetdiff-install' ./fleetdiff-install/fleetdiff --version ``` @@ -77,7 +78,7 @@ to dates or explicit UTC start/end intervals. The report shows request count versus tokens per request by model, the leading contributors, and usage completeness. Choose `--group-by team`, `user`, `end-user` or `session` for another view; add `--format json` for automation. -[Full command and export guide](examples/litellm-spend/README.md). +[Export guide](examples/litellm-spend/README.md); [compare recorded cache reuse](docs/CACHE.md). ## Try One Example @@ -85,7 +86,7 @@ Download the synthetic spend file and use the installed binary: ```sh curl --fail --location --proto '=https' --tlsv1.2 \ - https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.6.0/examples/litellm-spend/synthetic.csv \ + https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.7.0/examples/litellm-spend/synthetic.csv \ --output synthetic.csv ./fleetdiff-install/fleetdiff investigate --litellm-spend synthetic.csv \ --before-period 2026-10-07 --after-period 2026-10-08 @@ -119,14 +120,14 @@ The summary-file path also shows session concentration: ## Install The verified installer above selects the native binary for your machine. -For manual downloads, use the [v0.6.1 archives](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.6.1) +For manual downloads, use the [v0.7.0 archives](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.7.0) and [verification instructions](docs/OPERATIONS.md#install-and-verify). An administrator can verify and transfer artifacts through an approved internal mirror. With a security-patched Go 1.26 toolchain, install the same version from its module: ```sh -go install github.com/llm-measurement/fleetdiff/cmd/fleetdiff@v0.6.1 +go install github.com/llm-measurement/fleetdiff/cmd/fleetdiff@v0.7.0 ``` The executable goes to `GOBIN`, or `$(go env GOPATH)/bin` when unset. @@ -157,7 +158,7 @@ See [source builds and version reporting](docs/OPERATIONS.md#build-from-source). - [Questions and contributor rankings](docs/WORKFLOWS.md#questions-it-answers). - [How summary comparison works](docs/WORKFLOWS.md#how-it-works), [comparison contract](docs/COMPARISON.md) and [investigation contract](docs/INVESTIGATION.md). - [Operations](docs/OPERATIONS.md), [resource measurements](docs/BENCHMARKS.md) and [checks](docs/WORKFLOWS.md#built-to-be-checked). -- [FAQ](docs/FAQ.md), [security](SECURITY.md), [release notes](docs/releases/v0.6.1.md) and [release procedure](docs/RELEASING.md). +- [FAQ](docs/FAQ.md), [security](SECURITY.md), [release notes](docs/releases/v0.7.0.md) and [release procedure](docs/RELEASING.md). Questions or feedback: [open an issue](https://github.com/llm-measurement/fleetdiff/issues). See [Contributing](CONTRIBUTING.md) for checks and signed, signed-off commits. diff --git a/docs/CACHE.md b/docs/CACHE.md index f3f7b8b..2e62a5f 100644 --- a/docs/CACHE.md +++ b/docs/CACHE.md @@ -1,130 +1,134 @@ -# Cached-Token Share +# Compare Recorded Cache Reuse -Included in fleetdiff v0.5.0. Use the [verified release binary](OPERATIONS.md#install-and-verify) -or build from this source checkout. +Compare cache-read and cache-write shares before and after a change, then find +the keys, teams or sessions whose recorded uncached input grew. The expanded +view is available in **v0.7.0**. Versions v0.5.0 through v0.6.1 already provide +aggregate cache-read-share comparison for summary windows. -`fleetdiff investigate` appends `cache` after the existing six questions: -**Did caching get worse?** No new sketch or collector configuration is required. +## Try The Cache Drop -For each window, the recorded cached-token share is: +Build this checkout, then run the deterministic four-request example: -```text -cache_read_input_tokens / input_tokens -change in percentage points = 100 * (after share - before share) +```sh +go build -o ./fleetdiff-cache ./cmd/fleetdiff +./fleetdiff-cache investigate --litellm-spend examples/litellm-spend/cache-drop.csv \ + --before-period 2026-10-07 --after-period 2026-10-08 --group-by key ``` -Text describes this as **Share of recorded input tokens served from cache.** -The short explanation does not change the measurement or its evidence requirements. -JSON keeps the full explanation, including the distinction from request hit rate, -cost, savings, and provider billing. - -A decrease means a smaller recorded fraction of input tokens was cache-read; -an increase means a larger fraction. Neither establishes the cause. The JSON -`cache` object retains exact before/after input and cache-read totals, fractional -shares, `delta_percentage_points`, and `direction` (`decreased`, `increased`, or -`unchanged`). Direction and change are computed with exact rational arithmetic -before conversion to floating point. Text rounds percentages to two decimals, -so tiny changes can display as zero; inspect JSON and integer totals for detail. +Selected output: -This is **not a request cache-hit rate, cost, savings, or provider billing -claim**. Input provenance may be unknown or inferred: a numeric field is not -necessarily provider-reported. Even provider-reported declarations do not -establish billing equivalence. A changed workload mix can change this share. +```text +But cache reuse fell: cache-read share 92% -> 61% (uncached input +620). + Cache-read share of recorded input: 92% -> 61%. + Cache reads: 1,840 of 2,000 input tokens -> 1,220 of 2,000 (all requests report cache detail). + Cache-write share of recorded input: 5% -> 7.5%. + Cache writes: 100 of 2,000 input tokens -> 150 of 2,000 (all requests report cache detail). + Uncached input +620: cache-read share fell from 92% to 61%. + key-1: 80 -> 700 uncached input tokens; change +620. +``` -## Required Evidence +Total input and output stay flat. One synthetic key accounts for the added +uncached input. Use `--group-by team`, `user`, `end-user` or `session` for another +view. Aliases refer to the same identities across the total-token and cache +rankings in one report; `--show-hashes` also prints their keyed hashes. +Sessions remain scoped to their API key. -Both windows need complete declared observation intervals for every expected -producer and positive combined input-token totals. Partial observations remain -`cannot_determine` even with `--allow-partial`. Normal summary validation still -rejects invalid envelopes, overlaps, conflicting replays, and counter regression. +## Measurements And Coverage -Every supplied snapshot must contain the typed integer counters `requests`, -`input_tokens`, and `cache_read_input_tokens`, plus all ten reviewed quality -counters below. For each of `input` and `cache_read_input`: +The same quantities and wording apply to spend files and summary windows: -| Counter suffix in `token_observations.FIELD.STATE` | Required value | +| Measurement | Calculation on the same covered requests | | --- | --- | -| `reported` | Equal to that snapshot's `requests` | -| `missing` | Present and zero | -| `invalid` | Present and zero | -| `conflict` | Present and zero | -| `subset_violation` | Present and zero | - -These are the collector's exported accounting states. Optional cache detail -does **not** increment `missing` when absent, so `missing == 0` alone is -insufficient. An invalid source can coexist with a reported value. A violating -individual detail can also be hidden by otherwise valid aggregate totals. -Accordingly, both reported coverage and every quality state are checked in each -original snapshot before trusting the combined totals. Opposing producer errors -cannot cancel. Each snapshot also requires cache-read tokens no greater than -input tokens, and no positive input total with zero model attempts. - -An explicit zero cache total with complete quality evidence is a real zero. -Missing counters are unknown, never substituted with zero. Compatible older -snapshots lacking these quality counters produce `cannot_determine`; do not -reset their accounting fingerprint to manufacture compatibility. Different -counter schemas or accounting identities still fail the existing summary -compatibility checks, including mixed old/new schemas. This feature does not -normalize contracts or repair summaries. - -Missing output usage, output-quality errors, and tool failures do not by -themselves block this answer. Its numerator and denominator use input only. -The overall usage-coverage or provider-origin question may therefore be limited -while the recorded cached-token share is observed. Duplicate and cumulative -snapshots retain the existing selection and merge semantics; only selected -snapshots contribute to totals, but all supplied snapshots must pass cache -quality checks. - -Only the exact reviewed input/cache-read quality counter names are added to -report evidence. Unknown fields, states, and arbitrary counter names remain -omitted. Producer identifiers and metadata retain existing privacy behavior. -No raw prompts, user identifiers, or remote telemetry are needed. - -## Offline Demo - -From a source checkout with a local Go toolchain: +| Cache-read share of recorded input | Sum of cache-read tokens / sum of input tokens | +| Cache-write share of recorded input | Sum of cache-write tokens / sum of input tokens | +| Uncached input tokens | Input tokens minus cache-read tokens, including cache writes | + +LiteLLM exports get a breakdown per model. The [SQL export](../examples/litellm-spend/export.sql) +includes cache and reasoning fields by default, with `usage_details=false` to +omit them. Paths and normalization are pinned to the tested LiteLLM v1.104.0 +writer; consult the [verified path table](../examples/litellm-spend/README.md). +Normalized `prompt_tokens` already includes cache reads and writes on the tested +Messages path. Those details are subsets and are never added a second time. + +Each field has its own covered-input denominator and request count. Missing, +invalid or subset-violating details are excluded and counted separately. A read +can remain usable when the write field is absent, and conversely. Explicit zero +detail is a recorded zero; a missing detail or zero denominator has a null share. +The report counts recorded zero details with unknown origin separately: the +pinned Messages writer can fill absent detail with zero before export. +If neither period contains cache detail, text prints one line: +`Cache detail not recorded in this export.` + +The uncached-growth ranking composes the existing keyed frequent-items sketches +and their integer bounds. It uses read-covered requests with a group identity. +Candidates are ordered by the lower bound of their increase; uncertain positive +growth remains a range. Missing cache detail and missing identities stay outside +ranking weights and have separate coverage counts. Both periods need at least +one read-covered request before a growth comparison is available. + +A whole-model `Uncached input +X` explanation appears when both periods have +complete read coverage, read share fell, and uncached input grew by at least half +the net token increase (or total tokens stayed flat or fell). This is arithmetic +visibility. Increasing input volume or a changed workload mix can also increase +uncached tokens; the ranking alone does not establish a cache-share loss per key. + +With complete cache-read coverage, a second headline highlights a change of at +least five percentage points, or an uncached-input change of at least 100 tokens +and 20% relative to the before period. From zero uncached input, the 100-token +minimum applies. The comparison uses exact token ratios before formatting. +When cache-read share falls, the next step points to the provider's cache +diagnostics and the gateway route's preservation of cache markers. + +## Collector Summary Windows ```sh sh examples/cache.sh sh examples/cache.sh --json ``` -The script builds this checkout in a temporary directory. The build can download -Go dependencies, as the other source demos do; the resulting investigation runs -offline and makes no provider calls. With an already verified module cache and -toolchain, set `GOWORK=off GOTOOLCHAIN=local GOPROXY=off GOSUMDB=off` for an offline -build too. Existing `GOCACHE` and `GOMODCACHE` settings are preserved. - -The counter-only synthetic fixtures contain ten attempts and 1,000 input tokens -per window, starting at **2026-10-04 00:00 UTC** and **00:01 UTC**. Each window is -one minute long. Recorded cache-read tokens fall from 600 to 200: **60% to 20%, a -40-percentage-point decrease**. They contain no source-provenance declarations, -so provider origin correctly remains unknown. They are arithmetic fixtures, -not evidence of a real provider regression or savings. - -When user and session rankings are absent, the text report ends with: - -```text -Turn on user and session rankings (topk_keys) for more answers. -``` - -If only one ranking is missing, the hint names only that ranking. Configured -rankings with no recorded weight or incomplete snapshots are not presented as -disabled. Detailed coverage and provenance explanations remain in JSON. - -Regenerate into a new directory and compare without overwriting the fixtures: +The released collector exports aggregate read/write totals and per-field +observation-quality counters. Summary inputs must pass existing compatibility, +replay, interval and producer-coverage checks. Each original snapshot must report +input and the relevant cache field on every attempt, with zero missing, invalid, +conflict and subset-violation counters. Missing optional fields do not increment +the collector's missing counter, so reported coverage is also required. + +Read and write are validated independently. If partial cache detail exists, +summaries cannot recover the matching input denominator: that share stays null. +All supplied snapshots are checked for quality; only selected snapshots enter +totals. This prevents opposing producer errors or superseded snapshots from +hiding invalid accounting. +When summary counters cannot establish distinct excluded-request counts, those +counts stay null; the original overlapping quality counters remain in JSON. + +**Current data gap:** these summaries contain neither per-model cache counters +nor cache-weighted key/session rankings. The report names the gap rather than +inferring cache use from total-token rankings. No collector or wire-format +change is needed for the aggregate view. + +JSON adds `cache_view` to the summary cache question and `cache` to spend windows, +models and reports, with explicit null quantities for unknown observations. The +existing summary `cache` result remains available when its original full-coverage +requirements hold, including exact-rational direction and percentage-point change. + +## Limits + +These are recorded token shares, not request hit rates or cause diagnoses. +Usage origin can remain unknown, including gateway-filled values. Recorded spend +continues to be summed as supplied; cache loss gets no price or savings estimate. +Ranking bounds describe tracked contributors within the covered population. +Percentages are rounded for text; JSON retains integer totals and bounds. +Synthetic examples demonstrate arithmetic, not provider cache behavior. + +## Verification ```sh -GOWORK=off GOPROXY=off GOSUMDB=off GOTOOLCHAIN=local go run ./examples/cache -out /tmp/fleetdiff-cache-new -diff -ru examples/cache/data /tmp/fleetdiff-cache-new -go test ./internal/compare -run Cache -go test ./internal/cli ./examples/cache +go test ./internal/spend ./internal/compare ./internal/cli -run Cache +go test ./examples/cache ``` -Next, use complete real input/cache-read observations from the same accounting -contract. If quality is missing or conflicting, fix instrumentation first and -stop short of a cache-regression claim. For a valid decrease, inspect workload -and provider context separately before attributing cause or estimating money. +The pinned-image writer/DB checks and privacy sweeps are documented in +[LiteLLM testing](../examples/litellm-spend/TESTING.md). diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 5c336b0..44101d5 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -8,7 +8,7 @@ unprivileged. All input and output stays on your machine. The initial supported targets are Linux and macOS on AMD64 and ARM64. Native Windows binaries are not provided. Download a binary from the -[release page](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.6.1). +[release page](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.7.0). Go is not needed to run it. The commands below use `curl` for downloading and the [GitHub CLI](https://cli.github.com/) for provenance verification. @@ -20,7 +20,7 @@ subshell stops on any download or verification failure: ```sh ( set -eu -version=v0.6.1 +version=v0.7.0 case "$(uname -s)" in Linux) os=linux ;; Darwin) os=darwin ;; @@ -54,7 +54,7 @@ From a reviewed repository checkout, the same download and verification steps are available as a script (no Go compiler needed): ```sh -sh scripts/install.sh v0.6.1 ./fleetdiff-install +sh scripts/install.sh v0.7.0 ./fleetdiff-install ./fleetdiff-install/fleetdiff --version ``` @@ -65,7 +65,7 @@ curl --fail --location --proto '=https' --tlsv1.2 \ https://raw.githubusercontent.com/llm-measurement/fleetdiff/e75518e1b62ead77a6870cfeabd3ddae944a1e0a/scripts/install.sh \ --output fleetdiff-install.sh less fleetdiff-install.sh -sh fleetdiff-install.sh v0.6.1 ./fleetdiff-install +sh fleetdiff-install.sh v0.7.0 ./fleetdiff-install ./fleetdiff-install/fleetdiff --version ``` @@ -115,7 +115,7 @@ Install a published version with a currently patched Go 1.26 or later toolchain: go install github.com/llm-measurement/fleetdiff/cmd/fleetdiff@latest ``` -Use `@v0.6.1` instead of `@latest` to pin that release. The binary is installed +Use `@v0.7.0` instead of `@latest` to pin that release. The binary is installed under `GOBIN`, or `$(go env GOPATH)/bin` when unset; put that directory on `PATH`. This path uses Go module verification rather than the archive attestation above, and does not include unreleased checkout features. @@ -136,7 +136,7 @@ Check `--version` on the binary you actually run: | Build | Version | Revision | |---|---|---| | Unstamped checkout build above | Go's embedded module/VCS version, or `dev` when unavailable | `unknown` | -| Tagged `go install ...@v0.6.1` | `v0.6.1`, from Go's module metadata | `unknown` | +| Tagged `go install ...@v0.7.0` | `v0.7.0`, from Go's module metadata | `unknown` | | Verified release archive | Stamped release version | Stamped source revision | A checkout may report a pseudo-version and `+dirty`, depending on the Go toolchain diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 87dea87..0f17954 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -76,7 +76,7 @@ verify that every advertised version is downloadable. Do not replace immutable t or artifacts to correct a failed publication; fix forward with a new version if any release content must change. -## After Publishing v0.6.0 +## After Publishing v0.7.0 From an anonymous clone in a fresh directory outside any synced workspace: @@ -85,7 +85,7 @@ From an anonymous clone in a fresh directory outside any synced workspace: [OPERATIONS](OPERATIONS.md#install-and-verify). Confirm the signed tag and recorded build revision match the approved commit. 2. Run the README's verified installer into a new directory. Confirm the native - executable reports `v0.6.0`, then run the packaged-binary tests above. + executable reports `v0.7.0`, then run the packaged-binary tests above. 3. Set `FLEETDIFF_BIN` to that freshly installed executable and run the exact [pinned database test](../examples/litellm-spend/TESTING.md#pinned-database-integration). It exercises the real pinned writer, a SELECT-only SQL export and the released @@ -93,6 +93,13 @@ From an anonymous clone in a fresh directory outside any synced workspace: 4. Run the README's synthetic example with that executable and compare its selected output. Keep command logs, asset hashes, signature/attestation results, platform/version and test output in private `docs/local/` evidence. +5. Run the cache-drop fixture and inspect the shipped synthetic capture with + Go absent from PATH. Check the cache headline, coverage, keyed rankings and + unchanged capture bytes. Update the site's LiteLLM and trace-check guides to + the verified v0.7.0 binary, with the trace guide consuming the shipped capture. +6. Update collector recipes to the separately verified collector v0.3.2 images, + then publish the site's cache-check guide on fleetdiff v0.7.0. Keep those docs + held until their referenced artifacts are published and outsider-tested. The release workflow tests every native platform before publication. The local outsider run proves the post-publication download and database path on its own diff --git a/docs/media/sessions-transcript.txt b/docs/media/sessions-transcript.txt index fbf865a..14598e3 100644 --- a/docs/media/sessions-transcript.txt +++ b/docs/media/sessions-transcript.txt @@ -37,6 +37,7 @@ Which tracked users contribute tokens or model attempts? user-5 / top_users 100 -> 100 3.03% Did caching get worse? -Cached-token share unavailable: Required request, input-token, or cache-read-token counters are unavailable. Supply these counters; absence is not zero. +Cache detail not recorded in these summaries. + These summaries retain aggregate cache totals, not cache totals by model or key/session. Use request-level records for those breakdowns. Notes: Shares use ranked activity; missing keys are excluded. Aliases are local to each measurement. Flags need a share lower bound above 25% and complete coverage; check the session's traces. Use --format json for full details. diff --git a/docs/releases/v0.7.0.md b/docs/releases/v0.7.0.md new file mode 100644 index 0000000..bc84dc9 --- /dev/null +++ b/docs/releases/v0.7.0.md @@ -0,0 +1,62 @@ +# v0.7.0 Release Notes + +Compare recorded cache reuse before and after a change, and find the keys, +teams or sessions whose uncached input grew. `investigate` now includes cache +reads, cache writes and their request coverage for LiteLLM exports and collector +summary windows. Significant changes appear directly below the headline. + +## Upgrade + +The LiteLLM SQL export now includes version-pinned cache-read, cache-write and +reasoning columns by default. Fetch the v0.7.0 projection to get those fields; +set `-v usage_details=false` to retain the previous smaller export. Missing +writer paths stay NULL. The tested LiteLLM v1.104.0 Messages writer can fill +absent detail with zero, which remains recorded zero with unknown origin. +The [export guide](../../examples/litellm-spend/README.md) lists each call type's +verified behavior. + +This release adds a report section and changes the export default, hence the +minor-version increment. Existing exports remain readable. JSON adds explicit +cache quantities, coverage and uncached-growth rankings; unknown quantities are +null. Text uses human-readable coverage and omits zero exclusion counts. Use +JSON for integrations rather than parsing text. + +Collector summaries already contain aggregate cache totals and quality counts. +Their per-model and contributor cache breakdowns remain unavailable; the report +names that gap. No collector upgrade, wire-format change, hash-key rotation or +window reset is required. Token normalization, reported spend and identity +semantics remain unchanged, using llm-sketchkit v0.2.3 with x/text v0.41.0. + +## Read The Result + +The [cache example](../CACHE.md) keeps total input flat while cache-read share +falls from 92% to 61% and uncached input increases by 620 tokens. Both periods +need complete cache-read coverage for the second headline. It appears at a +five-percentage-point share change, or an uncached-input change of at least +100 tokens and 20% relative to the before period. From zero, the 100-token +minimum applies. + +Rankings retain keyed identities and deterministic bounds. Missing or invalid +cache details are excluded and counted separately. The report measures usage, +not the cause or price of cache loss; a falling share directs the next step to +provider cache diagnostics and the gateway route's preservation of cache markers. + +## Inspect Without Go + +`examples/inspect/synthetic-traces.json` is now checked in and reproduced byte +for byte in CI. Run it with the verified executable: + +```sh +curl --fail --location --proto '=https' --tlsv1.2 \ + https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.7.0/examples/inspect/synthetic-traces.json \ + --output synthetic-traces.json +./fleetdiff-install/fleetdiff inspect synthetic-traces.json +``` + +## Install And Verify + +Use the [verified installer](../OPERATIONS.md#install-and-verify) with `v0.7.0`. +Linux and macOS archives cover AMD64 and ARM64, with dependency licenses, +checksums, an SBOM and signed build provenance. The release workflow exercises +the packaged commands on all four platforms and the pinned LiteLLM writer, +read-only database export and importer on Linux AMD64. diff --git a/examples/inspect/README.md b/examples/inspect/README.md index dc8066f..a791f85 100644 --- a/examples/inspect/README.md +++ b/examples/inspect/README.md @@ -5,7 +5,20 @@ accounting, token coverage and origin, and the fields available for user, prompt and session questions. It reads locally; it does not start a receiver or contact your provider. -## Try It With Go Only +## Try It With A Binary + +With `fleetdiff` on your `PATH`, run from the repository root: + +```sh +fleetdiff inspect examples/inspect/synthetic-traces.json +``` + +No Go installation is needed. This ready-made OTLP capture is the deterministic +stock output of the example generator, with only synthetic identifiers and no +prompt or response content. `synthetic-spans.json` is the separate generator +source, not an OTLP capture for `inspect`. + +## Regenerate With Go From the repository root: @@ -19,6 +32,12 @@ synthetic spans, prints the inspection, and removes its temporary files. No collector, Python environment, credentials, or model calls are needed. It also works when invoked by absolute path from another directory. +To reproduce the shipped capture (CI checks exact bytes): + +```sh +go run ./examples/inspect --variant stock > examples/inspect/synthetic-traces.json +``` + Both variants contain four model attempts, a tool span, and an agent wrapper. The model spans carry 140 input and 30 output tokens. One attempt has no token fields; another has a normalized `0/0`. In the first variant that zero's origin diff --git a/examples/inspect/main_test.go b/examples/inspect/main_test.go index 6913a1d..cdd38c7 100644 --- a/examples/inspect/main_test.go +++ b/examples/inspect/main_test.go @@ -52,16 +52,33 @@ func TestSyntheticCapture(t *testing.T) { } } +func TestShippedCaptureMatchesGenerator(t *testing.T) { + want, err := capture("stock") + if err != nil { + t.Fatal(err) + } + got, err := os.ReadFile("synthetic-traces.json") + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, append(want, '\n')) { + t.Fatal("synthetic-traces.json differs from generator output; run go run ./examples/inspect --variant stock > examples/inspect/synthetic-traces.json from the repository root") + } +} + func TestGeneratedCaptureThroughCLI(t *testing.T) { - for _, variant := range []string{"stock", "provenance"} { + for _, variant := range []string{"stock", "provenance", "shipped"} { t.Run(variant, func(t *testing.T) { - data, err := capture(variant) - if err != nil { - t.Fatal(err) - } - path := filepath.Join(t.TempDir(), "private-capture.json") - if err := os.WriteFile(path, data, 0o600); err != nil { - t.Fatal(err) + path := "synthetic-traces.json" + if variant != "shipped" { + data, err := capture(variant) + if err != nil { + t.Fatal(err) + } + path = filepath.Join(t.TempDir(), "private-capture.json") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } } var out, errout bytes.Buffer if code := cli.Run([]string{"inspect", "--format", "json", path}, &out, &errout); code != 0 { diff --git a/examples/inspect/synthetic-traces.json b/examples/inspect/synthetic-traces.json new file mode 100644 index 0000000..f5e94af --- /dev/null +++ b/examples/inspect/synthetic-traces.json @@ -0,0 +1 @@ +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"synthetic-inspect-app"}}]},"scopeSpans":[{"scope":{"name":"fleetdiff.capture-example"},"spans":[{"attributes":[{"key":"app.prompt.template","value":{"stringValue":"SYNTHETIC_TEMPLATE_ONE"}},{"key":"enduser.id","value":{"stringValue":"SYNTHETIC_USER_ALICE"}},{"key":"gen_ai.conversation.id","value":{"stringValue":"SYNTHETIC_SESSION_ONE"}},{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.request.model","value":{"stringValue":"synthetic-model"}},{"key":"gen_ai.usage.cache_read.input_tokens","value":{"intValue":"25"}},{"key":"gen_ai.usage.input_tokens","value":{"intValue":"100"}},{"key":"gen_ai.usage.output_tokens","value":{"intValue":"20"}},{"key":"gen_ai.usage.reasoning.output_tokens","value":{"intValue":"5"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"chat synthetic-model","spanId":"0000000000000001","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"},{"attributes":[{"key":"app.prompt.template","value":{"stringValue":"SYNTHETIC_TEMPLATE_TWO"}},{"key":"enduser.id","value":{"stringValue":"SYNTHETIC_USER_BOB"}},{"key":"gen_ai.conversation.id","value":{"stringValue":"SYNTHETIC_SESSION_TWO"}},{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.request.model","value":{"stringValue":"synthetic-model"}},{"key":"gen_ai.usage.input_tokens","value":{"intValue":"40"}},{"key":"gen_ai.usage.output_tokens","value":{"intValue":"10"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"chat synthetic-model","spanId":"0000000000000002","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"},{"attributes":[{"key":"enduser.id","value":{"stringValue":"SYNTHETIC_USER_CAROL"}},{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.request.model","value":{"stringValue":"synthetic-model"}},{"key":"gen_ai.usage.input_tokens","value":{"intValue":"0"}},{"key":"gen_ai.usage.output_tokens","value":{"intValue":"0"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"chat synthetic-model","spanId":"0000000000000003","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"},{"attributes":[{"key":"enduser.id","value":{"stringValue":"SYNTHETIC_USER_DAN"}},{"key":"gen_ai.operation.name","value":{"stringValue":"chat"}},{"key":"gen_ai.request.model","value":{"stringValue":"synthetic-model"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"chat synthetic-model","spanId":"0000000000000004","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"},{"attributes":[{"key":"gen_ai.operation.name","value":{"stringValue":"execute_tool"}},{"key":"gen_ai.tool.name","value":{"stringValue":"SYNTHETIC_TOOL"}},{"key":"gen_ai.usage.input_tokens","value":{"intValue":"9000"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"execute_tool","spanId":"0000000000000005","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"},{"attributes":[{"key":"gen_ai.agent.name","value":{"stringValue":"SYNTHETIC_AGENT"}},{"key":"gen_ai.operation.name","value":{"stringValue":"invoke_agent"}},{"key":"gen_ai.usage.output_tokens","value":{"intValue":"9000"}}],"endTimeUnixNano":"1700000001000000000","kind":3,"name":"invoke_agent","spanId":"0000000000000006","startTimeUnixNano":"1700000000000000000","traceId":"00000000000000000000000000000001"}]}]}]} diff --git a/examples/litellm-spend/README.md b/examples/litellm-spend/README.md index c4f5b90..3ac6e34 100644 --- a/examples/litellm-spend/README.md +++ b/examples/litellm-spend/README.md @@ -79,9 +79,11 @@ Use `--group-by key|team|user|end-user|session`; model breakdowns are automatic. set your own strong secret in an environment variable and pass `--hash-secret-env NAME`. Add `--show-hashes` to display them. -### Optional Usage Details +### Usage Details (Default On) -Add `-v usage_details=true` to the `psql` command for these projection aliases: +The SQL includes these three projection aliases by default. Add +`-v usage_details=false` to the `psql` command to omit all three columns; +`-v usage_details=true` explicitly selects the default behavior. | Export alias | Path below `metadata.additional_usage_values` | | --- | --- | @@ -89,12 +91,52 @@ Add `-v usage_details=true` to the `psql` command for these projection aliases: | `cache_write_input_tokens` | `cache_creation_input_tokens` | | `reasoning_output_tokens` | `completion_tokens_details.reasoning_tokens` | -The pinned image's actual writer and SQL export are tested with values of 25, -10, and 5 respectively, and with absent details. See the +Missing metadata paths stay SQL `NULL` (empty CSV fields, JSONL `null`), never +invented zeros. Present zeros remain zeros. This describes the stored writer +metadata, not necessarily the provider's original usage: the pinned writer can +omit or synthesize details as shown below. These are **version-pinned paths for +LiteLLM v1.104.0**, not a cross-version or cross-provider guarantee. Recheck the +writer and export before applying this recipe to a different version. See the [pinned writer](https://github.com/BerriAI/litellm/blob/79645770fedc7ec2627e6468d31062f20f82aecc/litellm/proxy/spend_tracking/spend_tracking_utils.py#L732), [schema](https://github.com/BerriAI/litellm/blob/79645770fedc7ec2627e6468d31062f20f82aecc/schema.prisma#L646), and [integration checks](TESTING.md). +The following values are observed through the real pinned writer, PostgreSQL, +and default SQL export using synthetic responses. `Positive`, `Zero`, and +`Absent` describe the supplied usage details, not whether cache use occurred. + +| Path | Supplied details | Spend prompt | Cache read | Cache write | Reasoning | +| --- | --- | ---: | ---: | ---: | ---: | +| Chat Completions | Positive | 200 | 80 | 20 | 5 | +| Chat Completions | Zero | 200 | NULL | NULL | 0 | +| Chat Completions | Absent | 200 | NULL | NULL | NULL | +| Text completion | Positive | 200 | 80 | 20 | 5 | +| Text completion | Zero | 200 | NULL | NULL | 0 | +| Text completion | Absent | 200 | NULL | NULL | NULL | +| Anthropic Messages | Positive | 200 | 80 | 20 | 0 | +| Anthropic Messages | Zero | 100 | 0 | 0 | 0 | +| Anthropic Messages | Absent | 100 | 0 | 0 | 0 | +| Responses | Positive | 200 | 80 | NULL | NULL | +| Responses | Zero | 100 | NULL | NULL | NULL | +| Responses | Absent | 100 | NULL | NULL | NULL | + +Each row is checked for both stored spellings: `completion` / `acompletion`, +`text_completion` / `atext_completion`, `anthropic_messages` / +`aanthropic_messages`, and `responses` / `aresponses`. Chat/text probes supply +writer-ready usage dictionaries with nested `prompt_tokens_details` and +`completion_tokens_details`; this does not establish provider support for +those details on text completion. Messages and Responses use their native +logging conversions first. Messages uses the canonical adapter logging name +for both stored spellings; these are not live sync/async endpoint tests. + +The nested Chat/text cache aliases are copied only when positive. Messages +normalizes omitted cache fields to zero and text-only output to zero reasoning; +the export cannot recover the original absence. The Responses probe supplies no +cache write count, and even supplied `output_tokens_details.reasoning_tokens=5` does +not reach the pinned reasoning export path. Entirely missing usage leaves all +three export details `NULL` for every spelling, even though base token columns +default to zero. No missing-path `NULL` should be interpreted as zero cache use. + ### Claude Code And Codex Requests The importer reads `anthropic_messages` / `aanthropic_messages` and `responses` / @@ -145,7 +187,7 @@ the call types and asks for a supported request-level export. retain valid partial tokens. Unclear usage withholds the whole-period split; unchanged recorded totals do not prove unchanged real usage. - Prompt and completion tokens are added once; `total_tokens` is a cross-check. - Optional details are subsets, not extra totals or physical spend-log columns. + Usage details are subsets, not extra totals or physical spend-log columns. Missing details stay null, invalid casts fail the SQL export, and arbitrary metadata is not accepted as proof of provenance. These paths are version-pinned. - Recorded spend is source accounting, not an invoice. Default-zero cost is diff --git a/examples/litellm-spend/TESTING.md b/examples/litellm-spend/TESTING.md index 677cf9e..51485a0 100644 --- a/examples/litellm-spend/TESTING.md +++ b/examples/litellm-spend/TESTING.md @@ -28,7 +28,9 @@ PostgreSQL multi-platform index was resolved from the official `17-bookworm` image on October 10, 2026. Runtime versions, image digests, and the LiteLLM writer/schema source hashes are checked. There are no implicit pulls in tests. -Native Messages and Responses first pass through the pinned logging usage +Chat Completions and text completion use writer-ready usage dictionaries with +positive, explicit-zero, and absent nested cache/reasoning details. Native +Messages and Responses first pass through the pinned logging usage conversions, including the Responses standard-logging fallback. Messages uses the canonical logging operation used by the pinned adapters; the writer is exercised with both synchronous and asynchronous stored call-type names. @@ -39,7 +41,14 @@ JSON arrays for reader parity are generated locally from CSV, never by a database aggregate. [test-schema.sql](test-schema.sql) mirrors the relevant pinned spend-column types; it is not LiteLLM's entire migration suite. This tests the writer payload and export contract, not HTTP routing, the gateway's -asynchronous database writer, or a paid provider. +asynchronous database writer, provider support for text-completion cache fields, +or a paid provider. Both stored sync/async spellings are checked, not executed +as endpoint calls. The [observed behavior table](README.md#usage-details-default-on) +is checked against the same expected values as the real database tests. + +**Version-pin warning:** these observations apply only to LiteLLM v1.104.0 and +the checked image/source hashes. Re-run the writer and database probes before +trusting these metadata paths on another version or configuration. Both containers use `--network=none`, no published ports, no inherited provider credentials, and read-only roots. PostgreSQL uses a unique Docker named volume, @@ -61,15 +70,29 @@ manually; do not prune unrelated Docker resources. separately and make the selected-period comparison incomplete. Expected export membership comes from actual writer timestamps and the half-open UTC bounds, not a fixed row count or database row order. -- The real writer produces the optional usage detail metadata. SQL aliases +- The real writer produces the usage detail metadata. SQL aliases are compared to each actual payload's pinned metadata paths, including native usage details and absent values. Base token counts must remain unchanged; cache and reasoning details are not added to the writer's normalized totals. - No details are invented in the DB. + No details are invented in the DB. Default export and explicit + `usage_details=true` must match; `usage_details=false` omits all three detail + columns without changing the base projection or selected rows. +- The observed behavior table covers positive, zero, and absent details for + all eight supported stored names. Nested Chat/text zeros do not populate + top-level cache aliases; Messages normalizes absent cache fields to zero; + Responses omits zero cache aliases, cache write, and exported reasoning even + when native reasoning is supplied. Missing writer paths remain SQL NULL, + empty CSV fields, and literal JSONL null. Entirely missing usage is checked + independently from a usage object with missing details. +- Native Messages input 100 plus cache read 80 and creation 20 becomes inclusive + prompt 200 exactly once. Responses input 200 is already inclusive. Both retain + output 10 and total 210 through SQL and importer; subset counters must not + inflate recorded tokens. The normalizer's synthetic zero is not proof of a + provider-reported zero or independently observed cache behavior. - Only synthetic ignored content is added directly to the test table, proving that the projection excludes it. The default SQL invocation must create only CSV; setting `jsonl_file` must create only JSONL, with or without `csv_file`. - These checks cover both default and optional-detail projections. Records match + These checks cover both default-detail and opt-out projections. Records match across both SQL formats and the locally assembled JSON array. Empty intervals also produce valid files. A fixture-only check rejects database sorting, JSON-array aggregation, and multiple CSV `COPY` passes in the recipe. diff --git a/examples/litellm-spend/cache-drop.csv b/examples/litellm-spend/cache-drop.csv new file mode 100644 index 0000000..dc04722 --- /dev/null +++ b/examples/litellm-spend/cache-drop.csv @@ -0,0 +1,5 @@ +request_id,call_type,api_key,team_id,session_id,model,prompt_tokens,completion_tokens,total_tokens,spend,status,startTime,endTime,cache_read_input_tokens,cache_write_input_tokens,reasoning_output_tokens +cache-before-1,completion,SENTINEL_CACHE_KEY_A,SENTINEL_CACHE_TEAM_A,SENTINEL_CACHE_SESSION_A,SENTINEL_CACHE_MODEL,1000,100,1100,0.001,success,2026-10-07T12:00:00Z,2026-10-07T12:00:01Z,920,50,10 +cache-before-2,completion,SENTINEL_CACHE_KEY_B,SENTINEL_CACHE_TEAM_B,SENTINEL_CACHE_SESSION_B,SENTINEL_CACHE_MODEL,1000,100,1100,0.001,success,2026-10-07T12:01:00Z,2026-10-07T12:01:01Z,920,50,10 +cache-after-1,completion,SENTINEL_CACHE_KEY_A,SENTINEL_CACHE_TEAM_A,SENTINEL_CACHE_SESSION_A,SENTINEL_CACHE_MODEL,1000,100,1100,0.001,success,2026-10-08T12:00:00Z,2026-10-08T12:00:01Z,300,100,10 +cache-after-2,completion,SENTINEL_CACHE_KEY_B,SENTINEL_CACHE_TEAM_B,SENTINEL_CACHE_SESSION_B,SENTINEL_CACHE_MODEL,1000,100,1100,0.001,success,2026-10-08T12:01:00Z,2026-10-08T12:01:01Z,920,50,10 diff --git a/examples/litellm-spend/export.sql b/examples/litellm-spend/export.sql index 39ff8d0..822f7c6 100644 --- a/examples/litellm-spend/export.sql +++ b/examples/litellm-spend/export.sql @@ -7,7 +7,7 @@ \pset tuples_only on \if :{?usage_details} \else -\set usage_details false +\set usage_details true \endif BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY; SET LOCAL TIME ZONE 'UTC'; @@ -23,7 +23,8 @@ SELECT 1 / CASE WHEN THEN 1 ELSE 0 END AS valid_bounds \gset --- Opt-in, pinned-writer paths only. Missing details stay NULL, never zero. +-- Default-on, pinned-writer paths only; usage_details=false omits these columns. +-- Missing details stay NULL, never zero. SELECT CASE WHEN :'usage_details'::boolean THEN $details$ , (metadata #>> '{additional_usage_values,cache_read_input_tokens}')::bigint AS cache_read_input_tokens diff --git a/examples/litellm-spend/test_docs.py b/examples/litellm-spend/test_docs.py index 2c67791..9982924 100644 --- a/examples/litellm-spend/test_docs.py +++ b/examples/litellm-spend/test_docs.py @@ -7,6 +7,8 @@ import unittest from urllib.parse import unquote, urlsplit +from test_spend import WRITER_OBSERVATIONS + ROOT = Path(__file__).resolve().parents[2] @@ -18,6 +20,28 @@ def anchors(text): class DocumentationTests(unittest.TestCase): + def test_pinned_usage_table_matches_database_expectations(self): + readme = Path(__file__).with_name("README.md").read_text() + expected = [] + for family, (_, modes) in WRITER_OBSERVATIONS.items(): + for label, values in zip(("Positive", "Zero", "Absent"), modes.values()): + cells = [family, label, *("NULL" if value is None else str(value) + for value in values)] + expected.append("| " + " | ".join(cells) + " |") + actual = [line for line in readme.splitlines() + if any(line.startswith("| " + family + " |") + for family in WRITER_OBSERVATIONS)] + # The separate normalization table has a different column count. + actual = [line for line in actual if line.count("|") == 7] + self.assertEqual(actual, expected) + for name in ("README.md", "TESTING.md"): + content = Path(__file__).with_name(name).read_text() + self.assertIn("usage_details=false", content) + self.assertIn("usage_details=true", content) + self.assertIn("v1.104.0", content) + self.assertIn("\\set usage_details true", + Path(__file__).with_name("export.sql").read_text()) + def test_readme_shape_and_previous_anchors(self): readme = (ROOT / "README.md").read_text() self.assertLessEqual(len(readme.splitlines()), 200) @@ -40,7 +64,8 @@ def test_readme_shape_and_previous_anchors(self): def test_document_links(self): paths = [ROOT / "README.md", ROOT / "docs/WORKFLOWS.md", ROOT / "docs/OPERATIONS.md", ROOT / "docs/RELEASING.md", - ROOT / "docs/releases/v0.6.0.md", Path(__file__).with_name("README.md")] + ROOT / "docs/releases/v0.6.0.md", ROOT / "docs/releases/v0.7.0.md", + ROOT / "docs/CACHE.md", Path(__file__).with_name("README.md")] for path in paths: for link in re.findall(r"\]\(([^)\s]+)\)", path.read_text()): parsed = urlsplit(link) diff --git a/examples/litellm-spend/test_spend.py b/examples/litellm-spend/test_spend.py index f01231a..4fd7017 100644 --- a/examples/litellm-spend/test_spend.py +++ b/examples/litellm-spend/test_spend.py @@ -29,6 +29,29 @@ GROUPS = ("key", "team", "user", "end-user", "session") OPERATIONS = {"completion", "acompletion", "text_completion", "atext_completion", "anthropic_messages", "aanthropic_messages", "responses", "aresponses"} +DETAIL_PATHS = {"cache_read_input_tokens": ("cache_read_input_tokens",), + "cache_write_input_tokens": ("cache_creation_input_tokens",), + "reasoning_output_tokens": ("completion_tokens_details", "reasoning_tokens")} +# Expected (inclusive prompt, cache read, cache write, reasoning) from this pin. +# The documentation table is checked against these same DB-tested observations. +WRITER_OBSERVATIONS = { + "Chat Completions": (("completion", "acompletion"), { + "details_cached": (200, 80, 20, 5), + "details_zero": (200, None, None, 0), + "details_absent": (200, None, None, None)}), + "Text completion": (("text_completion", "atext_completion"), { + "details_cached": (200, 80, 20, 5), + "details_zero": (200, None, None, 0), + "details_absent": (200, None, None, None)}), + "Anthropic Messages": (("anthropic_messages", "aanthropic_messages"), { + "native_cached": (200, 80, 20, 0), + "native_uncached": (100, 0, 0, 0), + "native_absent_details": (100, 0, 0, 0)}), + "Responses": (("responses", "aresponses"), { + "native_cached": (200, 80, None, None), + "native_uncached": (100, None, None, None), + "native_absent_details": (100, None, None, None)}), +} def require(condition, message): @@ -68,6 +91,9 @@ def decode(data, encoding): for key in COLUMNS: if row.get(key) is None: row[key] = "" + for key in DETAIL_PATHS: + if key in row: + row[key] = None if row[key] in (None, "") else int(row[key]) return records @@ -139,6 +165,74 @@ def assert_arithmetic(report, records): require("recorded_volume_split" not in report, "unclear whole-period split must be withheld") +def assert_cache_arithmetic(report, records): + # Use CSV rows, not report counters, as the independent coverage oracle. + selected = [r for r in records if r["call_type"] in OPERATIONS and + r["startTime"][:10] in ("2026-10-07", "2026-10-08")] + models = {name: [r for r in selected if r["model"] == name] + for name in dict.fromkeys(r["model"] for r in selected)} + # Model aliases follow descending token delta, then first appearance. + ordered = sorted(models, key=lambda name: -sum( + (1 if r["startTime"].startswith("2026-10-08") else -1) * + (int(r["prompt_tokens"]) + int(r["completion_tokens"])) for r in models[name])) + require(len(report["models"]) == len(ordered), "cache model count differs") + populations = [(report, selected)] + for index, name in enumerate(ordered): + model = report["models"][index] + require(model["item"] == f"model-{index + 1}", "cache model alias order differs") + populations.append((model, models[name])) + for population, source in populations: + for side, day in (("before", "2026-10-07"), ("after", "2026-10-08")): + rows = [r for r in source if r["startTime"].startswith(day)] + window = population[side] + cache = window["cache"] + require(cache["requests"] == window["logged_model_requests"] == len(rows), + "cache request count differs from CSV population") + require(window["recorded_tokens"] == sum( + int(r["prompt_tokens"]) + int(r["completion_tokens"]) for r in rows), + "cache details were added to recorded tokens") + for kind, column in (("read", "cache_read_input_tokens"), + ("write", "cache_write_input_tokens")): + covered, reported, missing = [], 0, 0 + for row in rows: + value = row.get(column) + if value in (None, ""): + missing += 1 + continue + try: + tokens = int(value) + except ValueError: + continue + if 0 <= tokens <= 2**63 - 1: + reported += 1 + if tokens <= int(row["prompt_tokens"]): + covered.append((int(row["prompt_tokens"]), tokens)) + detail = cache[kind] + expected = { + "coverage_known": bool(rows), "reported_requests": reported, + "covered_requests": len(covered), "missing_detail_requests": missing, + "invalid_detail_or_input_requests": len(rows) - missing - len(covered), + "zero_origin_unknown_requests": sum(tokens == 0 for _, tokens in covered), + "covered_input_tokens": sum(v[0] for v in covered) if covered else None, + "tokens": sum(v[1] for v in covered) if covered else None, + } + require(all(detail[field] == value for field, value in expected.items()), + "cache coverage or totals differ from CSV rows") + denominator, tokens = expected["covered_input_tokens"], expected["tokens"] + share = tokens / denominator if denominator else None + require(detail["share_of_recorded_input"] == share, + "cache share used a different population denominator") + if kind == "read": + uncached = denominator - tokens if covered else None + require(cache["uncached_input_tokens"] == uncached, + "uncached input differs from covered CSV rows") + if population is report: + # Compare stable fields only; additive summary fields are allowed. + require(all(report["cache"][side][field] == cache[field] + for field in ("requests", "read", "write", "uncached_input_tokens")), + "top-level and window cache summaries differ") + + @contextmanager def database(): name = "fleetdiff-spend-" + uuid.uuid4().hex @@ -218,7 +312,7 @@ def seed_database(container, records): ''') -def export_database(container, start="2026-10-07T00:00:00Z", end="2026-10-09T00:00:00Z", details=False): +def export_database(container, start="2026-10-07T00:00:00Z", end="2026-10-09T00:00:00Z", details=None): exports = {} for fmt in ("csv", "jsonl"): directory = "/tmp/export-" + uuid.uuid4().hex @@ -228,8 +322,8 @@ def export_database(container, start="2026-10-07T00:00:00Z", end="2026-10-09T00: if fmt == "jsonl": # The JSONL selector must suppress CSV even when both are specified. arguments.extend(("-v", "jsonl_file=" + directory + "/spend.jsonl")) - if details: - arguments.extend(("-v", "usage_details=true")) + if details is not None: + arguments.extend(("-v", "usage_details=" + str(details).lower())) output = psql(container, *arguments, "-f", "/example/export.sql", reader=True) require(not output.strip(), "SQL export leaked output outside its selected file") files = docker("exec", container, "ls", "-1", directory).splitlines() @@ -421,6 +515,7 @@ def setUpClass(cls): seed_database(cls.db, cls.records) cls.exports = export_database(cls.db) cls.detail_exports = export_database(cls.db, details=True) + cls.plain_exports = export_database(cls.db, details=False) def test_jsonl_selector_does_not_require_csv_destination(self): directory = "/tmp/export-" + uuid.uuid4().hex @@ -450,7 +545,8 @@ def test_real_database_projection_and_utc_boundaries(self): "writer probe omits supported operation coverage") require(len(parsed[0]) == len(expected), "database export row count differs") for actual, original in zip(parsed[0], expected): - require(set(actual) == set(COLUMNS), "SQL selected an unexpected column") + require(set(actual) == set(COLUMNS) | set(DETAIL_PATHS), + "SQL selected an unexpected column") for field in COLUMNS: if field not in ("startTime", "endTime", "spend"): require(actual[field] == (original[field] if original[field] is not None else ""), @@ -471,21 +567,27 @@ def test_real_database_projection_and_utc_boundaries(self): empty = export_database(self.db, "2026-09-01T00:00:00Z", "2026-09-02T00:00:00Z") require(all(decode(empty[fmt], fmt) == [] for fmt in FORMATS), "empty SQL export is invalid") - def test_optional_aliases_come_from_actual_writer_metadata(self): - paths = {"cache_read_input_tokens": ("cache_read_input_tokens",), - "cache_write_input_tokens": ("cache_creation_input_tokens",), - "reasoning_output_tokens": ("completion_tokens_details", "reasoning_tokens")} + def test_default_aliases_and_opt_out_come_from_actual_writer_metadata(self): parsed = [] + plain_parsed = [] for fmt in FORMATS: records = sorted(decode(self.detail_exports[fmt], fmt), key=lambda r: r["request_id"]) + default = sorted(decode(self.exports[fmt], fmt), key=lambda r: r["request_id"]) + require(records == default, "default export differs from explicit usage_details=true") + plain_records = sorted(decode(self.plain_exports[fmt], fmt), + key=lambda r: r["request_id"]) + require(all(set(row) == set(COLUMNS) for row in plain_records), + "usage_details=false did not omit detail columns") + plain_parsed.append(plain_records) for row in records: - require(set(row) == set(COLUMNS) | set(paths), "SQL selected an unexpected detail column") - for alias in paths: - row[alias] = None if row[alias] in (None, "") else int(row[alias]) + require(set(row) == set(COLUMNS) | set(DETAIL_PATHS), + "SQL selected an unexpected detail column") parsed.append(records) require(parsed[0] == parsed[1] == parsed[2], "optional detail formats differ") + require(plain_parsed[0] == plain_parsed[1] == plain_parsed[2], + "opt-out export formats differ") originals = {r["request_id"]: r for r in self.records} - plain = {r["request_id"]: r for r in decode(self.exports["csv"], "csv")} + plain = {r["request_id"]: r for r in decode(self.plain_exports["csv"], "csv")} require(len(parsed[0]) == len(plain), "optional export row count differs") require({r["request_id"] for r in parsed[0]} == set(plain) <= set(originals), "optional export membership differs from writer records") @@ -495,17 +597,43 @@ def test_optional_aliases_come_from_actual_writer_metadata(self): metadata = originals[row["request_id"]]["metadata"] if isinstance(metadata, str): metadata = json.loads(metadata) - for alias, path in paths.items(): + for alias, path in DETAIL_PATHS.items(): value = (metadata or {}).get("additional_usage_values") for key in path: value = value.get(key) if isinstance(value, dict) else None require(row[alias] == (None if value is None else int(value)), "optional usage values differ from actual writer metadata") + def test_observed_cache_contract_for_every_supported_spelling(self): + require({op for operations, _ in WRITER_OBSERVATIONS.values() for op in operations} + == OPERATIONS, "cache observations omit a supported operation") + for fmt in FORMATS: + records = {r["request_id"]: r for r in decode(self.exports[fmt], fmt)} + for operations, modes in WRITER_OBSERVATIONS.values(): + for operation in operations: + for day in (7, 8): + for mode, expected in modes.items(): + row = records[f"synthetic-{day}-{operation}-{mode}"] + require(tuple(row[field] for field in + ("prompt_tokens", *DETAIL_PATHS)) == expected, + "pinned writer detail observation changed") + require(row["completion_tokens"] == 10 and + row["total_tokens"] == expected[0] + 10, + "inclusive prompt normalization was changed or counted twice") + missing = records[f"synthetic-{day}-{operation}-missing"] + require(all(missing[field] is None for field in DETAIL_PATHS), + "absent writer paths became zero") + if fmt == "jsonl": + raw = [json.loads(line) for line in self.exports[fmt].splitlines()] + require(all(row[field] is None for row in raw + if row["request_id"].endswith("-missing") for field in DETAIL_PATHS), + "JSONL must encode missing detail paths as literal null") + @unittest.skipUnless(CLI, "set FLEETDIFF_BIN for the DB-export-to-importer check") def test_real_database_exports_through_importer(self): with tempfile.TemporaryDirectory(prefix="SENTINEL_PATH_db-", dir="/tmp") as temp: - for exports in (self.exports, self.detail_exports): + for exports in (self.exports, self.plain_exports): + csv_records = list(csv.DictReader(io.StringIO(exports["csv"].decode("utf-8")))) paths = [] for fmt in FORMATS: path = Path(temp) / ("export." + fmt) @@ -518,8 +646,34 @@ def test_real_database_exports_through_importer(self): require(reports[0] == reports[1] == reports[2], "database export importer reports differ") assert_arithmetic(reports[0], decode(exports["json"], "json")) + for report in reports: + assert_cache_arithmetic(report, csv_records) for path in paths: - investigate(path, encoding="text", records=self.records) + result = investigate(path, encoding="text", records=self.records) + require((b"Whose uncached input grew?" in result.stdout) == + (exports is self.exports), + "uncached growth heading differs from exported detail availability") + + @unittest.skipUnless(CLI, "set FLEETDIFF_BIN for the DB-export-to-importer check") + def test_exported_csv_cache_invalid_details_do_not_become_missing_or_zero(self): + records = list(csv.DictReader(io.StringIO(self.exports["csv"].decode("utf-8")))) + # Mutate private export copies only; these are importer rejection probes, + # not claims that the pinned writer or SQL produced malformed counts. + for day in (7, 8): + for operation in ("completion", "text_completion", "anthropic_messages", "responses"): + mode = "native_cached" if operation in ("anthropic_messages", "responses") else "details_cached" + row = next(r for r in records if + r["request_id"] == f"synthetic-{day}-{operation}-{mode}") + row["cache_read_input_tokens"] = "not-a-token-count" + row["cache_write_input_tokens"] = str(int(row["prompt_tokens"]) + 1) + with tempfile.TemporaryDirectory(prefix="SENTINEL_PATH_cache-", dir="/tmp") as temp: + path = Path(temp) / "invalid-cache.csv" + with path.open("w", newline="") as output: + writer = csv.DictWriter(output, fieldnames=list(records[0])) + writer.writeheader() + writer.writerows(records) + report = investigate(path, records=records) + assert_cache_arithmetic(report, records) if __name__ == "__main__": diff --git a/examples/litellm-spend/writer_probe.py b/examples/litellm-spend/writer_probe.py index fb3648f..44e8f07 100644 --- a/examples/litellm-spend/writer_probe.py +++ b/examples/litellm-spend/writer_probe.py @@ -18,11 +18,12 @@ def require(condition, message): raise RuntimeError(message) -def native_response(call_type, cached, trace_id): +def native_response(call_type, mode, trace_id): """Exercise the pinned logging normalizers, then pass their output to DB writer.""" from litellm.litellm_core_utils.litellm_logging import Logging from litellm.types.llms.openai import ResponsesAPIResponse + cached = mode == "native_cached" # The pinned Messages adapters set the logging operation to the canonical # name. Exercise both stored writer spellings with that normalized usage. context = Logging(model="claude-sonnet-4-5" if "anthropic" in call_type else "gpt-4.1", @@ -39,14 +40,19 @@ def native_response(call_type, cached, trace_id): "usage": {"input_tokens": 100, "output_tokens": 10, "cache_read_input_tokens": 80 if cached else 0, "cache_creation_input_tokens": 20 if cached else 0}} + if mode == "native_absent_details": + del raw["usage"]["cache_read_input_tokens"] + del raw["usage"]["cache_creation_input_tokens"] else: + usage = {"input_tokens": 200 if cached else 100, "output_tokens": 10, + "total_tokens": 210 if cached else 110} + if mode != "native_absent_details": + usage.update(input_tokens_details={"cached_tokens": 80 if cached else 0}, + output_tokens_details={"reasoning_tokens": 5 if cached else 0}) raw = ResponsesAPIResponse(id="synthetic-native-response", object="response", created_at=0, status="completed", model="gpt-4.1", output=[], parallel_tool_calls=False, tool_choice="auto", tools=[], - usage={"input_tokens": 200 if cached else 100, "output_tokens": 10, - "total_tokens": 210 if cached else 110, - "input_tokens_details": {"cached_tokens": 80 if cached else 0}, - "output_tokens_details": {"reasoning_tokens": 5}}) + usage=usage) _, _, result = context._success_handler_helper_fn( result=raw, end_time=datetime(2026, 10, 7, 0, 0, 1, tzinfo=timezone.utc)) return result, context.model_call_details["standard_logging_object"] @@ -80,11 +86,18 @@ def payloads(): endTime=f"2026-10-{day:02}T19:00:01Z") cases.append((row["request_id"], row, "complete")) for operation in ("anthropic_messages", "aanthropic_messages", "responses", "aresponses"): - for mode in ("native_cached", "native_uncached", "missing", "failure_partial"): + for mode in ("native_cached", "native_uncached", "native_absent_details", + "missing", "failure_partial"): row = dict(base[0], request_id=f"synthetic-{day}-{operation}-{mode}", call_type=operation, startTime=f"2026-10-{day:02}T20:00:00Z", endTime=f"2026-10-{day:02}T20:00:01Z") cases.append((row["request_id"], row, mode)) + for operation in ("completion", "acompletion", "text_completion", "atext_completion"): + for mode in ("details_cached", "details_zero", "details_absent", "missing"): + row = dict(base[0], request_id=f"synthetic-{day}-{operation}-{mode}", + call_type=operation, startTime=f"2026-10-{day:02}T21:00:00Z", + endTime=f"2026-10-{day:02}T21:00:01Z") + cases.append((row["request_id"], row, mode)) for label, stamp in ( ("outside_before", "2026-10-06T23:59:59.999Z"), ("at_start", "2026-10-07T00:00:00Z"), @@ -118,8 +131,17 @@ def payloads(): ("prompt_tokens", "completion_tokens", "total_tokens")} response = {"usage": usage} expected = tuple(usage.values()) - if mode in ("native_cached", "native_uncached"): - response, standard = native_response(row["call_type"], mode == "native_cached", row["session_id"]) + if mode.startswith("details_"): + usage.update(prompt_tokens=200, completion_tokens=10, total_tokens=210) + expected = (200, 10, 210) + if mode != "details_absent": + usage["prompt_tokens_details"] = { + "cached_tokens": 80 if mode == "details_cached" else 0, + "cache_write_tokens": 20 if mode == "details_cached" else 0} + usage["completion_tokens_details"] = { + "reasoning_tokens": 5 if mode == "details_cached" else 0} + if mode.startswith("native_"): + response, standard = native_response(row["call_type"], mode, row["session_id"]) kwargs["standard_logging_object"] = standard response.id = label expected = (200, 10, 210) if mode == "native_cached" else (100, 10, 110) diff --git a/internal/cli/cache.go b/internal/cli/cache.go new file mode 100644 index 0000000..3bd851b --- /dev/null +++ b/internal/cli/cache.go @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package cli + +import ( + "fmt" + "io" + "math/big" + "slices" + "strings" + + "github.com/llm-measurement/fleetdiff/internal/compare" +) + +func cacheCount(v *uint64) string { + if v == nil { + return "unknown" + } + return spendNumber(*v) +} + +func cacheShare(v *float64) string { + if v == nil { + return "unknown" + } + return spendPercent(*v * 100) +} + +func cacheReadChange(a, b compare.CacheWindow) *big.Rat { + if !a.CompleteRead() || !b.CompleteRead() || a.Read.Input == nil || b.Read.Input == nil || a.Read.Tokens == nil || b.Read.Tokens == nil || *a.Read.Input == 0 || *b.Read.Input == 0 { + return nil + } + share := func(d compare.CacheDetail) *big.Rat { + return new(big.Rat).SetFrac(new(big.Int).SetUint64(*d.Tokens), new(big.Int).SetUint64(*d.Input)) + } + return new(big.Rat).Sub(share(b.Read), share(a.Read)) +} + +func renderCacheHeadline(out io.Writer, a, b compare.CacheWindow) { + change := cacheReadChange(a, b) + if change == nil { + return + } + delta, sign := *b.Uncached, "+" + if delta >= *a.Uncached { + delta -= *a.Uncached + } else { + delta, sign = *a.Uncached-delta, "-" + } + // A five-point share change or a 20% change of at least 100 tokens is material. + minimumChange := *a.Uncached / 5 + if *a.Uncached%5 != 0 { + minimumChange++ + } + material := delta >= 100 && delta >= minimumChange + if new(big.Rat).Abs(change).Cmp(big.NewRat(1, 20)) < 0 && !material { + return + } + lead := "Cache reuse held" + if change.Sign() < 0 { + lead = "But cache reuse fell" + } else if change.Sign() > 0 { + lead = "Cache reuse rose" + } + if delta == 0 { + sign = "" + } + fmt.Fprintf(out, "%s: cache-read share %s -> %s (uncached input %s%s).\n", lead, cacheShare(a.Read.Share), cacheShare(b.Read.Share), sign, spendNumber(delta)) +} + +func cacheCoverage(a, b compare.CacheWindow, ad, bd compare.CacheDetail) string { + if ad.Covered != nil && bd.Covered != nil && *ad.Covered == a.Requests && *bd.Covered == b.Requests { + return "all requests report cache detail" + } + return fmt.Sprintf("before: %s of %s requests report cache detail; after: %s of %s", cacheCount(ad.Covered), spendNumber(a.Requests), cacheCount(bd.Covered), spendNumber(b.Requests)) +} + +func renderCacheExclusions(out io.Writer, a, b compare.CacheDetail) { + for _, f := range []struct { + label string + a, b *uint64 + }{{"missing detail", a.Missing, b.Missing}, {"invalid detail or input", a.Invalid, b.Invalid}} { + if (f.a != nil && *f.a != 0) || (f.b != nil && *f.b != 0) { + fmt.Fprintf(out, " Excluded: %s %s -> %s requests.\n", f.label, cacheCount(f.a), cacheCount(f.b)) + } + } +} + +const cacheDiagnosticsNext = "Next: Use your provider's cache diagnostics (Anthropic, OpenAI) to find why, and check that your gateway route preserves cache markers." + +func renderCachePair(out io.Writer, a, b compare.CacheWindow, beforeTokens, afterTokens uint64) { + for _, f := range []struct { + label string + a, b compare.CacheDetail + }{{"Cache-read", a.Read, b.Read}, {"Cache-write", a.Write, b.Write}} { + if !f.a.HasDetail() && !f.b.HasDetail() { + fmt.Fprintf(out, " %s detail not recorded.\n", f.label) + continue + } + fmt.Fprintf(out, " %s share of recorded input: %s -> %s.\n", f.label, cacheShare(f.a.Share), cacheShare(f.b.Share)) + verb := "reads" + if f.label == "Cache-write" { + verb = "writes" + } + fmt.Fprintf(out, " Cache %s: %s of %s input tokens -> %s of %s (%s).\n", verb, cacheCount(f.a.Tokens), cacheCount(f.a.Input), cacheCount(f.b.Tokens), cacheCount(f.b.Input), cacheCoverage(a, b, f.a, f.b)) + renderCacheExclusions(out, f.a, f.b) + if f.a.Issue != "" || f.b.Issue != "" { + fmt.Fprintln(out, " Complete input and cache-detail observations are needed for this summary share.") + for _, side := range []struct { + name string + detail compare.CacheDetail + }{{"Before", f.a}, {"After", f.b}} { + var issues []string + for name, count := range side.detail.IssueCounts { + if count != 0 && !strings.HasSuffix(name, ".reported") { + issues = append(issues, strings.TrimPrefix(name, "token_observations.")+": "+spendNumber(count)) + } + } + slices.Sort(issues) + if len(issues) != 0 { + fmt.Fprintf(out, " %s quality evidence (categories may overlap): %s.\n", side.name, strings.Join(issues, "; ")) + } + } + } + if (f.a.ZeroOriginUnknown != nil && *f.a.ZeroOriginUnknown != 0) || (f.b.ZeroOriginUnknown != nil && *f.b.ZeroOriginUnknown != 0) { + fmt.Fprintf(out, " Recorded zero detail, origin unknown: %s -> %s requests.\n", cacheCount(f.a.ZeroOriginUnknown), cacheCount(f.b.ZeroOriginUnknown)) + } + } + if a.CompleteRead() && b.CompleteRead() && *b.Uncached > *a.Uncached && *b.Read.Share < *a.Read.Share { + growth := *b.Uncached - *a.Uncached + if afterTokens <= beforeTokens || growth >= (afterTokens-beforeTokens)/2+(afterTokens-beforeTokens)%2 { + fmt.Fprintf(out, " Uncached input +%s: cache-read share fell from %s to %s.\n", spendNumber(growth), cacheShare(a.Read.Share), cacheShare(b.Read.Share)) + } + } +} + +func renderCacheRankings(out io.Writer, v compare.CacheView) { + fmt.Fprintln(out, "\nWhose uncached input grew?") + if v.Rankings == nil { + fmt.Fprintln(out, " Cache-read detail is needed in both periods to compare uncached growth.") + return + } + fmt.Fprintln(out, " Ranked by recorded uncached-input growth on read-covered requests; volume and workload mix can also change it.") + for _, m := range v.Rankings.Movers { + label := m.Item + if m.Hash != "" { + label += " (" + m.Hash + ")" + } + fmt.Fprintf(out, " %s: %s -> %s uncached input tokens; change %s.\n", label, spendInterval(m.Before), spendInterval(m.After), signedInterval(m.Delta)) + } + if len(v.Rankings.Movers) == 0 { + fmt.Fprintln(out, " No tracked candidates with possible uncached-input growth.") + } + fmt.Fprintf(out, " Cache-read coverage: %s.\n", cacheCoverage(v.Before, v.After, v.Before.Read, v.After.Read)) + renderCacheExclusions(out, v.Before.Read, v.After.Read) + if v.Before.MissingIdentity+v.After.MissingIdentity != 0 { + fmt.Fprintf(out, " Read-covered requests without group identity: %s -> %s.\n", spendNumber(v.Before.MissingIdentity), spendNumber(v.After.MissingIdentity)) + } +} diff --git a/internal/cli/cache_view_test.go b/internal/cli/cache_view_test.go new file mode 100644 index 0000000..7f3c32b --- /dev/null +++ b/internal/cli/cache_view_test.go @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package cli + +import ( + "bytes" + "os" + "strings" + "testing" + + "github.com/llm-measurement/fleetdiff/internal/compare" + "github.com/llm-measurement/fleetdiff/internal/spend" +) + +func TestCacheDropFixtureTextJSONAndPrivacy(t *testing.T) { + for _, group := range []string{"key", "team", "session"} { + for _, format := range []string{"text", "json"} { + for _, hashes := range []bool{false, true} { + args := []string{"investigate", "--litellm-spend", "../../examples/litellm-spend/cache-drop.csv", "--before-period", "2026-10-07", "--after-period", "2026-10-08", "--group-by", group, "--format", format} + if hashes { + args = append(args, "--show-hashes") + } + var out, diagnostic bytes.Buffer + if code := Run(args, &out, &diagnostic); code != 0 { + t.Fatal(code, diagnostic.String()) + } + if strings.Contains(out.String()+diagnostic.String(), "SENTINEL_") { + t.Fatal("cache fixture leaked") + } + if format == "text" { + lines := strings.Split(out.String(), "\n") + if lines[1] != "But cache reuse fell: cache-read share 92% -> 61% (uncached input +620)." { + t.Fatalf("cache change must be the second headline: %s", out.String()) + } + if !strings.Contains(out.String(), cacheDiagnosticsNext) || strings.Contains(out.String(), "Excluded:") || strings.Contains(out.String(), "Uncached input tokens (read-covered requests)") || strings.Contains(out.String(), "unsupported rows: 0") || strings.Contains(out.String(), "outside periods: 0") { + t.Fatal(out.String()) + } + guide, err := os.ReadFile("../../docs/CACHE.md") + if err != nil { + t.Fatal(err) + } + if group == "key" && !hashes { + excerpt := strings.SplitN(strings.SplitN(string(guide), "```text\n", 2)[1], "```", 2)[0] + for _, line := range strings.Split(strings.TrimSuffix(excerpt, "\n"), "\n") { + if !strings.Contains(out.String(), line+"\n") { + t.Fatalf("guide excerpt differs: %q", line) + } + } + } + for _, want := range []string{"Cache-read share of recorded input: 92% -> 61%.", "Cache-write share of recorded input: 5% -> 7.5%.", "Uncached input +620: cache-read share fell from 92% to 61%.", "Whose uncached input grew?", group + "-1", "80 -> 700 uncached input tokens; change +620.", "No usage problems found."} { + if !strings.Contains(out.String(), want) { + t.Fatalf("missing %q: %s", want, out.String()) + } + } + } else if !strings.Contains(out.String(), `"weight_unit": "attributed-recorded-uncached-input-tokens"`) { + t.Fatal(out.String()) + } + } + } + } +} + +func testCacheWindow(input, read uint64) compare.CacheWindow { + zero, one, uncached := uint64(0), uint64(1), input-read + w := compare.CacheWindow{Requests: 1, Read: compare.CacheDetail{CoverageKnown: true, Reported: 1, Covered: &one, Missing: &zero, Invalid: &zero}, Uncached: &uncached} + w.Read.SetTotals(input, read) + return w +} + +func TestCacheHeadlineThresholds(t *testing.T) { + for _, tt := range []struct { + name string + a, b compare.CacheWindow + want string + }{ + {"five points down", testCacheWindow(100, 92), testCacheWindow(100, 87), "But cache reuse fell: cache-read share 92% -> 87% (uncached input +5).\n"}, + {"five points up", testCacheWindow(100, 87), testCacheWindow(100, 92), "Cache reuse rose: cache-read share 87% -> 92% (uncached input -5).\n"}, + {"below five points", testCacheWindow(100, 92), testCacheWindow(100, 88), ""}, + {"material unchanged share", testCacheWindow(1000, 500), testCacheWindow(1200, 600), "Cache reuse held: cache-read share 50% -> 50% (uncached input +100).\n"}, + {"material decrease", testCacheWindow(1000, 500), testCacheWindow(800, 400), "Cache reuse held: cache-read share 50% -> 50% (uncached input -100).\n"}, + {"below absolute floor", testCacheWindow(100, 50), testCacheWindow(200, 100), ""}, + {"below relative floor", testCacheWindow(2000, 1000), testCacheWindow(2200, 1100), ""}, + {"from zero", testCacheWindow(10000, 10000), testCacheWindow(10000, 9900), "But cache reuse fell: cache-read share 100% -> 99% (uncached input +100).\n"}, + {"small from zero", testCacheWindow(10000, 10000), testCacheWindow(10000, 9999), ""}, + {"missing before", compare.CacheWindow{Requests: 1}, testCacheWindow(100, 50), ""}, + {"missing after", testCacheWindow(100, 50), compare.CacheWindow{Requests: 1}, ""}, + {"zero input", testCacheWindow(0, 0), testCacheWindow(100, 50), ""}, + } { + t.Run(tt.name, func(t *testing.T) { + var out bytes.Buffer + renderCacheHeadline(&out, tt.a, tt.b) + if out.String() != tt.want { + t.Fatalf("got %q, want %q", out.String(), tt.want) + } + }) + } +} + +func TestCachePartialCoverageAndExclusions(t *testing.T) { + a, b := testCacheWindow(100, 92), testCacheWindow(100, 61) + a.Requests, b.Requests = 2, 3 + missing := uint64(1) + a.Read.Missing = &missing + var out bytes.Buffer + renderCacheHeadline(&out, a, b) + if out.Len() != 0 { + t.Fatal("partial population must not imply a whole-period cache change") + } + renderCachePair(&out, a, b, 100, 100) + for _, want := range []string{"before: 1 of 2 requests report cache detail; after: 1 of 3", "Excluded: missing detail 1 -> 0 requests."} { + if !strings.Contains(out.String(), want) { + t.Fatal(out.String()) + } + } + if strings.Contains(out.String(), "Excluded: invalid") || strings.Contains(out.String(), "Uncached input +") { + t.Fatal(out.String()) + } + // Unknown counts remain in JSON; text prints only known nonzero exclusions. + out.Reset() + renderCacheExclusions(&out, compare.CacheDetail{}, compare.CacheDetail{}) + if out.Len() != 0 { + t.Fatal(out.String()) + } +} + +func TestIncompleteSpendKeepsCoverageFirst(t *testing.T) { + r := spend.Report{Incomplete: true} + r.Before.Cache, r.After.Cache = testCacheWindow(100, 92), testCacheWindow(100, 61) + var out bytes.Buffer + renderSpend(&out, r) + if !strings.HasPrefix(out.String(), "Comparison incomplete:") || strings.Contains(out.String(), "But cache reuse fell:") || strings.Contains(out.String(), cacheDiagnosticsNext) { + t.Fatal(out.String()) + } +} + +func TestNoCacheDetailOneLine(t *testing.T) { + var out, diagnostic bytes.Buffer + code := Run([]string{"investigate", "--litellm-spend", "../../examples/litellm-spend/synthetic.csv", "--before-period", "2026-10-07", "--after-period", "2026-10-08"}, &out, &diagnostic) + if code != 0 || strings.Count(out.String(), "Cache detail not recorded in this export.") != 1 || strings.Contains(out.String(), "Whose uncached input grew?") { + t.Fatal(code, out.String(), diagnostic.String()) + } +} + +func TestCacheInvalidAndUnknownCoverageRemainVisible(t *testing.T) { + invalid, zero, raw := uint64(10), uint64(0), uint64(600) + for _, detail := range []compare.CacheDetail{ + {CoverageKnown: true, Invalid: &invalid, Covered: &zero, Missing: &zero}, + {RecordedTokens: &raw, Issue: "unknown coverage"}, + {Issue: "invalid observations", IssueCounts: map[string]uint64{"token_observations.cache_read_input.invalid": 10}}, + } { + var out bytes.Buffer + w := compare.CacheWindow{Requests: 10, Read: detail} + renderCachePair(&out, w, w, 1000, 1000) + if !strings.Contains(out.String(), "Cache-read share of recorded input: unknown -> unknown.") || strings.Contains(out.String(), "Cache-read detail not recorded.") { + t.Fatal(out.String()) + } + } +} + +func TestCacheZeroOriginAndImprovedShareWording(t *testing.T) { + zero, one := uint64(0), uint64(1) + a := compare.CacheWindow{Requests: 1, Read: compare.CacheDetail{CoverageKnown: true, Reported: 1, Covered: &one, Missing: &zero, Invalid: &zero, ZeroOriginUnknown: &one}, Uncached: &zero} + a.Read.SetTotals(1000, 0) + var out bytes.Buffer + renderCachePair(&out, a, a, 1000, 1000) + if !strings.Contains(out.String(), "Recorded zero detail, origin unknown: 1 -> 1 requests.") { + t.Fatal(out.String()) + } + b := a + x, y := uint64(100), uint64(150) + a.Uncached, b.Uncached = &x, &y + a.Read.SetTotals(1000, 900) + b.Read.SetTotals(3000, 2850) + out.Reset() + renderCachePair(&out, a, b, 1000, 3000) + renderCacheRankings(&out, compare.CacheView{Before: a, After: b, Rankings: &compare.Concentration{}}) + if strings.Contains(out.String(), "share fell") || strings.Contains(out.String(), "Who lost cache reuse") || !strings.Contains(out.String(), "Whose uncached input grew?") { + t.Fatal(out.String()) + } +} diff --git a/internal/cli/investigation_text.go b/internal/cli/investigation_text.go index 21705ab..8e6f4a3 100644 --- a/internal/cli/investigation_text.go +++ b/internal/cli/investigation_text.go @@ -17,6 +17,15 @@ import ( func renderInvestigation(out io.Writer, r compare.Investigation, threshold float64) { headline := investigationHeadline(r) fmt.Fprintln(out, headline) + cacheFell := false + for _, q := range r.Questions { + if q.ID == "cache" && q.CacheView != nil { + v := q.CacheView + renderCacheHeadline(out, v.Before, v.After) + change := cacheReadChange(v.Before, v.After) + cacheFell = change != nil && change.Sign() < 0 + } + } volumeShown, attributed, flagged := false, false, false for _, q := range r.Questions { if v := q.Volume; v != nil { @@ -106,7 +115,21 @@ func renderInvestigation(out io.Writer, r compare.Investigation, threshold float continue } fmt.Fprintln(out, "\nDid caching get worse?") - if c := q.Cache; c != nil { + if v := q.CacheView; v != nil { + if !v.Before.HasDetail() && !v.After.HasDetail() { + fmt.Fprintln(out, "Cache detail not recorded in these summaries.") + } else { + var beforeTokens, afterTokens uint64 + for _, counter := range r.Evidence.Counters { + if counter.Name == "input_tokens" || counter.Name == "output_tokens" { + beforeTokens += counter.Before + afterTokens += counter.After + } + } + renderCachePair(out, v.Before, v.After, beforeTokens, afterTokens) + } + fmt.Fprintln(out, " "+v.GroupingGap) + } else if c := q.Cache; c != nil { fmt.Fprintf(out, "Cached-token share: %.2f%% -> %.2f%% (%+.2f percentage points; %s).\n", c.BeforeShare*100, c.AfterShare*100, c.DeltaPercentagePoints, c.Direction) fmt.Fprintf(out, " Recorded cache-read/input tokens: %d/%d -> %d/%d.\n", c.BeforeCacheReadInputTokens, c.BeforeInputTokens, c.AfterCacheReadInputTokens, c.AfterInputTokens) for _, entry := range []struct { @@ -149,6 +172,9 @@ func renderInvestigation(out io.Writer, r compare.Investigation, threshold float if len(missingRankings) > 0 { fmt.Fprintf(out, "Turn on %s rankings (topk_keys) for more answers.\n", strings.Join(missingRankings, " and ")) } + if cacheFell { + fmt.Fprintln(out, "\n"+cacheDiagnosticsNext) + } } func intervalText(v compare.Interval) string { diff --git a/internal/cli/investigation_text_test.go b/internal/cli/investigation_text_test.go index e88621d..4896e9e 100644 --- a/internal/cli/investigation_text_test.go +++ b/internal/cli/investigation_text_test.go @@ -87,10 +87,12 @@ func TestCacheInvestigationTextAndJSON(t *testing.T) { t.Fatal("cache replaced the existing headline", out.String()) } for _, want := range []string{ - "Cached-token share: 60.00% -> 20.00% (-40.00 percentage points; decreased).", - "Recorded cache-read/input tokens: 600/1000 -> 200/1000.", - "Before: 2026-10-04T00:00:00Z for 1m0s.", "After: 2026-10-04T00:01:00Z for 1m0s.", - "Share of recorded input tokens served from cache.", + "Cache-read share of recorded input: 60% -> 20%.", + "Cache reads: 600 of 1,000 input tokens -> 200 of 1,000 (all requests report cache detail).", + "But cache reuse fell: cache-read share 60% -> 20% (uncached input +400).", + cacheDiagnosticsNext, + "Cache-write detail not recorded.", + "not cache totals by model or key/session", "Turn on user and session rankings (topk_keys) for more answers.", } { if !strings.Contains(out.String(), want) { diff --git a/internal/cli/spend.go b/internal/cli/spend.go index 7551f84..c155f8a 100644 --- a/internal/cli/spend.go +++ b/internal/cli/spend.go @@ -59,6 +59,7 @@ func renderSpend(out io.Writer, r spend.Report) { fmt.Fprintf(out, "Analyzed calls only: %s -> %s recorded tokens; %s -> %s logged model requests.\n", spendNumber(r.Before.Tokens), spendNumber(r.After.Tokens), spendNumber(r.Before.Requests), spendNumber(r.After.Requests)) } else { fmt.Fprintln(out, spendHeadline(r)) + renderCacheHeadline(out, r.Before.Cache, r.After.Cache) renderSpendModelContribution(out, r) fmt.Fprintf(out, "Logged model requests: %s -> %s.\n", spendNumber(r.Before.Requests), spendNumber(r.After.Requests)) } @@ -86,6 +87,14 @@ func renderSpend(out io.Writer, r spend.Report) { if m.Volume != nil { renderSpendVolume(out, m.Volume) } + if m.Before.Cache.HasDetail() || m.After.Cache.HasDetail() { + renderCachePair(out, m.Before.Cache, m.After.Cache, m.Before.Tokens, m.After.Tokens) + } else if r.Before.Cache.HasDetail() || r.After.Cache.HasDetail() { + fmt.Fprintln(out, " Cache detail not recorded for this model.") + } + } + if !r.Before.Cache.HasDetail() && !r.After.Cache.HasDetail() { + fmt.Fprintln(out, " Cache detail not recorded in this export.") } if !r.Incomplete && r.After.Tokens > r.Before.Tokens { fmt.Fprintln(out, "\nWho drove the increase?") @@ -136,6 +145,22 @@ func renderSpend(out io.Writer, r spend.Report) { fmt.Fprintln(out, " No tracked contributors with attributable recorded tokens.") } fmt.Fprintf(out, " Attributed tokens: %s -> %s.\n", spendNumber(r.Before.AttributedTokens), spendNumber(r.After.AttributedTokens)) + if r.Before.Cache.HasDetail() || r.After.Cache.HasDetail() { + renderCacheRankings(out, r.Cache) + for _, c := range r.CallTypes { + if !c.Analyzed { + continue + } + for _, side := range []struct { + name string + counts spend.CallTypeCounts + }{{"before", c.Before}, {"after", c.After}} { + if side.counts.Requests != 0 && side.counts.CacheReadDetailRecorded == 0 && side.counts.CacheWriteDetailRecorded == 0 { + fmt.Fprintf(out, " %s (%s): cache detail not recorded on %s requests.\n", c.Name, side.name, spendNumber(side.counts.Requests)) + } + } + } + } fmt.Fprintln(out, "\nHow complete is the export?") problems := r.Incomplete if r.Incomplete { @@ -197,7 +222,14 @@ func renderSpend(out io.Writer, r spend.Report) { } else { fmt.Fprintln(out, " No usage problems found.") } - fmt.Fprintf(out, " Rows read: %s; unsupported rows: %s; outside periods: %s.\n", spendNumber(r.Rows), spendNumber(r.ExcludedRows), spendNumber(r.OutsidePeriods)) + fmt.Fprintf(out, " Rows read: %s", spendNumber(r.Rows)) + if r.ExcludedRows != 0 { + fmt.Fprintf(out, "; unsupported rows: %s", spendNumber(r.ExcludedRows)) + } + if r.OutsidePeriods != 0 { + fmt.Fprintf(out, "; outside periods: %s", spendNumber(r.OutsidePeriods)) + } + fmt.Fprintln(out, ".") if r.Incomplete { fmt.Fprintln(out, " Whole-period per-request split unavailable while selected call types remain unanalyzed.") } else if r.Volume == nil { @@ -215,6 +247,8 @@ func renderSpend(out io.Writer, r spend.Report) { } if r.Incomplete { fmt.Fprintln(out, "\nNext: review unsupported call types in your local LiteLLM logs; analyze those rows before drawing a whole-export token-change conclusion.") + } else if change := cacheReadChange(r.Before.Cache, r.After.Cache); change != nil && change.Sign() < 0 { + fmt.Fprintln(out, "\n"+cacheDiagnosticsNext) } else if a.Unclear != 0 || b.Unclear != 0 || a.MissingIdentity != 0 || b.MissingIdentity != 0 { fmt.Fprintln(out, "\nNext: review the leading contributors and flagged records in your local LiteLLM logs.") } else { diff --git a/internal/compare/cache.go b/internal/compare/cache.go index 170151a..cd95457 100644 --- a/internal/compare/cache.go +++ b/internal/compare/cache.go @@ -10,7 +10,7 @@ import ( ) func cacheQuestion(r Report, before, after []summary.Envelope) Question { - q := Question{ID: "cache", Question: "Did caching get worse?", Status: "cannot_determine"} + q := Question{ID: "cache", Question: "Did caching get worse?", Status: "cannot_determine", CacheView: summaryCacheView(r, before, after)} if !r.Complete { q.Answer = "An expected producer or observation interval is incomplete. Supply complete intervals for every expected producer." return q diff --git a/internal/compare/cache_summary.go b/internal/compare/cache_summary.go new file mode 100644 index 0000000..2a2307d --- /dev/null +++ b/internal/compare/cache_summary.go @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package compare + +import "github.com/llm-measurement/llm-sketchkit/go/sketchkit/summary" + +func summaryCacheView(r Report, before, after []summary.Envelope) *CacheView { + v := &CacheView{GroupingGap: "These summaries retain aggregate cache totals, not cache totals by model or key/session. Use request-level records for those breakdowns."} + for side, w := range []*CacheWindow{&v.Before, &v.After} { + c := map[string]uint64{} + for _, counter := range r.Counters { + c[counter.Name] = counter.Before + if side == 1 { + c[counter.Name] = counter.After + } + } + w.Requests = c["requests"] + for _, f := range []struct { + name string + detail *CacheDetail + }{{"cache_read_input", &w.Read}, {"cache_write_input", &w.Write}} { + d := f.detail + d.Reported = c["token_observations."+f.name+".reported"] + _, d.CoverageKnown = c["token_observations."+f.name+".reported"] + d.IssueCounts = make(map[string]uint64) + for _, name := range []string{"input", f.name} { + for _, state := range []string{"reported", "missing", "invalid", "conflict", "subset_violation"} { + key := "token_observations." + name + "." + state + if count, present := c[key]; present { + d.IssueCounts[key] = count + } + } + } + docs := before + if side == 1 { + docs = after + } + complete := r.Complete + for _, doc := range docs { + if !summaryCacheFieldComplete(doc.Counters, f.name) { + complete = false + } + } + input, hasInput := c["input_tokens"] + tokens, hasTokens := c[f.name+"_tokens"] + if hasTokens { + d.RecordedTokens = &tokens + } + if complete && hasInput && hasTokens && tokens <= input { + covered, missing, invalid := w.Requests, uint64(0), uint64(0) + d.Covered, d.Missing, d.Invalid = &covered, &missing, &invalid + d.SetTotals(input, tokens) + } else { + d.Issue = "Complete input and cache-detail observations are needed; summaries cannot recover the covered-row denominator." + } + } + if w.Read.Input != nil { + n := *w.Read.Input - *w.Read.Tokens + w.Uncached = &n + } + } + return v +} + +func summaryCacheFieldComplete(c map[string]uint64, field string) bool { + requests, ok := c["requests"] + if !ok { + return false + } + input, inputOK := c["input_tokens"] + tokens, tokensOK := c[field+"_tokens"] + if !inputOK || !tokensOK || tokens > input || requests == 0 && input != 0 { + return false + } + for _, name := range []string{"input", field} { + for _, state := range []string{"reported", "missing", "invalid", "conflict", "subset_violation"} { + n, present := c["token_observations."+name+"."+state] + if !present || (state == "reported" && n != requests) || (state != "reported" && n != 0) { + return false + } + } + } + return true +} diff --git a/internal/compare/cache_view.go b/internal/compare/cache_view.go new file mode 100644 index 0000000..b07f671 --- /dev/null +++ b/internal/compare/cache_view.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package compare + +import "strings" + +// CacheDetail uses a denominator from the same observations as its numerator. +// Nil quantities mean unknown; explicit zero detail remains a measured zero. +type CacheDetail struct { + CoverageKnown bool `json:"coverage_known"` + Issue string `json:"issue,omitempty"` + IssueCounts map[string]uint64 `json:"issue_counts,omitempty"` + Reported uint64 `json:"reported_requests"` + // Summary quality categories overlap; their sum is not a distinct-request + // count. Nil counts are unknown, while pointers to zero are measured zeros. + Covered *uint64 `json:"covered_requests"` + Missing *uint64 `json:"missing_detail_requests"` + Invalid *uint64 `json:"invalid_detail_or_input_requests"` + // RecordedTokens preserves raw summary totals, including invalid subsets. + // It is not a covered-population numerator. + RecordedTokens *uint64 `json:"recorded_tokens,omitempty"` + // Nil means the origin of recorded zeros cannot be reconstructed, not that + // all zeros are verified. Summaries do not retain this request count. + ZeroOriginUnknown *uint64 `json:"zero_origin_unknown_requests"` + Input *uint64 `json:"covered_input_tokens"` + Tokens *uint64 `json:"tokens"` + Share *float64 `json:"share_of_recorded_input"` +} + +func (d CacheDetail) HasDetail() bool { + if d.Reported != 0 || d.Invalid != nil && *d.Invalid != 0 || d.RecordedTokens != nil && (!d.CoverageKnown || *d.RecordedTokens != 0) { + return true + } + for name, count := range d.IssueCounts { + if count != 0 && strings.HasPrefix(name, "token_observations.cache_") && + (strings.HasSuffix(name, ".invalid") || strings.HasSuffix(name, ".conflict") || strings.HasSuffix(name, ".subset_violation")) { + return true + } + } + return false +} + +type CacheWindow struct { + Requests uint64 `json:"requests"` + Read CacheDetail `json:"read"` + Write CacheDetail `json:"write"` + Uncached *uint64 `json:"uncached_input_tokens"` + MissingIdentity uint64 `json:"read_covered_requests_without_group_identity"` +} + +type CacheView struct { + Before CacheWindow `json:"before"` + After CacheWindow `json:"after"` + Rankings *Concentration `json:"uncached_growth_rankings"` + GroupingGap string `json:"grouping_gap,omitempty"` +} + +func (d *CacheDetail) SetTotals(input, tokens uint64) { + d.Input, d.Tokens = &input, &tokens + if input != 0 { + share := float64(tokens) / float64(input) + d.Share = &share + } +} + +func (w CacheWindow) HasDetail() bool { return w.Read.HasDetail() || w.Write.HasDetail() } + +// CompleteRead is required for a whole-population change statement, not for +// displaying a clearly labeled covered-population measurement. +func (w CacheWindow) CompleteRead() bool { + return w.Requests > 0 && w.Read.Covered != nil && *w.Read.Covered == w.Requests && w.Uncached != nil && w.Read.Share != nil +} diff --git a/internal/compare/cache_view_test.go b/internal/compare/cache_view_test.go new file mode 100644 index 0000000..381cd86 --- /dev/null +++ b/internal/compare/cache_view_test.go @@ -0,0 +1,191 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package compare + +import ( + "encoding/json" + "strings" + "testing" +) + +func TestSummaryCacheWriteAndUncached(t *testing.T) { + a, b, o := cacheWindows(t) + for _, window := range [][]map[string]uint64{{a[0].Counters}, {b[0].Counters}} { + c := window[0] + c["cache_write_input_tokens"] = 100 + for _, state := range []string{"reported", "missing", "invalid", "conflict", "subset_violation"} { + c["token_observations.cache_write_input."+state] = 0 + } + c["token_observations.cache_write_input.reported"] = c["requests"] + } + q, _ := cacheAnswer(t, a, b, o) + v := q.CacheView + if v == nil || *v.Before.Write.Share != .1 || *v.Before.Uncached != 400 || *v.After.Uncached != 800 || v.Rankings != nil || v.GroupingGap == "" { + t.Fatal(v) + } + b[0].Counters["token_observations.cache_write_input.reported"] = 9 + q, _ = cacheAnswer(t, a, b, o) + if q.CacheView.After.Write.Share != nil || q.CacheView.After.Read.Share == nil { + t.Fatal("write gap spoiled read or became zero") + } + b[0].Counters["cache_read_input_tokens"] = 1001 + q, _ = cacheAnswer(t, a, b, o) + if q.CacheView.After.Read.Share != nil || q.CacheView.After.Uncached != nil || q.CacheView.Before.Read.Share == nil { + t.Fatal("subset violation masked") + } +} + +func TestSummaryCacheQualityEvidenceAndUnknownCounts(t *testing.T) { + for _, field := range []string{"cache_read_input", "cache_write_input"} { + for _, problem := range []string{"malformed", "subset", "overlap", "partial", "missing quality"} { + t.Run(field+"/"+problem, func(t *testing.T) { + a, b, o := cacheWindows(t) + for _, counters := range []map[string]uint64{a[0].Counters, b[0].Counters} { + counters[field+"_tokens"] = 200 + for _, state := range []string{"reported", "missing", "invalid", "conflict", "subset_violation"} { + counters["token_observations."+field+"."+state] = 0 + } + counters["token_observations."+field+".reported"] = counters["requests"] + counters["token_observations."+field+".PRIVATE_SENTINEL"] = 1 + } + c := b[0].Counters + switch problem { + case "malformed": + c[field+"_tokens"] = 0 + c["token_observations."+field+".reported"] = 0 + c["token_observations."+field+".invalid"] = 10 + case "subset": + c[field+"_tokens"] = 1001 + c["token_observations."+field+".subset_violation"] = 1 + case "overlap": + c["token_observations.input.invalid"] = 6 + c["token_observations."+field+".invalid"] = 6 + c["token_observations."+field+".conflict"] = 6 + case "partial": + c["token_observations."+field+".reported"] = 9 + case "missing quality": + for _, counters := range []map[string]uint64{a[0].Counters, b[0].Counters} { + delete(counters, "token_observations."+field+".invalid") + } + } + q, _ := cacheAnswer(t, a, b, o) + d := q.CacheView.After.Read + if field == "cache_write_input" { + d = q.CacheView.After.Write + } + if d.Covered != nil || d.Missing != nil || d.Invalid != nil || d.ZeroOriginUnknown != nil { + t.Fatalf("unknown request counts became measured counts: %+v", d) + } + if d.Input != nil || d.Tokens != nil || d.Share != nil || d.Issue == "" || !d.HasDetail() { + t.Fatalf("invalid or incomplete detail became a valid measurement or disappeared: %+v", d) + } + if d.RecordedTokens == nil || *d.RecordedTokens != c[field+"_tokens"] { + t.Fatal("raw summary totals were not preserved") + } + for _, name := range []string{"input", field} { + for _, state := range []string{"reported", "missing", "invalid", "conflict", "subset_violation"} { + key := "token_observations." + name + "." + state + got, present := d.IssueCounts[key] + want, expected := c[key] + if present != expected || got != want { + t.Fatalf("raw evidence %s: got %d/%t, want %d/%t", key, got, present, want, expected) + } + } + } + data, err := json.Marshal(d) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "PRIVATE_SENTINEL") || len(d.IssueCounts) > 10 { + t.Fatal("raw quality evidence escaped its bounded allowlist") + } + var wire map[string]json.RawMessage + if err := json.Unmarshal(data, &wire); err != nil { + t.Fatal(err) + } + for _, key := range []string{"covered_requests", "missing_detail_requests", "invalid_detail_or_input_requests", "zero_origin_unknown_requests"} { + if string(wire[key]) != "null" { + t.Fatalf("%s must serialize as unknown, got %s", key, wire[key]) + } + } + }) + } + } +} + +func TestSummaryCacheLegacyTotalsRemainVisible(t *testing.T) { + for _, tokens := range []uint64{0, 600} { + a, b, o := cacheWindows(t) + for _, counters := range []map[string]uint64{a[0].Counters, b[0].Counters} { + counters["cache_read_input_tokens"] = tokens + for key := range counters { + if strings.HasPrefix(key, "token_observations.") { + delete(counters, key) + } + } + } + q, _ := cacheAnswer(t, a, b, o) + for _, w := range []CacheWindow{q.CacheView.Before, q.CacheView.After} { + d := w.Read + if !w.HasDetail() || !d.HasDetail() || d.RecordedTokens == nil || *d.RecordedTokens != tokens { + t.Fatal("old summary totals were mistaken for absent detail") + } + if d.CoverageKnown || d.Covered != nil || d.Missing != nil || d.Invalid != nil || d.Share != nil || d.ZeroOriginUnknown != nil || len(d.IssueCounts) != 0 { + t.Fatalf("old summary supplied false coverage or zero-origin evidence: %+v", d) + } + } + } +} + +func TestSummaryCacheCompleteCountsAndAbsentDetail(t *testing.T) { + a, b, o := cacheWindows(t) + for _, counters := range []map[string]uint64{a[0].Counters, b[0].Counters} { + counters["cache_read_input_tokens"] = 0 + } + q, _ := cacheAnswer(t, a, b, o) + for _, w := range []CacheWindow{q.CacheView.Before, q.CacheView.After} { + d := w.Read + if !w.CompleteRead() || !d.HasDetail() || d.Covered == nil || *d.Covered != 10 || d.Missing == nil || *d.Missing != 0 || d.Invalid == nil || *d.Invalid != 0 { + t.Fatalf("complete summary lost exact counts: %+v", d) + } + if d.Tokens == nil || *d.Tokens != 0 || d.ZeroOriginUnknown != nil { + t.Fatal("recorded zero was lost or its origin invented") + } + if w.Write.HasDetail() { + t.Fatal("absent write counters became recorded detail") + } + } + for _, counters := range []map[string]uint64{a[0].Counters, b[0].Counters} { + counters["token_observations.cache_read_input.reported"] = 0 + } + q, _ = cacheAnswer(t, a, b, o) + if q.CacheView.Before.HasDetail() || q.CacheView.After.HasDetail() { + t.Fatal("initialized zero counter with known absent observations became recorded detail") + } +} + +func TestCacheDetailExactCountsAndPresence(t *testing.T) { + covered, missing, invalid, zeroUnknown := uint64(0), uint64(0), uint64(2), uint64(0) + d := CacheDetail{CoverageKnown: true, Covered: &covered, Missing: &missing, Invalid: &invalid, ZeroOriginUnknown: &zeroUnknown} + if !d.HasDetail() || !(CacheWindow{Read: d}).HasDetail() { + t.Fatal("invalid spend detail was mistaken for absent detail") + } + data, err := json.Marshal(d) + if err != nil { + t.Fatal(err) + } + var wire map[string]json.RawMessage + if err := json.Unmarshal(data, &wire); err != nil { + t.Fatal(err) + } + for key, want := range map[string]string{"covered_requests": "0", "missing_detail_requests": "0", "invalid_detail_or_input_requests": "2", "zero_origin_unknown_requests": "0"} { + if string(wire[key]) != want { + t.Fatalf("exact %s: got %s, want %s", key, wire[key], want) + } + } + invalid, missing = 0, 2 + if d.HasDetail() { + t.Fatal("missing-only spend detail became recorded detail") + } +} diff --git a/internal/compare/compare.go b/internal/compare/compare.go index ddd4e7b..2800bfa 100644 --- a/internal/compare/compare.go +++ b/internal/compare/compare.go @@ -119,6 +119,11 @@ var counterUnits = map[string]string{ "token_observations.input.invalid": "observations", "token_observations.cache_read_input.invalid": "observations", "token_observations.input.conflict": "observations", "token_observations.cache_read_input.conflict": "observations", "token_observations.input.subset_violation": "observations", "token_observations.cache_read_input.subset_violation": "observations", + "token_observations.cache_write_input.reported": "observations", + "token_observations.cache_write_input.missing": "observations", + "token_observations.cache_write_input.invalid": "observations", + "token_observations.cache_write_input.conflict": "observations", + "token_observations.cache_write_input.subset_violation": "observations", } var sketchKinds = map[string]string{ diff --git a/internal/compare/investigate.go b/internal/compare/investigate.go index 596f42e..c11c951 100644 --- a/internal/compare/investigate.go +++ b/internal/compare/investigate.go @@ -28,6 +28,7 @@ type Question struct { Volume *VolumeChange `json:"volume,omitempty"` Contributors []Contributor `json:"contributors,omitempty"` Cache *CacheChange `json:"cache,omitempty"` + CacheView *CacheView `json:"cache_view,omitempty"` } type CacheChange struct { diff --git a/internal/spend/cache.go b/internal/spend/cache.go new file mode 100644 index 0000000..532e07a --- /dev/null +++ b/internal/spend/cache.go @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package spend + +import ( + "github.com/llm-measurement/fleetdiff/internal/accounting" + "github.com/llm-measurement/fleetdiff/internal/compare" +) + +func validCacheField(obs accounting.Observation, field string) bool { + return obs.Quality[field+"/reported"] == 1 && obs.Quality[field+"/invalid"] == 0 && + obs.Quality[field+"/conflict"] == 0 && obs.Quality[field+"/subset_violation"] == 0 +} + +func addCache(w *compare.CacheWindow, row Row, obs accounting.Observation, q Quality) error { + w.Requests++ + for _, f := range []struct { + name, column, metric string + detail *compare.CacheDetail + }{ + {"cache_read_input", "cache_read_input_tokens", "cache_read_input_tokens_total", &w.Read}, + {"cache_write_input", "cache_write_input_tokens", "cache_write_input_tokens_total", &w.Write}, + } { + d := f.detail + d.CoverageKnown = true + if d.ZeroOriginUnknown == nil { + d.ZeroOriginUnknown = new(uint64) + d.Covered, d.Missing, d.Invalid = new(uint64), new(uint64), new(uint64) + } + d.Reported += obs.Quality[f.name+"/reported"] + if row.Values[f.column] == "" { + *d.Missing++ + continue + } + if !validCacheField(obs, "input") || !validCacheField(obs, f.name) { + *d.Invalid++ + continue + } + input, tokens := obs.Metrics[accounting.Prefix+"input_tokens_total"], obs.Metrics[accounting.Prefix+f.metric] + if tokens == 0 { + *d.ZeroOriginUnknown++ + } + if d.Input != nil { + var err error + input, err = accounting.AddBounded(*d.Input, input) + if err != nil { + return err + } + tokens, err = accounting.AddBounded(*d.Tokens, tokens) + if err != nil { + return err + } + } + *d.Covered++ + d.SetTotals(input, tokens) + } + if w.Read.Input != nil { + uncached := *w.Read.Input - *w.Read.Tokens + w.Uncached = &uncached + } + if validCacheField(obs, "input") && validCacheField(obs, "cache_read_input") { + w.MissingIdentity += q.MissingIdentity + } + return nil +} diff --git a/internal/spend/cache_test.go b/internal/spend/cache_test.go new file mode 100644 index 0000000..78a5f33 --- /dev/null +++ b/internal/spend/cache_test.go @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: Apache-2.0 +// Code authors: Vijay and Codex + +package spend + +import ( + "encoding/json" + "fmt" + "math" + "strings" + "testing" + + "github.com/llm-measurement/fleetdiff/internal/compare" +) + +func cacheRow(id, day, key, read, write string) map[string]string { + r := sample(id, day, key, 1000, 100) + r["cache_read_input_tokens"], r["cache_write_input_tokens"] = read, write + return r +} + +func TestCacheDropUsesCoveredInputAndExistingMappings(t *testing.T) { + for _, op := range []string{"completion", "acompletion", "text_completion", "atext_completion", "anthropic_messages", "aanthropic_messages", "responses", "aresponses"} { + t.Run(op, func(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "sensitive-key", "920", "50"), cacheRow("b", "2026-09-02", "sensitive-key", "610", "100")} + for _, row := range rows { + row["call_type"] = op + } + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + a, b := r.Before.Cache, r.After.Cache + if !a.CompleteRead() || !b.CompleteRead() || *a.Read.Share != .92 || *b.Read.Share != .61 || *a.Write.Share != .05 || *b.Write.Share != .1 || *a.Uncached != 80 || *b.Uncached != 390 { + t.Fatalf("cache: %+v -> %+v", a, b) + } + if r.Before.Tokens != 1100 || r.After.Tokens != 1100 { + t.Fatal("cache counted twice") + } + m := r.Cache.Rankings.Movers[0] + if m.Before.Lower != 80 || m.After.Upper != 390 || m.Delta.Lower != 310 || m.Item != r.Rankings.Movers[0].Item { + t.Fatal(m) + } + data, _ := json.Marshal(r) + if strings.Contains(string(data), "sensitive-key") { + t.Fatal("identity leak") + } + }) + } +} + +func TestCacheMissingInvalidAndZero(t *testing.T) { + for _, tc := range []struct { + name, read, write, input string + covered, missing, invalid uint64 + }{ + {"absent", "", "", "1000", 0, 1, 0}, + {"zero", "0", "0", "1000", 1, 0, 0}, + {"subset", "1001", "20", "1000", 0, 0, 1}, + {"invalid", "bad", "20", "1000", 0, 0, 1}, + {"negative", "-1", "20", "1000", 0, 0, 1}, + {"too large", "9223372036854775808", "20", "1000", 0, 0, 1}, + {"missing input", "20", "20", "", 0, 0, 1}, + {"empty input", "0", "0", "0", 1, 0, 0}, + } { + for _, side := range []int{0, 1} { + t.Run(fmt.Sprintf("%s/%d", tc.name, side), func(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "key", "920", "50"), cacheRow("b", "2026-09-02", "key", "610", "100")} + rows[side]["cache_read_input_tokens"], rows[side]["cache_write_input_tokens"], rows[side]["prompt_tokens"] = tc.read, tc.write, tc.input + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + w := r.Before.Cache + if side == 1 { + w = r.After.Cache + } + if *w.Read.Covered != tc.covered || *w.Read.Missing != tc.missing || *w.Read.Invalid != tc.invalid { + t.Fatal(w.Read) + } + if tc.covered == 0 && (w.Read.Share != nil || w.Uncached != nil || r.Cache.Rankings != nil) { + t.Fatal("unknown became zero") + } + if tc.input == "0" && w.Read.Share != nil { + t.Fatal("zero denominator") + } + if tc.read == "0" && *w.Read.ZeroOriginUnknown != 1 { + t.Fatal("recorded cache zero lost its unknown origin") + } + if tc.covered == 0 { + b, _ := json.Marshal(w) + if !strings.Contains(string(b), `"uncached_input_tokens":null`) { + t.Fatal(string(b)) + } + } + }) + } + } +} + +func TestCachePartialCoverageDoesNotUseWholeDenominator(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "key", "920", "50"), cacheRow("b", "2026-09-02", "key", "610", ""), cacheRow("c", "2026-09-02", "key", "", "100")} + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + w := r.After.Cache + if *w.Read.Share != .61 || *w.Write.Share != .1 || *w.Read.Missing != 1 || *w.Write.Missing != 1 || w.CompleteRead() || *w.Uncached != 390 { + t.Fatal(w) + } + if r.Cache.Rankings.Movers[0].Delta.Lower != 310 { + t.Fatal("missing rows entered ranking") + } +} + +func TestCacheRankingBoundsAndStableAliases(t *testing.T) { + opts := options(t) + opts.ShowHashes = true + rows := []map[string]string{} + for i := 0; i < 1500; i++ { + rows = append(rows, cacheRow(fmt.Sprintf("b%d", i), "2026-09-01", fmt.Sprint(i), "900", "0"), cacheRow(fmt.Sprintf("a%d", i), "2026-09-02", fmt.Sprint(i), "500", "0")) + } + p := writeRows(t, rows) + r, err := Investigate(p, opts) + if err != nil { + t.Fatal(err) + } + if r.Cache.Rankings.BeforeMaxError == 0 && r.Cache.Rankings.AfterMaxError == 0 { + t.Fatal("expected approximate ranking") + } + for _, m := range r.Cache.Rankings.Movers { + if m.Before.Lower > 100 || m.Before.Upper < 100 || m.After.Lower > 500 || m.After.Upper < 500 || m.Delta.Lower > 400 || m.Delta.Upper < 400 { + t.Fatal(m) + } + } + // Equal total weights but unequal cache losses: aliases still refer to the same keys. + rows = []map[string]string{cacheRow("1", "2026-09-01", "a", "900", "0"), cacheRow("2", "2026-09-01", "b", "900", "0"), cacheRow("3", "2026-09-02", "a", "800", "0"), cacheRow("4", "2026-09-02", "b", "100", "0")} + opts.SecretEnv = "" + for i := 0; i < 5; i++ { + r, err = Investigate(writeRows(t, rows), opts) + if err != nil { + t.Fatal(err) + } + if r.Cache.Rankings.Movers[0].Item != "key-2" || r.Cache.Rankings.Movers[0].Delta != (compare.Interval{Lower: 800, Upper: 800}) { + t.Fatal(r.Cache.Rankings) + } + } +} + +func TestCacheGroupsAndMissingIdentity(t *testing.T) { + for _, group := range []string{"key", "team", "session", "user", "end-user"} { + t.Run(group, func(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "secret", "920", "50"), cacheRow("b", "2026-09-02", "secret", "610", "100")} + for _, row := range rows { + row["team_id"], row["user"], row["end_user"] = "private-team", "private-user", "private-enduser" + } + opts := options(t) + opts.GroupBy = group + r, err := Investigate(writeRows(t, rows), opts) + if err != nil { + t.Fatal(err) + } + if len(r.Cache.Rankings.Movers) != 1 || r.Cache.Rankings.Movers[0].Delta.Lower != 310 { + t.Fatal(r.Cache) + } + }) + } +} + +func TestCacheMissingIdentityExclusion(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "key", "920", "50"), cacheRow("b", "2026-09-02", "", "610", "100")} + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + if r.After.Cache.MissingIdentity != 1 || *r.After.Cache.Uncached != 390 || len(r.Cache.Rankings.Movers) != 0 { + t.Fatal("identity-free usage must enter totals but not ranking", r.Cache) + } +} + +func TestCacheWriteInvalidAndAggregateOverflow(t *testing.T) { + for _, value := range []string{"bad", "-1", "1001", "9223372036854775808"} { + rows := []map[string]string{cacheRow("a", "2026-09-01", "key", "920", "50"), cacheRow("b", "2026-09-02", "key", "610", value)} + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + if *r.After.Cache.Write.Invalid != 1 || r.After.Cache.Write.Share != nil || !r.After.Cache.CompleteRead() { + t.Fatal("invalid write must not become zero or hide usable read", r.After.Cache) + } + } + rows := []map[string]string{sample("a", "2026-09-01", "key", math.MaxInt64, 0), sample("b", "2026-09-01", "key", 1, 0), sample("c", "2026-09-02", "key", 1, 0)} + for _, row := range rows { + row["cache_read_input_tokens"], row["cache_write_input_tokens"] = "0", "0" + } + if _, err := Investigate(writeRows(t, rows), options(t)); err == nil { + t.Fatal("cache aggregate overflow must fail") + } +} + +func TestUncachedGrowthCanAccompanyImprovedShare(t *testing.T) { + rows := []map[string]string{cacheRow("a", "2026-09-01", "key", "900", "0"), cacheRow("b", "2026-09-02", "key", "2850", "0")} + rows[1]["prompt_tokens"], rows[1]["total_tokens"] = "3000", "3100" + r, err := Investigate(writeRows(t, rows), options(t)) + if err != nil { + t.Fatal(err) + } + if *r.After.Cache.Read.Share <= *r.Before.Cache.Read.Share || r.Cache.Rankings.Movers[0].Delta.Lower != 50 { + t.Fatal(r.Cache) + } +} diff --git a/internal/spend/coverage.go b/internal/spend/coverage.go index 719ed76..7d65dae 100644 --- a/internal/spend/coverage.go +++ b/internal/spend/coverage.go @@ -44,6 +44,14 @@ func recordCallType(r *Report, index map[string]int, row Row, side int) error { c = &r.CallTypes[i].After } c.Requests++ + for _, field := range []struct { + name string + count *uint64 + }{{"cache_read_input_tokens", &c.CacheReadDetailRecorded}, {"cache_write_input_tokens", &c.CacheWriteDetailRecorded}} { + if row.Values[field.name] != "" { + *field.count++ + } + } invalid, missing := false, false for _, f := range []struct { name string @@ -88,16 +96,22 @@ func recordCallType(r *Report, index map[string]int, row Row, side int) error { // avoids an exact identity map growing with every user; its digest checks that // aliases and totals describe the same file contents. func orderCandidates(path string, r *Report, key, expected []byte) error { - if len(r.Rankings.Movers) == 0 { + groups := []*compare.Concentration{&r.Rankings} + if r.Cache.Rankings != nil { + groups = append(groups, r.Cache.Rankings) + } + if len(r.Rankings.Movers) == 0 && (r.Cache.Rankings == nil || len(r.Cache.Rankings.Movers) == 0) { return nil } order := map[uint64]int{} - for _, m := range r.Rankings.Movers { - h, err := strconv.ParseUint(m.Hash, 16, 64) - if err != nil { - return errors.New("invalid internal candidate hash") + for _, g := range groups { + for _, m := range g.Movers { + h, err := strconv.ParseUint(m.Hash, 16, 64) + if err != nil { + return errors.New("invalid internal candidate hash") + } + order[h] = 0 } - order[h] = 0 } digest := sha256.New() enc := json.NewEncoder(digest) @@ -129,17 +143,25 @@ func orderCandidates(path string, r *Report, key, expected []byte) error { return errors.New("export changed while reading; use a completed immutable export") } strength := func(v compare.Interval) int64 { return max(v.Upper, -v.Lower) } - slices.SortFunc(r.Rankings.Movers, func(a, b compare.Mover) int { - x, y := strength(a.Delta), strength(b.Delta) - if x > y { - return -1 - } - if x < y { - return 1 - } - ha, _ := strconv.ParseUint(a.Hash, 16, 64) - hb, _ := strconv.ParseUint(b.Hash, 16, 64) - return order[ha] - order[hb] - }) + for _, g := range groups { + slices.SortFunc(g.Movers, func(a, b compare.Mover) int { + x, y := strength(a.Delta), strength(b.Delta) + if g == r.Cache.Rankings { + x, y = a.Delta.Lower, b.Delta.Lower + } + if x > y { + return -1 + } + if x < y { + return 1 + } + ha, _ := strconv.ParseUint(a.Hash, 16, 64) + hb, _ := strconv.ParseUint(b.Hash, 16, 64) + return order[ha] - order[hb] + }) + } + if r.Cache.Rankings != nil { + r.Cache.Rankings.Movers = slices.DeleteFunc(r.Cache.Rankings.Movers, func(m compare.Mover) bool { return m.Delta.Upper <= 0 }) + } return nil } diff --git a/internal/spend/investigate.go b/internal/spend/investigate.go index 376ad4e..14d98fc 100644 --- a/internal/spend/investigate.go +++ b/internal/spend/investigate.go @@ -39,19 +39,27 @@ type aggregate struct { window Window cost big.Rat top *frequentitems.Sketch + cacheTop *frequentitems.Sketch distinct *hllpp.Sketch } func newAggregate(p Period, rankings bool) *aggregate { a := &aggregate{window: Window{Period: p, Counters: map[string]uint64{}, RecordedSpend: "0"}} + for _, d := range []*compare.CacheDetail{&a.window.Cache.Read, &a.window.Cache.Write} { + d.Covered, d.Missing, d.Invalid, d.ZeroOriginUnknown = new(uint64), new(uint64), new(uint64), new(uint64) + } if rankings { a.top, _ = frequentitems.New(frequentitems.ProfileSmall, sketchhash.UserV1, sketchhash.HMACSHA25664) + a.cacheTop, _ = frequentitems.New(frequentitems.ProfileSmall, sketchhash.UserV1, sketchhash.HMACSHA25664) a.distinct, _ = hllpp.New(hllpp.ProfileSmall, sketchhash.UserV1, sketchhash.HMACSHA25664) } return a } func (a *aggregate) add(row Row, obs accounting.Observation, q Quality) error { + if err := addCache(&a.window.Cache, row, obs, q); err != nil { + return err + } if err := accounting.MergeCounts(a.window.Counters, obs.Metrics); err != nil { return err } @@ -275,6 +283,12 @@ func Investigate(path string, opts Options) (Report, error) { if present { h := binary.BigEndian.Uint64(group[:8]) w.distinct.AddHash(h) + if validCacheField(a, "input") && validCacheField(a, "cache_read_input") { + weight := a.Metrics[accounting.Prefix+"input_tokens_total"] - a.Metrics[accounting.Prefix+"cache_read_input_tokens_total"] + if e = w.cacheTop.AddHash(h, int64(weight)); e != nil { + return errors.New("cache ranking weight exceeds supported range") + } + } // Match collector token-weight ranking: both primary fields must be present. if a.Metrics[accounting.Prefix+"missing_token_usage_total"] == 0 { weight := a.Metrics[accounting.Prefix+"total_tokens_total"] @@ -354,6 +368,16 @@ func Investigate(path string, opts Options) (Report, error) { return Report{}, err } r.Rankings.WeightUnit = "attributed-recorded-tokens" + r.Cache = compare.CacheView{Before: r.Before.Cache, After: r.After.Cache} + cacheRanks, err := compare.RankCandidates("uncached_"+opts.GroupBy, windows[0].cacheTop, windows[1].cacheTop) + if err != nil { + return Report{}, err + } + cacheRanks.WeightUnit = "attributed-recorded-uncached-input-tokens" + r.Cache.Rankings = &cacheRanks + if r.Before.Cache.Read.Covered == nil || r.After.Cache.Read.Covered == nil || *r.Before.Cache.Read.Covered == 0 || *r.After.Cache.Read.Covered == 0 { + r.Cache.Rankings = nil + } if err := orderCandidates(path, &r, key, secondDigest.Sum(nil)); err != nil { return Report{}, err } @@ -361,10 +385,22 @@ func Investigate(path string, opts Options) (Report, error) { r.Increase = increase(r.Rankings, r.Before.Tokens, r.After.Tokens) } r.Rankings.Movers = r.Rankings.Movers[:min(opts.Top, len(r.Rankings.Movers))] - for i := range r.Rankings.Movers { - r.Rankings.Movers[i].Item = fmt.Sprintf("%s-%d", opts.GroupBy, i+1) - if !opts.ShowHashes { - r.Rankings.Movers[i].Hash = "" + groups := [][]compare.Mover{r.Rankings.Movers} + if r.Cache.Rankings != nil { + r.Cache.Rankings.Movers = r.Cache.Rankings.Movers[:min(opts.Top, len(r.Cache.Rankings.Movers))] + groups = append(groups, r.Cache.Rankings.Movers) + } + aliases := map[string]string{} + for _, movers := range groups { + for i := range movers { + m := &movers[i] + if aliases[m.Hash] == "" { + aliases[m.Hash] = fmt.Sprintf("%s-%d", opts.GroupBy, len(aliases)+1) + } + m.Item = aliases[m.Hash] + if !opts.ShowHashes { + m.Hash = "" + } } } r.Notes = []string{ diff --git a/internal/spend/report.go b/internal/spend/report.go index cda48db..58ae32b 100644 --- a/internal/spend/report.go +++ b/internal/spend/report.go @@ -10,15 +10,16 @@ import ( ) type Window struct { - Period Period `json:"period"` - Requests uint64 `json:"logged_model_requests"` - Tokens uint64 `json:"recorded_tokens"` - Counters map[string]uint64 `json:"counters"` - Quality Quality `json:"quality"` - RecordedSpend string `json:"recorded_spend_usd"` - AttributedTokens uint64 `json:"attributed_tokens,omitempty"` - Distinct float64 `json:"distinct_groups_estimate,omitempty"` - DistinctNominalRSE float64 `json:"distinct_nominal_rse,omitempty"` + Cache compare.CacheWindow `json:"cache"` + Period Period `json:"period"` + Requests uint64 `json:"logged_model_requests"` + Tokens uint64 `json:"recorded_tokens"` + Counters map[string]uint64 `json:"counters"` + Quality Quality `json:"quality"` + RecordedSpend string `json:"recorded_spend_usd"` + AttributedTokens uint64 `json:"attributed_tokens,omitempty"` + Distinct float64 `json:"distinct_groups_estimate,omitempty"` + DistinctNominalRSE float64 `json:"distinct_nominal_rse,omitempty"` } type Model struct { @@ -36,6 +37,7 @@ type IncreaseShare struct { } type Report struct { + Cache compare.CacheView `json:"cache"` Incomplete bool `json:"comparison_incomplete"` CallTypes []CallType `json:"call_types"` Schema string `json:"schema"` @@ -67,6 +69,8 @@ type CallType struct { // Raw column sums describe coverage only; unsupported mappings are never mixed // into analyzed token totals. Unknown call-type names receive local aliases. type CallTypeCounts struct { + CacheReadDetailRecorded uint64 `json:"cache_read_detail_recorded_requests"` + CacheWriteDetailRecorded uint64 `json:"cache_write_detail_recorded_requests"` Requests uint64 `json:"requests"` PromptTokens uint64 `json:"prompt_tokens"` CompletionTokens uint64 `json:"completion_tokens"` diff --git a/scripts/install.sh b/scripts/install.sh index 4905d91..82c1c57 100644 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -6,11 +6,11 @@ umask 077 usage() { printf '%s\n' 'Usage: sh scripts/install.sh [VERSION [NEW_DIRECTORY]]' \ - 'Defaults: v0.6.1 ./fleetdiff-install. Requires curl, gh, shasum, and tar.' + 'Defaults: v0.7.0 ./fleetdiff-install. Requires curl, gh, shasum, and tar.' } case "${1:-}" in --help|-h) usage; exit 0 ;; esac if [ "$#" -gt 2 ]; then usage >&2; exit 2; fi -version=${1-v0.6.1} +version=${1-v0.7.0} destination=${2-./fleetdiff-install} case "$version" in ''|*[!a-zA-Z0-9.-]*) printf '%s\n' 'Supply a release version such as v0.3.0.' >&2; exit 2 ;; diff --git a/scripts/install_test.go b/scripts/install_test.go index 905f0f1..f3de1d3 100644 --- a/scripts/install_test.go +++ b/scripts/install_test.go @@ -15,7 +15,7 @@ import ( "testing" ) -const installTestVersion = "v0.6.1" +const installTestVersion = "v0.7.0" func TestInstallDefaultVersion(t *testing.T) { root, env := installEnvironment(t, "Darwin", "arm64", "darwin_arm64", "")