Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ jobs:
FLEETDIFF_BIN="$PWD/bin/fleetdiff" FLEETDIFF_SPEND_DOCKER=1 python3 -m unittest discover -s examples/litellm-spend -p 'test_*.py' -v
- name: Offline capture accounting contract
run: |
go run ./examples/inspect --variant stock > "$RUNNER_TEMP/inspect-synthetic-traces.json"
cmp examples/inspect/synthetic-traces.json "$RUNNER_TEMP/inspect-synthetic-traces.json"
(cd testdata/inspect-contract/v1 && shasum -a 256 -c SHA256SUMS)
go test ./internal/inspect -run '^TestInspectAccountingContract$' -count=1
sh examples/inspect.sh
Expand Down Expand Up @@ -107,7 +109,7 @@ jobs:
go test ./internal/compare -run '^$' -fuzz '^FuzzCanonicalSummary$' -fuzztime 10s -parallel 2
go test ./internal/compare -run '^$' -fuzz '^FuzzEmbeddedSketch$' -fuzztime 10s -parallel 2
go test ./internal/compare -run '^$' -fuzz '^FuzzScanSummary$' -fuzztime 10s -parallel 2
go test ./internal/diagnose -run '^$' -fuzz '^FuzzDiagnose$' -fuzztime 10s -parallel 2
go test ./internal/diagnose -run '^$' -fuzz '^FuzzDiagnose$' -fuzztime 10000x -parallel 2 -timeout 2m
go test ./internal/inspect -run '^$' -fuzz '^FuzzTraceRecord$' -fuzztime 10s -parallel 2
- name: Two operators through the released collector
if: runner.os == 'Linux'
Expand Down
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,21 @@

Notable user-visible changes are recorded here, newest first.

## Unreleased
## [0.7.0] - 2026-10-11

See the [release notes](docs/releases/v0.7.0.md) for the SQL export default and
report upgrade.

- Compare recorded cache-read and cache-write shares and uncached input on
LiteLLM exports, with keyed uncached-growth rankings and separate coverage.
Include version-pinned cache and reasoning columns in the SQL export by
default, preserving absent writer paths as NULL.
- Extend summary cache comparisons with independent read/write coverage and
uncached totals; identify unavailable model and contributor breakdowns.
- Ship a deterministic synthetic OTLP capture for binary-only inspection,
checked byte for byte against its generator in CI.
- Lead with significant recorded cache changes, show request coverage in words,
and direct cache-read-share drops to provider diagnostics and route checks.

## [0.6.1] - 2026-10-11

Expand Down
19 changes: 10 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,15 @@ whether the usage records are complete.

Use the [read-only SQL projection](examples/litellm-spend/export.sql) and your
existing PostgreSQL access. The [export guide](examples/litellm-spend/README.md#export-request-level-rows)
covers connection settings and optional cache fields. Once your approved
covers connection settings and version-pinned cache fields. Once your approved
read-only connection is named `litellm_readonly`, this example exports 16 UTC days:

```sh
umask 077
export_dir="$HOME/litellm-export"
mkdir "$export_dir"
curl --fail --location --proto '=https' --tlsv1.2 \
https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.6.0/examples/litellm-spend/export.sql \
https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.7.0/examples/litellm-spend/export.sql \
--output "$export_dir/export.sql"
psql 'service=litellm_readonly' -Xq -v ON_ERROR_STOP=1 \
-v start_utc=2026-09-23T00:00:00Z -v end_utc=2026-10-09T00:00:00Z \
Expand All @@ -48,13 +48,14 @@ The SQL writes CSV in one pass and excludes prompts and responses. Keep the
export directory private. Existing request-level JSON and JSONL work too.

<a id="2-install-v060"></a>
<a id="2-install-v061"></a>

### 2. Install v0.6.1
### 2. Install v0.7.0

Run this in Bash with `curl`, `gh`, `shasum` and `tar` available:

```sh
bash -o pipefail -c 'curl --fail --location --proto "=https" --tlsv1.2 https://raw.githubusercontent.com/llm-measurement/fleetdiff/7773f212e4ae6e800f066b394e5507c0df8ada5f/scripts/install.sh | sh -s -- v0.6.1 ./fleetdiff-install'
bash -o pipefail -c 'curl --fail --location --proto "=https" --tlsv1.2 https://raw.githubusercontent.com/llm-measurement/fleetdiff/7773f212e4ae6e800f066b394e5507c0df8ada5f/scripts/install.sh | sh -s -- v0.7.0 ./fleetdiff-install'
./fleetdiff-install/fleetdiff --version
```

Expand All @@ -77,15 +78,15 @@ to dates or explicit UTC start/end intervals.
The report shows request count versus tokens per request by model, the leading
contributors, and usage completeness. Choose `--group-by team`, `user`, `end-user`
or `session` for another view; add `--format json` for automation.
[Full command and export guide](examples/litellm-spend/README.md).
[Export guide](examples/litellm-spend/README.md); [compare recorded cache reuse](docs/CACHE.md).

## Try One Example

Download the synthetic spend file and use the installed binary:

```sh
curl --fail --location --proto '=https' --tlsv1.2 \
https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.6.0/examples/litellm-spend/synthetic.csv \
https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.7.0/examples/litellm-spend/synthetic.csv \
--output synthetic.csv
./fleetdiff-install/fleetdiff investigate --litellm-spend synthetic.csv \
--before-period 2026-10-07 --after-period 2026-10-08
Expand Down Expand Up @@ -119,14 +120,14 @@ The summary-file path also shows session concentration:
## Install

The verified installer above selects the native binary for your machine.
For manual downloads, use the [v0.6.1 archives](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.6.1)
For manual downloads, use the [v0.7.0 archives](https://github.com/llm-measurement/fleetdiff/releases/tag/v0.7.0)
and [verification instructions](docs/OPERATIONS.md#install-and-verify).
An administrator can verify and transfer artifacts through an approved internal mirror.

With a security-patched Go 1.26 toolchain, install the same version from its module:

```sh
go install github.com/llm-measurement/fleetdiff/cmd/fleetdiff@v0.6.1
go install github.com/llm-measurement/fleetdiff/cmd/fleetdiff@v0.7.0
```

The executable goes to `GOBIN`, or `$(go env GOPATH)/bin` when unset.
Expand Down Expand Up @@ -157,7 +158,7 @@ See [source builds and version reporting](docs/OPERATIONS.md#build-from-source).
- <a id="questions-it-answers"></a><a id="user-and-session-contributors"></a> [Questions and contributor rankings](docs/WORKFLOWS.md#questions-it-answers).
- <a id="how-it-works"></a><a id="use-your-own-data"></a><a id="reading-the-results"></a> [How summary comparison works](docs/WORKFLOWS.md#how-it-works), [comparison contract](docs/COMPARISON.md) and [investigation contract](docs/INVESTIGATION.md).
- <a id="built-to-be-checked"></a> [Operations](docs/OPERATIONS.md), [resource measurements](docs/BENCHMARKS.md) and [checks](docs/WORKFLOWS.md#built-to-be-checked).
- <a id="docs"></a> [FAQ](docs/FAQ.md), [security](SECURITY.md), [release notes](docs/releases/v0.6.1.md) and [release procedure](docs/RELEASING.md).
- <a id="docs"></a> [FAQ](docs/FAQ.md), [security](SECURITY.md), [release notes](docs/releases/v0.7.0.md) and [release procedure](docs/RELEASING.md).

Questions or feedback: [open an issue](https://github.com/llm-measurement/fleetdiff/issues).
See [Contributing](CONTRIBUTING.md) for checks and signed, signed-off commits.
Expand Down
Loading
Loading