Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/aa-sync-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: AA Sync Service Test

on:
push:
branches: [main]
paths:
- "services/aa-sync/**"
- "libraries/python/**"
pull_request:
paths:
- "services/aa-sync/**"
- "libraries/python/**"

jobs:
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: astral-sh/setup-uv@v7
with:
version: latest
- uses: actions/setup-python@v6
with:
python-version: "3.12"
- name: Install dependencies
working-directory: services/aa-sync
run: uv sync --group dev
- name: Run tests
working-directory: services/aa-sync
run: uv run pytest
- name: Run lint
working-directory: services/aa-sync
run: uv run ruff check .
63 changes: 63 additions & 0 deletions .github/workflows/aa-sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Artificial Analysis Sync

on:
workflow_dispatch:
schedule:
# Daily, off-peak. The whole capture is two requests now that the leaderboard
# payload supersedes the per-provider pages, so a daily cadence is nearly free.
- cron: "17 4 * * *"

jobs:
sync:
runs-on: ubuntu-latest
permissions:
contents: write
# The sync alerts on failure by opening or commenting on an issue itself;
# see services/aa-sync/src/tokenpricing_aa/alerting.py.
issues: write
steps:
- uses: actions/checkout@v6
- uses: astral-sh/setup-uv@v7
with:
version: latest
- uses: actions/setup-python@v6
with:
python-version: "3.12"
- name: Install dependencies
working-directory: services/aa-sync
run: uv sync
- name: Refresh Artificial Analysis dataset
working-directory: services/aa-sync
env:
# Consumed by alerting.deliver_via_github_issue. Each of the three
# failure modes (payload-not-found, request-blocked, schema-drift)
# opens its own named issue, or comments on the open one that matches.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: uv run tokenpricing-aa-sync sync
- name: Upload raw capture
# ~12MB of source HTML, kept out of git but retained so that a failed
# parse can be diagnosed against its exact input.
if: always()
uses: actions/upload-artifact@v4
with:
name: aa-raw-capture
path: .capture/artificial-analysis/
retention-days: 30
if-no-files-found: ignore
- name: Prune old history snapshots
run: |
mapfile -t history_files < <(ls -1t database/history/artificial-analysis-*.json 2>/dev/null || true)
if [ "${#history_files[@]}" -gt 26 ]; then
printf '%s\0' "${history_files[@]:26}" | xargs -0 rm -f --
fi
- name: Commit refreshed data
run: |
if git diff --quiet; then
echo "No Artificial Analysis changes"
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add database/current database/history
git commit -m "chore: refresh Artificial Analysis dataset"
git push
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,6 @@ package-lock.json

!services/dashboard/package-lock.json
!docs/package-lock.json

# Artificial Analysis raw HTML capture (~60MB/run, uploaded as a CI artifact)
.capture/
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ The canonical skill source in this repository is `skills/tokenpricing/SKILL.md`.

Pricing data is now synchronized directly inside this repository from OpenRouter and LiteLLM, normalized into the tokenpricing schema, and published as the canonical database every six hours.

A separate daily dataset (`database/current/artificial-analysis.json`) adds provider × model × reasoning-effort serving data — benchmark scores, throughput, latency and Openness Index — sourced from [Artificial Analysis](https://artificialanalysis.ai). See [docs/database.md](docs/database.md#artificial-analysis-dataset).

## Repository Structure

```
Expand All @@ -151,7 +153,8 @@ tokenpricing/
├── services/
│ ├── dashboard/ Vite + React pricing explorer
│ ├── notifier/ Webhook notification service
│ └── sync/ Canonical database sync pipeline
│ ├── sync/ Canonical database sync pipeline
│ └── aa-sync/ Artificial Analysis acquisition (daily)
├── libraries/
│ ├── python/ Python SDK + CLI (PyPI)
│ └── typescript/ TypeScript SDK (npm)
Expand All @@ -178,6 +181,7 @@ Each library and service is self-contained. See the individual READMEs for setup
## Credits

- Canonical database: this repository (`database/current/prices.json`), synchronized from OpenRouter and LiteLLM and inspired by the original [LLMTracker](https://github.com/MrUnreal/LLMTracker) implementation
- Provider × model performance and Openness Index data (`database/current/artificial-analysis.json`): [Artificial Analysis](https://artificialanalysis.ai) — benchmark, latency and throughput figures are third-party measurements produced by Artificial Analysis, not vendor-published and not measured by tokenpricing

## License

Expand Down
Loading