Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 22 additions & 38 deletions docs/analytics-storage.md
Original file line number Diff line number Diff line change
@@ -1,58 +1,42 @@
# Analytics Storage Migration (SQLite -> PostgreSQL)
# Analytics Storage

## Goal

Move analytics persistence to PostgreSQL for production workloads while keeping existing `/api/analytics` behavior stable.
Analytics data is persisted in PostgreSQL. The SQLite (`better-sqlite3`) dependency
was removed after the PostgreSQL migration was completed (see issue #334).

## Data model

### `analytics_vault_summary`

- Single-row summary keyed by `id=1`
Single-row summary keyed by `id=1`:

- `total_vaults`, `active_vaults`, `completed_vaults`, `failed_vaults`
- `total_locked_capital`, `active_capital`, `success_rate`
- `last_updated`

### `analytics_vault_daily_rollups`

- Daily materialized rollup keyed by `bucket_date`
- Same aggregate fields as summary table
- Supports historical analytics and backfill verification

## Runtime storage modes

Configure with environment variables:
Daily materialized rollup keyed by `bucket_date` with the same aggregate fields.
Supports historical analytics and backfill verification.

- `ANALYTICS_STORAGE=postgres` enables PostgreSQL reads for analytics summary.
- `ANALYTICS_DUAL_WRITE=true` enables dual-write of summary/rollups to PostgreSQL while SQLite remains active.
## Runtime storage mode

Recommended migration rollout:
Set `ANALYTICS_STORAGE=postgres` to enable PostgreSQL reads for the analytics summary
endpoint. When unset the endpoint returns empty aggregates (safe default for environments
without a live database).

1. Deploy with `ANALYTICS_DUAL_WRITE=true`, `ANALYTICS_STORAGE` unset.
2. Run backfill once and compare summary parity.
3. Switch read path to `ANALYTICS_STORAGE=postgres`.
4. Keep dual-write briefly for safety.
5. Disable dual-write after confidence window.
## Backfill

## Backfill strategy

- Use `backfillAnalyticsStorage()` from `src/db/database.ts`.
- It initializes analytics tables in PostgreSQL and recomputes:
- global summary row (`analytics_vault_summary`)
- daily rollups (`analytics_vault_daily_rollups`) from `vaults.created_at`.
Call `backfillAnalyticsStorage()` from `src/db/database.ts` to initialise the
PostgreSQL tables and recompute all aggregates from the `vaults` table.

## Validation checklist

1. Run migration: `npm run migrate:latest`
2. Trigger summary recompute/backfill.
3. Verify row counts and totals:
- SQLite `vault_analytics_summary` vs PostgreSQL `analytics_vault_summary`
4. Run contract tests:
- `npm test -- tests/analytics.test.ts`
- `npm test -- tests/jobs.test.ts`

## Security / privacy considerations

- Analytics responses remain aggregate-only.
- No additional PII fields are emitted by `/api/analytics`.
- Audit logs and privacy logger behavior are unchanged.
2. Set `ANALYTICS_STORAGE=postgres` and trigger a summary recompute.
3. Verify row counts and totals against the `vaults` table.
4. Run contract tests: `npm test -- tests/analytics.test.ts`

## Security / privacy

Analytics responses remain aggregate-only. No PII is emitted by `/api/analytics`.
Audit logs and privacy-logger behaviour are unchanged.
4 changes: 4 additions & 0 deletions eslint.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ export default [
name: 'ts-node',
message: 'ts-node is dropped in favour of tsx and ts-jest. Please do not import or use it.',
},
{
name: 'better-sqlite3',
message: 'SQLite has been removed. Use PostgreSQL via src/db/pool.ts instead.',
},
],
},
],
Expand Down
2 changes: 0 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@
"@stellar/stellar-sdk": "^14.5.0",
"argon2": "^0.31.0",
"bcryptjs": "^3.0.3",
"better-sqlite3": "^11.10.0",
"cors": "^2.8.6",
"csv-stringify": "^6.6.0",
"date-fns": "^4.1.0",
Expand All @@ -52,7 +51,6 @@
"@eslint/js": "^9.13.0",
"@redocly/cli": "^2.29.2",
"@types/bcryptjs": "^2.4.6",
"@types/better-sqlite3": "^7.6.13",
"@types/cors": "^2.8.19",
"@types/debug": "^4.1.13",
"@types/express": "^4.17.21",
Expand Down
Loading
Loading