Skip to content

feat(storage): implement time-series compaction with downsampling and retention - #269

Merged
levibliz merged 1 commit into
RiftCore00:mainfrom
daddygokings-art:feat/timeseries-compaction
Aug 18, 2026
Merged

feat(storage): implement time-series compaction with downsampling and retention#269
levibliz merged 1 commit into
RiftCore00:mainfrom
daddygokings-art:feat/timeseries-compaction

Conversation

@daddygokings-art

@daddygokings-art daddygokings-art commented Aug 18, 2026

Copy link
Copy Markdown

Summary

Implements time-series data compaction with automated downsampling, retention policies, and continuous aggregates for cost-effective long-term historical storage of location events.

Changes

StorageAdapter Interface Extension (src/storage/adapter.js)

  • Added CompactionResult, TierStatus, CompactionStatus type definitions
  • Extended QueryOptions with optional resolution parameter (raw | 1m | 1h | 1d | auto)
  • Added compact() and getCompactionStatus() to the interface

MemoryAdapter (src/storage/memory.js)

  • In-memory implementation of compaction, downsampling, and tiered query
  • 1-minute downsample with median lat/lon calculation
  • 1-hour aggregate with haversine distance calculation
  • 1-day rollup from hourly data
  • Resolution-aware queryRoom() with auto-resolution selection

PostgresAdapter (src/storage/postgres.js)

  • Declarative partitioning by day on location_events (PG16)
  • New tables: location_events_1m, location_events_1h, location_events_1d
  • Auto-partition creation trigger for new inserts
  • Compaction pipeline: partition drop (instant DDL), 1m downsampling, 1h/1d aggregation
  • Transparent query routing across tiers
  • Background compaction timer (configurable interval)
  • Compaction status tracking in compaction_status table
  • All compaction queries use ON CONFLICT DO UPDATE for idempotency

Factory (src/storage/index.js)

  • Extended with new configuration options
  • NoneAdapter updated with compact() and getCompactionStatus()

Migration Script (src/storage/migration-compaction.sql)

  • For existing deployments: migrates to partitioned table, creates downsample/aggregate tables, helper functions

Configuration

  • STORAGE_RAW_RETENTION_DAYS (default: 7)
  • STORAGE_1M_RETENTION_DAYS (default: 90)
  • STORAGE_1H_RETENTION_DAYS (default: 365)
  • STORAGE_1D_RETENTION_DAYS (default: 2555)
  • STORAGE_COMPACTION_INTERVAL_MS (default: 300000)
  • STORAGE_COMPACTION_BATCH_SIZE (default: 10000)
  • TIMESCALEDB_ENABLED (default: false)

Docker Compose

  • Updated Postgres image to timescale/timescaledb:latest-pg16
  • Added compaction-related environment variables

Tests (tests/storage-compaction.test.js)

  • 32 tests covering interface contract, resolution routing, compaction pipeline, tier statistics, multi-room compaction, edge cases
  • PostgreSQL integration tests (skipped without DATABASE_URL)

Retention Policy

Tier Resolution Default Retention Data
raw Per-event 7 days Every location update
1m 1 minute 90 days Median lat/lon, avg altitude, max speed
1h 1 hour 1 year Avg/min/max speed, distance, bounding box
1d 1 day 7 years Daily rollups for compliance

Query Routing

queryRoom(roomId, { from, to, limit, resolution }) where resolution can be:

  • raw: query raw events
  • 1m: 1-minute downsampled data
  • 1h: hourly aggregates
  • 1d: daily rollups
  • auto (default): selects finest resolution fitting within limit

Verification

  • All storage tests pass (60 passed, 4 PG-integration skipped)
  • Lint passes for all modified files (3 pre-existing errors in unmodified files)
  • Backwards compatible: existing callers using queryRoom(roomId) get raw data
  • No new npm dependencies required
  • Compaction is idempotent and resumable

Closes #257

… retention

Add automated downsampling, retention policies, and continuous aggregates
for cost-effective long-term historical storage of location events.

- Extend StorageAdapter interface with compact() and getCompactionStatus()
- Add resolution parameter to queryRoom() for tiered data access
  (raw, 1m, 1h, 1d, auto)
- Implement declarative partitioning by day on location_events
- Add location_events_1m, _1h, _1d downsample/aggregate tables
- Implement compaction pipeline: partition drop, 1m downsampling,
  1h aggregates, 1d rollups, tier retention trimming
- Add transparent query routing with auto-resolution selection
- Implement MemoryAdapter compaction for testing without PostgreSQL
- Add migration-compaction.sql for existing deployments
- Update docker-compose.yml for TimescaleDB support
- Add compaction configuration env vars
- Add comprehensive test suite (storage-compaction.test.js)

Closes RiftCore00#257
Closes RiftCore00#254
Closes RiftCore00#252
Closes RiftCore00#251
@daddygokings-art
daddygokings-art force-pushed the feat/timeseries-compaction branch from 48280fe to eea095d Compare August 18, 2026 00:30
@levibliz
levibliz merged commit 5480bb1 into RiftCore00:main Aug 18, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants