feat: add contract event archival layer for historical analytics

[lajay-faith]

Summary

Implements #504

Adds a persistent archival layer for contract events, enabling long-term analytics beyond live event subscriptions.

Features

✅ Persistent Storage - Archive events from live subscriptions to any storage backend
✅ Historical Queries - Search archived events with filters, pagination, and ordering
✅ Time Series Analytics - Aggregate events over time intervals
✅ Pluggable Storage - Replaceable storage adapters for any database or object storage
✅ Deduplication - Automatic handling of duplicate events
✅ Error Isolation - Storage failures don't corrupt live event streams
✅ Comprehensive Tests - Full coverage of archival functionality

Implementation Details

Architecture

• Event ingestion separated from storage concerns
• Storage adapter interface allows any backend (PostgreSQL, MongoDB, S3, etc.)
• Batch processing for efficient storage
• Deduplication prevents duplicate events
• Error recovery without interrupting live streams

Components

• EventArchiveStorage - Interface for pluggable storage backends
• InMemoryEventArchiveStorage - In-memory implementation for testing
• EventArchivalManager - Coordinates event persistence and querying
• Query System - Filters, pagination, ordering, and aggregation
• Time Series - Group events into time buckets for analytics

Key Features

• Filter by contract ID, event type, topics, timestamp range, ledger range
• Deterministic pagination with offset/limit
• Sorting by timestamp, ledger, contract ID, or event type
• Event rate calculations
• Time series aggregation
• Count by event type

Acceptance Criteria

✅ Contract events can be persisted through a storage adapter
✅ Archived records retain event type, contract, topics, ledger, and timestamp
✅ queryContractEvents(filters) supports historical retrieval
✅ Queries support time ranges, contract IDs, topics, and event types
✅ Results support deterministic pagination and ordering
✅ Aggregation utilities can calculate event counts and rates over time
✅ Storage failures do not silently corrupt the live event stream
✅ Tests cover persistence, duplicate events, pagination, filtering, and recovery
✅ Storage implementation remains replaceable

Files Changed

New Files

• src/soroban/eventArchival/types.ts - Type definitions and interfaces
• src/soroban/eventArchival/inMemoryStorage.ts - In-memory storage implementation
• src/soroban/eventArchival/eventArchivalManager.ts - Main archival coordinator
• src/soroban/eventArchival/index.ts - Public API exports
• src/tests/contractEventArchival.test.ts - Comprehensive tests
• docs/contract-event-archival.md - Complete documentation

Modified Files

• src/soroban/index.ts - Export archival module

Usage Example

import {
  EventArchivalManager,
  InMemoryEventArchiveStorage,
} from "sorokit-core";

// Create storage adapter
const storage = new InMemoryEventArchiveStorage();

// Create archival manager
const manager = new EventArchivalManager(storage, {
  batchSize: 50,
  deduplicate: true,
});

// Start archiving contract events
const subscription = await manager.archiveContractEvents(
  "CONTRACT_ADDRESS",
  undefined,
  {
    horizonUrl: "https://horizon-testnet.stellar.org",
    intervalMs: 1500,
  }
);

// Query archived events
const results = await manager.queryArchivedEvents({
  contractIds: ["CONTRACT_ADDRESS"],
  fromTimestamp: Date.now() - 86400000, // Last 24 hours
  eventTypes: ["transfer", "mint"],
  limit: 100,
  orderBy: "timestamp",
  order: "desc",
});

// Get time series analytics
const stats = await manager.getEventAggregation({
  contractIds: ["CONTRACT_ADDRESS"],
  fromTimestamp: Date.now() - 86400000,
}, 3600000); // 1 hour buckets

console.log("Total events:", stats.data.total);
console.log("Events per second:", stats.data.rate);
console.log("Time series:", stats.data.timeSeries);

// Stop archiving
subscription.data.unsubscribe();

Custom Storage Adapters

Applications can implement the EventArchiveStorage interface for production use:

class PostgresStorage implements EventArchiveStorage {
  async store(events) {
    // Store in PostgreSQL
  }
  
  async query(query) {
    // Query with SQL
  }
  
  async aggregate(query, intervalMs) {
    // Aggregate with SQL
  }
  
  // ... other methods
}

Recommended backends:

• PostgreSQL (with JSON support)
• MongoDB
• TimescaleDB
• ClickHouse
• Amazon S3
• Google BigQuery

Testing

All tests pass with comprehensive coverage:

• ✅ Event storage and persistence
• ✅ Querying with multiple filter combinations
• ✅ Pagination and deterministic ordering
• ✅ Time range and ledger range filtering
• ✅ Topic filtering (exact match and regex)
• ✅ Time series aggregation
• ✅ Event rate calculations
• ✅ Deduplication logic
• ✅ Error handling and recovery
• ✅ Storage statistics

npm test -- contractEventArchival.test.ts

Documentation

Complete documentation added:

• docs/contract-event-archival.md - Full usage guide, API reference, examples
• Covers storage adapters, queries, aggregations, best practices
• Migration guide from live subscriptions

Benefits

Before (Live Subscriptions Only)

• ❌ No historical data access
• ❌ Limited to in-memory filtering
• ❌ Data lost on restart
• ❌ No analytics capabilities

After (With Archival)

• ✅ Full historical data access
• ✅ Powerful query capabilities
• ✅ Persistent storage
• ✅ Time series analytics
• ✅ Event rate tracking
• ✅ Compliance/audit logging

Use Cases

1. Analytics Dashboards - Historical trends and real-time data
2. Audit Logging - Compliance and security monitoring
3. Event Replay - Reconstruct historical state
4. Performance Monitoring - Track event rates and anomalies
5. Data Science - Export events for ML/analysis
6. Debugging - Review past events to diagnose issues

Design Principles

• Separation of Concerns - Event ingestion independent of storage
• Pluggable Storage - Any backend via adapter interface
• Error Isolation - Storage failures don't affect live streams
• Backward Compatible - Live subscriptions continue working
• No Vendor Lock-in - Storage implementation is replaceable
• Performance - Batch processing and efficient queries

Notes for Reviewers

• Archival is completely opt-in and backward compatible
• Storage adapter interface allows flexibility for any backend
• In-memory storage is for testing/development only
• Production apps should implement persistent storage adapters
• Comprehensive tests ensure reliability
• Documentation includes migration guide and best practices

Closes #504

[drips-wave]

@lajay-faith Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits