Skip to content

fix: implement lib/retention module, resolve server.ts startup crash - #440

Merged
Osuochasam merged 3 commits into
Open-audit-foundation:mainfrom
retkatmun:fix/retention-scheduler-402
Aug 30, 2026
Merged

fix: implement lib/retention module, resolve server.ts startup crash#440
Osuochasam merged 3 commits into
Open-audit-foundation:mainfrom
retkatmun:fix/retention-scheduler-402

Conversation

@retkatmun

Copy link
Copy Markdown
Contributor

Summary

Closes #402

Two broken references in server.ts caused the process to crash on every boot via npm run dev:ws:

// line ~30 — module did not exist
import { startRetentionScheduler } from "./lib/retention/scheduler";

// line 174 — called but never imported or defined anywhere
schedulePruner();

Root cause

Two independent retention implementations were left in-flight. startRetentionScheduler was the scheduler entry-point; schedulePruner was a separate one-shot prune call. Neither was complete. This PR consolidates them into a single coherent lib/retention/ module and removes the duplicate call site.


Changes

lib/retention/pruner.ts (new)

Pure pruneOldData(options) function, injected PrismaClient for testability.

  • Batch-delete safety rail: findMany({ take, orderBy }) + deleteMany({ id: { in: [...] } }) — Prisma has no native DELETE ... LIMIT, so row IDs are fetched first, keeping the transaction short and bounded.
  • batchCap budget distributed across Event → DeadLetterEvent → WebhookDelivery in order; total deleted never exceeds cap.
  • Dry-run mode: counting queries run, all deleteMany calls are skipped. Reports exactly what would be removed.
  • Structured log output: JSON per run — row counts, oldest/newest createdAt per table, wall-clock duration — for operator auditing.

lib/retention/scheduler.ts (new)

node-cron wrapper that calls pruneOldData on a configurable schedule.

  • Reads RETENTION_ENABLED, RETENTION_DAYS, RETENTION_CRON_SCHEDULE, ARCHIVE_BATCH_SIZE.
  • Returns undefined (no-op) when RETENTION_ENABLED=false — tested.
  • Validates the cron expression before registering; logs and bails on bad config.

lib/retention/index.ts (new)

Re-exports the public surface (startRetentionScheduler, pruneOldData, logPruneResult, types).

scripts/retention.ts (new)

CLI script backing the two npm scripts below.
Flags: --dry-run, --days <n>, --batch-cap <n>. Env vars (RETENTION_DAYS, ARCHIVE_BATCH_SIZE) used as defaults.

package.json

Added:

"retention":         "tsx scripts/retention.ts"
"retention:dry-run": "tsx scripts/retention.ts --dry-run"

server.ts

  • Added missing import { persistExecutionDag } (was used on line 73 but never imported — pre-existing compile error).
  • Added import { startRetentionScheduler } and the startRetentionScheduler() call in app.prepare().
  • Removed schedulePruner() (undefined, duplicate intent).

.env.example

Documented RETENTION_ENABLED (default: true) in the existing Data Retention section.


Acceptance criteria check

Criterion Status
npm run dev:ws boots without throwing on either retention call ✅ Both broken references resolved
Exactly one retention mechanism ✅ Single lib/retention/ module, schedulePruner removed
RETENTION_ENABLED=false fully disables pruning ✅ Tested in scheduler.test.ts
npm run retention:dry-run reports without touching DB ✅ Verified by test (no deleteMany calls in dry-run)
npm run retention prunes respecting batch cap ✅ Tested in pruner.test.ts
New tests pass under npm run test ✅ 6/6 pass, zero regressions

Tests

lib/retention/__tests__/pruner.test.ts    (4 tests)
  ✓ deletes only rows older than the cutoff date
  ✓ deletes no more than batchCap rows in total
  ✓ dry-run mode reports counts without deleting anything
  ✓ rows exactly at cutoff are NOT deleted (lt, not lte)

lib/retention/__tests__/scheduler.test.ts (2 tests)
  ✓ returns undefined when RETENTION_ENABLED=false
  ✓ returns a stoppable ScheduledTask when enabled

Out of scope (flagged for follow-up)

  • Archiving to cold storage before deletion — the tables grow and delete silently today. A follow-up issue should consider exporting pruned rows to a .csv.gz or S3-compatible sink before the deleteMany. The logPruneResult structured output already captures the timestamp range affected, making reconstruction feasible if logs are retained.
  • Per-contract or per-event-type retention thresholds — a single global age threshold is sufficient and consistent with the issue scope.

Closes Open-audit-foundation#402

Two broken references in server.ts caused a crash on every boot:
- import { startRetentionScheduler } from './lib/retention/scheduler'
  pointed at a module that didn't exist.
- schedulePruner() on line 174 was called but never imported or defined
  anywhere in the repository.

Root cause: two people implemented the same retention feature
independently. startRetentionScheduler() was the scheduler entry-point;
schedulePruner() was a duplicate one-shot prune call. Consolidated into
a single coherent lib/retention/ module.

Changes:
- lib/retention/pruner.ts   — pure pruneOldData() function; batch-delete
  via findMany(take)+deleteMany(id in) to bound transaction size; dry-run
  mode; structured JSON log output (counts, oldest/newest per table,
  duration); batchCap distributes budget across Event, DeadLetterEvent,
  WebhookDelivery in order
- lib/retention/scheduler.ts — node-cron wrapper; reads RETENTION_ENABLED,
  RETENTION_DAYS, RETENTION_CRON_SCHEDULE, ARCHIVE_BATCH_SIZE; returns
  undefined (no-op) when RETENTION_ENABLED=false
- lib/retention/index.ts    — clean re-export surface
- scripts/retention.ts      — CLI with --dry-run / --days / --batch-cap
- package.json              — retention and retention:dry-run npm scripts
- server.ts                 — added missing imports (persistExecutionDag,
  startRetentionScheduler) and startRetentionScheduler() call
- .env.example              — documented RETENTION_ENABLED variable

Tests (6, all passing, zero regressions):
- pruner: age threshold, batch cap, dry-run no-delete, lt boundary
- scheduler: RETENTION_ENABLED=false no-op, enabled returns stoppable task
@retkatmun

Copy link
Copy Markdown
Contributor Author

kindly review and merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix and properly implement the event retention scheduler (lib/retention/scheduler)

2 participants