fix: make agent source scans persistent and streaming

[hijzy]

Summary

Fixes #311 by replacing the full-memory agent source scan pipeline with a persistent, streaming implementation.

Previously, large scans loaded and sorted all messages in memory, which could cause OOM. This PR stages messages in a per-job SQLite store and processes them with bounded batches and keyset pagination.

Changes

• Added a shared scan core used by App/backend and Memory standalone.
• Unified streaming behavior for Codex, Cursor, and all other adapters.
• Added resumable checkpoints and crash-safe scan state.
• Added deterministic splitting for oversized turns and a 64 MiB record limit.
• Preserved existing adapter APIs, HTTP/SSE behavior, watermarks, and deduplication.
• Added migration support for legacy scan journals and standalone v1 state.
• Added bounded result payloads with pagination for large scans.

Testing

• App/backend and Memory test suites pass.
• Shared core tests pass.
• Typecheck and backend lint pass.

Closes #311