Build durable extraction backend with recovery, OAuth and MCP

[EncHawk]

The previous extraction loop ran inside HTTP requests, relied on process-local locks/pub-sub, and simulated healing with a shared flag. This change makes Lippy a backend-only service with persisted asynchronous execution, actual fixture selector repair, and documented Bright Data Scraper Studio integration.

Behavior

• REST run requests enqueue immediately and return a durable run ID. PostgreSQL workers use row locks, expiring leases, fencing tokens, idempotency keys, bounded retries/deadlines, and scheduling.
• State changes and events commit together. Independent SSE/MCP subscribers replay the event log from sequence cursors across process restarts.
• Bright Data creation, generation, batch extraction, polling, repair, and empty-body approval responses use the documented API shapes. Interrupted non-idempotent submissions pause for reconciliation instead of silently duplicating jobs.
• Schema/identity constraints and deterministic semantic normalization/change limits gate accepted data. Suspicious changes require review; exhausted repairs preserve the last accepted result.
• Google OAuth uses PKCE, one-time state, nonce and signed ID-token validation. Hashed, revocable API tokens and ownership checks protect REST and MCP.
• Removes the frontend and undocumented Parallel AI diff endpoint. Adds PostgreSQL migrations, local Docker setup, a complete .env.example, a runnable fixture demo, CI and an architecture/failure walkthrough.

Validation

• 29 unit and PostgreSQL integration tests passed, including competing workers, expired-lease fencing, duplicate enqueue, bounded repair, event replay, ownership and preservation of accepted data.
• Lint, TypeScript check and production build passed.
• HTTP demo passed: baseline → changed HTML → repaired selectors → re-extraction → suspicious-value quarantine.
• Actual authenticated MCP client passed all six tools; SSE cursor replay and API-token revocation passed.
• Both migrations applied successfully to an isolated PostgreSQL database.

Configuration and limits

Live Bright Data and Google sign-in still require credentials from .env.example; live-account integration is not claimed as verified. Semantic verification is contract-rule-based, not general AI truth verification. Upstream repair is approved/saved before local re-extraction; there is no isolated candidate scraper or automatic upstream rollback. The event log supports pub/sub readers, not outbound webhook delivery.

Existing databases must baseline the original schema before the second migration. Legacy history is preserved and unowned contracts are paused. Dedicated collector IDs are now unique per contract. See README for the migration procedure.

Compatible dependency updates were applied, including Next.js 15.5.25. npm audit still reports transitive advisories in Prisma/deepmerge-ts, Vitest/mocker and Next's PostCSS; major upgrades/overrides were not forced as part of this feature PR.