Skip to content

fix(server): migration 050 no longer stalls startup on large databases #234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
badcuban merged 1 commit into from
Sep 6, 2026
+26 −25
Merged

fix(server): migration 050 no longer stalls startup on large databases #234

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ Threadlines is dense and flat. When building or changing any user-facing surface
- Comments describe how a thing is used: concise notes above functions and classes, not line-by-line narration. When you change code, keep its comments in sync.
- Be careful with destructive actions (deleting files, dropping data, rewriting history) that the developer did not explicitly request.
- No continuously repainting animations; they peg the GPU on high-refresh displays.
- A migration that rewrites existing rows must be timed against a database with real history (hundreds of thousands of events) before it merges. In-memory tests cannot tell a two-second backfill from one that takes hours on a user's install.
- If a rule here fights the task in front of you, say so loudly and get sign-off before breaking it.

## Reference Repos
Expand Down
50 changes: 25 additions & 25 deletions apps/server/src/persistence/Migrations/050_ProjectionTranscriptEventSequence.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,29 @@ const decodeActivity = Schema.decodeUnknownOption(
Schema.fromJsonString(ThreadActivityAppendedPayload),
);

/** Order transcript entries by their first durable event, even after a clock correction. */
/**
* Order transcript entries by their first durable event, even after a clock correction.
*
* Each backfill is an UPDATE ... FROM so SQLite materializes the first-event lookup once.
* A correlated subquery over the CTE re-scans the whole event log per projected row,
* which took hours on a multi-gigabyte database and stalled startup.
*/
export default Effect.gen(function* () {
const sql = yield* SqlClient.SqlClient;

yield* sql`ALTER TABLE projection_thread_messages ADD COLUMN event_sequence INTEGER`;
yield* sql`
WITH first_events AS (
UPDATE projection_thread_messages
SET event_sequence = first_events.first_sequence
FROM (
SELECT stream_id, json_extract(payload_json, '$.messageId') AS item_id,
MIN(sequence) AS first_sequence
FROM orchestration_events
WHERE event_type IN ('thread.message-sent', 'thread.follow-up-accepted')
GROUP BY stream_id, json_extract(payload_json, '$.messageId')
)
UPDATE projection_thread_messages
SET event_sequence = (
SELECT first_sequence FROM first_events
WHERE stream_id = projection_thread_messages.thread_id
AND item_id = projection_thread_messages.message_id
)
) AS first_events
WHERE first_events.stream_id = projection_thread_messages.thread_id
AND first_events.item_id = projection_thread_messages.message_id
`;
yield* sql`
CREATE INDEX idx_projection_thread_messages_event_order
Expand All @@ -36,19 +40,17 @@ export default Effect.gen(function* () {

yield* sql`ALTER TABLE projection_thread_proposed_plans ADD COLUMN event_sequence INTEGER`;
yield* sql`
WITH first_events AS (
UPDATE projection_thread_proposed_plans
SET event_sequence = first_events.first_sequence
FROM (
SELECT stream_id, json_extract(payload_json, '$.proposedPlan.id') AS item_id,
MIN(sequence) AS first_sequence
FROM orchestration_events
WHERE event_type IN ('thread.proposed-plan-upserted')
GROUP BY stream_id, json_extract(payload_json, '$.proposedPlan.id')
)
UPDATE projection_thread_proposed_plans
SET event_sequence = (
SELECT first_sequence FROM first_events
WHERE stream_id = projection_thread_proposed_plans.thread_id
AND item_id = projection_thread_proposed_plans.plan_id
)
) AS first_events
WHERE first_events.stream_id = projection_thread_proposed_plans.thread_id
AND first_events.item_id = projection_thread_proposed_plans.plan_id
`;
yield* sql`
CREATE INDEX idx_projection_thread_proposed_plans_event_order
Expand All @@ -57,19 +59,17 @@ export default Effect.gen(function* () {

yield* sql`ALTER TABLE projection_thread_activities ADD COLUMN event_sequence INTEGER`;
yield* sql`
WITH first_events AS (
UPDATE projection_thread_activities
SET event_sequence = first_events.first_sequence
FROM (
SELECT stream_id, json_extract(payload_json, '$.activity.id') AS item_id,
MIN(sequence) AS first_sequence
FROM orchestration_events
WHERE event_type IN ('thread.activity-appended')
GROUP BY stream_id, json_extract(payload_json, '$.activity.id')
)
UPDATE projection_thread_activities
SET event_sequence = (
SELECT first_sequence FROM first_events
WHERE stream_id = projection_thread_activities.thread_id
AND item_id = projection_thread_activities.activity_id
)
) AS first_events
WHERE first_events.stream_id = projection_thread_activities.thread_id
AND first_events.item_id = projection_thread_activities.activity_id
`;
yield* sql`
CREATE INDEX idx_projection_thread_activities_event_order
Expand Down
Loading