Skip to content

feat: redesign UPLOADS_UPDATE to send only changed items#107

Merged
DaxServer merged 4 commits into
mainfrom
feat/uploads-update-partial
Jun 28, 2026
Merged

feat: redesign UPLOADS_UPDATE to send only changed items#107
DaxServer merged 4 commits into
mainfrom
feat/uploads-update-partial

Conversation

@DaxServer

Copy link
Copy Markdown
Owner

Instead of polling all uploads every 2s and JSON-diffing the full list, the backend now tracks a lastUpdateTime and queries only rows changed since that timestamp. This mirrors the OptimizedBatchStreamer pattern used for the batches list.

  • Backend: adds getLatestUploadUpdateTime, getUploadsByBatchChangedSince, and countActiveUploadsInBatch to UploadService; rewrites startUploadStream to use the lastUpdateTime pattern; removes the unused UPLOAD_DONE_STATUSES constant
  • WS schema: adds partial: boolean to UploadsUpdateSchema
  • Frontend: matches uploads by id (was: key) in the merge; stats still recomputed from the full batchUploads list after each merge

— Claude Sonnet 4.6

…ates)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
backend/src/core/handler.ts Upload streaming now sends deduplicated partial updates and completes when no active uploads remain.
backend/src/db/dal/uploads.ts Upload queries now support changed-since polling and active upload counts.
backend/src/types/ws.ts Upload update messages now require a partial boolean.
frontend/src/composables/useCollections.ts Upload update merging now matches rows by id and refreshes batch stats from merged state.
frontend/src/App.vue The beta banner was removed from the app shell.

Reviews (4): Last reviewed commit: "fix: deduplicate via sentState map to ha..." | Re-trigger Greptile

Comment thread backend/src/core/handler.ts Outdated
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@DaxServer

Copy link
Copy Markdown
Owner Author

Fixed Greptile P1 (same-second skips) in ec7b46f: changed getUploadsByBatchChangedSince to use gte (>=) and widened the poll guard to current >= lastUpdateTime. lastUpdateTime only advances on strict > to avoid infinite re-polls when nothing newer has arrived. Re-sends within the same second are idempotent on the frontend.

— Claude Sonnet 4.6

Comment thread backend/src/core/handler.ts Outdated
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@DaxServer

Copy link
Copy Markdown
Owner Author

Fixed in 4c50e81. Replaced the lastUpdateTime + gte approach with a (updated_at, id) cursor: query updated_at > since OR (updated_at = since AND id > lastMaxId), then advance both pointers to the last row returned. No re-sends are possible — each row is only returned once regardless of timestamp precision — and same-second updates are caught via the id tiebreaker. Also dropped the separate getLatestUploadUpdateTime call (one fewer DB round-trip per poll).

— Claude Sonnet 4.6

.from(uploadRequests)
.where(
and(
eq(uploadRequests.batchid, batchId),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cursor Skips Earlier Rows

The (updated_at, id) cursor still misses later changes to a row that was already behind the current id boundary. For example, after the stream sends rows through (T, id=10), a later status change to id=5 can still have updated_at = T when timestamps share the same second. This predicate then requires id > 10 for rows at T, so the changed upload is excluded from every later poll and the UI can stay on the old status.

Fix in Claude Code

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@DaxServer

Copy link
Copy Markdown
Owner Author

Fixed in ee19ba3. The (updated_at, id) cursor was correct for append-only streams but breaks when existing rows re-update within the same second after the cursor advanced past their id.

The new approach drops the id tiebreaker and queries with updated_at >= cursorTime (minId=0) on every poll. To avoid re-sending rows that haven't changed, each poll filters through a sentState: Map<number, string> that tracks the last-sent status|error|success key per upload id. Only rows where the serialised state differs are emitted. cursorTime still advances to the max updated_at seen, so the re-scan window shrinks as uploads complete.

— Claude Sonnet 4.6

@DaxServer DaxServer merged commit 38e50a5 into main Jun 28, 2026
9 checks passed
@DaxServer DaxServer deleted the feat/uploads-update-partial branch June 28, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant