Skip to content

feat: show vectorization status of memories + stats count + force re-vectorize#140

Merged
rahilp merged 12 commits into
mainfrom
138-show-vectorization-status-of-memories-+-stats-count-and-force-re-vectorize
Jun 9, 2026
Merged

feat: show vectorization status of memories + stats count + force re-vectorize#140
rahilp merged 12 commits into
mainfrom
138-show-vectorization-status-of-memories-+-stats-count-and-force-re-vectorize

Conversation

@rahilp

@rahilp rahilp commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

  • Surfaces the vectorization state of every memory as a three-state badge on Recent cards: Vectorized (check), Vectorizing… (clock, within grace window), Not indexed (red warning, past grace window)
  • Adds unvectorized count and vectorize_grace_ms to /stats, and exposes vector_ids on /list responses
  • Adds POST /vectorize-pending endpoint that re-embeds up to 25 past-grace unvectorized entries per call, returning { processed, failed, remaining }
  • Adds a Vectorize now repair action to the stats menu that loops /vectorize-pending until all failed memories are re-indexed, then refreshes the UI

How it works

vector_ids = '[]' is the de-facto failure signal — it's permanent when storeEntry throws silently inside ctx.waitUntil. A configurable grace window (VECTORIZE_GRACE_MS, default 5 min) distinguishes in-flight embeds from actual failures, so only past-grace rows are counted as failed and eligible for re-embedding.

Test Plan

  • Capture a memory → Recent card shows Vectorizing… badge, flips to Vectorized after reload once embedding completes
  • Back-date a row past the grace window (UPDATE entries SET vector_ids='[]', created_at=created_at-600000) → card shows Not indexed
  • Stats menu shows the failed count and Vectorize now button; click → repairs entries, count drops to 0, cards flip to Vectorized
  • Confirm in-grace rows are skipped by /vectorize-pending and not counted in unvectorized
  • npm test → 285/285 passing

rahilp added 11 commits June 8, 2026 18:22
Add configurable grace window for distinguishing in-flight embeddings from
permanent failures, enabling proper vectorization status tracking.
Adds vector_ids field to the SELECT clause in buildEntryFilterQuery so /list
endpoint includes vectorization status for each entry. This enables the frontend
to compute the badge state for the recent card (Task 5).
Adds endpoint to re-embed memories that silently failed vectorization
(vector_ids still "[]" past the grace window), processing up to 25 entries
per call and returning processed/failed/remaining counts for UI looping.
- Add error logging in the /vectorize-pending catch block to catch silent errors
- Assert remaining count in partial failure test when entry lacks vectors
…rors

Add two safeguards to runVectorize:
1. Check res.ok and throw on 4xx/5xx to surface HTTP errors instead of silently failing
2. Break the loop if a batch processes zero entries while remaining > 0 to prevent infinite spinning during Worker AI outages

This prevents infinite loops when entries keep failing to vectorize and ensures error states are properly surfaced to the user.
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 81.7% 567 / 694
🔵 Statements 82.52% 666 / 807
🔵 Functions 81.48% 88 / 108
🔵 Branches 76.35% 394 / 516
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/index.ts 81.34% 74.3% 80% 80.58% 106-153, 173, 444, 451, 492, 720, 831, 837, 846, 854-874, 937-940, 966, 973, 974, 985, 1007-1015, 1023-1025, 1077, 1118, 1131, 1177-1399, 1408-1412, 1425-1450, 1469, 1511, 1549, 1583, 1749-1760, 1822, 1832
Generated in workflow #134 for commit 916c374 by the Vitest Coverage Report Action

@rahilp rahilp merged commit be21e20 into main Jun 9, 2026
1 check passed
@rahilp rahilp deleted the 138-show-vectorization-status-of-memories-+-stats-count-and-force-re-vectorize branch June 9, 2026 01:13
@rahilp rahilp added this to the v1.9.0 milestone Jun 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Show vectorization status of memories (+ stats count and force re-vectorize)

1 participant