Make SQLite the default local store by jaydestro · Pull Request #25 · jaydestro/content-scout

jaydestro · 2026-07-20T18:44:03Z

Summary

create .local/state/content-scout.db automatically during web UI setup using built-in node:sqlite
migrate reports, JSON sidecars, social drafts, configs, sentiment overrides, run history, normalized dashboard records, and asset metadata into versioned SQLite storage
serve report listings/reads, search, and normalized dashboard data from SQLite after startup reconciliation
add backup, restore, corruption recovery, export, verification, retention, benchmarks, Setup status UI, and cross-platform CI coverage

Storage contract

SQLite is local-only and gitignored; no database server, credentials, or connection string
Markdown/JSON remain deterministic import/export and archive formats
generated images and browser captures stay on disk; SQLite stores metadata and relationships, not large blobs
Node.js 22.13+ is required for built-in node:sqlite
FTS5 is used where Node's bundled SQLite provides it; an identical portable literal fallback is automatic elsewhere

Migration and rollback

first startup applies transactional migrations and imports existing workspace artifacts idempotently
source files are never deleted during migration
node tools/storage.mjs verify|backup|restore|export|retention provides recovery and portability controls
corrupt databases are quarantined and rebuilt from retained files

Validation

161 tests passed locally, 0 failed
real workspace: schema 8 / WAL, 257 artifacts, 29 report sidecars, 101 sentiment overrides, 712 asset records
normalized parity: 48 reports, 1,162 items, 582 conversations, 585 creators, 115 sources
Express integration covers setup, reports, search, sentiment/competitor/creator/source parity, persisted runs, source-directories-offline reads, and restart hydration
schema 1–4 upgrade fixtures, forced WAL contention, corruption recovery, backup/restore, retention, and path-ignore behavior tested
Windows, macOS, and Ubuntu SQLite matrix passed on Node 22.13
Web UI suite, lint, and GitGuardian passed
benchmarks documented in docs/SQLITE-BENCHMARKS.md

Closes #20

Create and migrate a gitignored local SQLite database during setup, persist normalized reports and operational state, route hot reads/search through SQLite, add recovery/retention tooling, cross-platform tests, benchmarks, and Setup status UI. Closes the implementation requirements tracked in #20 pending CI.

Use FTS5 where available and preserve identical literal search behavior on Node SQLite bundles that omit the module, including Linux and macOS CI.

Capture Express child stdout and stderr in the cross-platform integration test so Linux startup failures include the underlying exception.

Treat stdin EPIPE during optional model discovery as a soft failure so headless Linux startup remains healthy when the Copilot CLI exits early.

Copilot

Pull request overview

This PR makes SQLite the default local operational store for Content Scout, wiring the web UI and CLI tooling to create, migrate, import, and serve report/search/dashboard data from a local .local/state/content-scout.db using Node’s built-in node:sqlite.

Changes:

Added a SQLite-backed artifact store with migrations (schema v8), reconciliation/import, search (FTS5 + fallback), run history, normalized index persistence, and retention/backup/export tooling.
Updated the web UI server to initialize and query via the SQLite store for listings/search/dashboard hydration, plus added a storage status endpoint and Setup UI status panel.
Raised the Node.js requirement to 22.13+ and expanded CI coverage with a cross-platform SQLite test matrix and benchmarks/docs.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/web-ui/test/sqlite-store.test.js	New unit/integration coverage for SQLite store behaviors (import/search/backup/retention/migrations/etc.).
tools/web-ui/test/sqlite-server.test.js	New end-to-end Express test verifying SQLite initialization, offline reads, and restart hydration.
tools/web-ui/test/copilot-models.test.js	Test ensuring ACP model discovery treats EPIPE as a soft failure.
tools/web-ui/server.js	Swaps listings/search/runs/index hydration to SQLite-backed store and adds `/api/storage/status`.
tools/web-ui/README.md	Documents Node 22.13+ requirement and storage commands/behavior.
tools/web-ui/public/styles.css	Adds styles for Setup storage status panel.
tools/web-ui/public/index.html	Adds Setup “Local storage” status section + refresh control.
tools/web-ui/public/app.js	Fetches and renders SQLite storage status in Setup.
tools/web-ui/package.json	Bumps Node engine to `>=22.13.0` and adds `npm run storage`.
tools/web-ui/lib/copilot-models.mjs	Exposes injectable ACP spawn + timeout and handles stdin EPIPE as soft failure.
tools/storage.mjs	New CLI for storage init/import/status/verify/backup/restore/export/retention.
tools/lib/sqlite-store.mjs	New SQLite store implementation: migrations, reconciliation, search, retention, backups, normalized snapshots.
tools/lib/report-index.mjs	Allows injecting stored report JSON + sentiment overrides to avoid filesystem reads.
tools/lib/paths.mjs	Adds canonical `CONTENT_SCOUT_DB_FILE` and allows `SCOUT_REPO_ROOT` override.
tools/benchmark-storage.mjs	New benchmark harness comparing legacy file search vs SQLite + server cold path timings.
README.md	Documents Node requirement + first-start SQLite import behavior.
docs/SQLITE-BENCHMARKS.md	New benchmark documentation and interpretation notes.
docs/ARCHITECTURE.md	Adds local storage architecture section describing SQLite contract and commands.
CHANGELOG.md	Adds 0.32.0 entry detailing SQLite default store, commands, validation notes.
.github/workflows/ci.yml	Updates Node version and adds cross-platform SQLite storage job.

+      const fullPath = path.join(absoluteDirectory, name);
+      const stat = statSync(fullPath);
+      const relativePath = posixPath(path.relative(this.repoRoot, fullPath));
+      seen.add(relativePath);
+      const prior = existing.get(relativePath);
+      if (prior && Number(prior.mtime_ms) === Math.trunc(stat.mtimeMs) && Number(prior.size_bytes) === stat.size) {
+        unchanged++;
+        continue;
+      }
+      const content = readFileSync(fullPath, 'utf8');


+      const fullPath = path.join(absoluteDirectory, name);
+      const stat = statSync(fullPath);
+      seen.add(name);
+      const prior = existing.get(name);
+      if (prior && Number(prior.mtime_ms) === Math.trunc(stat.mtimeMs) && Number(prior.size_bytes) === stat.size) {
+        unchanged++;
+        continue;
+      }
+      const content = readFileSync(fullPath, 'utf8');


+    const placeholders = kinds.map(() => '?').join(', ');
+    const useFts = !options.regex && this.fts5Enabled;
+    const parameters = options.regex || !useFts
+      ? kinds
+      : [quotedFtsQuery(value), value, ...kinds];
+    const sql = options.regex || !useFts
+      ? `SELECT kind, name, path, mtime_ms, content FROM artifacts WHERE kind IN (${placeholders}) ORDER BY mtime_ms DESC`
+      : `
+          WITH candidates(id) AS (
+            SELECT rowid FROM artifacts_fts WHERE artifacts_fts MATCH ?
+            UNION
+            SELECT id FROM artifacts WHERE instr(lower(content), lower(?)) > 0
+          )
+          SELECT kind, name, path, mtime_ms, content
+          FROM artifacts
+          WHERE id IN (SELECT id FROM candidates) AND kind IN (${placeholders})
+          ORDER BY mtime_ms DESC
+        `;


+  const kind = path.resolve(dir) === path.resolve(SOCIAL_DIR) ? 'social-posts' : 'reports';
+  const stored = artifactStore.readArtifact(kind, name);
+  if (!stored) throw new Error('not found');
+  const raw = stored.content;


Co-authored-by: jaydestro <2974195+jaydestro@users.noreply.github.com>

jaydestro added 4 commits July 20, 2026 14:43

fix(storage): fall back when bundled SQLite lacks FTS5

4da1932

Use FTS5 where available and preserve identical literal search behavior on Node SQLite bundles that omit the module, including Linux and macOS CI.

test(storage): surface server startup failures

eb25742

Capture Express child stdout and stderr in the cross-platform integration test so Linux startup failures include the underlying exception.

fix(web-ui): handle Copilot ACP closed pipes

f173b05

Treat stdin EPIPE during optional model discovery as a soft failure so headless Linux startup remains healthy when the Copilot CLI exits early.

jaydestro mentioned this pull request Jul 20, 2026

Make SQLite the default local store during initial setup #20

Closed

34 tasks

jaydestro requested a review from Copilot July 20, 2026 20:44

Copilot started reviewing on behalf of jaydestro July 20, 2026 20:44 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Copilot started work on behalf of jaydestro July 20, 2026 23:30 View session

Copilot AI and others added 2 commits July 20, 2026 23:35

Fix SQLite review feedback

accf81a

Co-authored-by: jaydestro <2974195+jaydestro@users.noreply.github.com>

Refine SQLite reconciliation cleanup

9c91085

Co-authored-by: jaydestro <2974195+jaydestro@users.noreply.github.com>

Copilot finished work on behalf of jaydestro July 20, 2026 23:39

jaydestro merged commit e120125 into main Jul 22, 2026
11 checks passed

jaydestro deleted the feature/sqlite-default-store branch July 22, 2026 14:51

Navigation Menu

Make SQLite the default local store - #25

Make SQLite the default local store#25
jaydestro merged 6 commits into
mainfrom
feature/sqlite-default-store

jaydestro commented Jul 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jaydestro commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Storage contract

Migration and rollback

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jaydestro commented Jul 20, 2026 •

edited

Loading