refactor: replace Mongo and Redis/BullMQ with SQLite-backed stores - #171
Merged
Merged
Conversation
Collapses the external datastore dependencies down to a single Docker container (Neo4j). Documents and the job queue now live in SQLite files under ~/.plumbline. Document store (Mongo -> SQLite): - Delete @bb/mongo, the mongo_uri config key, and DbProviderType.Mongo. - Port the concept-graph enrichment ledger to db-core/db/sqlite; the strategy now goes through the @bb/db facade instead of importing a driver directly. - Move KnowledgeNotFoundError out of mongo-errors.ts into knowledge-errors.ts; it is provider-agnostic. - Re-point the remaining direct @bb/mongo imports (ingest-core, neo4j, ingest-strategies, cli) at @bb/db / @bb/db-core. - DELETE /api/v1/repos/<id> renames mongoDeleted -> dbDeleted. Queue (Redis/BullMQ -> Honker over SQLite): - Delete @bb/redis and @bb/queue-bullmq, the redis_url config key, QueueProviderType.Bullmq, and the Redis* error classes. - Honker is now the only queue provider; queue_db_path defaults to queue.db and is seeded in both infra presets. Upgrade path: the config schema is strict, so a config.json carrying mongo_uri or redis_url would fail to parse outright. Both are now retired keys, dropped on read and rewritten out of the file; stored db_provider "mongo" and queue_provider "bullmq" migrate to sqlite and honker. Indexed data is NOT migrated - repos need re-indexing, and in-flight jobs should be drained before upgrading. Also restores @ladybugdb/core so @bb/ladybug builds against the real native addon rather than a throwing stub. The graph provider default is unchanged (neo4j): embedded mode is not usable yet - upsertFileNode replaces rather than accumulates files, fetchFileMetadata throws on a null collect(), and the keywords search channel hits a MAX binder exception. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapses the external datastore dependencies down to a single Docker container (Neo4j). Documents and the job queue now live in SQLite files under ~/.plumbline.
Document store (Mongo -> SQLite):
Queue (Redis/BullMQ -> Honker over SQLite):
Upgrade path: the config schema is strict, so a config.json carrying mongo_uri or redis_url would fail to parse outright. Both are now retired keys, dropped on read and rewritten out of the file; stored db_provider "mongo" and queue_provider "bullmq" migrate to sqlite and honker. Indexed data is NOT migrated - repos need re-indexing, and in-flight jobs should be drained before upgrading.
Also restores @ladybugdb/core so @bb/ladybug builds against the real native addon rather than a throwing stub. The graph provider default is unchanged (neo4j): embedded mode is not usable yet - upsertFileNode replaces rather than accumulates files, fetchFileMetadata throws on a null collect(), and the keywords search channel hits a MAX binder exception.
Pull Request Guide
Follow these conventions so every PR is easy to navigate, review, and merge.
1. Branch Naming
General rules
!,@,#,$,%, etc.Branch prefixes
feature/feature/user-authenticationbugfix/bugfix/login-issuehotfix/hotfix/payment-gateway-errorimprovement/improvement/refactor-auth-modulerelease/release/v1.2.0chore/chore/update-dependenciesdocs/docs/update-readme2. PR Title
Every PR title must start with a status tag followed by the branch purpose.
Status tags
[WIP][TEST][DONE]Update the tag as the PR progresses — move from
[WIP]→[TEST]→[DONE].Examples
[WIP] feature/user-authentication — initial Bitbucket OAuth wiring[TEST] bugfix/login-issue — fix wrong org routing on re-login[DONE] chore/update-dependencies — bump fastify to 5.6[DONE] docs/update-readme3. PR Description
Every PR description must include these sections:
What changed
A clear bulleted list of the concrete changes. Not a copy of the commit log — a human-readable summary.
Why
The reason for the change. Link the ticket / issue / conversation that prompted it. If it's a bug fix, describe the bug.
How to test
Step-by-step instructions a reviewer can follow to verify the change end-to-end. Include:
Screenshots (frontend PRs only)
Required for any PR that touches UI. Include:
Use the GitHub image upload (drag and drop into the description box) — do not link to external image hosts.
4. Example PR Description
5. Checklist Before Marking
[DONE]Thanks for keeping the review process fast and the git history readable.