feat: pgvector embedding, extraction & shared knowledge search system#1137
Closed
reski-rukmantiyo wants to merge 93 commits intonextlevelbuilder:devfrom
Closed
feat: pgvector embedding, extraction & shared knowledge search system#1137reski-rukmantiyo wants to merge 93 commits intonextlevelbuilder:devfrom
reski-rukmantiyo wants to merge 93 commits intonextlevelbuilder:devfrom
Conversation
…r matching for prefixed model names
… for WhatsApp media handling
…ovements Squash-merge of 6 commits: - WhatsApp group agent override configuration support - Store methods for channel instances - Improved WhatsApp group agent routing and recursive config coercion - Block reply tracking restricted to intermediate tool calls - WhatsApp group management and display in channel list - Human-readable names for WhatsApp group overrides Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ommits squashed) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nt fallback logic
…xtraction pipeline (18 commits squashed) Squashed from feat/agent-listen-only-whatsapp branch covering: - WhatsApp listen-only mode for silent knowledge graph extraction - Per-group require_mention and agent ID overrides - listen_raw_messages storage and background extraction worker - Raw message listing API and UI dashboard with detail dialog - ListenBuffer refactoring for real-time raw message storage - Group refresh functionality and UI sync status indicator - Group entity and participation relations in extraction prompt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ved debugging of agent system prompts
… and extraction support
…tering and new UI configuration section
…d PostgreSQL backends
…ities and relations
…el approval messages
…er-message feat: WhatsApp media caption buffer, reconnect watchdog, KG event_time, Telegram reconnect
…king fix: KG shared mode scoping, ClearAll, raw message ResetProcessed, and UI workspace sharing
…snt-works fix: KG event_time not returned in entity queries + ambiguous user_id in recursive traversal
…ssion-error fix: allow workspace path exemption for skill execution
Merge W3: WhatsApp media, KG event_time, Telegram reconnect, bugfixes
…invalidating stale sessions on tool timeouts
Resolve merge conflicts in version.go and schema.go: - Keep both heartbeat FK migration (25→26) and MCP health checks (27→28) - Bump SchemaVersion to 28 for SQLite Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…removewriter, and /writers commands and update file write permissions.
…mpt handling and system interaction
…es, and enable fail-open for file write permissions when no rules exist.
… and dynamic agent resolution
…ent dates in search, and improve knowledge graph tool output formatting.
…ated match logic, and increased result limits
… and ranking logic
…port and real-time channel notifications
… and list remaining pending requests
…d priority metadata in extractor prompt
…ed data processing
…ogging to listen_raw_message store
…ion worker robustness
…g, and add pq.StringArray support for chunk IDs
…rmats, and increment schema version
…message statistics display
… group recovery for embedding and extraction workers
…nd vector search to support dynamic query parameters
…e gateway agent framework
…chunk storage scoping, and implement embedding evaluation tools.
… FTS to OR logic, and update shared knowledge search to scan all scopes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full pgvector embedding and extraction pipeline with shared knowledge search, embedding management UI, and worker system for WhatsApp raw message processing.
Changes by commit (chronological)
23834d6b) — chunk raw messages, embed via workers, store inraw_message_chunkstablef20fc646) — retry failed extractions with status tracking and error logging inlisten_raw_messagestored71cc003) — recover from truncated JSON in extraction, improve WhatsApp worker robustnessfead736a) — full embedding CRUD API, management page in web UI9ec4b594) — update to 768 dims, improve response parsing, chunk ID array supportf3b04fd4) — 768 dimension support, multiple response formats, schema v570b083196) — replace processed filter with extraction status, add statistics display645170af) — support extracted/failed states in UI678b4c97) — dynamic polling intervals, concurrent embedding/extraction, abandoned group recoveryd9a32eda) — re-embed functionality with UI integration847c51b8) — defer FTS/vector scopeClause evaluation for dynamic query params522286ff) — newshared_knowledge_searchtool, integrated into gateway agent framework86098a6f) — date range extraction for shared knowledge, refactor chunk scoping, embedding eval tools44c40f5a) — day-based message grouping for embeddings, FTS OR logic, scan all scopes in shared knowledge searchKey additions
whatsapp/embedding_worker.go,whatsapp/extract_worker.gopg/raw_message_chunks.go,listen_raw_messages.go(PG + SQLite)http/embeddings.go,tools/shared_knowledge_search.go,tools/raw_message_search.go,tools/date_extract.goknowledgegraph/extractor.go, testsmemory/embeddings.go, testspages/embeddings/(new page),raw-messages/updates, i18n (en/vi/zh)version.gobumpchunk_evaluator_test.go,embeddings_test.go,date_extract_test.go,extractor_test.goStats
Test plan
go test -v -tags integration ./tests/integration/🤖 Generated with Claude Code