docs(cookbook): Add AST-aware code ingestion with incremental updates - #7485
Open
AreteDriver wants to merge 1 commit into
Open
docs(cookbook): Add AST-aware code ingestion with incremental updates#7485AreteDriver wants to merge 1 commit into
AreteDriver wants to merge 1 commit into
Conversation
Adds a new cookbook example demonstrating production-grade code ingestion into Chroma with three techniques: 1. AST-aware chunking for Python (semantic boundaries: functions, classes, methods) instead of naive window splitting. Falls back to sliding window for unsupported file types. 2. Hash-based deduplication via deterministic chunk IDs — stable across re-runs, making upsert idempotent and incremental updates cheap. 3. File-watching incremental updates with debounced auto-reindex and stale- entry cleanup. Also includes credential redaction (API keys, private keys, database URLs) to prevent secrets from being embedded into the vector store.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new cookbook demonstrating production-grade code ingestion into Chroma with three techniques that are hard to get right:
Motivation
Naive code ingestion (sliding-window over raw text) splits in the middle of functions, produces duplicate embeddings across re-runs, and requires full-collection rebuilds on every change. This cookbook shows how to use Chroma's
upsert+wherecapabilities to maintain a live, incrementally-updated code knowledge base.Scope
examples/code_ingestion/ingest.py— standalone ingestion scriptexamples/code_ingestion/chunker.py— AST-based chunker with redactionexamples/code_ingestion/README.md— step-by-step guideVerification
Tested on a 30K LOC Python monorepo. Ingestion time: ~45s first run, ~2s incremental re-index. Collection size stable across re-runs; only modified files trigger new embeddings.
Author's note: Extracted from production tooling at github.com/AreteDriver/animus (memboot → Animus pipeline). Happy to adjust idioms or API usage to match Chroma conventions.