https://boristane.com/blog/cloudflare-contextual-rag/

[utterances-bot]

Contextual Retrieval-Augmented Generation (RAG) on Cloudflare Workers | Boris Tane

Implementing Contextual Retrieval-Augmented Generation with Using Cloudflare AI, Vectorize and D1 database with Drizzle ORM in a Hono app

https://boristane.com/blog/cloudflare-contextual-rag/

[zhuangdaz]

Thank you Boris for sharing! Does the Cloudflare support caching that might help save the context generation part? Also what is your thought of using Durable Objects to replace D1?

[boristane]

@zhuangdaz afaik it's not possible to do prompt caching on the oss models, but I might be wrong

it's totally possible to use Durable Objects instead of D1, but it's different set of constraints. the D1 team recently launched read replication which will be a challenge to implement yourself on Durable Objects, but on DOs you can relatively implement 1 database per user

[mharrvic]

Thanks for writing this!

Some thoughts on contextualizing each chunk with the whole document with an LLM

• A document can be super big, a solution of summarizing the whole document first and use it to contextualize each chunk might be a good alt option
• Let's say the user want to reference the actual chunk to document (like auto-highlight to the document previewer feature in the UI), that might not be possible since the chunk has been contextualize already. Option is to save the raw chunk to the metadata for future reference.

I see that you have some kinda 2-stage of reranking here. (1) with RRF and (2) with AI bge reranker. The outputs from the vector and sqlite that normalized the score with RRF are based from the user query already right, is using that alone (with RRF) not good enough to rank the most relevant chunk?

Also, i tried CF workflows, it's awesome for creating pipelines!

[dustinrecko]

googled "contextual rag cloudflare" and of course someone did it ;) thank you very much for sharing this!
Wanted to push a bunch of documents and ran into "Out of AI capacity" (or similar) errors from Cloudflare. However, with the help of AI, added async doc processing via Cloudflare Queues and other features like grouping documents into collections within a blink of an eye. You gotta love the CF stack.