Does RocketRag use the internal knowledge of the models?

[rosegarden-coder]

Apologies for writing a question is the issue board.

I am curious to learn if RocketRag also integrates the internal knowledge of models to answer questions? My concern is that since we are mostly running small sized models locally, it will be prone to hallucinate if it draws upon the internal knowledge.

Is there a way to strictly make it answer questions only based on the files supplied to it?

The design ideas behind this project is good and I think many other RAG projects are overly cumbersome.

[AleksanderObuchowski]

You can adjust the behavior of llm with context_message_template and system_message
If you look at llm.py you will see:

context_message_template = """
Question: {question}

Here are some context documents that might be relevant
{context}
"""

system_message = """
You are a helpful assistant. Answer the user question.
"""

If you want the model to rely solely on documents you could modify this message e.g setting it to "Answer only based on the provided documents...
you can do it post init of the llm:

from rocketrag import RocketRAG
from rocketrag.vectors import SentenceTransformersVectorizer
from rocketrag.chonk import ChonkieChunker
from rocketrag.llm import LLamaLLM
from rocketrag.loaders import KreuzbergLoader

# Configure high-performance components
vectorizer = SentenceTransformersVectorizer(
    model_name="minishlab/potion-multilingual-128M"  # Fast multilingual model
)

chunker = ChonkieChunker(
    method="semantic",  # Semantic chunking for better context
    embedding_model="minishlab/potion-multilingual-128M",
    chunk_size=512
)

llm = LLamaLLM(
    repo_id="unsloth/gemma-3n-E2B-it-GGUF",
    filename="*Q8_0.gguf"  # Quantized for speed
)

llm.system_message = "<your system message>"
llm.context_message_template = "<your message template>"

loader = KreuzbergLoader()  # Ultra-fast document processing

rag = RocketRAG(
    vectorizer=vectorizer,
    chunker=chunker,
    llm=llm,
    loader=loader
)

Just remember to keep {context} variable placeholder in context_message_template as it will be automatically populated with retrieved chunks

[rosegarden-coder]

@AleksanderObuchowski Thank you for the reply. I will test it out.

Can I suggest that answering solely and only based on the user documents be made the default behavior?

Since we are most likely using small-sized models locally, which tend to hallucinate a lot -- defeating the purpose of RAG, its answers can only come from the papers and documents we feed it.