Build a question-answering assistant over your own documents. It answers with citations and refuses when the indexed documents do not contain the answer.
New to this? GETTING-STARTED.md builds the project from scratch and explains each design decision.
dotnet run --project src/CustomLlm.Cli -- ingest data --embedder hashing
dotnet run --project src/CustomLlm.Cli -- ask "how many days of annual leave" --embedder hashing --no-llmPeople asking for "a custom LLM trained on my data" usually want one of three quite different things. Choosing wrong wastes a lot of time, so here they are honestly:
| What it does | What it costs | When it is right | |
|---|---|---|---|
| 1. RAG (this repo's default) | Finds relevant passages at question time and puts them in the prompt | Minutes. CPU is fine | You want factual answers about your documents, with citations |
| 2. Custom model definition | Bakes a system prompt and parameters into a named model | Seconds | You want to change tone, format or persona |
| 3. Fine-tuning export | Writes JSONL training data for LoRA tooling | Hours, and a GPU | You want to change how the model behaves in a way prompting cannot |
If your goal is "answer questions about my documents", you want option 1. Not because fine-tuning is hard, but because it is the wrong tool: fine-tuning teaches a model a style, not a fact table. Facts change; retraining every time a document changes is absurd when you could simply re-index in seconds.
flowchart LR
DATA[("Your data")]
DATA --> P1["<b>1 · RAG</b><br/>index it, retrieve at<br/>question time"]
DATA --> P2["<b>2 · Custom model</b><br/>ollama create<br/>with a system prompt"]
DATA --> P3["<b>3 · Fine-tune</b><br/>JSONL → LoRA"]
P1 --> R1["cites its sources<br/>refuses when unsure<br/>update = re-index<br/><br/><b>minutes, CPU</b>"]
P2 --> R2["shapes tone and format<br/><b>invents facts</b><br/>cannot cite<br/><br/><b>seconds</b>"]
P3 --> R3["changes deep behaviour<br/>still cannot cite<br/>retrain to update<br/><br/><b>hours, GPU</b>"]
R1 --> USE1["Use for<br/><b>facts about your documents</b>"]
R2 --> USE2["Use for<br/><b>persona and format</b>"]
R3 --> USE3["Use for<br/><b>behaviour prompting cannot reach</b>"]
classDef store fill:#0d3b66,stroke:#0d3b66,color:#fff
classDef good fill:#1b5e20,stroke:#1b5e20,color:#fff
classDef mid fill:#7c4a03,stroke:#7c4a03,color:#fff
class DATA store
class R1,USE1 good
class R2,R3 mid
This repository supports all three paths, and is honest about what each delivers.
During development, create-model built a real Ollama model whose system prompt described this
coffee company's knowledge base. Asked a question that is nowhere in that knowledge base:
The information you're asking about can be found in document [2], which states: "Water has a standard atmospheric pressure boiling point at exactly 100 degrees Celsius (°C) or 212 degrees Fahrenheit (°F)." Therefore, the answer is that water boils at 100°C or 212°F. [2]
There is no document [2] containing that. The model invented the quote and the citation. The
system prompt was stored correctly — ollama show confirms it — but instructions are not
guarantees.
The same question through the RAG path:
A: I don't have anything in the indexed documents that answers that.
That is the entire argument. Retrieval can refuse, because it knows what it actually found.
Prerequisites: .NET 10. Ollama is optional; hashing embeddings keep the whole pipeline runnable without models, network or GPU.
dotnet restore
dotnet build -c Release
dotnet run -c Release --project tests/CustomLlm.Tests --no-build
dotnet run -c Release --project src/CustomLlm.Cli --no-build -- ingest data --embedder hashing
dotnet run -c Release --project src/CustomLlm.Cli --no-build -- ask "how many days of annual leave" --embedder hashing --no-llmFor semantic embeddings and generated answers:
ollama pull all-minilm
ollama pull phi3
dotnet run --project src/CustomLlm.Cli -- ingest data --embedder ollama
dotnet run --project src/CustomLlm.Cli -- ask "how many days of annual leave do employees get" --embedder ollama| Command | What it does |
|---|---|
ingest <path> |
Chunk, embed and index a folder or file |
ask <question> |
Retrieve passages, then answer with citations |
search <question> |
Show retrieval results and score components |
info |
Describe the current index |
create-model <name> |
Write an Ollama Modelfile and optionally build it |
export |
Write JSONL training data for fine-tuning tools |
Useful flags: --index, --embedder {auto,ollama,hashing}, --embed-model, --ollama-url,
--chunk-size, --overlap, --top-k, --model, --no-llm, --show-context, --out,
--persona, --base, and --write-only.
dotnet run --project src\CustomLlm.Cli -- ingest C:\path\to\my-notes
dotnet run --project src\CustomLlm.Cli -- ask "what did we decide about pricing"| Extension | How it is read |
|---|---|
.md, .markdown, .txt |
One document per file |
.json |
A top-level array becomes one document per element, named file.json#0, file.json#1, ... A single wrapped array ({"items": [...]}) is unwrapped. Anything else is one document |
.jsonl, .ndjson |
One document per line, named file.jsonl#1, #2, ... numbered as your editor shows them |
JSON records are flattened into readable key: value lines rather than fed in raw, because embedding
models were trained on prose and braces carry no meaning:
{"id":"TKT-1041","customer":{"name":"Priya"},"tags":["hardware","grinder"]}becomes
id: TKT-1041
customer.name: Priya
tags: hardware, grinder
Field names are kept because they are genuine context — subject: Grinder jams embeds better than
the bare value. Nulls are dropped, since a line reading resolution: null is noise that embeds.
Splitting an array into one document per element is the point of JSON support: a citation reading
support-tickets.json#3:1 sends you to a specific record you can open and check, whereas "somewhere
in tickets.json" tells you nothing.
For PDFs or Word files, convert them first (pandoc, pdftotext) — deliberately not built in, so
the dependency list stays honest. GETTING-STARTED.md
shows how to add a format in about five lines.
flowchart TD
subgraph INGEST["INGEST — once per corpus change"]
direction LR
SRC["Your files<br/><code>.md .txt .json .jsonl</code>"]
LOAD["<b>load</b><br/>walk, name<br/>flatten JSON to<br/>one doc per record"]
CHUNK["<b>chunk</b><br/>whole sentences + overlap<br/>keeps source:line"]
EMB1["<b>embed</b><br/>one batched call"]
SRC --> LOAD --> CHUNK --> EMB1
end
IDX[("<b>index.json</b> — vectors + provenance + embedder name")]
EMB1 --> IDX
subgraph QUERY["QUERY — per question"]
direction LR
Q["Question"]
EMB2["<b>embed</b> question<br/>same model as index"]
SEARCH["<b>search</b><br/>0.75 cosine + 0.25 keyword"]
Q --> EMB2 --> SEARCH
end
IDX --> QUERY
SEARCH --> FLOOR{"top score at least 0.25?"}
FLOOR -->|no| REFUSE["<b>refuse</b><br/>nothing in the indexed<br/>documents answers that"]
FLOOR -->|yes| PROMPT["<b>prompt</b> — numbered passages,<br/>cite every claim, NOT_IN_CONTEXT escape"]
PROMPT --> LLM["LLM"] --> AUDIT["<b>audit citations</b>"]
AUDIT --> ANSWER["<b>Answer</b> + handbook.md:19"]
AUDIT -.->|number never supplied| W1["warn: fabricated"]
AUDIT -.->|wording mismatch| W2["warn: mis-numbered"]
classDef store fill:#0d3b66,stroke:#0d3b66,color:#fff
classDef good fill:#1b5e20,stroke:#1b5e20,color:#fff
classDef stop fill:#7f1d1d,stroke:#7f1d1d,color:#fff
classDef warn fill:#7c4a03,stroke:#7c4a03,color:#fff
class IDX store
class ANSWER good
class REFUSE stop
class W1,W2 warn
The two phases are deliberately separate. Ingest is the slow part and runs only when your documents change. Query is fast, and touches nothing but the index — which is why updating what the system knows is a re-index measured in seconds, not a retrain measured in hours.
Every component, in the order it is actually called:
sequenceDiagram
autonumber
actor U as You
participant CLI as CLI<br/>(customllm ask)
participant PIPE as Pipeline
participant EMB as IEmbedder<br/>(hashing or all-minilm)
participant IDX as VectorIndex<br/>(index.json)
participant GEN as Generation
participant LLM as IChatClient<br/>(phi3)
U->>CLI: customllm ask "how much annual leave?"
CLI->>IDX: VectorIndex.Load(indexPath)
IDX-->>CLI: chunks + vectors + embedder name
CLI->>PIPE: Ask(question, index, embedder, chat, topK=4)
PIPE->>PIPE: Retrieve(question, index, embedder, topK=4)
PIPE->>IDX: EnsureCompatible(embedder)
Note over PIPE,IDX: Refuses if the index was built with a<br/>different model. Cross-model cosine is<br/>arithmetic without meaning.
IDX-->>PIPE: ok
PIPE->>EMB: EmbedOne(question)
EMB-->>PIPE: unit vector
PIPE->>IDX: Search(question, vector, topK=4)
Note over IDX: 0.75 x cosine + 0.25 x keyword overlap
IDX-->>PIPE: ranked passages with source:line
PIPE->>GEN: AnswerQuestion(question, passages, chat)
alt top score below 0.25
GEN-->>PIPE: refusal, grounded = false
Note over GEN: The corpus has no answer.<br/>Never ask the model to improvise.
else passages look relevant
GEN->>GEN: BuildPrompt(question, passages)
Note over GEN: [1] (handbook.md:19) Everyone receives 27 days...<br/>[2] (products.md:1) Meridian is 60 percent...
GEN->>LLM: Complete(system rules, numbered passages)
LLM-->>GEN: "Employees get 27 days [1]."
GEN->>GEN: StripTemplateArtifacts(reply)
GEN->>GEN: check cited numbers exist
GEN->>GEN: check wording matches cited passage
GEN-->>PIPE: answer + citations + any warnings
end
PIPE-->>CLI: answer
CLI-->>U: A: Employees get 27 days [1].<br/>Sources: handbook.md:19
Three things in that sequence are easy to miss and matter a lot:
The compatibility check happens before any expensive question work. Querying an index with a different embedder than built it produces no error — just silently meaningless scores. Failing loudly here saves an afternoon of misdiagnosis.
The model is never asked to improvise. If the best passage is too weak, the LLM is not called at all. You cannot hallucinate from a prompt you never sent.
The model's output is checked, not trusted. The prompt asks for honest citations; the audit verifies them. Instructions are not guarantees.
Six decisions worth knowing about:
One document per JSON record. A top-level array becomes file.json#0, #1, … rather than one
blob, so a citation points at a record you can actually open and check.
Sentence-aware chunking with overlap. Chunks are packed with whole sentences up to a size budget, and each chunk repeats a little of the previous one. Overlap matters because a fact sitting on a boundary would otherwise be split across two chunks and retrievable in neither.
Hybrid retrieval. Score is 75% cosine similarity, 25% keyword overlap. Pure vector search is weak on rare literal tokens — part numbers, error codes, surnames — because embeddings smooth them away. Pure keyword search misses paraphrase. Together they cover each other.
A relevance floor. If the best passage scores below 0.25, the system refuses rather than letting the model improvise from thin context.
Citation auditing. Instructions are not guarantees, so citations are checked afterwards:
- invalid — the answer cited a passage number that was never supplied
- weak — the cited passage shares little wording with the claim, so the number is probably wrong even where the content is right
That second check exists because phi3 answered a question correctly from passage [1] and then cited [2]. A citation nobody verifies is decoration.
Embedder identity is recorded in the index. Querying an index built by a different model is refused outright. Cosine similarity between vectors from two different models is arithmetic without meaning, and the symptom is not an error — it is quietly terrible retrieval.
The suite has 120 deterministic tests. They use xUnit v3 on Microsoft.Testing.Platform, so run
them with the project rather than dotnet test — the latter reports "Zero tests ran" with this
layout on .NET 10:
dotnet run -c Release --project tests/CustomLlm.TestsThey cover document loading, JSON array splitting, JSONL line numbering, flattening, chunking, deterministic hashing embeddings, vector-index round trips, hybrid ranking, grounded refusal, citation auditing, Modelfile creation, JSONL export and end-to-end RAG. Because they use the hashing embedder and a stub chat client throughout, they need no network, no model and no GPU — which is also how CI runs them.
MIT. See LICENSE.