kai — Koncrete AI

Deterministic LLM Pipelines: a framework and experimental toolkit for guaranteeing that the same input produces the same final output artifact, even when intermediate LLM generations differ.

Research Questions

1. What conditions are required for end-to-end determinism? — We formalize the pipeline as prompt → retrieval → generation → transformation → validation → output and identify where nondeterminism enters.
2. Where does nondeterminism actually enter the pipeline? — LLM sampling (even at temperature=0), retrieval ordering, timestamp injection, floating-point variance across hardware.
3. How do normalization + validation eliminate variance? — We propose a deterministic convergence loop that repeatedly applies generate → normalize → validate until output stabilizes.

Formal Model

Pipeline Stages

Input → [Prompt Template] → [Retrieval] → [LLM Generation] → [Normalization] → [Validation] → Output
                                                ↑                                      |
                                                └──────── convergence loop ────────────┘

Determinism Levels

Level	Definition	Example
Strong	Bitwise identical output (sha256(a) == sha256(b))	JSON with sorted keys, normalized whitespace
Weak	Semantically identical output (same meaning, different surface form)	"32 years old" vs "age: 32"

Convergence Loop

The loop feeds output back through generation → normalization → validation until:

• Two consecutive iterations produce the same artifact hash (converged), or
• Max iterations reached (did not converge)

This is the core contribution: normalization + validation as determinism-forcing functions.

Results (Real LLM — temp=0.7, no seed)

Model	Experiment	n	Unique Outputs	Strong Determinism	Convergence
GPT-4o	JSON extraction	42	1	100%	100%
GPT-4.1-mini	JSON extraction	20	1	100%	100%
GPT-4o	Code generation	20	3	80%	100%
GPT-4.1-mini	Code generation	19	2	94.7%	100%
GPT-4.1-mini	Summarization	20	19	10%	0%

Key finding: determinism is a function of output structure. Tasks with canonical forms (JSON) achieve perfect determinism through normalization. Free-form text remains irreducibly nondeterministic.

Experiments

Three built-in experiments test the framework across task types:

Experiment	Task
json_extraction	Extract structured JSON from text
code_generation	Generate a Python function from spec
summarization	Summarize text in 2 sentences

Quick Start

# Install
pip install -e ".[dev,llm]"

# Run with real LLM via GitHub Models (free)
export GITHUB_TOKEN=ghp_...
kai run json_extraction --github -t 0.7 --no-seed -n 20
kai run code_generation --github -t 0.7 --no-seed -n 20
kai run summarization --github -t 0.7 --no-seed -n 20

# Analyze results and generate figures
kai analyze results/json_extraction

# Run simulated experiments (fully reproducible, no API key)
pip install -e ".[dev]"
python -m kai.run_experiments

Project Structure

src/kai/
├── __init__.py          # Package root
├── model.py             # Formal model: Artifact, PipelineStage, DeterminismLevel
├── pipeline.py          # Pipeline engine with convergence loop
├── stages.py            # Concrete stages: LLM generation, normalization, validation
├── simulator.py         # Simulated LLM stages for reproducible experiments
├── experiment.py        # Experiment runner (N runs, trace collection)
├── experiments.py       # Pre-built experiment configs (real API)
├── run_experiments.py   # Main script: runs all experiments with simulator
├── analysis.py          # Metrics computation and figure generation
└── cli.py               # CLI entrypoint
paper/
└── paper.tex            # Full research paper (LaTeX)

Generated Figures

After running kai analyze, the results directory contains:

• fig_hash_frequency_raw.png — Bar chart of raw output hash frequency
• fig_hash_frequency_semantic.png — Bar chart of semantic hash frequency
• fig_determinism_comparison.png — Strong vs. weak determinism rates
• fig_convergence_iterations.png — Distribution of convergence loop iterations
• fig_cumulative_variance.png — Cumulative unique outputs over runs

Why This Matters

• Coding agents (GitHub Copilot, Cursor, internal tools) need deterministic outputs for reproducible builds and test stability
• Bridges theory and real systems — formalizes what practitioners do informally
• Strong fit for industry conferences (ICSE, ESEC/FSE, ASE) and ML systems venues (MLSys, NeurIPS Systems)

Paper

The full research paper is in paper/paper.tex. It includes:

• Formal model with definitions of strong/weak determinism
• Taxonomy of nondeterminism sources (5 categories)
• Deterministic convergence loop algorithm with convergence proof
• Experimental evaluation on real LLM (GPT-4.1-mini, temp=0.7)
• Simulated experiments in appendix for reproducibility

Compile with: cd paper && pdflatex paper.tex && pdflatex paper.tex

License

MIT