Docs · Source · pip install aither-kvcache · The Aither World
The Aither World is an operating system for agents — a Linux you can hand to one, the runtimes it works in, and the tools it works with. awnix is the Linux underneath it; aitherkvcache is one of its 64 bricks — each installs on its own, runs offline, and needs no account.
Start here: Quantize the cache on one model you already serve and measure the headroom.
Near-optimal KV cache compression for LLM inference. Two compression engines:
-
TurboQuant — Vector quantization (Zandieh et al., arXiv:2504.19874). 2-4 bit, 3.8-7.1× compression vs FP16. No calibration data. Works on streaming tokens.
-
TriAttention (v2.0) — Spectral KV compression via trigonometric series. Retains top RoPE frequency pairs, scores via trig series without materializing full K/V. 14–26× compression at F=8–16. Calibration-dependent — read this before using it: which frequency pairs carry the energy is a property of the model, and off-profile the ranking degrades while reconstruction still looks fine. Measured on an uncalibrated model at the F=12 default: cosine 0.91, but mean top-32 attention overlap 0.41 over 64 query directions. Profiles ship for Qwen3.5, Nemotron, DeepSeek-R1 and Llama 3.1; anything else now emits a
RuntimeWarninginstead of falling back silently. -
KVTransfer (NEW in v2.4) — Cross-model KV transfer. Convert one model's KV cache into another model's so the receiving model skips prefill entirely. Per-(layer, head) linear maps fitted by ridge; source layers chosen per target layer by held-out R². Same model family only — the two models must share a tokenizer.
pip install aither-kvcache # core library
pip install aither-kvcache[vllm] # + vLLM plugin (v0.15+)
pip install aither-kvcache[triton] # + fused GPU kernels
pip install aither-kvcache[transfer] # + cross-model KV transfer
pip install aither-kvcache[all] # everythingfrom aither_kvcache import TurboQuant
tq = TurboQuant(head_dim=128, bits=4, device="cuda")
packed, norms = tq.encode(kv_vectors) # [..., 128] float16 -> [..., 64] uint8 + [...] f32
decoded = tq.decode(packed, norms) # [..., 64] uint8 + [...] f32 -> [..., 128] float16from aither_kvcache.triattention import TriAttention, TriAttentionConfig
# Configure: 12 frequency pairs, 4-bit coefficients → ~10× compression
config = TriAttentionConfig(
head_dim=128, num_freqs=12, coeff_bits=4,
num_kv_heads=8, num_query_heads=32,
rope_base=1_000_000.0, # Qwen3.5 RoPE base
)
tri = TriAttention(config, device="cuda")
# Encode K/V to spectral representation (pre-RoPE keys)
k_enc, v_enc = tri.encode_kv(keys, values)
# keys: [B, S, num_kv_heads, head_dim] → spectral: 26 bytes/token vs 256 FP16
# Decode: score via trig series, accumulate values
output = tri.decode_step(query, k_enc, v_enc, query_pos, key_positions)from aither_kvcache.triattention.calibration import get_config_for_model
config = get_config_for_model("Qwen3.5-8B", coeff_bits=4)
# Per-layer frequency schedule: early layers get more frequencies,
# middle layers are most spectrally concentrated.
print(config.summary())
# TriAttention Config (qwen3.5)
# head_dim=128, num_freqs=12, coeff_bits=4
# heads: 32Q / 8KV (GQA 4:1)
# storage: 26 bytes/token (vs 256 FP16)
# compression: 9.8× per K/V tensorTransformer attention with RoPE is naturally a trigonometric series in position difference Δ = n − m:
score(q, k, m, n) = (1/√d) Σᵢ [cᵢ cos(Δθᵢ) + sᵢ sin(Δθᵢ)]
where cᵢ = q₂ᵢk₂ᵢ + q₂ᵢ₊₁k₂ᵢ₊₁ (pair dot product) and θᵢ = base^{-2i/d} (RoPE frequency).
Most key energy concentrates in a few frequency pairs. By retaining only the top-F pairs (by energy E_i = k₂ᵢ² + k₂ᵢ₊₁²) and quantizing coefficients to 4-bit, we store 26 bytes per token instead of 256 — a ~10× reduction with bounded approximation error.
| Mode | Coeff Bits | Bytes/Token | Compression vs FP16 |
|---|---|---|---|
| F=12, int4 | 4 | 26 | 9.85× |
| F=12, int8 | 8 | 38 | 6.74× |
| F=16, int4 | 4 | 34 | 7.53× |
| F=8, int4 | 4 | 18 | 14.2× |
Prefill is the tax you pay before a model says anything, and a KV cache only works on the model that produced it. Route a conversation to a different model and the accumulated cache becomes dead weight. KVTransfer fits a map that converts one model's cache into another's, so the receiving model can skip prefill.
# 1. capture aligned KV from both models over one corpus
python -m aither_kvcache.kvtransfer.capture \
--source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
--out ./cap --corpus ./my-corpus --seq-len 256 --train-seqs 64 --val-seqs 24
# 2. fit the mapper pack
python -m aither_kvcache.kvtransfer.fit --capture ./cap --out ./pack --top-k 8
# 3. measure what the RECEIVING model does with a translated cache, and record it
python -m aither_kvcache.kvtransfer.transfer \
--pack ./pack --source Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B \
--corpus ./held-out --sequences 24 --cuts 32 64 96 128 160 192 224 --recordfrom aither_kvcache.kvtransfer import load_pack
from aither_kvcache.kvtransfer.transfer import translate_cache
pack = load_pack("./pack", live_source=src_geom, live_target=tgt_geom)
layers = translate_cache(src_k, src_v, pack, positions) # -> the target's KV cacheload_pack refuses any pack that carries no downstream acceptance evidence. That is
the core design decision here, and it is not defensive programming — it is the only thing
standing between an interesting R² and a broken deployment.
A converted cache does not fail loudly. An approximate one makes the receiving model produce fluent, on-topic, confidently wrong text: no exception, no error status, no unhealthy process, nothing in a log. Every cheap signal is green.
So a pack must carry a measurement, and the measurement has four arms:
| arm | what it is |
|---|---|
reference |
the target prefills the context itself — the ceiling |
translated |
the target is handed the converted cache — the candidate |
control |
the map is run on a different document — catches a mapper ignoring its input |
nocontext |
no cache at all — the floor |
The control arm is the one that matters. A mapper that has quietly learned the target's
average key/value statistics scores respectably against reference alone. Without a
floor, "68% agreement" is unreadable — it could be excellent, or exactly what a dead
mapper gets.
A minimum sample size is enforced too. Top-1 agreement is a proportion, so a mean over a few dozen positions has a standard error near ten percentage points — and once recorded it is indistinguishable from a mean over ten thousand.
- The two models must tokenize identically. The map sends position i to position i; different tokenizers mean row i is not the same token and the fit is regressing misaligned data. Nothing downstream can see this — it reads as "this pair transfers poorly" rather than "these rows do not correspond". Enforced by vocabulary digest, and by comparing token ids per document at capture time.
- The RoPE schedule must be exactly reproducible. Keys carry a position rotation, so
the map is fitted in position-free space: stripped with the source schedule, re-applied
with the target's (not a no-op — the two models often disagree on
rope_theta).default,linearandyarnare implemented; anything else is refused rather than approximated.
check_pair() separates definitional refusals from merely-unproven regimes — mismatched
KV-head counts, differing head dimension, cross-family pairs. The latter are recorded as
flags and decided by measurement, because a gate keyed on "heads must match" would refuse
the very experiment that tests whether heads need to match.
Models such as DeepSeek V2/V3/V4 cache a compressed latent plus a decoupled RoPE key
rather than per-head K and V. That is a different cache layout, not an obstacle — the
latent is dense, and often smaller than a conventional cache for the same context.
KVGeometry.roles() returns ("c", "kr") for those models, and only kr carries a
rotation because the latent is position-free by construction.
The latent is captured at the compression projection rather than from past_key_values:
HuggingFace materialises per-head K/V before caching, so reading the cache would yield
tensors a real serving engine never stores, and a mapper fitted to them could not drive a
production deployment while looking entirely correct end to end.
With the TQ-patched vLLM fork, TurboQuant is a native KV cache dtype. One flag, no hooks, no env vars:
vllm serve your-model --kv-cache-dtype tq-t4ncSupported dtypes:
| Dtype | Bits | Compression | Notes |
|---|---|---|---|
tq-t4nc |
4 | 3.8x | Recommended — best quality/compression tradeoff |
tq-t3nc |
3 | 4.9x | More aggressive |
tq-t35nc |
3.5 | 4.4x | Hybrid split-group quantization |
tq-t2nc |
2 | 7.1x | Maximum compression |
Production Docker example:
services:
vllm:
image: aither-vllm-tq:latest
command: >
python -m vllm.entrypoints.openai.api_server
--model your-model
--kv-cache-dtype tq-t4nc
--gpu-memory-utilization 0.85
--max-model-len 40960
--compilation-config '{"cudagraph_mode":"piecewise","max_cudagraph_capture_size":16}'
--max-num-seqs 16
--enable-prefix-cachingBoundary layer protection: First/last 2 layers auto-use FlashAttn (auto)
to preserve embedding quality and output mixing.
Production results (RTX 5090, Nemotron-8B-AWQ, tq-t4nc):
| Metric | Value |
|---|---|
| KV cache | 250,928 tokens (19.1 GiB) |
| Max concurrency (40K ctx) | 6.1x simultaneous |
| CUDA graphs | 10 piecewise, 6s capture |
| Model load | 6.4 GiB, 15s |
For stock vLLM without the TQ fork:
pip install aither-kvcache[vllm]
vllm serve your-model --attention-backend CUSTOMFor older vLLM or when you need monkey-patching:
import os
os.environ["AITHER_TQ_MODE"] = "tq4-primary"
os.environ["AITHER_TQ_BITS"] = "4"
from aither_kvcache.vllm import apply_tq_patches, apply_tq_hooks
apply_tq_patches(bits=4) # BEFORE vLLM starts
from vllm import LLM
llm = LLM(model="your-model", gpu_memory_utilization=0.90)
apply_tq_hooks() # AFTER model loadsSplit-group quantization with QJL residual encoding:
from aither_kvcache import HybridTurboQuant
htq = HybridTurboQuant(head_dim=128, mode="tq35", device="cuda")
htq.calibrate_uniform()
packed = htq.encode(kv_vectors)
decoded = htq.decode(packed)| Mode | Avg Bits | Strategy |
|---|---|---|
| tq35 | 3.5 | 50% dims @ 4-bit + 50% @ 3-bit (MSE + QJL) |
| tq25 | 2.5 | 25% dims @ 3-bit + 75% @ 2-bit (MSE + QJL) |
All modes register torch.library.custom_op for zero-graph-break decode.
Requires PyTorch 2.4+. Falls back to @torch.compiler.disable on older versions.
If you manage your own KV cache, drop encode() where you write and decode() where you read:
from aither_kvcache import TurboQuant
tq = TurboQuant(head_dim=128, bits=4, device="cuda")
# Write to cache: compress
packed, norms = tq.encode(key_proj) # [batch, heads, 128] -> [batch, heads, 64] uint8
# Read from cache: decompress
key_restored = tq.decode(packed, norms) # -> [batch, heads, 128] float16Works with block-structured caches (like vLLM's). Handles arbitrary batch dimensions:
# Compress a block of 16 tokens across 8 heads
block = cache[block_idx] # [16, 8, 128]
packed, norms = tq.encode(block) # [16, 8, 64] uint8 + [16, 8] f32
restored = tq.decode(packed, norms) # [16, 8, 128]Compute attention directly from compressed data without ever decompressing:
from aither_kvcache.fused_attention import TQPagedAttention
attn = TQPagedAttention(tq, num_query_heads=32)
output = attn.forward(
query, k_packed, k_norms, v_packed, v_norms,
block_tables, context_lens,
)The math: rotate the query forward once, dot-product in the rotated domain against codebook-decoded values, accumulate weighted values in the rotated domain, rotate back once. Two matrix multiplies total regardless of context length.
Uses fused Triton kernels on GPU (Ampere through Blackwell). Falls back to PyTorch reference on CPU.
Set AITHER_TQ_FORCE_TRITON=1 on Blackwell (SM_120) GPUs -- validated on RTX 5090 at 26 tok/s.
tq = TurboQuant(head_dim=128, bits=4)
print(tq.validate(num_vectors=50000))python -m turboquant.benchFor head_dim=128:
| Bits | Bytes/vector | vs FP16 | vs FP8 |
|---|---|---|---|
| 4 | 68 | 3.8x | 1.9x |
| 3 | 52 | 4.9x | 2.5x |
| 2 | 36 | 7.1x | 3.6x |
| Bits | MSE | Theory Lower | Theory Upper | Ratio to LB |
|---|---|---|---|---|
| 4 | 0.0095 | 0.0039 | 0.0184 | 2.4x |
| 3 | 0.0345 | 0.0156 | 0.0736 | 2.2x |
| 2 | 0.1175 | 0.0625 | 0.2945 | 1.9x |
- Normalize: extract L2 norm, project onto unit sphere
- Rotate: multiply by a fixed random orthogonal matrix (data-oblivious). Makes each coordinate ~N(0, 1/d).
- Quantize: each coordinate via precomputed Lloyd-Max codebook
- Pack: indices into uint8 bytes
- Store: packed bytes + float32 norm
Decoding reverses steps 4-1.
class TurboQuant:
def __init__(self, head_dim=128, bits=4, seed=42, device="cuda", ...)
def encode(self, x: Tensor) -> Tuple[Tensor, Tensor]
def decode(self, packed: Tensor, norms: Tensor) -> Tensor
def validate(self, num_vectors=10000) -> dict
def benchmark(self, num_vectors=32768) -> dict
def compression_ratio(self) -> float
def memory_report(self, seq_len, num_layers=32, num_kv_heads=8) -> dict
class HybridTurboQuant:
def __init__(self, head_dim=128, mode="tq35", seed=42, device="cuda")
def calibrate_uniform(self, num_kv_heads=1)
def calibrate(self, sample_vectors: Tensor)
def encode(self, x: Tensor) -> Tensor # packed only (norms embedded)
def decode(self, packed: Tensor) -> Tensor
def validate(self, num_vectors=10000) -> dict
def compression_ratio(self) -> float
@staticmethod
def packed_dim_for_mode(head_dim: int, mode: str) -> int
class TQPagedAttention:
def __init__(self, tq: TurboQuant, num_query_heads: int)
def forward(self, query, k_packed, k_norms, v_packed, v_norms,
block_tables, context_lens, block_size=16) -> TensorStandard KV cache eviction (LRU/FIFO) doesn't know the difference between your system
prompt and throwaway generation tokens. KVCacheGraph builds a relationship graph over
physical KV cache blocks so eviction decisions understand what the blocks actually mean.
from aither_kvcache import KVCacheGraph, GraphEvictionAdvisor, EdgeType
# 1. Create the graph — protect system prompt blocks from eviction
graph = KVCacheGraph(protected_sources={"system", "tools"})
# 2. Register blocks as they enter the KV cache
graph.add_block(0, "system", importance=0.95, token_range=(0, 16))
graph.add_block(1, "system", importance=0.90, token_range=(16, 32))
graph.add_block(2, "user", importance=0.60, token_range=(32, 48))
graph.add_block(3, "assistant", importance=0.30, token_range=(48, 64))
# 3. Feed attention patterns — edges form automatically
graph.on_attention_step([0, 1, 2, 3]) # track co-attendance
graph.on_temporal_sequence([2, 3]) # sequential generation
graph.on_prefix_hit("req_42", [0, 1]) # prefix cache reuse
# 4. Ask who to evict (system blocks are structurally protected)
victims = graph.suggest_eviction(n_blocks=2)
# -> returns least-connected, lowest-importance, non-protected blocks
# 5. Ask what to prefetch from cold tier
graph.on_spill([3]) # block 3 moved to DDR5
prefetch = graph.suggest_prefetch(active_block_idxs=[0, 1, 2])
# -> returns spilled blocks that are graph-neighbors of active setFor hot inference loops where you can't afford graph queries on the decode path:
from aither_kvcache import GraphEvictionAdvisor
advisor = GraphEvictionAdvisor(graph, interval=0.5, max_stale=2.0)
advisor.start() # background thread recomputes rankings every 0.5s
# Hot decode path — lock-free, zero overhead:
candidates = advisor.get_eviction_candidates(n=16) # pre-computed list or None
prefetch = advisor.get_prefetch_candidates([0, 1], n=8) # graph neighbor lookup
advisor.stop()The advisor pre-computes eviction rankings on a background thread. The decode path reads an atomically-swapped reference — no lock, no mutex, no blocking. If the ranking goes stale (>2s), returns None and the caller falls back to FIFO.
The suggest_eviction() method scores every non-protected, non-spilled block:
score = age × 0.01 # older = more evictable
− degree × 5.0 # more graph connections = keep
− edge_weight × 2.0 # stronger edges = keep
− importance × 20.0 # higher importance = keep
− hit_count × 3.0 # more prefix cache hits = keep
Protected source labels are excluded entirely — they cannot be eviction candidates.
| Edge Type | Created By | Meaning |
|---|---|---|
PREFIX_SHARE |
on_prefix_hit() |
Blocks reused across requests |
CO_ATTEND |
on_attention_step() |
Blocks frequently attended together |
SEMANTIC |
add_block(embedding=...) |
Similar key vector embeddings (cosine > 0.8) |
TEMPORAL |
on_temporal_sequence() |
Consecutive in same generation |
SPILL_LINK |
on_spill() / on_warm() |
Hot ↔ cold tier tracking |
The graph has no vLLM dependency. It works with any paged KV cache:
- Call
add_block()when blocks are allocated - Call
remove_block()when blocks are freed - Call
on_attention_step()with active block indices each decode step - Call
suggest_eviction()when you need to free VRAM - Call
suggest_prefetch()to warm cold-tier blocks preemptively
class KVCacheGraph:
def __init__(self, protected_sources={"system"}, coattend_threshold=3,
semantic_threshold=0.8)
def add_block(self, block_idx, source_label, importance, token_range,
embedding=None) -> KVBlockNode
def remove_block(self, block_idx) -> None
def add_edge(self, source, target, edge_type, weight=1.0) -> Optional[KVEdge]
def on_attention_step(self, active_block_idxs: List[int]) -> None
def on_prefix_hit(self, request_id: str, block_idxs: List[int]) -> None
def on_spill(self, block_idxs: List[int]) -> None
def on_warm(self, block_idxs: List[int]) -> None
def on_temporal_sequence(self, block_idxs: List[int]) -> None
def suggest_eviction(self, n_blocks, protect_sources=None) -> List[int]
def suggest_prefetch(self, active_block_idxs, max_suggestions=16) -> List[int]
def neighbors(self, block_idx, edge_type=None, max_depth=1) -> Set[int]
def subgraph(self, block_idxs) -> Dict
def get_stats(self) -> Dict
class GraphEvictionAdvisor:
def __init__(self, graph=None, interval=0.5, max_stale=2.0, eviction_batch=256)
def start(self) -> None
def stop(self) -> None
def get_eviction_candidates(self, n: int) -> Optional[List[int]]
def get_prefetch_candidates(self, active_block_idxs, n=8) -> Optional[List[int]]
def get_stats(self) -> Dict
def reorder_by_ranking(block_indices: List[int], ranked: List[int]) -> List[int]| Model | Layers | KV Heads | FP16 | FP8 | TQ4 (4-bit) | TQ3 (3-bit) | TQ2 (2-bit) |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 4.0 GB | 2.0 GB | 1.1 GB | 0.8 GB | 0.6 GB |
| Mistral 7B v0.3 | 32 | 8 | 4.0 GB | 2.0 GB | 1.1 GB | 0.8 GB | 0.6 GB |
| Qwen2.5 14B | 40 | 8 | 5.0 GB | 2.5 GB | 1.3 GB | 1.0 GB | 0.7 GB |
| Llama 3.1 70B | 80 | 8 | 10.0 GB | 5.0 GB | 2.7 GB | 2.0 GB | 1.4 GB |
| Qwen2.5 72B | 80 | 8 | 10.0 GB | 5.0 GB | 2.7 GB | 2.0 GB | 1.4 GB |
| Context | FP16 | FP8 | TQ4 | TQ3 | TQ2 |
|---|---|---|---|---|---|
| 8K | 1.0 GB | 512 MB | 272 MB | 208 MB | 144 MB |
| 32K | 4.0 GB | 2.0 GB | 1.1 GB | 0.8 GB | 0.6 GB |
| 128K | 16.0 GB | 8.0 GB | 4.3 GB | 3.3 GB | 2.3 GB |
| Integration | Single Request | 5x Concurrent | CUDA Graphs |
|---|---|---|---|
| Hook mode (recommended) | 40 tok/s | 120 tok/s | 7/7 captured |
| Plugin mode (CUSTOM backend) | 23.6 tok/s | 120 tok/s | 7/7 captured |
| Baseline (FP8, no TQ) | 45 tok/s | 130 tok/s | 7/7 captured |
Hook mode reaches ~89% of baseline FP8 throughput while storing 3.8x more KV cache blocks.
Shows maximum tokens that fit in KV cache VRAM after model weights.
| Model | Weights | FP8 | TQ4 | TQ3 | TQ2 |
|---|---|---|---|---|---|
| Llama 3.1 8B (util=0.90) | ~5 GB | 353K | 665K | 869K | 1.26M |
| Qwen2.5 14B (util=0.90) | ~9 GB | 247K | 466K | 609K | 880K |
| Llama 3.1 70B (util=0.90) | ~37 GB | N/A | N/A | N/A | N/A |
70B requires multi-GPU or offloading — KV savings still apply per-GPU.
| Mode | Avg Bits | MSE | Compression vs FP16 | Compression vs FP8 |
|---|---|---|---|---|
| TQ4 | 4.0 | 0.0095 | 3.8x | 1.9x |
| tq35 | 3.5 | 0.0130 | 4.4x | 2.2x |
| TQ3 | 3.0 | 0.0345 | 4.9x | 2.5x |
| tq25 | 2.5 | 0.0520 | 5.8x | 2.9x |
| TQ2 | 2.0 | 0.1175 | 7.1x | 3.6x |
All MSE values within 2.7x of the information-theoretic lower bound (matches paper).
Run python -m aither_kvcache.bench to reproduce on your hardware.
See notebooks/vllm_quickstart.ipynb for a step-by-step
walkthrough covering installation, validation, vLLM integration, and graph-aware eviction.
@article{zandieh2025turboquant,
title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
journal={arXiv preprint arXiv:2504.19874},
year={2025}
}Measured results live in BENCHMARKS.md: kernel numbers, and the
distilled aither-code-embed-0.6b scored against general-purpose embedders on real
documents (p@1 0.893 vs 0.821 for Qwen3-Embedding-0.6B and nomic-embed-text-v1.5, on 26
real document chunks and 28 queries). Rerun the embedder comparison on your own corpus
with awembed compare.
- GitHub Discussions — Questions, ideas, show & tell
- GitHub Issues — Bug reports and feature requests
CC BY 4.0
Native integration via TQ-patched fork. Upstream PR: vllm-project/vllm#39008.
# TQ fork (works now):
vllm serve your-model --kv-cache-dtype tq-t4nc
# Stock vLLM (plugin mode):
pip install aither-kvcache[vllm]
vllm serve your-model --attention-backend CUSTOMThis repo is one piece of a connected set. All public, MIT/BSL-licensed:
| repo | what it is | pages |
|---|---|---|
| awrecover | Labelled snapshots with an all-or-nothing restore | docs |
| awshare | Publish an artifact and fetch it back verified | docs |
| awseal | Sign an artifact so a stranger can verify it | docs |
| awnode | Lightweight local gateway — your apps to backends you chose | docs |
| awnix | A bootable, immutable Linux base for agent-run machines | docs |
| awdk | Build AI agent fleets — 3 lines, any backend | docs |
| awskills | Free agent skills, scripts & automations | docs |
| AitherZero | PowerShell 7+ automation framework | docs |
| awgit | Semantic version control on top of git | docs |
| awgraph | Code knowledge graph for AI agents | docs |
| aitherkvcache | Near-optimal KV cache quantization | docs |
| awrelay | Agent-to-agent messaging over any chat server | docs |
| awm | A small world model (LeWM JEPA + MLP) to bootstrap your own | docs |
| AitherConnect | Browser extension: federated AI search & desktop bridge | — |
| homebrew-tap | brew tap aitherium/tap |
— |
Built by Aitherium.
Standalone tools that share one idea: replace something you would otherwise have to trust with something you can check.
Each installs on its own, works offline, and needs no account.
| instead of trusting | you check | |
|---|---|---|
| awdk | a framework's idea of how your agents should run | one loop you can read, pointed at a backend you already pay for |
| awskills | that an agent knows your procedure | the procedure written down, versioned, and loadable by any agent |
| awpack | that the pack you want shipped inside somebody's SDK, under whatever licence that SDK happens to carry | the pack as its own versioned artifact, with its own licence, that any agent runtime can install |
| awm | that memory stayed in its lane | tenant:user:project scopes, so a write cannot cross a boundary |
| awdesk | that the agent is somewhere behind a browser tab | a tray icon, a face on your desktop, and the decision card that pops when it needs you |
| awnode | a vendor's cloud with every prompt | a local gateway routing to backends you chose |
| awgraph | that grep found everything | an AST + tree-sitter call graph an agent can traverse |
| awgit | that no one else is editing this file | a lease, refused at commit time if you do not hold it |
| awdelphi | one agent's confident take on a decision | the round trace, the anonymity, and who dissents |
| awclassify | a filename, a folder, or whoever last touched it | doc_type, visibility, audience and topics, with the evidence lines that decided each |
| awtoll | that your tooling is saving you context | the measured token cost of each tool call, and what the alternative cost |
| awseal | that the artifact came from who you think | an Ed25519 seal — the key that verifies is not the key that forges |
| awshare | that the download is intact | content-addressed bundles, verified on fetch |
| awnest | that there is a person on the other end | a verdict with evidence, where "we could not tell" is not "yes" |
| awrena | a leaderboard someone can edit, and votes nobody counted | a scored duel with both answers kept, and a result bound to them |
| awnboard | a share link anyone who sees it can use | an invitation addressed to one person, for one gate, revocable |
| awnix | that the box is what you left it as | an immutable image you built, with atomic rollback |
| awrecover | that the restore worked | a restore that fully lands or does not land at all |
| awstorage | a du you ran last month, and a peers file that says 3 TB free | an inventory snapshot per node with a diff since the last one, and each tree classified re-fetchable or not |
| awrelay | a SaaS in the middle of your agents | findings, alerts and coordination over your own transport |
| awask | that anyone read the paragraph where you asked | the ask itself, with a button that steers the session that raised it |
| awmail | a mailbox somebody else can read | mail your agents send and receive over your own server |
| awswarm | that a model either fits your GPU or it doesn't run at all | a placement plan and an acquisition-probability estimate before you spend on a run |
| awfind | one vendor's idea of the web | results from whichever providers you configured |
| awbrowse | that the page said what you were told | the render, the DOM and the requests it made |
| awvoice | that a cloud vendor may hold your audio | a transcript and a wav from a service you host |
| awvision | a filename and a caption somebody wrote | what a model actually reports about the pixels |
| awscreen | a selector that was true when the page was written | the elements actually rendered, by what they look like |
| awbeads | that a layout your users built survives the next deploy | the arrangement as data you can read back, diff, and hand to another surface |
| awbonsai | that inference always means a request left the machine | a WebGPU model answering on the tab's own GPU, with a consent record logged before it ever loaded |
| gawbbonet | the model to keep a 300-message campaign coherent by itself | campaign facts recalled from scoped memory you can list and edit |
| aitherkvcache (you are here) | a vendor's quantisation defaults | sub-byte KV cache kernels you can benchmark yourself |
| awrtifact | a hand-rolled split script and a hand-edited worker manifest | byte-verified parts in a release, served with Range + CORS, sizes asserted by a live gate |
| AitherZero | a pile of scripts nobody has numbered | numbered, discoverable automation with declarative playbooks |
| AitherConnect | what a page tells your browser to do | a federated search and desktop bridge you host |
| awreason | a confident paragraph | the phases it went through, and every tool call it made to get there |
| awrecurse | that everything you pasted in was actually read | which slices it opened, and what it concluded from each |
| awprism | the first explanation that fits | the ranked alternatives, and the observation that separates them |
| awrepl | what the agent believes the value is | the value, printed from the live session |
| awreport | that the report you pasted carried no token in it | a redacted report, and the duplicate it merged into instead of filing twice |
| awresearch | a summary of pages nobody opened | every claim against the source it came from |
| awfocus | twelve terminal tabs and a bad memory | one command that names every session, finds any transcript, and opens or steers the one you want |
| awgym | that a world model learned anything from the games it saw | transitions captured from real play, fed back, and the retrodiction score falling on grids it never saw |
| awpredict | a model because it trained without erroring | its prediction against a self-updating lookup, on the rows that are actually novel |
| awevolve | that your optimisation loop is finding anything | every version it kept, the score that version earned, and the edit that produced it |
| awsh | that you already know the name of the command | what it decided your line meant, before it acts on it |
| awrise | that a scheduled agent ran at all, and ran exactly once | a durable record of every wake -- fired, skipped, overlapped or timed out -- each with its reason |
| awkno | that the docs site is up, or that you remember the family | the whole ecosystem in your terminal, with no network at all |
| awwall | that a service only talks to the hosts you think it talks to | an explicit egress allowlist, where a denial names the rule that denied it |
| awembed | a general-purpose embedder that has never seen your code | a held-out split of whole directories, scored teacher vs student vs int8 |
| awtax | a closed tax app's sealed file you can never read again | a plain, provider-neutral schema of every figure, with the page it came from |
| awsettings | that you will remember to re-approve the same thing on every box you work from | one profile, unioned rather than overwritten, with the credentials left behind |
| awavatar | a cloud 3D vendor's opaque task id | a manifest with a sha256, a licence and a rig-audit verdict per file |
awnix is the ground floor — A Linux you can hand to an agent — immutable base, capabilities included.
Every repository here is public. Each publishes an aither-manifest.json beside its page, so any surface can read every sibling's — the network is browsable from any node in it.
| repo | what it is | pages |
|---|---|---|
| awdk | Build AI agent fleets — 3 lines, any backend, local or cloud | docs |
| awskills | Portable agent skills — self-contained procedures an agent loads on demand | docs |
| awpack | First-party agent packs — the ones we build, versioned and installable on their own | docs |
| awm | A portable, scoped agent memory | docs |
| awdesk | Aither World Desk -- the desktop body of AitherOS Online: tray, avatars, decision cards, the Living Desktop as an overlay | docs |
| awnode | A lightweight local gateway — bridges your apps to the AI backends you chose | docs |
| awrun | A priority-aware queue and dispatcher for agentic runs and ad-hoc CI builds. It also judges whether the runner pool is big enough for the queue it is draining, and can ask a host to grow it -- reserving capacity is zero-sum, so a saturated pool needs more of it, not a different share of it | docs |
| awgraph | A semantic code graph for agents — AST + tree-sitter, call graphs | docs |
| awgit | Semantic version control on top of git — edit-ops and leases | docs |
| awdelphi | Anonymous multi-round expert panels — a converged answer with a trace | docs |
| awclassify | Classify any document -- what it is, who may read it, who it is for, what it is about | — |
| awtoll | What every tool call costs you in context, measured from your own transcripts | docs |
| awseal | Sign an artifact so a stranger can verify it | docs |
| awshare | Publish an artifact and fetch it back verified | docs |
| awdit | An append-only audit trail whose gaps are DETECTABLE | docs |
| awbac | Role-based access control that fails closed and explains itself | docs |
| awiam | Who is this caller? A directory and session store that fails honestly | docs |
| awtunnel | Reach a service that has no public address | docs |
| awnest | Prove there is a human before you let them into the nest | docs |
| awrena | Put two agents head to head and get a verdict you can check | docs |
| awnboard | A front gate you can put in front of anything, and hand someone the key to | docs |
| awnix | A Linux you can hand to an agent — immutable base, capabilities included | docs |
| awrecover | Labelled snapshots with an all-or-nothing restore | docs |
| awstorage | Every drive on every node, indexed, classified and diffed -- so you can see what you own before you delete it | docs |
| awrelay | Portable agent messaging — findings, alerts, coordination | docs |
| awask | Your agent asks you a question — and acts on your answer | docs |
| awmail | Give an agent an email address — send, and actually receive | docs |
| awnet | The agentic web — agents host a mesh, and agents join one | docs |
| awswarm | Run one model too big for any single GPU across a pool of small ones | — |
| awfind | A portable search client — query, results, ranking | docs |
| awbrowse | A portable browser client — navigate, console, network, DOM, screenshot | docs |
| awvoice | Hear and speak — transcribe audio, synthesize a voice | docs |
| awvision | See an image — describe it, ask it a question, compare two | docs |
| awscreen | See this machine — what is on screen, and where to click it | docs |
| awkit | Render an agent panel from a tool result — one component, any React app | — |
| awbeads | A spatial canvas for a page — arrange things, connect them, and keep the arrangement | — |
| awbonsai | Run a real model in the visitor's own browser — no server round trip, no upload | — |
| awknowledge | How to run a coding agent so the result survives — the laws, with evidence | docs |
| awbrain | Your history as a wiki of linked markdown — claims pinned to the evidence | — |
| gawbbonet | GobboNet campaigns with a real agent brain — scoped memory, graph recall | docs |
| aitherkvcache (you are here) | Near-optimal KV cache quantization for LLM inference — sub-byte compression | docs |
| awrtifact | Deliberately chunk artifacts into GitHub release assets — the productized aitherkvcache mirror lane | docs |
| AitherZero | PowerShell 7+ automation framework — numbered, self-describing scripts | docs |
| AitherConnect | Browser extension — federated AI search, page context, and the Living OS overlay | docs |
| awreason | A portable reasoning client — sessions, phases, thoughts, and the chain that produced the answer | docs |
| awrecurse | Answer a question over a context far larger than the window — recursively, with the trace kept | docs |
| awprism | Turn a failure into ranked hypotheses — and say what would confirm each one | docs |
| awrepl | A REPL an agent can actually use — state that survives between turns | docs |
| awreport | File a bug report that has already scrubbed your secrets and collapsed the duplicate | — |
| awresearch | Ask a research question, get a cited report you can check | docs |
| awfocus | See, search and steer every Claude session from one command | docs |
| awgym | An ARC training gym — a game a world model can watch, and six roles that play through it | docs |
| awpredict | Predict what your environment does next, and how surprised you were | docs |
| awevolve | Point an agent at a file and a command that scores it, and let it improve | — |
| awsh | Your terminal answers you -- type a question where a command would go | docs |
| awrise | Wake an agent on a schedule, let it do one thing, and put it back to sleep | docs |
| awkno | The man page for the Aither World — every brick, stack and law, offline | docs |
| awwall | Say what a workload may reach, and watch everything else fail closed | docs |
| awrouter | OpenRouter for your own fleet: pick a model backend by cost/latency/ capability, fail over, fit the context window, stream. Standalone, OpenAI-compatible, no Aither-specifics required to be valuable | — |
| awembed | Train an embedding model that knows your corpus, and prove it beats the big one | docs |
| awtax | Turn any tax PDF -- returns, W-2, 1099, statements, even scans -- into structured data you can check | docs |
| awflow | A deterministic workflow runtime — chain agent calls with journal replay and budget control | docs |
| awsettings | Your agent's permissions and config, following you to the next machine | docs |
| awavatar | One character spec in, a rigged, animated, multi-style avatar pack out | docs |