liminallm

liminallm is an experiment in what a chatgpt-like system looks like if you stop hard-coding product logic and let the model help evolve itself.

the core bet: small models, deeply adapted. a small self-hosted model with behavior baked into lora weights beats a small model begging through a long system prompt — weights survive context pressure, free the window for the user's actual content, and cost nothing per token. a frontier model can help as an offline teacher, but inference never depends on one.

it’s a small kernel wrapped around:

• a frozen base llm (jax — the primary training and serving framework)
• per-user persona adapters + per-skill lora adapters trained on pooled cluster data
• the adapter ladder: skills are born as prompts and only earn weights when the data justifies it — and an eval gate agrees
• emergent “skills” from clusters + preference events
• self-describing artifacts (workflows, routing policies, tools)
• notebooklm-style grounding over filesystem-backed files
• boring infra: postgres + redis + filesystem
• artifacts and adapter payloads live as JSON/weights on the shared filesystem

the code is just the glue. everything interesting lives as data.

---

what it does (conceptually)

feedback loop (at a glance)

User Feedback → Embeddings → Clustering → Skill Discovery
     ↑                                            ↓
Router Updates ← Eval Gate ← Adapter Training ← Prompt-Mode Skill

• chatgpt-like web ui

  • multi-user, password + pluggable auth
  • conversations, history, summaries
  • text first; voice later

• deep behavioral memory (the adapter ladder)

  • per-user persona adapters (lora): small, low-stakes — tone and format
  • skill adapters born from usage: “when problems like this show up, start with this debugging workflow”
  • every skill starts as a prompt (instructions distilled from cluster labels + highly-rated exemplars) — useful immediately on any backend
  • once a cluster pools enough positive feedback across users, a jax training job runs; one user’s thumbs are too sparse to train weights on
  • trained weights only ship if a holdout eval gate measures real improvement; a failed gate leaves the skill on the prompt rung. nothing regresses.
  • passing the gate is the only thing that makes weights servable: the adapter's promoted version number is the authority, and serving reads exactly that version's weights. a file on disk, a latest pointer, the newest directory — none of them mean an adapter graduated.
  • a graduated skill speaks once, not twice: where its weights apply it is carried by them, and where they cannot (an api backend) its prompt carries it instead
  • optionally, a teacher model distills raw chat transcripts into clean training exemplars first
  • continuous micro-training jobs in jax, only on adapters, never on the base model

• natural factual memory

  • user files in the filesystem (/users/{id}/files)
  • ingestion → chunking → embeddings in postgres (pgvector)
  • notebooklm-style: bind “contexts” (collections of files/folders) to a chat and ask questions grounded in that corpus

• a notes vault with a witness

  • notes link to each other with [[title]]; links become a graph you can see
  • the model can search your vault mid-chat (note_search) and cite what you once wrote
  • the witness puts two dated notes side by side and asks how they relate — agrees, contradicts, or the position quietly moved. contradiction isn’t the goal; it’s one honest result of the process
  • when a position has moved, the report shows the trail: the chain of links between the two thoughts, with dates
  • a vault-wide sweep runs the same process over the strongest pairs across everything you’ve written
  • uploaded files stay chat-scoped by default; a file joins the vault only when you promote it (one click), because permanent cross-chat memory should be a decision, not a side effect
  • promoted pdfs and images get fleeced for content: text layer → pypdf, images and scans → ocr, then model vision. install tesseract-ocr + pip install 'liminallm[ocr]' — technically optional, practically required

• context that fits the model you actually run

  • the prompt budget comes from the serving model's real window — asked of the provider (gemini and vllm both report it), else a known-family table, else a conservative default; model_context_window overrides when discovery guesses wrong
  • recent turns go verbatim; older ones are folded into a rolling digest kept on the conversation, so a long chat degrades to “remembers less precisely” instead of “forgets entirely”
  • the digest is written off the hot path and never blocks a reply; the window is the same whether redis is up or down

• an openai-compatible responses api for agents

  • POST /v1/responses speaks the responses dialect, so any agent framework can point its base url here and get the kernel's whole enrichment stack — personas, skill adapters, hybrid rag, notes, memory — behind what looks like a plain model endpoint. a weak local model plus this kernel presents as a much richer model; the caller changes nothing but the base url.
  • stateful: previous_response_id continues the conversation server-side; pass context_id (a liminallm extension) on the first turn to ground the whole thread in a knowledge context
  • streaming: stream: true returns sse response.* events (created → tool items as they run → text deltas → completed), with the reply's id stable from the first event to the persisted message
  • serves what the turn learned, not just the text: server-side searches appear as file_search_call/web_search_call output items; grounding snippets, the full tool trace and active adapters ride under a namespaced liminallm key; usage includes reasoning/cached token details when the upstream reports them, and real totals from our own tokenizer on the local jax path
  • auth via api keys (sk-liminal-…): mint, list, and revoke from the settings tab in the web ui, or at /v1/auth/api-keys with a logged-in session. keys are valid only on the agent surfaces (/v1/responses, /v1/mcp) — a leaked key can chat and search and nothing else, and in particular cannot mint or revoke keys. only a sha-256 of the key is stored; the plaintext is shown exactly once, at mint time.
  • agent conversations appear in the web ui like any other chat, tagged “api” in the sidebar
  • the kernel's internal tool loop (retrieval, notes, the reranker's out-of-band scoring) rides the provider tool-call transport wherever one exists — including the local jax backend via its advertised <tool_call> channel — so agents get the same grounded answers on every backend

• an mcp server for everyone else's agents

  • POST /v1/mcp speaks the model context protocol (streamable http, revision 2025-06-18): initialize, list tools, call tools — stateless, json responses, batching rejected as the spec now requires
  • two tools, both read-only, both the kernel's own retrieval: note_search over the notes vault and knowledge_search over knowledge contexts, the exact services the internal agent loop uses
  • read-only is the point: nothing here can carry data off the box, so an injected document has no egress to abuse, and every result names its own text as document content — not instructions
  • same api keys as the responses api; the roadmap (resources, prompts, oauth, and an mcp client under the kernel's taint discipline) lives in the spec so growth is a decision, not drift

• small kernel, big data

  • kernel only knows how to:
    • auth users
    • run workflows (graphs)
    • run routing policies
    • call the llm with optional lora adapters
    • talk to postgres / redis / filesystem
  • everything else (domains, skills, behaviors, tools, routing rules) is expressed as artifacts:
    • adapter.lora
    • workflow.chat
    • policy.routing
    • tool.spec
    • context.knowledge
    • etc.

• emergent domains & skills

  • no hard-coded DEBUGGING, WRITING, whatever
  • we cluster preference events in embedding space
  • llm labels clusters (“kernel panic debugging”, “multi-tenant billing schema design”, …)
  • when a cluster is big + consistently positive, we auto-create a prompt-mode skill adapter tied to that cluster — weights come later, gated on data volume and a passing eval

• router as data, not code

  • routing policies are artifacts (policy.routing) with a tiny expression language:
    • conditions over embeddings, clusters, safety flags
    • actions: activate/deactivate adapters, scale weights, etc.
  • the router engine is dumb and stable; policy is editable data
  • a gate is an activation first and a strength second: weight 0 means the adapter is absent from the turn — no weights, no prompt, nothing sent to a provider, nothing in the kv cache key, and nothing claimed in what the turn reports it used. above zero it scales where scaling is defined; prompt text has no half-measure, so it goes in once, unchanged.

• llm as architect (under guardrails)

  • a config-ops api lets the llm propose patches to:
    • routing policies
    • workflows
    • adapter metadata
  • patches are stored, validated, can be auto- or human-approved, and are fully versioned

---

architecture (stack)

• language / runtime

  • python (services, api, orchestration)
  • jax + optax (base model, lora training, eval gates) — install with pip install -e ".[train]"
  • the local serving path is a real plain-jax decoder — rmsnorm, rope, grouped-query attention with a kv cache, swiglu — loading config.json + *.safetensors straight from the model directory (no torch, no flax). incremental decode is tested to reproduce a full recompute, and a lora adapter at B=0 is tested to change nothing. with no checkpoint on disk it falls back to a synthetic stand-in and says so in the log — that path moves tokens, it does not answer questions. training uses that same forward pass, so an adapter is fitted to the model that will serve it: the loss is taken over the real decoder with the lora matrices inside its attention projections, and weights only load onto the base they declare.
  • conversations reuse their own kv prefix across turns (content-addressed, adapter-keyed, strict-prefix only), so the reused prefill shows up honestly as cached_tokens in usage
  • remote multi-lora servers (lorax / vllm-style, openai-compatible) as an optional scale-out serving path; same artifacts, config change only

• storage

  • postgres
    • users, auth, conversations, messages
    • artifacts & versions
    • semantic clusters
    • knowledge chunks (with pgvector)
    • preference events, training jobs, router state
  • redis
    • sessions
    • rate limiting
    • hot conversation summaries
    • router and workflow scratch state
  • filesystem
    • /shared/models – frozen base model weights
    • /users/{id}/files – user docs
    • /users/{id}/adapters – per-user lora weight files
    • /users/{id}/artifacts – generated notebooks, exports, etc.

• services (logically)

  • auth service
  • chat orchestrator
  • artifact service
  • workflow engine
  • router service
  • llm inference (jax + lora)
  • knowledge / rag service
  • preference + training service
  • clusterer + skill discovery
  • configops (patch proposals / approvals)

for v1 these can all live in one python app with clear module boundaries.

---

current status

• early design / prototyping
  • do not treat as production-ready
  • interfaces & schemas are expected to change
• the training loop is real: jax + optax lora training with causal-lm sft batches, holdout eval, and a promotion gate — a skipped or regressed run never ships weights
• skill adapters follow the ladder end-to-end: prompt-mode birth → pooled-data training job → eval-gated graduation to hybrid
• goal is to keep:
  • implementation minimal
  • all "product behavior" in data (artifacts / policies / workflows)
  • evolution driven by usage + llm suggestions, not constant code surgery

---

quick start

See INSTALL.md — Docker on Linux, Linux without Docker, or OpenBSD.

acceptance criteria (ready to test)

Before QA begins, verify:

Criterion	How to Verify
Health check	curl http://localhost:8000/healthz returns {"status": "healthy"}
Chat UI loads	Open http://localhost:8000/ in browser
User signup	Sign up via UI or POST /v1/auth/signup
User login	Log in via UI or POST /v1/auth/login
Send message	Create conversation and send via /v1/chat
Admin protected	Regular user gets 403 on /v1/admin/settings
Admin access	Admin user gets 200 on /v1/admin/settings
Tests pass	./scripts/run_tests.sh passes on fresh install
Bootstrap works	python scripts/bootstrap_admin.py creates admin

Run the automated smoke test:

./scripts/smoke_test.sh http://localhost:8000

---

deployment

• Installation in INSTALL.md; operations and backend lanes in docs/DEPLOYMENT.md
• Configuration architecture documented in docs/CONFIGURATION.md
• Testing guide in TESTING.md

implementation completeness (prototype)

• implemented
  • file upload endpoint writing to the shared filesystem and ingesting chunks into RAG contexts with configurable chunk sizes; default retrieval runs against pgvector with shared deterministic embeddings (optional in-process hybrid fallback for dev/test)
  • workflow execution with branching/parallel scheduling across workflow.chat graphs
  • router policies with a sandboxed evaluation engine (limited adapter gating usage)
  • pluggable model backend that can target external API fine-tune IDs or local JAX+LoRA adapter application
  • filesystem-backed LoRA adapter training that turns preference events into new adapter versions
  • preference capture with clustering + skill adapter promotion and routing integration
  • hardened auth + multi-tenant isolation: OAuth provider mapping, session revocation on password resets, error envelopes with stable error.code, ownership-enforced artifact and conversation access (including workflows/tools), adapter checksum + path validation, and email verification flows
• MFA with TOTP enrollment (otpauth URL), session gating, and login verification
• email verification tokens with /v1/auth/request_email_verification and /v1/auth/verify_email
• tenant-scoped conversation history enforcement in workflows and tool invocations
• HMAC-signed JWT access tokens with refresh rotation, tenant-aware sessions, and admin-only config endpoints
• preference UI and rich routing feedback loop
• LLM-as-architect auto-patch generation
• voice interface
• admin UI for patch approval
• chat and admin frontends prompt for MFA codes when required and revoke sessions on logout

---

getting started (high level)

note: this is intentionally vague; exact commands depend on how you wire the codebase.

1. bring your infra
   • postgres (with pgvector installed)
   • redis
   • filesystem path accessible to the app

• gpu / tpu for jax model if you expect to train adapters
• backend selection is single-sourced from the SQL deployment config (editable from the web console when wired); env vars only override if you set them explicitly
• set model_backend to local_gpu_lora in the admin console to target the local JAX+LoRA path instead of external API fine-tune IDs; leave the default to use the OpenAI-style plug. The JAX backend (LocalJaxLoRABackend in liminallm/service/model_backend.py) loads adapters from the filesystem, tokenizes prompts, runs a JAX forward pass, and enforces conservative shapes; it requires a JAX runtime and optionally a Transformers tokenizer for decode parity. Provider keys are admin settings, with <PROVIDER>_API_KEY as an environment fallback.

frontend (chat + admin)

• A minimal, ChatGPT-style UI now lives in /frontend and is served by the FastAPI app at / with static assets mounted at /static/*.
• Authenticate with /v1/auth/login; the UI stores the issued bearer token/tenant ID locally and uses it for /v1/chat, /v1/conversations, and other API calls.
• The admin console is separate at /admin and is guarded by the admin role (FastAPI enforces the role before serving the HTML). It surfaces config patch proposal/approval flows backed by /v1/config/* endpoints, tenant-scoped user administration (list/add/delete, role changes), adapter visibility, and a read-only inspector for database objects.

tests

• Run scripts/run_tests.sh to mirror CI defaults; it compiles the code and executes pytest. the suite spins up a throwaway postgres cluster and a throwaway redis (tests/harness.py) and applies sql/schema.sql, so tests exercise the same store and cache production runs — set TEST_DATABASE_URL / TEST_REDIS_URL to point at existing services instead.

adapters: local LoRA vs remote fine-tune IDs vs prompt-distilled

• Router policies pick an adapter; the inference backend decides whether that means applying LoRA weights locally, swapping to a remote fine-tuned model ID, or injecting distilled prompt instructions on top of a black-box API.

• Each adapter.lora artifact carries a backend field describing where inference happens:

  {
    "kind": "adapter.lora.remote",
    "provider": "zhipu",
    "backend": "api",
    "base_model": "glm-4-air",
    "remote_model_id": "glm-4-air-ft-2025-11-01-u123-debug",
    "region": "cn-beijing",
    "cluster_id": "…",
    "applicability": {
      "natural_language": "u123: kernel panic debugging skill on GLM-4-Air",
      "embedding_centroid": []
    }
  }

  {
    "kind": "adapter.lora.local",
    "backend": "local",
    "provider": "aliyun",
    "base_model": "qwen2.5-32b-instruct",
    "cephfs_dir": "/users/u123/adapters/{id}",
    "rank": 8,
    "layers": [0, 1, 2, 3],
    "matrices": ["attn_q", "attn_v"],
    "cluster_id": "…"
  }

  {
    "kind": "adapter.lora.prompt",
    "backend": "prompt",
    "provider": "api_only",
    "base_model": "glm-4-air",
    "prompt_instructions": "for kernel issues: reproduce → bisect → log inspection; keep replies terse",
    "cluster_id": "…",
    "applicability": {
      "natural_language": "prompt-distilled skill for kernel debugging",
      "embedding_centroid": []
    }
  }

• Remote adapters send requests to OpenAI-compatible fine-tuned model IDs (e.g., Zhipu BigModel or Alibaba DashScope). Local adapters resolve to filesystem-backed LoRA weights and are composable. Prompt-distilled adapters inject behavior as system messages without changing model IDs so you can still steer API-only providers.

• “Model-ID adapters” (fine-tuned endpoints) map 1:1 to model strings on providers like OpenAI/Azure (fine-tuned deployments), Vertex AI Gemini, or Bedrock custom models. Switching behavior = switching the model string; composition happens at routing time, not inside a single call.

• “Adapter-ID adapters” (multi-LoRA / adapter servers) surface adapter_id parameters on Together AI Serverless Multi-LoRA, LoRAX-style servers, or SageMaker adapter inference components. The backend keeps the base model string and passes adapter_id for one-or-more adapters per request when supported.

• Hybrid patterns (local adapter-enabled “controller” + external API “executor”) flow through the same artifacts: the controller uses a local LoRA backend to plan, then the API backend executes with prompt or remote-model adapters.

2. configure env — one variable

   • DATABASE_URL – postgres dsn. that is the configuration.

   four others exist and none of them are settings you tune: BUILD_SHA (stamped by the build), TEST_MODE (the test harness), EMBEDDING_VECTOR_DIM (a property of the schema you applied, shared with scripts/migrate.sh), and EXTRACT_READER_PLUGINS (imports python modules, so making it settable from a web form would mean remote code execution).

   everything else — the model, credentials, rate limits, ttls, cors, smtp, the signing key — lives in the database and is edited from the admin console at /admin.html, applied to every replica without a restart. changing an smtp password should not require redeploying the app. for declarative deploys, seed on first boot with INSTANCE_SETTINGS_JSON='{"model_backend": "stub"}'.

3. migrate db

   • run the alembic / migration tool to create tables described in the spec.
   • if you ran earlier builds, delete ${SHARED_FS_ROOT}/state/training_pg.json after upgrading to purge legacy MFA secrets (secrets are now sourced solely from the user_mfa_secret table).

4a. preference_event → adapter dataset → tokenized batches

• preference_event rows (positive feedback) capture context_embedding, score, and optional context_text; they are clustered per-user to build adapter personas.
• the training service reconstructs prompts from recent messages, appends any provided context snippet, and uses corrected text as targets while tracking cluster centroids.
• dataset rows are written to ${SHARED_FS_ROOT}/users/{user_id}/adapters/{adapter_id}/jobs/{job_id}/dataset.jsonl.
• tokenized batches carry shapes for the downstream JAX/Optax loop (padding + masks, no base-model update), and training metadata records batch shapes + cluster summaries.
• adapter metadata and params are stored under ${SHARED_FS_ROOT}/users/{user_id}/adapters/{adapter_id}/v####/.

4. start services

   • run the api server (http + websocket for streaming)
   • run a background worker for:
     • ingestion / embeddings
     • clustering
     • adapter training
     • configops patch application

5. open the web ui

   • sign up / log in
   • create a conversation
   • upload a few files, create a knowledge context, and attach it to a chat
   • start talking to see basic chat + rag behavior
   • enable preference capture + adapters once that’s wired

---

roadmap (rough)

• minimal chat with postgres-backed conversations
• file upload + filesystem + rag over pgvector chunks
• artifacts for workflows + tools (no adapters yet)
• preference events + single persona adapter per user
• semantic clustering + skill adapters
• router policies as data + simple editor
• configops api + llm-generated patches
• mobile / voice clients (optional layer)

---

license

MIT

testing

See TESTING.md for comprehensive testing documentation.

# The suite starts its own throwaway Postgres; no setup needed
./scripts/run_tests.sh

# Full integration test with Docker
docker compose -f docker-compose.test.yml up --build
./scripts/smoke_test.sh

API endpoints

Key endpoints (Bearer access token required):

• POST /v1/auth/signup → returns session + signed access/refresh tokens
• POST /v1/auth/login → returns tokens, with MFA gating when enabled
• POST /v1/auth/refresh → rotates refresh tokens
• POST /v1/chat → creates conversation + LLM reply
• POST /v1/responses → the same turn in OpenAI's Responses shape, for agents (api key or session auth; stream: true for SSE)
• POST /v1/mcp → MCP server (note_search + knowledge_search) for MCP-speaking agents
• POST /v1/auth/api-keys → mint an agent-surface api key (list with GET, revoke with DELETE /v1/auth/api-keys/{id})
• GET /v1/artifacts → lists data-driven workflows/policies
• GET /v1/admin/settings → admin-only system settings

Admin endpoints (/v1/admin/*, /v1/config/*) require admin role.

operational hardening

• local rate limits now fall back to in-process counters when Redis is unavailable (TEST_MODE), covering auth and chat flows
• uploads are capped by max_upload_bytes to prevent unbounded in-memory reads