LARA

Lightweight Additive Residual Adaptation: post-training for frozen language models, with small adaptations that can be combined as a Mixture of Behaviors (MoBs).

---

Paper · Slides · Video walkthrough · Demo

---

Contents

• What is LARA?
• Why LARA?
• How it works
• Install
• Train a behavior
• Turn the adaptation up or down
• Mixture of Behaviors (MoBs)
• Routing
• Small model, specialized behavior
• What a behavior looks like on disk
• API
• Examples
• Tests
• Notebooks
• Results
• Deep dive
• Citation
• License

What is LARA?

LARA adapts a frozen language model without modifying its weights, producing a small behavior artifact (a few MB) that sits on top of the base model. Multiple behaviors can be loaded simultaneously, mixed, scaled, and routed per token.

capability        = base model
behavior          = learned adaptation
selection         = router
strength          = runtime scaling
combination       = composition

Because the base weights remain frozen, LARA is naturally suited to continuous learning. New behaviors can be added over time without interfering with existing ones, effectively eliminating catastrophic forgetting. The same residual stream mechanism also provides a straightforward path for multimodal adaptation: learned corrections can steer a frozen text model to align with visual or audio token representations without retraining the core model.

📖 The bigger picture

Foundation models made a different approach to AI development possible: build a general model first, then tailor it for particular purposes afterwards. LARA is a post-training method for that second step.

What it adds over a weight-space adapter is a scale you can turn at inference, and the ability to hold many behaviors resident on one base and pick between them per token.

The larger system built around LARA is Mixture of Behaviors (MoBs). A MoB is a collection of independently trained behaviors sharing one frozen base. Behaviors can be trained with SFT, DPO, GRPO or other post-training methods. At inference they can be selected, scaled, pinned or composed, with hard or soft routing.

This turns post-training into a modular layer around a foundation model.

Why LARA?

LoRA showed that useful adaptation does not require updating every parameter. LARA places the correction in the residual stream rather than in the weight matrices, making behaviors independent, composable, and runtime-selectable.

	LoRA	LARA
Where it acts	Weight matrices	Residual stream
Can blend multiple adapters per token?	No (must merge weights)	Yes (weighted sum)
Adapter size (SFT)	~2.2M parameters	~2.4M parameters
Adapter size (DPO/GRPO, 1 LARA layer)	~2.2M parameters	~0.4M parameters
Runtime scaling	Fixed at merge time	Adjustable via gamma

Note: Qwen3-1.7B has 28 total layers. LARA inserts adapters at a subset of them (e.g., 6 evenly spaced for SFT, or a single middle layer for DPO/GRPO).

How it works

h = h + gamma * (alpha / rank) * up(down(layer_norm(h)))

down projects to rank r, up projects back. up starts at zero, so an untrained adapter contributes nothing. At rank 128 over six layers of a 1.5B model: ~2.4M trainable parameters, a few MB on disk.

⚙️ Technical details

The base model is loaded once and stays frozen. Each behavior is a separate file, typically a few megabytes. That makes it practical to keep several behaviors on the same model rather than producing a full model for every specialization.

Because no weights are modified, behaviors compose. A small router reads the frozen hidden state and produces a per-token distribution over the behaviors in the bank, and their corrections are blended by weight. There is no limit to how many you install.

Install

pip install git+https://github.com/pfekin/LARA.git

Or from a clone:

git clone https://github.com/pfekin/LARA.git
cd LARA
pip install -e .

Python 3.9+, torch 2.0+. Examples also need transformers, datasets, accelerate and trl:

pip install -e ".[examples]"

Train a behavior

from transformers import AutoModelForCausalLM, Trainer, TrainingArguments
from lara import LARA

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")

lara = LARA(model, layers=6, rank=128)     # base frozen, 2.4M trainable
print(lara.num_trainable())

trainer = Trainer(model=model, args=TrainingArguments(...), train_dataset=ds)
trainer.train()

lara.save("behaviors/code", route_samples=prompts[:200])

layers=6 spreads six adapters evenly over the depth. Pass a list for exact control. LARA works with HF Trainer, TRL's DPOTrainer, GRPOTrainer, or custom loops.

Turn the adaptation up or down

lara.gamma = 0.0     # the frozen base
lara.gamma = 0.5     # halfway
lara.gamma = 1.0     # the trained behavior

Useful range: ~0 to 2.0.

Mixture of Behaviors (MoBs)

A MoB is a bank of behaviors sharing one frozen base, with a router that selects or combines them per token.

from lara import Bank

bank = Bank(model, tokenizer)
bank.add("code",   "behaviors/code")
bank.add("math",   "behaviors/math")
bank.add("polite", "behaviors/polite")
bank.fit_router()

out = model.generate(**inputs)            # routing happens inside the forward pass

_{Five behaviors on Qwen3-1.7B. Sizes vary with rank and layer count.}

Add behaviors incrementally:

bank.add("legal", "behaviors/legal")
bank.fit_router(mode="append")            # keeps existing rows, fits the new one

Save and reload:

bank.save("mybank/")
bank = Bank.load("mybank/", model, tokenizer)

Routing

bank.top_k = None    # blend all of them by weight (default)
bank.top_k = 2       # blend the two highest
bank.top_k = 1       # hard selection, one behavior per token

Override the router:

with bank.pin("code"):                        # one behavior
    out = model.generate(**inputs)

with bank.pin({"code": 1.0, "polite": 0.4}):  # custom blend
    out = model.generate(**inputs)

with bank.disabled():                         # frozen base
    out = model.generate(**inputs)

Per-behavior scales:

bank.set_gamma("polite", 0.6)

Inspect routing:

bank.route_weights(input_ids)
# {'code': 0.71, 'math': 0.04, 'polite': 0.25}

Small model, specialized behavior

Prompt: Name a common over-the-counter pain reliever that reduces fever but does NOT increase bleeding risk.

	Without medical behavior	With medical behavior
Qwen3‑1.7B output	"... ibuprofen."	"Acetaminophen (Tylenol) ..."
Correct?	❌	✅

A lightweight learned behavior corrects a domain-specific distinction without changing the base model. Measurements across five behaviors and four base models are in research.md.

What a behavior looks like on disk

behaviors/code/
  adapter.safetensors     # the projections, a few MB
  config.json             # layers, rank, alpha, base model id
  route_samples.jsonl     # short texts for routing

API

LARA(model, layers=6, rank=128, alpha=128)

Method	Description
lara.gamma	Scale applied at inference
lara.num_trainable()	Parameter count
lara.save(path, route_samples=..., method=...)	Write the behavior
LARA.from_pretrained(model, path)	Load a saved behavior
lara.disabled()	Context manager: frozen base
lara.detach()	Remove adapters and hooks

Bank(model, tokenizer, top_k=None)

Method	Description
bank.add(name, path)	Add a behavior
bank.remove(name)	Drop one
bank.fit_router(mode="refit" | "append", steps=300)	Train the router
bank.top_k	How many behaviors apply per token
bank.pin(name_or_dict)	Context manager: override router
bank.disabled()	Context manager: frozen base
bank.gamma, bank.set_gamma(name, g)	Per-behavior scales
bank.route_weights(input_ids)	Mean routing weight per behavior
bank.save(path), Bank.load(path, model, tokenizer)	Save/load the bank

Examples

Runnable end to end on one GPU:

• examples/01_finetune.py — HF Trainer
• examples/02_dpo.py — TRL's DPOTrainer
• examples/03_bank.py — loads both, fits router, generates under routing

Tests

python tests/test_lara.py

Runs without downloading a model.

Results

For SFT, LARA matches LoRA at equal parameter counts. For DPO and GRPO, a single adapter in the middle of the network is enough. A much smaller artifact than a comparable LoRA. Behaviors transfer across base models and across quantization, down to binary weights.

Full benchmarks, gamma sweeps, and routing tables are in research.md, alongside the benchmark code and instructions to rerun it.

Deep dive

🧠 Why MoBs matter

With conventional fine-tuning, specialization tends to produce another model:

base model
   │
   ├── fine-tune → model A
   ├── fine-tune → model B
   └── fine-tune → model C

With LARA and MoBs the base capability is shared and the learned behaviors stay independent. This becomes more useful as the number of adaptations grows: a bank of behaviors occupies megabytes of adapter storage rather than one multi-gigabyte model per specialization.

Behaviors also need not be mutually exclusive. A system can combine them at inference: a mathematical behavior together with a tutoring behavior and a preferred writing style, without training a full model for that combination.

🧩 Modular cognition

The MoB architecture can be read as a form of modular cognition. Different learned behaviors encode different ways of using the same underlying model:

base capability
      │
      ├── planner
      ├── verifier
      ├── critic
      ├── domain specialist
      ├── tutor
      └── personal style

A router can select among them or combine them.

The analogy with mixture-of-experts is useful, but the target is different. A conventional MoE routes among experts that are part of one model, mainly to increase capacity. MoBs route among lightweight adaptations over a shared base, to make post-training modular. That difference makes behaviors closer to software components than to further copies of a model.

📱 Why this matters for small local models

Local AI changes the constraints around customization. A small model running on a phone or PC has less room for long system prompts, extended histories and large RAG contexts than a frontier model in the cloud. Those techniques remain useful, but they become an expensive way of telling a small model how to behave.

LARA offers another way to express customization: learn the behavior once and keep it in a small residual-stream adaptation rather than spelling it out in a long prompt at every interaction. A model can acquire a user's preferred writing style, a tutor's teaching method, a company's house style, or a domain-specific way of reasoning without reconstructing all of that from the context window each time.

This does not replace RAG. RAG provides information the model does not have. A behavior changes how the model uses information it already has. The two are complementary. The point is that behavior need not consume context.

The small size of the artifacts is what makes this practical at the edge. A device can carry one base model and a bank of specialized behaviors, and a new specialization is distributed as a small adapter rather than another copy of the base. That matters where storage, memory and network access are constrained, and it allows personal adaptations to stay on the device rather than being sent to a server.

Citation

@article{ekin2026laralightweightadaptersresidual,
      title={LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment},
      author={Pascal Ekin and Hyosun Choi and Wei Jie},
      year={2026},
      eprint={2607.28669},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.28669},
}

License

---