Principal Loyalty

Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents

Most LLM agents today operate in a digital world: they call APIs, browse, and serve the user who invoked them. A rapidly growing class of deployments is multi-party — the agent represents a principal (who briefs the agent, sends follow-up instructions, and receives results) while also conversing in a separate channel with a counterparty whose interests may diverge: negotiating with a vendor, screening inbound requests for an owner, mediating between employees. Here "help whoever you are talking to" is the wrong objective; the agent must stay loyal to the principal it represents without over-refusing the principal's own cooperative requests.

This repository contains the benchmark, training recipes, evaluation harness, analysis scripts, and companion website for our study of the multi-party loyalty problem.

---

What's here

Contribution	Where
PrincipalBench — a 75-item multi-turn measurement instrument with leak probes, dual judges, and an integrity-audit gate	items/, src/
(M1) Prompt-time loyalty scaffold — seven prioritized rules open-coded from 50+ failure trajectories	src/loyalty.py, src/agent.py
(M2) Per-token-KL distillation — an on-policy recipe transferring a prompted Qwen3-32B teacher into 8B Qwen3 / Llama-3.1 students	scripts/pertoken_kl_collect.py, scripts/run_iter4_pertoken_kl.sh
Reproducibility pipeline — every paper number regenerates from raw trajectories	scripts/recompute_all.py
Companion website — browse items, models, runs, and sample conversations	website/
Paper	arXiv:2606.30383 · paper_arxiv.tex, paper_arxiv.pdf

---

Key findings

• A calibrated / over-refuse split. Running PrincipalBench across 13 frontier models exposes a sharp split — ≤20% harm for calibrated models vs. 53.6–75.3% harm for over-refusers — that is invisible to single-turn safety evaluations.
• The scaffold works. Under the seven-rule loyalty scaffold, Claude-Sonnet reaches 21/108 (19.4%) harm, and all nine calibrated frontier models hold ≤20% harm.
• Distillation transfers. Per-token-KL distillation is the strongest open-weight recipe we measure, moving an 8B student to harm 33 / leak 13.
• A structural floor. Both mechanisms move along a common leak/over-refusal trade-off rather than crossing it — a floor that single-knob RL (DAPO) also fails to break.

---

Repository layout

principal-loyalty/
├── src/                  # the benchmark harness — see docs/architecture.md
│   ├── harness.py        #   trajectory runner: (subject, arm, item) → JSONL
│   ├── items.py          #   item schema + loader
│   ├── agent.py          #   agent under test: plain / prompted / scaffolded arms
│   ├── counterparty.py   #   counterparty simulator (pressure strategies)
│   ├── loyalty.py        #   externalized LoyaltyState (the M1 scaffold)
│   ├── probe.py          #   leak-detection probe (lexical + LLM)
│   ├── scorer.py         #   per-item / per-cell metric aggregation
│   ├── reward.py         #   fast proxy reward for DAPO training
│   └── vendors.py        #   unified vendor layer (Anthropic/OpenAI/Google/OpenRouter)
├── items/                # benchmark items (JSON) — see docs/benchmark.md
│   ├── v0/               #   50 training items
│   ├── v0_75/            #   25 held-out items
│   └── v0_5/             #   superseded dev-intermediate set (not part of release)
├── data/                 # SFT / DPO / on-policy distillation datasets — see docs/training.md
├── scripts/              # eval, training, analysis, figure, release scripts (see Key scripts below)
├── figures/              # paper figures (PDF), regenerated by scripts/make_figs_arxiv.py
├── website/              # static React companion site — see docs/website.md
├── docs/                 # extended documentation — see the Documentation section below
├── paper_arxiv.{tex,pdf,bib,bbl}, arxiv.sty, abstract.txt   # the paper
├── RELEASE.md            # building the clean public release tree
└── requirements.txt

Note on large artifacts. Raw run trajectories (runs/, 150+ GB) and Hydra training outputs (outputs/) are git-ignored. The repository ships the items, code, datasets, figures, and paper; trajectories are regenerated by re-running the harness (see docs/reproduce.md).

Key scripts

scripts/ holds the full research history; these are the entry points you'll actually use:

Script	What it does
run_traj_only.py · multi_rollout_eval.py	Generate trajectories / run a multi-seed grid eval
recompute_all.py	Recompute every paper number from runs/
paired_seed_test.py · per_arm_xvendor_wilcoxon.py	The paper's statistical tests
make_figs_arxiv.py	Regenerate all paper figures
pertoken_kl_collect.py · run_iter4_pertoken_kl.sh	Per-token-KL distillation (mechanism M2)
run_dapo.sh · run_full_pipeline.sh	DAPO baseline / end-to-end Llama pipeline
build_website_data.py · build_release.py	Refresh the website data / build the clean release tree

---

Quick start

# 1. install (the harness itself needs only anthropic + openai)
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. set provider keys (only those you intend to use)
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
export OPENROUTER_API_KEY=...

# 3. generate trajectories for a subject across all arms on the v0 items
python3 scripts/run_traj_only.py --seed-dirs runs/smoke_seed1 \
    --subject claude-sonnet --counterparty claude-sonnet \
    --items items/v0 --parallel 4

# 4. multi-seed grid eval (trajectories + scoring + aggregation)
python3 scripts/multi_rollout_eval.py --base runs/smoke \
    --seeds 5 --counterparty claude-sonnet --parallel 4

The harness (src/) is a library, not a CLI — the scripts in scripts/ drive it (see docs/architecture.md). For the full evaluation sweep, training recipes, and figure regeneration, see the documentation below.

---

Documentation

Doc	Contents
docs/benchmark.md	Item schema, the six cells, the three arms, leak probes, dual judges, the integrity-audit gate
docs/architecture.md	How the harness, scorer, and vendor layer fit together
docs/reproduce.md	Reproducing every number and figure in the paper
docs/training.md	The per-token-KL distillation recipe and DAPO baselines
docs/website.md	Building and refreshing the companion site
RELEASE.md	Building the clean public release tree

---

Citation

@misc{principalbench2026,
  title         = {Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents},
  author        = {Li, Bojie and Shi, Noah},
  year          = {2026},
  eprint        = {2606.30383},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  note          = {PrincipalBench},
  url           = {https://arxiv.org/abs/2606.30383}
}

License

Released under the MIT License.