Evals

[mlund01]

Intro

Evaluating the performance and expectations of a mission is essential to ensuring consistency across runs and protecting against drift. Because squadron is purpose built to go through a series of logical gateways and creates structured data outputs, evals naturally fit into the model nicely.

Goal

Great evals test outcomes. They are not intended to assess the reasoning or logic used to accomplish the outcomes. Evals will focus on evaluating these outcomes. Based on eval results, developers can debug their own missions by going deeper into the reason and logic to understand how it got to those outcomes.

Solution Proposal

Add a config-driven evaluation (eval) framework. This framework will allow users of squadron to create test cases in the form of LLM-as-a-judge workflows, complete with measures to rate the success, quality, or categorize/label aspects of missions based on predefined criteria.

On the squadron side, there will be an "eval" agent, designed similar to commander or regular agent, but with it's own set of tools and own system prompts.

Evals abilities

• to create test cases that measure a variety of conditions, to do LLM-as-a-judge sort of assessments (can be true/false, categories, or ranges)
• to have these run 1 every x mission runs
• to run an eval based on conditions targeting historical runs of a mission (largely date based, or "last 5 runs" or something like that)
• to focus on actions, outputs, datasets, and final answers. The Judges will not have access to the session itself, just the actions and data it called or created along the way
• to scope by a task, or full mission
• to verify certain conditionals where triggered

Supporting abilities

• trigger an eval, which simply means run the mission, then run the eval
• eval results analytics page for each eval group (for command center)

[nstang43]

Really like the direction here — outcome-focused evals that don't second-guess reasoning is the right call, and scoping judges to actions/outputs/datasets (not session internals) is a clean separation.

I want to push on two things: broadening the grader toolkit beyond LLM-as-a-judge, and then a bigger idea about what evals unlock once you have them.

---

Expanding the grader model

The proposal leans heavily on LLM-as-a-judge as the primary eval mechanism. For a system like Squadron that produces structured data and goes through explicit logical gateways, there's a big opportunity to layer in deterministic/code-based graders as the foundation before reaching for an LLM judge.

Anthropic published a solid framework for this in their "Demystifying evals for AI agents" post (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — the mental model is three tiers of graders:

1. Code-based graders (fast, cheap, reproducible)

• Schema validation on structured outputs
• Value range / threshold checks
• Exact match or regex on expected fields
• State verification ("did the expected record get created?")
• Tool call verification ("were the right tools called with the right params?")
• Transcript metrics (turns taken, token usage)

For Squadron specifically, these feel like the natural first layer — missions produce structured data, conditionals fire or don't, datasets get created. You can write hard assertions against all of that without burning LLM tokens.

2. Model-based graders (flexible, captures nuance)

• This is the LLM-as-a-judge layer the proposal describes
• Best for open-ended outputs where "correct" is subjective
• Needs periodic calibration against human judgment to stay trustworthy

3. Human graders (gold standard, used sparingly)

• Not for running at scale, but for validating that your automated graders (both code and LLM) are actually measuring what you think

A few other concepts from Anthropic's framework worth considering:

→ Capability vs. regression evals — Capability evals ask "what can this agent do well?" and should start at a low pass rate (giving you a hill to climb). Regression evals ask "does it still do what it used to?" and should be near 100%. The distinction matters for how you interpret scores and when you alert. Capability evals that reach high pass rates can "graduate" into the regression suite.

→ Non-determinism — Agents produce different outputs across runs. The "run 1 every X mission runs" approach might want to account for this. Anthropic uses pass@k (at least one success in k tries) and pass^k (all k succeed) as complementary metrics. Which one matters depends on whether you care about "can it do this at all" vs. "does it do this reliably every time."

→ Eval-driven development — Building evals before the mission can fulfill them, then iterating until it does. Basically TDD for agents. Flips evals from retrospective assessment to a development driver.

→ Grader calibration — If using LLM judges, periodically validating their judgments against human reviewers to make sure the judge isn't drifting.

---

The bigger idea: evals as an objective function for automated optimization

Once you have evals producing quantitative scores, you have an objective function. And once you have an objective function, you can optimize against it.

Squadron's HCL configs define missions declaratively — the configs essentially ARE a search space. Every mission has tunable knobs: model choice per agent/task (Sonnet vs Haiku vs Opus), temperature, max_tokens, system prompt variations, tool availability, retry strategies, conditional thresholds, few-shot example selection. Today, developers hand-tune all of this. But with a solid eval layer in place, you could wire up an optimizer to explore that space automatically.

Three approaches worth looking at:

Optuna (https://optuna.org) — general-purpose Bayesian optimization. Handles conditional search spaces natively (maps well to Squadron's conditional logic), can efficiently prune unpromising configurations early, and — the killer feature — supports multi-objective optimization. Imagine optimizing a mission for quality AND cost AND latency simultaneously, with Optuna building a Pareto front so the developer can choose where they want to operate on the tradeoff curve.

DSPy (https://dspy.ai) — purpose-built for prompt optimization. Treats LLM applications as optimizable programs where prompts are "weights" that can be tuned. Their MIPROv2 optimizer runs your program many times, collects high-scoring traces, uses those to draft better instructions, then searches for the best combination. For Squadron, this could mean automatically discovering better system prompts and few-shot examples for each agent in a mission.

GEPA (https://github.com/gepa-ai/gepa) — reflective prompt evolution. Uses LLMs to iteratively improve prompts based on eval feedback, and can evolve entire programs including control flow. Notably integrated into DSPy and MLflow.

The practical wins would be things like:

• Discovering that Task 3 works better with Haiku than Sonnet (cheaper AND faster with no quality loss)
• Finding that a specific system prompt variation improves correctness by 15%
• Identifying the temperature sweet spot for each agent in the pipeline
• Trading off cost vs. quality on a Pareto front so users pick their operating point

This doesn't need to be in v1 of the eval framework — but if the eval config schema is designed with quantitative scores and composable graders from the start, optimization becomes a natural extension later. The eval layer becomes the foundation for a "compile my mission" capability.

---

None of this contradicts the proposal — more like expanding the grader toolkit so LLM-as-a-judge is one option in the eval config rather than the only one, and planting a flag for where evals can go once the foundation is solid. The structured nature of Squadron's outputs is actually a strength here — lots of things you can assert deterministically before needing a judge.