Test: Validate single Foundry agent evaluation

[placerda]

Objective

Validate that a single Foundry agent can be evaluated end-to-end with AgentOps and produce stable reports, thresholds, and comparison output.

Scope

Use a remote Foundry agent target with type: agent, hosting: foundry, execution_mode: remote, and endpoint.kind: foundry_agent. For the bundle, default to conversational_agent_baseline for a plain assistant/Q&A agent, or use agent_workflow_baseline if the chosen agent already uses tools. Foundry agents are a supported remote target, and agent_mode is a valid Foundry-only dimension.

Tasks

• Run agentops init
• Create a dedicated run config for the selected Foundry agent
• Configure agent_id, judge model, and AZURE_AI_FOUNDRY_PROJECT_ENDPOINT
• Execute agentops eval run
• Review results.json and report.md
• Make one small instruction/config change and rerun
• Run agentops eval compare --runs <baseline>,latest
• Log bugs, usability issues, and docs gaps

Acceptance Criteria

• A Foundry agent can be evaluated successfully end-to-end
• The run produces results.json and report.md
• Threshold behavior is clear and repeatable
• Comparison output is understandable and useful for regression analysis
• Follow-up backlog items are captured

Out of Scope

• Improving the agent itself
• Adding new evaluators in this sprint
• CI/CD integration for this scenario

[Dongbumlee]

End-to-end validation results

Executed all 8 tasks against a real FoundryAgent on https://aifappframework.services.ai.azure.com/api/projects/models with judge model gpt-5.1. Both bundles tested.

Acceptance criteria — all met ✅

Criterion	Result
Foundry agent evaluated successfully end-to-end	✅ 3 cloud-eval runs completed (eval_53a8…, eval_a3c0…, eval_fd26…)
results.json and report.md produced	✅ for all 3 runs
Threshold behavior clear and repeatable	✅ exit 2 on FAIL, exit 0 on PASS, deterministic across reruns
Comparison output understandable	✅ comparison.md + comparison.json produced; clear delta column
Follow-up backlog items captured	✅ see below

What was run

1. agentops init (workspace already present)
2. run-foundry-conversational.yaml → conversational_agent_baseline → FAIL (exit 2; latency missed)
3. run-foundry-tools.yaml → agent_workflow_baseline → FAIL (exit 2; tool scores low)
4. Small change: relaxed bundle copy with avg_latency_seconds <= 30 (was 10)
5. run-foundry-conversational-v2.yaml → PASS (exit 0)
6. agentops eval compare --runs foundry-conv-baseline,foundry-conv-relaxed → NO REGRESSIONS

Foundry Studio links

• conv-baseline: evalrun_228a36912fa04fb382f7a4d722cd317e
• tools-baseline: evalrun_e44a33a67ddc453f938eb3642a2a28ff
• conv-relaxed: evalrun_1488ef3d4d084dea959a27becf39d62f

Headline metrics

conversational_agent_baseline (5 rows, FoundryAgent, gpt-5.1)

• Coherence 4.80 / Fluency 4.00 / Relevance 4.80 / Similarity 4.40 — all PASS (>= 3)
• avg_latency_seconds 22.47s — FAIL (<= 10)

agent_workflow_baseline (5 rows, same agent)

• TaskCompletion 0.4 / ToolCallAccuracy 2.0 / IntentResolution 3.6 / TaskAdherence 1.0 / ToolSelection 0.2 / ToolInputAccuracy 0.8 — all FAIL (>= 3) except IntentResolution which still missed (4/5 items)
• This strongly suggests FoundryAgent does not have tool definitions registered, so applying this bundle without an agent that uses tools is misleading.

---

Bugs / usability issues / docs gaps

🐛 Bug 1 — report.md shows empty agent "Response" for cloud-eval runs

Every "### Row N" block has **Response:** followed by nothing. The Foundry cloud-eval pipeline does produce per-row outputs (we see them via "Processed output item N/5"), but they don't propagate into the per-row response cells in report.md. Severely limits debug-ability of "why did this row score X?".

• Repro: any hosting: foundry, execution_mode: remote run; see .agentops/results/foundry-conv-baseline/report.md lines 92, 106, 120…
• Severity: medium

🐛 Bug 2 — Per-row latency is faked, not measured

Latency in cloud evaluation is estimated from total eval duration (112.4s / 5 rows ≈ 22.47s per row). Every row receives the same latency value. This breaks trend/outlier analysis and makes the avg_latency_seconds metric misleading at the row level.

• Source: backends/foundry_backend.py (cloud-eval branch)
• Severity: medium — consider either parsing real per-call timing from the OpenAI Evals API response or marking the metric explicitly as "run-level estimate".

🐛 Bug 3 — Comparison report shows baseline thresholds for all runs

agentops eval compare against runs that use different bundles displays the baseline's threshold across all columns. Example: relaxed run had avg_latency_seconds <= 30 (and PASSED its own threshold at 14.04s), but comparison.md shows it as 14.04 (-8.43, improved) Missed against <= 10.

• Status column says PASS, per-evaluator column says Missed → confusing.
• Fix idea: per-run column should display each run's own threshold.
• Severity: medium

⚠️ Issue 4 — agent_id without version silently accepted, behavior undocumented

Per docs the format is <name>:<version> (e.g. my-agent:3). Used bare FoundryAgent with no version; cloud eval succeeded. Need to clarify in docs:

• Does this default to latest version?
• Should the runner log the resolved version for reproducibility?

⚠️ Issue 5 — agentops config validate is a documented but unimplemented stub

agentops config validate --help shows "Validate configuration files (planned)." with no flags. Either hide the command until implemented or accept -c <path> and exercise the existing Pydantic models — implementation should be a few lines.

📝 Docs gap 6 — conversational_agent_baseline default latency threshold is unrealistic for Foundry

Default is avg_latency_seconds <= 10, but Foundry cloud-eval latency includes the entire judge-evaluator pipeline. Realistic per-row totals are 20-30s. Anyone running the smoke dataset against a real Foundry agent will see immediate FAIL on the latency rule. Either:

• Raise the default to e.g. 30s for Foundry,
• Or split into separate agent_latency_seconds (model-only) and eval_latency_seconds (full pipeline) metrics,
• Or document this prominently in the Foundry tutorial.

📝 Docs gap 7 — results.json schema version unset

Project conventions (CONTRIBUTING.md, AGENTS.md) say results files include a version field. Top-level schema_version is missing/None in the produced results.json. Tighten the runner to always emit it.

📝 Docs gap 8 — agent_workflow_baseline requires tool-equipped agent (silently scores ~0 otherwise)

Running this bundle against an agent without registered tool definitions yields scores like ToolSelection 0.2, ToolCallAccuracy 2.0 with no warning. Tutorial should call out: "This bundle is only meaningful if the Foundry agent has the matching tool definitions registered." Optionally, the runner could emit a warning when tool_definitions are present in the dataset but the agent under test doesn't expose any tools.

---

Artifacts for repro

.agentops/run-foundry-conversational.yaml      # baseline run config
.agentops/run-foundry-tools.yaml               # tools run config
.agentops/run-foundry-conversational-v2.yaml   # small-change rerun
.agentops/bundles/conversational_agent_relaxed.yaml  # tweaked bundle (latency 10 -> 30)
.agentops/results/foundry-conv-baseline/
.agentops/results/foundry-tools-baseline/
.agentops/results/foundry-conv-relaxed/
.agentops/results/compare-conv/

These are local-only test artifacts; happy to either commit the run configs (they're a useful starter for future Foundry-agent regression testing) or delete them — let me know which is preferred.

[Dongbumlee]

Follow-up issues opened

#	Issue	Title
1	#110	Foundry cloud-eval: agent response missing from report.md
2	#113	Foundry cloud-eval: per-row latency is faked (run-level avg duplicated)
3	#112	agentops eval compare: per-run column uses baseline's threshold for all runs
4	#111	Document agent_id resolution when version is omitted
5	#115	Implement or hide 'agentops config validate' (currently a stub)
6	#114	Default avg_latency_seconds threshold (10s) is unrealistic for Foundry cloud eval
7	#116	results.json schema_version is missing/None
8	#117	agent_workflow_baseline silently scores ~0 against tool-less agents