Automate agent usage reporting with categorized PR data, median metrics, and per-repository breakdowns

[fullsend-ai-triage]

Note

This issue was automatically generated from meeting notes by the scribe agent.
Please review, edit, and add any missing context before prioritizing.

Problem

Agent performance data is currently collected through manual scraping, which is inefficient and limits analytical richness. The team needs automated reporting that surfaces categorized PR data, median-based metrics (more representative than averages for skewed cost distributions), per-repository usage breakdowns, and visualizations to support operational decision-making.

Options considered

• Continue manual data collection and report assembly — does not scale and is error-prone as agent adoption grows across repositories
• Build automated reporting on top of the OpenTelemetry tracing pipeline (Forward OTEL env vars to all managed workflow agents, not just triage #5528) — provides consistent, real-time data once tracing covers all agents, but depends on resolving current data quality issues (telemetry: MLflow trace columns (Name/Session/User/Source) blank; token & cost omit cache tokens #5361)
• Extend the existing retro agent analysis to extract and aggregate PR categorization and merge metrics — leverages existing agent infrastructure

Acceptance criteria

• Agent usage report includes merged PR metrics categorized by agent type (code, review, fix, retro, etc.)
• Median cost values are computed and displayed alongside mean values for more representative performance analysis
• Per-repository usage breakdown is available with PR count metrics
• Distribution visualization is generated for agent action categories
• Report methodology accounts for self-referential usage and includes appropriate disclaimers

Related

• Builds on Forward OTEL env vars to all managed workflow agents, not just triage #5528 (OTEL env var forwarding to all agents — provides the automated data pipeline)
• Relates to telemetry: MLflow trace columns (Name/Session/User/Source) blank; token & cost omit cache tokens #5361 (MLflow trace data quality — prerequisite for reliable automated reporting)
• Relates to feat(eval): add rework rate tracking for agent PRs #5516 (rework rate tracking — complementary metric for agent PR analysis)
• Source: Meeting notes

[fullsend-ai-prioritize]

RICE Priority Score: 0.25

Score breakdown

Dimension	Score	Reasoning
Reach	1	This is an internal operational reporting tool for the team managing agent performance across repositories. It affects the ops/engineering team that currently does manual scraping, not end-users or external customers. Score of 1 reflects a moderate number of internal stakeholders who rely on this data for decision-making.
Impact	1	Automates a manual, error-prone scraping process and adds richer analytics (median metrics, categorized PRs, visualizations). However, the existing manual process is a functioning workaround. This is a clear workflow improvement but not blocking core functionality.
Confidence	0.5	The issue has well-structured acceptance criteria and clear options considered, but it was auto-generated from meeting notes with a disclaimer to review and add missing context before prioritizing. It has no labels, no milestone, no assignees, and no human comments. Dependencies on #5528 and #5361 add uncertainty about readiness and true scope.
Effort	2	Significant implementation across multiple components: automated data pipeline integration, PR categorization logic, median/mean metric computation, per-repo breakdowns, distribution visualizations, and self-referential usage handling. Dependencies on the OTEL tracing pipeline (#5528) and data quality fixes (#5361) add coordination overhead.

Formula: (1 x 1 x 0.5) / 2 = 0.25

[fullsend-ai-triage]

Meeting update — 2026-08-18

Relevant to this issue: The team introduced a PR statistics dashboard that surfaces fix frequencies and provides visibility into failed runs, directly advancing automated reporting of categorized PR data.

• A PR dashboard has been introduced with data smoothing adjustments for clearer visualization of issue counts
• The dashboard scope is explicitly limited to current repositories — no plan to extend to external sets
• Color mismatches between text and graph elements were identified and assigned for fix

Next steps: Fix color mismatches on the analytics dashboard between text labels and graph elements.

Meeting notes

[ascerra]

The color mismatch is being resolved here fullsend-ai/metrics#11

I don't believe what we have in the metrics issues to cover pr delivery type covers every part of this issue but it does cover some of it (categorized PR data) but its important to note that this is for the fullsend and agents repos PRs and issues being delivered by any means (humans and agents). I believe this ticket is more scoped around agent metrics and what the agents are doing.