METRICS-02] Track rework rate and change failure rate as leading instability indicators

[paruff]

[METRICS-02] Track rework rate and change failure rate as leading instability indicators

Labels: observability, process, p1-critical, effort-small
Assignee: 🤖 Copilot
Phase: Phase 1 — AI Foundation
DORA 2025 Source: “Rework rate” added to DORA core metrics; “AI increases throughput, also increases instability” finding
Effort: Small (< 1 day)
Prerequisites: DEVOPS-01 (CI pipeline), OBS-01 (Sentry)

---

Why This Is New in 2025

The 2025 DORA report introduces rework rate as a new core metric alongside the traditional four (deployment frequency, lead time, change failure rate, time to restore). The finding: AI-assisted teams that don’t track rework see throughput gains consumed by hidden instability. Rework rate is the earliest warning signal that AI output quality is degrading or that the copilot-instructions.md needs updating.

“Teams should track rework rate as a leading indicator of hidden instability.” — 2025 DORA Report, Engineering Leaders analysis

The 2025 report also reinforces change failure rate tracking — but with a new urgency: AI increases the volume of changes, which statistically increases the number of failures unless quality gates are proportionally strengthened.

---

The Three Metrics to Track

Metric	Definition	Target	Tool
Rework rate	Lines reverted or substantially changed within 14 days of authoring, as % of total lines authored	< 10%	scripts/churn-report.sh (extend METRICS-01)
Change failure rate	% of deployments that cause a user-visible bug, crash, or rollback	< 5%	Sentry error spike detection
PR revision rate	% of PRs that require at least one revision request before merge	< 25%	GitHub Insights + weekly manual tally

---

Acceptance Criteria

Rework Rate Dashboard (extend METRICS-01)

• scripts/churn-report.sh updated to also output:
  • Lines authored in period
  • Lines reverted in period (git log grep for “revert” commits)
  • Rework rate as percentage
  • Trend indicator: ↑ / → / ↓ vs prior month
• docs/METRICS.md updated with rework rate definition, target threshold (< 10%), and interpretation guide:
  • 0–5%: healthy — Copilot output is landing well
  • 5–10%: watch — check for pattern drift in copilot-instructions.md
  • 10–20%: act — run PROCESS-02 tuning immediately
  • 20%: stop adding features — fix the instructions first

Change Failure Rate via Sentry

• Sentry alert configured: if error count spikes > 3× baseline within 2 hours of a deployment, post a GitHub issue automatically (or Slack webhook if team uses Slack)
• docs/RUNBOOKS.md updated with “Change Failure Response” runbook:

1. Identify the deploy that caused the spike (git log)
2. Revert the merge commit: git revert -m 1 <merge-sha>
3. File a bug issue with Sentry error link and rework PR
4. Root cause: was it a Copilot pattern failure? Update copilot-instructions.md

Weekly Snapshot Script

• scripts/weekly-metrics.sh created that outputs a single-screen summary:

  === Goald Weekly Metrics ===
  Rework rate (14d):        X%    [target: <10%]
  PRs merged this week:     N
  PRs requiring revision:   N     [target: <25%]
  Sentry errors (7d):       N     [vs prior week: ±N%]
  Test coverage:            X%    [target: >80%]
  

• Script is documented in docs/RUNBOOKS.md under “Weekly Review”

• Added to package.json: "metrics": "bash scripts/weekly-metrics.sh"

---

Files to Create / Change

• scripts/churn-report.sh — extend with rework rate
• scripts/weekly-metrics.sh (create)
• docs/METRICS.md — add rework rate, change failure rate definitions and targets
• docs/RUNBOOKS.md — add Change Failure Response runbook and Weekly Review cadence
• package.json — add metrics script