delta mode: emit a versioned change_risk block (change-level risk score)

[stephenc222]

Problem

hotspots analyze --mode delta returns per-function deltas and per-file aggregates, but no change-level (commit- or PR-level) score.
Consumers (hosted API, CI comments, an internal PR-risk vs. defect study) each re-derive one
from deltas[], and three pilot findings make naive re-derivation unreliable:

1. Floating-point noise: functions are reported modified when LRS differs only by ~1e-15 (e.g. 3.9 → 3.9000000000000004).
2. modified appears on functions the change likely did not edit (7 modified on a 1-line PR, mitchellh/cli@25f4c7c).
3. function_id embeds the absolute clone path (/private/tmp/pilot-cli/./cli.go::Run), so IDs don't join across runs or machines.

Proposal

Add an additive, optional top-level change_risk object to the delta JSON (bump schema_version minor; no existing
field changes):

"change_risk": {
  "scope": { "kind": "range", "base": "<parent sha>", "head": "<commit sha>" },
  "score": { "kind": "sum_positive_delta_lrs", "version": "1.0.0", "value": 8.4 },
  "components": {
    "max_delta_lrs": 3.1,
    "sum_positive_delta_lrs": 8.4,
    "new_critical_functions": 1,
    "changed_functions": 6
  },
  "inputs": { "functions_scored": 6, "noise_epsilon": 1e-9 },
  "tool_version": "1.x.y"
}

change_risk is deliberately not PR-specific: delta mode compares any two commits, so the CLI fills scope.kind
with commit or range. Callers that know more (e.g. a hosted API) may set scope.kind to pull_request and add
the PR reference. scope.kind is an open enum (commit, range, pull_request, release). The PR-scoped
instance is the concrete case this was motivated by (informally, a "PR score").

Definitions (frozen per version)

• Scored function set: functions with status in {added, modified, removed} and |Δlrs| > noise_epsilon or a band transition, and whose line range overlaps a changed hunk (when git diff is available; otherwise skip the overlap filter and say so in inputs).
• sum_positive_delta_lrs: sum over the scored set of max(0, after.lrs - before.lrs); an added function contributes its full after.lrs.
• max_delta_lrs: max of the same per-function values.
• new_critical_functions: scored functions whose after.band is critical and before was absent or not critical.
• changed_functions: size of the scored set.
• noise_epsilon: 1e-9 default, echoed in the output.

Related fixes (same PR or split)

• Emit function_id (and file paths in aggregates) relative to the repo root.
• Include tool_version in the delta output (snapshots already have it).
• Suppress status: modified when metrics and LRS are unchanged within epsilon (report as unchanged).

Why in the CLI

The definition of the score should be versioned with the analysis that produces its inputs: one source of truth
for CI, the hosted API and research. Downstream may still compute their own from deltas[]; this makes the
default well-defined and reproducible.

Non-goals

• No claim that the score predicts defects. It is unvalidated (under evaluation in an internal study); change_risk carries no tier, percentile or validation claim.
• No PR number/merge SHA (the CLI doesn't know them; the caller adds those).

Acceptance criteria

• change_risk present in --mode delta --format json; absent from other modes.
• Golden tests: no-op change → value: 0, changed_functions: 0 (noise suppressed); single-function edit; added function; band transition into critical.
• Byte-identical output across runs/machines (relative IDs, deterministic ordering).
• schema_version bumped (minor); older consumers unaffected.
• Docs: docs/hotspots-cli-contract.md updated with the definitions above.

Notes

• The study spec fixes sum_positive_delta_lrs as its primary predictor before any scoring; changing definitions later requires a new version.
• CLI 1.26.0 was used in the pilot; confirm behaviour on the latest release before implementing.

Framing (added 2026-09-20)

change_risk is a derived summary of deltas[], not new source data: a PR is a grouping of per-function records.
deltas[] stays the source of truth and must remain complete in the output. Any future score definition must be
recomputable from deltas[] alone. Downstream stores can keep per-function rows keyed with PR grouping fields
(pr_number, merge_sha) added by the caller, with change_risk as an optional derived block.

Weighting and aggregation (added 2026-09-20)

change_risk.score declares how it was built so alternatives can be added without breaking consumers:

"change_risk": {
  "scope": { "kind": "range", "base": "…", "head": "…" },
  "score": {
    "kind": "weighted_aggregate", "version": "1.0.0", "value": 8.4,
    "term": "positive_delta_lrs",
    "weights": { "scheme": "uniform" },
    "aggregation": { "operator": "sum" }
  },
  "components": { "...": "..." }
}

• term: per-function quantity aggregated (default positive_delta_lrs).
• weights.scheme: uniform (default), and later e.g. fan_in, loc, band; scheme params echoed in the output.
• aggregation.operator: sum (default), max, top_k_mean (with k), lp_norm (with p).
• Default remains uniform + sum. Additional schemes/operators are additive; deltas[] stays complete so any variant is recomputable from it.

[stephenc222]

Correction and rescoping after checking main and our validation notes. Two things in the original description
were wrong or incomplete:

1. A PR-level score already exists. hotspots diff emits pr_risk_score (feat: add PR-wide risk score to diff aggregates #201, v1.40.0): the net LRS delta
   summed across all new/modified/deleted functions. This issue claimed the CLI had no such score. That was
   incomplete: it was checked against analyze --mode delta only.
2. The default proposed here (sum_positive_delta_lrs) is the same shape that has already tested poorly.
   Internal validation found that PR-level rollups of function-level risk scores (including the shipped
   pr_risk_score) do not beat raw files-changed as a defect-discrimination signal. So no scalar formula should be the
   default until one is shown to beat size baselines.

Rescoped proposal

Add an additive, optional change_risk object to the delta output that describes the whole changeset (for a PR,
the net merged diff) as parts, with the scalar demoted to one declared summary:

"change_risk": {
  "scope":     { "kind": "range", "base": "…", "head": "…" },
  "exposure":  { "files_changed": 6, "functions_changed": 9, "functions_added": 2,
                 "functions_removed": 0, "lines_added": 140, "lines_deleted": 33 },
  "coverage":  { "files_analyzed": 5, "lines_analyzed": 150,
                 "unsupported": { "languages": [], "non_source": 1 } },
  "intensity": { "max_delta_lrs": 3.1, "sum_positive_delta_lrs": 8.4, "net_delta_lrs": 1.4,
                 "new_critical_functions": 1, "mean_delta_lrs_per_changed_function": 0.93 },
  "shape":     { "top1_share": 0.37, "files_with_regression": 3 },
  "score":     { "kind": "…", "version": "1.0.0", "value": null,
                 "validation": { "status": "unvalidated" } }
}

• net_delta_lrs is exactly today's pr_risk_score; it is unchanged and is not renamed or removed.
• score.value may be null until a scalar is validated. The parts are the deliverable; consumers (and our
  own studies) can compute any candidate from them.
• Exposure and coverage make size and analyzability visible, so a score is never read as covering more of the
  changeset than it did.
• A PR is a scope kind: the CLI fills commit/range; callers may add pull_request and its reference.

Same-PR fixes still wanted (unchanged from the original description)

• Repo-relative function_id and aggregates paths (absolute clone paths currently leak into output).
• Suppress status: modified when metrics and LRS are unchanged within epsilon.
• tool_version in delta output.

Acceptance (revised)

• change_risk parts present and deterministic in delta JSON; score.value may be null.
• Noise-suppression, added-function, band-transition, and coverage golden tests.
• pr_risk_score output unchanged.
• Docs updated; validation.status documented.

[stephenc222]

Validation status of the proposed score (research results, pre-registered).

We tested max_delta_lrs (and sum_positive_delta_lrs, max_after_lrs, changed_functions, new_critical_functions) against defect-linked PR-units, with PR size as a pre-registered baseline and size-stratified analysis (3 repos: scikit-learn, vscode, vite; 360 PR-units, 89 defect-linked; deltas derived from v1.40.0 structural snapshots, identical to diff on 47/47 units checked).

• max_delta_lrs fails the gate. Pooled size-stratified OR 1.05 (95% CI 0.39-2.81). It does not beat max_after_lrs (level vs change), and adds nothing over a history-based composite.
• It behaves like a change-count proxy: changed_functions gets the same within-stratum AUC (0.578 vs 0.579), and the median score is 0 in every repo.
• The modest rank signal that remains disappears when fix/revert PRs are excluded (AUC 0.476), so it comes from bugfix PRs themselves, not from risk in the changed code.
• Related earlier result: a per-file-risk PR aggregate lost to raw n_files (F132). A cheap history-based alternative (recent-activity burst) also failed confirmation on 6 fresh repos. Full write-up is in the research PR: https://github.com/Stephen-Collins-tech/hotspots-research/pull/91

Caveat: 3 repos and 89 positives is underpowered, so this is "not validated," not "shown false."

Suggestion: keep the non-goal exactly as written ("no claim that the score predicts defects") and make sure components always carries size (changed_functions, and ideally files_changed/lines_changed) so consumers can control for it. In our data PR size (n_files) was the strongest simple flag.

[stephenc222]

Overlap with #201's aggregates.pr_summary.pr_risk_score.

#201 (shipped in v1.40.0) defines pr_risk_score as the signed net LRS delta (deleted functions subtract, modified add delta.lrs). This issue proposes sum_positive_delta_lrs (positive parts only, added contributes full after.lrs). Two differently-defined "PR risk" numbers in the same output will confuse consumers.

Suggest reconciling before #202 lands: either expose #201's quantity as a named component (e.g. net_delta_lrs) inside change_risk.components, or version it into change_risk.score and deprecate the separate field. In the study data the two positive-part variants were nearly identical anyway (typical unit has a single scored function), so the empirical difference is small; the naming/versioning is what matters.

[stephenc222]

Implementation notes from running this at scale (~900 PR-units).

1. Runtime. hotspots diff <base> <head> --auto-analyze took ~19s per PR on scikit-learn, 5+ minutes on vscode, and >15 minutes on elasticsearch. It has no --touch-mode option and appears to spend the time on per-function git-log touch metrics, which don't feed LRS. analyze --mode snapshot --touch-mode none --no-persist --all-functions on vscode takes ~6s, and deltas derived from two such snapshots matched diff's scored function sets exactly (47/47 units). A structural-only path for delta mode would keep change_risk cheap in the common case.
2. Still slow on some trees. Even with --touch-mode none, snapshots took 4+ minutes on zed (Rust) and elasticsearch (Java). Worth profiling separately.
3. Hunk overlap. Snapshot functions carry a start line; diff's deltas[] has no line info, so the proposed hunk-overlap filter can't be applied from deltas[] alone. Emitting line ranges (or a per-function touched_by_hunk flag) would make that filter reproducible downstream. We did not test a hunk-overlap variant.
4. Zero-inflation. After the epsilon filter, the median max_delta_lrs was 0 in all three repos (most PRs have no scored function). Percentile- or tercile-based consumers need to handle ties; exposing changed_functions: 0 (already proposed) is what lets them separate the empty case.
5. Paths. Confirmed: function_id and file paths embed the absolute clone path in both diff and snapshot output, so they don't join across runs until they are repo-relative.
6. Weights/operators. The proposed weights.scheme and aggregation.operator variants (fan_in, top_k_mean, lp_norm, ...) are untested. None has validation evidence yet, so it's better to ship the uniform+sum/max defaults labeled as descriptive and add variants only with their own evaluation.