Very verbose comments on PRs

[juanmichelini]

See for example

https://github.com/OpenHands/software-agent-sdk/pull/3497/changes#diff-11cea73f79bdd101b3b9fd57ce0a795426d1548e602e51d3bf84be4e55abe82d

In that PR, suggested fix is 3 lines, for that it creates 19 lines of comments. Those 19 lines are noise, they are okay as a PR description, but should not live in code. Reasons:

• they are redundant, repeating things that can be noticed in the code.
• they are redundant, repeating things that can be notice in the PR description or with git blame.
• they describe non local part of the system without any mechanism to be up to date, so they can drift with time.

Over all comments should be used strategically for unintuitive things, for anything else code should be structured in such a way that meaning is clear.

[juanmichelini]

@OpenHands look at the last 100 PRs in software-agent-sdk see if you can find a pattern of which PRs have very verbose comments and what model/agent/settings are using.

For reference PR 3497 was done with Opus-4-7 and OpenHands SaaS.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[juanmichelini]

@juanmichelini I scanned the last 100 PRs (#3366 → #3497) and joined them with the OpenHands Cloud API (/api/v1/app-conversations) and the Co-Authored-By trailers in the merge commits to attribute model/agent. Here's what fell out.

TL;DR

The verbose-comment pattern is overwhelmingly correlated with Claude Opus 4.7 / 4.8, on top of both OpenHands SaaS and the Claude Code CLI flow. Of 27 PRs in the last 100 that introduced a consecutive block of ≥10 added comment lines, 26 carry an AI signature (OpenHands footer, app.all-hands.dev/conversations/... link, or Co-Authored-By: Claude Opus 4.x trailer). The remaining 1 (#3399) is a security-pin change where the comments are arguably load-bearing.

The verbose comments cluster heavily on fix(...) and root-cause investigative PRs — exactly the ones where the agent had to reason about why something failed.

Method

For each PR I extracted:

• additions, classified line-by-line into code / comment / blank (Python #/docstring, JS/TS //)
• the longest consecutive run of added comment lines per PR (signal for "comment paragraph dropped above small code change")
• comment_added / (code_added + comment_added) ratio
• AI signals: PR-body footer, embedded app.all-hands.dev/conversations/<id> URL, and Co-Authored-By: trailers in commit messages
• For PRs with a conversation URL, I queried GET /api/v1/app-conversations?ids=... to read llm_model, agent_kind, accumulated_cost

Scripts and raw JSON are throwaway analysis (kept locally), but the headline tables below are reproducible from the GitHub API + Cloud API.

Top 25 PRs by longest added consecutive comment block

PR	Author	API llm_model	Commit Co-Authored-By: Claude	AI?	code+	cmt+	block
#3433	simonrosenberg	–	Opus 4.8	AI	324	170	48
#3426	neubig	–	–	AI	804	211	20
#3420	simonrosenberg	–	Opus 4.8	AI	221	97	19
#3403	rbren	–	–	AI	1226	457	19
#3409	simonrosenberg	–	Opus 4.7	AI	56	78	18
#3407	simonrosenberg	–	Opus 4.7	AI	213	119	17
#3399	Sanjays2402	–	–	–	27	17	17
#3422	simonrosenberg	–	Opus 4.8	AI	217	139	16
#3452	simonrosenberg	–	Opus 4.8	AI	929	384	15
#3390	simonrosenberg	–	Opus 4.7	AI	504	293	15
#3483	simonrosenberg	–	Opus 4.8	AI	131	62	14
#3410	chuckbutkus	–	–	AI	509	206	14
#3464	simonrosenberg	–	Opus 4.8	AI	223	119	13
#3465	simonrosenberg	–	Opus 4.8	AI	185	67	12
#3436	simonrosenberg	–	Opus 4.8	AI	72	16	12
#3400	VascoSch92	–	–	AI	4	12	12
#3493	simonrosenberg	–	Opus 4.8	AI	51	29	11
#3482	juanmichelini	litellm_proxy/claude-opus-4-7	–	AI	91	56	11
#3462	simonrosenberg	–	Opus 4.8	AI	132	75	11
#3444	xingyaoww	–	–	AI	72	34	11
#3497	juanmichelini	litellm_proxy/claude-opus-4-7	–	AI	20	31	10
#3492	simonrosenberg	–	Opus 4.8	AI	191	100	10
#3490	simonrosenberg	–	Opus 4.8	AI	186	108	10
#3489	VascoSch92	–	–	AI	15	10	10
#3477	juanmichelini	–	–	AI	14	16	10

The "API llm_model" column is null for most rows because their PR descriptions don't link the conversation — but the Co-Authored-By: Claude Opus 4.x trailer in the merge commit makes the model unambiguous in those cases.

Distribution

• AI-labeled PRs in the last 100: 76 / 100 (footer, convo URL, or Claude co-author).
• Verbose PRs (block ≥ 10): 27. AI-labeled: 26. Non-AI: 1.
• llm_model from the Cloud API (16 conversations queryable):
  • litellm_proxy/claude-opus-4-7 — 14
  • litellm_proxy/glm-5.1 — 2
• Claude versions in Co-Authored-By trailers across the 100 PRs:
  • Claude Opus 4.8 — 20
  • Claude Opus 4.7 — 9
• Claude versions in the verbose subset (block ≥ 10):
  • Claude Opus 4.8 — 12
  • Claude Opus 4.7 — 3
  • (the rest are OpenHands SaaS Opus-4.7 PRs without a Co-Author trailer, plus a few unattributed AI-footer PRs)
• Authors of verbose PRs: simonrosenberg (15), juanmichelini (4), VascoSch92 (2), neubig (2), xingyaoww (1), chuckbutkus (1), rbren (1), Sanjays2402 (1).

What the verbose comments look like

Sampling #3497, #3440, #3429, #3433, #3409, #3407 and #3477, the comment paragraphs share a clear shape:

1. A prose justification of why a tiny change was made — typically duplicating the PR's "Why" section verbatim, including URLs (StepFun docs, OpenAI proxy registry behaviour, FastMCP nesting counters, ACP SSE corruption).
2. Decision matrices / "Action / Do nothing" tables baked into # comments in .github/run-eval/ADDINGMODEL.md and resolve_model_config.py.
3. Cross-references to other PRs/issues with deep technical context (e.g. "see step-3.7-flash eval: 0% Critic Acceptance because StepFun 10 RPM rate-limit exhausts the SDK retry budget #3496", "see commit b0fedcd"), which will drift the moment those references move.
4. Restatements of code that is right next to the comment — e.g. "the SDK's lookup … does an exact string match of model.removeprefix("litellm_proxy/")" written immediately above the line that hardcodes the alias.

This is exactly the failure mode #3498 describes: PR-description material being written into the source.

Settings shared by the worst offenders

For the PRs where I could read the Cloud-side conversation (juanmichelini's OpenHands SaaS):

• agent_kind = openhands
• llm_model = litellm_proxy/claude-opus-4-7
• metrics.context_window = 1_000_000
• accumulated_cost for the verbose ones is in the $9–$20 range (long, root-cause-style sessions: fix(step-3.7-flash): stretch retry envelope to weather StepFun 10 RPM cap #3497 $9.6, fix(llm): skip legacy thinking block for claude-opus-4-8 #3440/Revert "fix(llm): strip temperature/top_p for claude-opus-4-8 (#3427)" #3441 $19.6, fix(llm): match LiteLLM proxy model_info by litellm_params.model too #3429 $11.2, fix(acp): classify upstream-gateway SSE bytes-repr corruption distinctly #3482 ~$X), versus $1–$2 for the small "Register" PRs that produced no extra comments (Register claude-opus-4-8 in Claude Code ACP provider #3491, Register claude-opus-4-8 in SDK model lists #3424).

For simonrosenberg's ACP PRs, the agent isn't OpenHands SaaS — the Co-Authored-By: Claude Opus 4.7/4.8 (1M context) <noreply@anthropic.com> trailer is the Claude Code CLI signature. So the verbosity is not OpenHands-SaaS-specific — Claude Opus 4.7/4.8 produces the same shape of comments under Claude Code too.

Hypothesis

• The trigger isn't the agent harness — it's Claude Opus 4.7/4.8 + a long investigative session with a 1M context window. When the model has spent hundreds of turns reasoning out a root cause, it appears to treat the eventual code change as a place to "anchor" that reasoning, instead of trusting the PR description / git history.
• "Register a model" and "Revert" sessions (short, low-context) produced clean, minimal diffs from the same model/agent (Register claude-opus-4-8 in Claude Code ACP provider #3491, Register claude-opus-4-8 in SDK model lists #3424, Revert "fix(llm): strip temperature/top_p for claude-opus-4-8 (#3427)" #3441). The verbosity correlates with session length / investigation depth, not with the model alone.
• The pattern reproduces across at least two distinct agent harnesses (OpenHands SaaS and Claude Code), which strongly suggests the corrective action belongs in the system prompt / coding guidelines fed to the model, not in any one agent harness.

Suggested next step (not done in this issue)

Add an explicit "comments policy" item to the SDK contributor guidelines (e.g. AGENTS.md <CODE> section) that we then feed to whichever agent is opening PRs against this repo. Something like:

Comments are reserved for non-obvious behavioural notes that aren't expressible in code structure or naming. Do not mirror the PR description, decision matrices, or external links into source comments. If the reasoning has to live somewhere durable, link it from the PR body and let the source speak for itself.

That's a one-line policy change to the contributor guide; happy to open a follow-up PR if you want.

---

This comment was generated by an AI agent (OpenHands) on behalf of @juanmichelini.

[openhands-ai]

Summary

The user's request — "look at the last 100 PRs in software-agent-sdk see if you can find a pattern of which PRs have very verbose comments and what model/agent/settings are using" — was an investigation/analysis task, not a code change. I answered it by posting findings on the issue:

#3498 (comment)

What I did

1. Pulled the last 100 PRs (Add qwen3.6-plus to verified models #3366 → fix(step-3.7-flash): stretch retry envelope to weather StepFun 10 RPM cap #3497) via the GitHub API.
2. For every PR, diffed the additions and computed the longest consecutive added-comment block (Python #/docstring + JS/TS //), comment/code ratio, etc. — to score "verbosity".
3. For PRs whose body linked an app.all-hands.dev/conversations/<id> URL, called GET /api/v1/app-conversations?ids=... on OpenHands Cloud to read llm_model, agent_kind, metrics.accumulated_cost, and context window.
4. For PRs with no Cloud URL but a Co-Authored-By: Claude Opus 4.x … trailer (Claude Code CLI signature), extracted the model from commit messages.
5. Cross-tabulated verbosity vs. model / agent / author.

Headline answer

• 26 of 27 PRs with a ≥10-line consecutive added comment block carry an AI signature. The single non-AI exception (fix(security): pin ACP npx launchers to reviewed versions (closes #3357) #3399) is a security pin.
• The model is Claude Opus 4.7 / 4.8 across the verbose set, on two distinct harnesses:
  • OpenHands SaaS: 14/16 queryable Cloud conversations used litellm_proxy/claude-opus-4-7 with agent_kind = openhands and a 1M context window.
  • Claude Code CLI: 12× Opus 4.8 + 3× Opus 4.7 via Co-Author trailers (mostly @simonrosenberg's ACP PRs).
• Trigger is model + long investigative session, not the harness. Short, low-context sessions from the same model (e.g. "Register a model" Register claude-opus-4-8 in Claude Code ACP provider #3491/Register claude-opus-4-8 in SDK model lists #3424 or revert Revert "fix(llm): strip temperature/top_p for claude-opus-4-8 (#3427)" #3441, $1–$2 cost) produced clean, minimal diffs; the verbose ones are root-cause "fix(...)" PRs ($9–$20 cost).
• Authors of verbose PRs: simonrosenberg (15), juanmichelini (4), VascoSch92 (2), neubig (2), xingyaoww (1), chuckbutkus (1), rbren (1), Sanjays2402 (1).
• The verbose comments duplicate the PR description's "Why" section, bake decision matrices/URLs into code, and cross-reference other PRs — i.e. exactly the four drift-prone categories called out in the issue body.

What I did not do

• No code/config changes, no branch, no PR. The user's request was strictly analysis, and the issue's previous AI-authored PR (fix(step-3.7-flash): stretch retry envelope to weather StepFun 10 RPM cap #3497) is itself the motivating example of the problem, so opening another bot PR with code changes would have been off-task. I ended the comment with a concrete (one-line) follow-up suggestion — adding a "comments policy" to AGENTS.md <CODE> — explicitly offered for a separate PR if @juanmichelini wants it.

Faithfulness checklist

• ✅ Focused on the latest comment (the analysis request) over the issue body.
• ✅ Used the GitHub API (not the browser) for PR data.
• ✅ Used the OpenHands Cloud V1 API to attribute llm_model / agent_kind / settings, including the 1M context-window detail.
• ✅ Disclosed the comment was generated by an AI agent on behalf of the user, per the external-services guideline.
• ✅ No throwaway analysis scripts committed to the repo — they live only under /tmp.

[juanmichelini]

@OpenHands please add an explicit "comments policy" item to the SDK contributor guidelines (e.g. AGENTS.md section)

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

Addressed issue #3498 by adding an explicit "comments policy" item to the SDK contributor guidelines, as requested in the key comment from @juanmichelini.

PR opened (draft): #3502

Checklist against the request

• Key comment addressed: Added an explicit "comments policy" item to AGENTS.md under the <CODE> section (the exact location requested).
• Policy reflects the issue's reasoning: Covers the three concerns the reporter raised — comments that restate code, comments that duplicate PR-description/git blame context, and comments that describe non-local parts of the system without a sync mechanism — plus the positive guidance to comment on genuinely unintuitive things.
• Concrete callback to the cited PR: The "3-line change shouldn't produce 19 lines of comments" line directly mirrors the example from the issue body.
• No tests required: Per the guidelines, documentation-only changes don't need new tests.
• Pre-commit run: uv run pre-commit run --files AGENTS.md — passes (hooks correctly skip Markdown).
• Branch naming: Uses the openhands/ prefix (openhands/comments-policy-agents-md).
• Commit message: Clear, signed with Co-authored-by: openhands <openhands@all-hands.dev>.
• PR description: Follows the repo's PULL_REQUEST_TEMPLATE.md (Why / Summary / Issue Number / How to Test / Type / Notes), includes Closes #3498, and contains the required AI-disclosure note.
• Draft state: Opened as a draft per repo convention (not marked ready for review).

Conciseness

The diff is a single, focused 5-line addition to AGENTS.md — no extraneous files, no formatting churn, no unrelated edits. Nothing to revert.

Note on scope

I did not change the earlier <CODE> bullets or touch package-level AGENTS.md files, since the request was specifically to add a single policy item at the root AGENTS.md <CODE> section. If you'd like the policy mirrored into the package-level guidance (e.g. openhands-sdk/openhands/sdk/AGENTS.md) or want me to mark the PR ready for review, just say the word.

[juanmichelini]

Fix did not work, see recent PR #3521
2 line code fix, but 7 lines of comments

[juanmichelini]

@OpenHands please create two PRs.

1. in this repo Very verbose comments on PRs #3498 modify the system prompt to only write the tests when the code is doing something unintuitive, similar to docs(agents): add explicit comments policy to AGENTS.md  #3502 but modifying openhands-sdk/openhands/sdk/agent/prompts/system_prompt.j2 also lets keep it short and sweet

2. create a PR in the extensions repo https://github.com/OpenHands/extensions/ modify https://github.com/OpenHands/extensions/tree/main/skills/code-review so that it suggests removing unnecessary comments.

[openhands-ai]

@juanmichelini your session has expired. Please login again at OpenHands Cloud and try again.

[juanmichelini]

@OpenHands please create two new PRs.

1. in this repo Very verbose comments on PRs #3498 modify the system prompt to only write the tests when the code is doing something unintuitive, similar to docs(agents): add explicit comments policy to AGENTS.md docs(agents): add explicit comments policy to AGENTS.md  #3502 but modifying openhands-sdk/openhands/sdk/agent/prompts/system_prompt.j2 also lets keep it short and sweet

2. create a PR in the extensions repo https://github.com/OpenHands/extensions/ modify https://github.com/OpenHands/extensions/tree/main/skills/code-review so that it suggests removing unnecessary comments.