Repository navigation
Discussion: guidelines for LLM-generated PR reviews #25317
Description
Is your feature request related to a problem or challenge?
LLM-generated PR reviews are becoming more common in DataFusion, and they're often helpful. But I've noticed some hidden issues with them, and I'd like to start a discussion to exchange opinions.
Once we've reached some agreement, we can document it as an AI review policy, similar to https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions.
Issue
LLM-generated reviews are sometimes hard to parse. Human-written review feedback is usually easy to understand: it gets to the point in one or two sentences. AI reviews (say, today's Codex/Claude with the best models in default mode) can be hard to understand: they throw a verbose amount of detail at you, and you have to spend time reconstructing the idea behind it.
This consumes the contributor's time on interpretation. One potential consequence is that it encourages contributors to let AI address the review feedback entirely. Such a loop would degrade the codebase quality very quickly, since today's LLM agents still can't handle medium-complexity tasks in DataFusion well.
Proposed guidelines:
- AI reviews are always encouraged, but the contributor can address them selectively and skip the ones that are hard to parse or overly verbose.
(I believe some of them are easy to understand directly; only some are not.) - Reviewers are encouraged to do the interpretive labor: first understand the LLM-generated review, then express it in an understandable way.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Activity
2010YOUY01 commentedon Sep 15, 2026
Another idea that came to mind is a checklist-based review. 🤔
I remember someone previously proposed a contributor PR checklist, but it got pushed back because such a checklist tends to grow very long, and eventually no one reads it.
However, this seems worth maintaining if we can trigger LLM reviews cheaply. The idea: we maintain a checklist for PRs (e.g., the testing policy), and the AI reviewer only points out violations of the checklist. Since each item maps to a known rule, these reviews are much easier to understand than open-ended ones. And because the checklist is read by the LLM rather than by every contributor, it can afford to be long.
kumarUjjawal commentedon Sep 15, 2026
Thank you @2010YOUY01 for starting the discussion. I totally agree with your point. I think a simple bulleted list just like in https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions should also work for the AI reviews.
comphead commentedon Sep 15, 2026
Checklist is a good idea for the review, I personally have a skill that has predefined checklist what to check, we can come up with best practices for datafusion specific review skill which may include:
alamb commentedon Oct 2, 2026
I tried to address this as part of
Specifically here is my proposal
AI-assisted reviews
The same standard applies to AI-generated reviews as to AI-generated code: you
should have read and understand anything you post. Raw AI review output is often
verbose with unnecessary details, and it can take substantial effort to figure
out what is actually being asked.
If you review PRs with the help of an AI tool, read all comments first, remove
detail that is unnecessary or you don't understand, and explain the rest in your
own words so that each comment makes a clear, specific request.
Update contributor guidelines regarding AI spam, reviews, code (apach…
Update contributor guidelines regarding AI spam, reviews, code (apach…