[DISCUSSION] Rethinking Code Review and Testing in the Agent Era

@Xuanwo

Motivation

It seems large PRs generated by AI is becoming a norm nowadays. SQL query engine is highly structured machine that the output is predictable from the given inputs. I think it is possible that we could build a testing framework that every AI generated code changes could run against the testing machine and we have highly confidence to ship the changes.

#23738 is an example recently which has 4500+ LoC diffs from @Xuanwo. I believe such trends will continue, presenting an opportunity for us to review how we can conduct code reviews in agent era.

What we expect after running testing framework

Correctness
No Performance regression
Clean crate separation (logical plan stuffs doesn't go to physical layer and vice versa) Great crate design so it has clear responsibility and not ends up the mess.

What we have

Correctness - sqllogictest / datafusion-fuzzer
Performance - benchmark bot
Documentation to tell AI the purpose of each crate / component

What are the GAPs

Correctness coverage
I think we need more test coverage for correctness that gives us high confidence to ship the code if the changes pass the test (e.g., catching silent semantic bugs in complex logic without relying solely on manual test writing).
Benchmarking for agents
I'm thinking of more benchmark coverage that is easy to run and reproduce locally (assume most contributors run benchmark in their laptop) and also cost efficient for benchmark bot.
Architecture & System Design
It should be carefully reviewed by reviewers (human). We might need tools to help us easily understand whether changes fit our system design or maybe we need spec in details to tell AI to follow the purpose of each components

Objective

This discussion is to gather ideas for how code review could be like in agent era, and what gaps are left for us to achieve the goals. I hope we could ship large PRs without carefully review by human but with high quality code one day.

@jayzhan211

Thanks for filing this @jayzhan211

It seems large PRs generated by AI is becoming a norm nowadays. SQL query engine is highly structured machine that the output is predictable from the given inputs.

Yes, 100% agree.

What are the GAPs

Another gap I think is in how do we find what is most importnt to review (aka use our review bandwidht)

[DISCUSSION] PR Review Process / States / "How to find the PRs in the list that need review" #21038

Correctness coverage
I think we need more test coverage for correctness that gives us high confidence to ship the code if the changes pass the test (e.g., catching silent semantic bugs in complex logic without relying solely on manual test writing).

I think the sqllogictesting framework is the best we have in this regard. Running more such tests I think is scalable and adds good end to end coverage.

I have found LLms are quite good at creating sqllogictests, so maybe we can spend more time having them expand our existing coverage. For example I have been trying to do this with

SQL / slt coverage for IN lists #23307

Benchmarking for agents
I'm thinking of more benchmark coverage that is easy to run and reproduce locally (assume most contributors run benchmark in their laptop) and also cost efficient for benchmark bot.

This is good -- I tink our bench.sh script is reasonable for this, and @Omega359 's work in this ticket I think will help

[EPIC] Add SQL based benchmarking #21706

Architecture & System Design
It should be carefully reviewed by reviewers (human). We might need tools to help us easily understand whether changes fit our system design or maybe we need spec in details to tell AI to follow the purpose of each components

I think the idea of formalizing / documenting (for humans and agents) what we are looking for in reviews would probably help a lot. We already have https://github.com/apache/datafusion/blob/main/AGENTS.md and https://github.com/apache/datafusion/tree/main/.ai/skills (for ffi that @timsaucer added).

Perhaps we can add something similar for performance related or other skills (e.g. that if changing join logic tests should always include nulls, etc).

Another idea could be to add a review skill that tries to explain what we are looking for in each PR

Objective

This discussion is to gather ideas for how code review could be like in agent era, and what gaps are left for us to achieve the goals. I hope we could ship large PRs without carefully review by human but with high quality code one day.

Another idea could be to add a review skill that tries to explain what we are looking for in each PR

Use this at work, very effective. The skill can be used to catch cases which are easy to spot and more general in nature.

Another idea could be to add a review skill that tries to explain what we are looking for in each PR

Use this at work, very effective. The skill can be used to catch cases which are easy to spot and more general in nature.

Awesome -- here is an initial attempt at documenting it

Docs: Add PR review guide #24051

Docs: Update PR template to ask for user-visible rationale (apache#24053

We could add cargo-mutants to the checklist an agent follows after making changes. Here are the docs. Essentially, it injects small changes into the source code (a flipped comparison, a swapped return value) and checks whether the tests catch them. It gives some assurance that the tests are robust, if a mutant is uncaught then the agent and developer can iterate.

Something like, cargo mutants --in-diff <diff> --package <crate> keeps the scope to just the local change set so it can be run before a PR.

Something like, cargo mutants --in-diff --package keeps the scope to just the local change set so it can be run before a PR.

I am a big fan of "ablation testing" -- where you remove the new code that is added and run the tests and the tests should fail.

This ensures:

the tests cover the change
The change is reachable other than programatically

We could add cargo-mutants to the checklist an agent follows after making changes. Here are the docs. Essentially, it injects small changes into the source code (a flipped comparison, a swapped return value) and checks whether the tests catch them. It gives some assurance that the tests are robust, if a mutant is uncaught then the agent and developer can iterate.

Something like, cargo mutants --in-diff <diff> --package <crate> keeps the scope to just the local change set so it can be run before a PR.

This is a good idea, I have thought about it and there are two challenges if we want to implement it:

Identifying and fixing all existing code for mutation tests failure can be too expensive, so ideally we want to only mutate the PR diff
The following case seems quite common in datafusion, but it's a false positive for mutation tests. We need a way to skip them

# Mutating the `cond` won't fail any test, and it's expected
if cond:
    fast_path()
else:
    fallback()

Identifying and fixing all existing code for mutation tests failure can be too expensive, so ideally we want to only mutate the PR diff

I was experimenting this locally and I agree with you here. I will do some experiment maybe ask codex to suggest the best method without the overhead.

Docs: Add PR review guide (apache#24051)

Docs: Update PR template to ask for user-visible rationale (apache#24053

Docs: Add PR review guide (apache#24051)

There is more nuance to using cargo mutants than I initially thought. The constraint is runtime.

Against unit tests it's fine. The suite is fast, so a contributor can run mutants on their diff and get through it. Against slt it isn't. slt is slow enough that running it once per mutant would take hours to over a day to run.

A fix could be a coverage map from each mutant to the slt files that touch its path, so you only run those. That cuts runtime, but now that map needs to be maintained. I don't think that is worth the trouble.

Running mutants against unit tests only is much faster but some "uncaught" mutants are actually caught by slt tests and others are actual gaps. Differentiating them needs a certain amount of codebase familiarity.
A possible solution to this issue would be to have only the surviving mutants run against the slt suite. Running cargo mutants would have two phases.

Phase 1 — fast filter (unit tests only, on the diff):

git diff <merge-base> > pr.diff
cargo mutants --in-diff pr.diff -j 4 --timeout-multiplier 3 -- --lib
# survivors are listed in mutants.out/missed.txt (file:line + the mutation)

Phase 2 — run slt on only the survivors:

# 1. pull the surviving file:line locations out of phase 1's output
#    (from mutants.out/missed.txt or outcomes.json)
# 2. build a narrowed diff containing only those lines  (survivors.diff)
# 3. re-run mutants on just those, with a test command that includes slt:
cargo mutants --in-diff survivors.diff -j 4 --timeout-multiplier 3 \
    --test-workspace true -- --test sqllogictests

Phase 2 could still take quite a bit of time depending on how good the existing unit suite is. If there are too many survivors then this falls back to being too much to do.

This feels like a decent option for ensuring correctness coverage but could be too heavyweight to have an agent/contributor/both run for every PR. Everyone please let me know your thoughts!

Maybe you could describe what Brain Scanner is and how it helps DataFusion?

@kevin-lozada-santos

@jayzhan211 your discussion has gotten very concrete around correctness gaps, agent-era review skills, and the cost of mutation and sqllogictest coverage. I’m validating a small tool called Brain Scanner on real agent-heavy projects: a free, async two-session scan that maps the repo, shows source-linked findings, and asks whether the result is useful. Would a short note inviting 1–2 contributors to try it on a public or sanitized fixture be on-topic here, or is there a preferred place in the DataFusion community for that kind of validation? No code or credentials are needed, and I’m asking before posting anything broader.

Hi @kevin-lozada-santos I appreciate that you are trying to help but this post seems like a promotion which is not ideal for a discussion.

Completely optional, and if this thread is not the right place, no worries.
This is probably not the right place

Since DataFusion is all open source, why not just run your tool and then open an [DISCUSSION] issue with a link for anyone to check it if they wanted. Better yet, perhaps review your tool's results first and see if you can find some insight of value that is worth a ticket or something

Navigation Menu

Repository navigation

[DISCUSSION] Rethinking Code Review and Testing in the Agent Era #23839

Motivation

What we expect after running testing framework

What we have

What are the GAPs

Objective

What are the GAPs

Objective

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[DISCUSSION] Rethinking Code Review and Testing in the Agent Era #23839

Description

Motivation

What we expect after running testing framework

What we have

What are the GAPs

Objective

Activity

alamb commented on Aug 2, 2026

What are the GAPs

Objective

kumarUjjawal commented on Aug 2, 2026

alamb commented on Aug 2, 2026

HairstonE commented on Aug 5, 2026

alamb commented on Aug 5, 2026

2010YOUY01 commented on Aug 10, 2026

kumarUjjawal commented on Aug 10, 2026

HairstonE commented on Aug 18, 2026

jayzhan211 commented on Sep 18, 2026

kumarUjjawal commented on Sep 18, 2026

alamb commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions