Support INSERT OVERWRITE for MemTable

[osipovartem]

Which issue does this PR close?

• Part of [EPIC] Complete DML Support (MERGE, INSERT OVERWRITE, TRUNCATE) #19617.

Rationale for this change

DataFusion parses and plans INSERT OVERWRITE, but MemTable rejects every insert operation except append. Users therefore cannot replace the contents of an in-memory table with the standard overwrite operation.

What changes are included in this PR?

• allow InsertOp::Overwrite in MemTable::insert_into
• let MemSink replace each target partition after the input stream completes successfully
• continue rejecting the distinct InsertOp::Replace operation

What is the testing strategy for this PR?

Three unit tests verify that overwrite replaces existing rows, distributes multiple replacement batches across multiple target partitions, and clears the table for empty input. A separate test confirms that InsertOp::Replace remains unsupported.

Validated with:

• cargo test -p datafusion --lib test_insert_overwrite -- --nocapture
• cargo clippy -p datafusion -p datafusion-datasource -p datafusion-catalog --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check

Are there any user-facing changes?

Yes. INSERT OVERWRITE now replaces all existing data in a MemTable. There are no breaking public API changes.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.82609% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.92%. Comparing base (133111f) to head (192b1f8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/core/src/datasource/memory_test.rs	87.37%	4 Missing and 9 partials ⚠️
datafusion/catalog/src/memory/table.rs	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24969    +/-   ##
========================================
  Coverage   81.92%   81.92%            
========================================
  Files        1134     1134            
  Lines      426000   426111   +111     
  Branches   426000   426111   +111     
========================================
+ Hits       349000   349091    +91     
- Misses      56304    56313     +9     
- Partials    20696    20707    +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@osipovartem,

Thanks for working on this. The change looks good overall, and I like that INSERT OVERWRITE is enabled for MemTable while Replace remains explicitly unsupported. The buffering before modifying the target also makes the overwrite behavior safer if the input stream fails.

I have one non-blocking test coverage suggestion below.

[kosiew]

Could we add an overwrite test with multiple target partitions and multiple input batches? The current tests cover a single target partition, while MemSink::write_all distributes batches round-robin and then replaces each target partition independently. A test that asserts every partition contains only its replacement batches would give us coverage for the full overwrite loop.

[osipovartem]

Added in e8c94de. The new test uses two target partitions with distinct existing rows and four replacement batches, then verifies the round-robin batch distribution and that both partitions contain only replacement data. The targeted tests and the PR clippy command pass locally.

[kosiew]

@osipovartem,

Thanks for the follow-up. I reviewed the latest changes and the additional overwrite coverage looks good.

The earlier request from @kosiew is addressed by e8c94de. The new test starts with distinct data across two target partitions, feeds four replacement batches, and confirms that each output partition contains only its round-robin assigned replacement values: [10, 30] and [20, 40].

I also verified the author's follow-up against the current HEAD, and the Codecov report does not point to any new actionable issue.

I did not find any additional problems in the follow-up changes. Looks good to me.

[kosiew]

🚀
@osipovartem
Thank you for your contribution.

[kosiew]

@osipovartem, thanks for the update. I took another look and the previous comment has been addressed. Commit e8c94dea6 now covers two distinct target partitions with four replacement batches, and verifies the expected round-robin outputs [10, 30] and [20, 40] while confirming the previous rows are gone.

I didn't find any new issues in the follow-up changes. The latest HEAD update is a merge from main and doesn't introduce any additional branch-side changes related to this feature.

Looks good to me. Thanks!

[kosiew]

🚀
@osipovartem
Thank you for your contribution.