Delta kernel conversion target by vaibhavk1992 · Pull Request #801 · apache/incubator-xtable

vaibhavk1992 · 2026-02-06T16:46:08Z

What is the purpose of the pull request

This PR migrates
XTable's Delta Lake integration from Delta Standalone to
Delta Kernel for writers

Brief change log

Added unit test for delta kernel data file updates
Added unit tests for conversion target

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added TestConversionController to verify the change.
Manually verified the change by running a job locally.

…itions,big fixes

the-other-tim-brown · 2026-02-23T15:14:34Z

@vaibhavk1992 can you update the PR description?

the-other-tim-brown

@vaibhavk1992 can you complete a self-review of the code first to clean up a bit?

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileUpdatesExtractor.java

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelDataFileUpdatesExtractor.java

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelSchemaExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileUpdatesExtractor.java

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelSync.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileUpdatesExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java

Merged 7 upstream commits: - f991e31 Parquet Source: snapshot sync fixes (apache#806) - 4307565 Parquet source: column stats support (apache#805) - 5c25674 Remove wildcard imports and enforce with spotless (apache#809) - fe7215e add .sdkmanrc to .gitignore (apache#793) - abbf4b7 fix(iceberg): nested comments (apache#797) - 8e58367 Remove redundant getSnapshotAt calls (apache#791) - 8cab6a2 fix(delta): avoid NPE for binary in map/array (apache#795) Resolved conflicts: - TestDeltaKernelSchemaExtractor.java: kept StructField import needed for new tests

Fixed wildcard imports in Delta Kernel test files to comply with spotless rules enforced in upstream commit 5c25674.

The spotless:apply command removed wildcard imports but didn't add back all necessary specific imports. Added missing imports: TestDeltaKernelReadWriteIntegration.java: - Static assertions (assertEquals, assertTrue, assertFalse, assertNotNull) - java.util.* (Random, UUID, List, Map, Set, Arrays, Collections, etc.) TestDeltaKernelSync.java: - Static assertions (including fail) - java.util.* (Random, UUID, List, Map, Set, Arrays, Collections, etc.) TestDeltaKernelDataFileUpdatesExtractor.java: - Static assertions (assertEquals, assertTrue, assertFalse, assertNotNull) - java.util.* (List, Arrays, Collections) All tests now compile successfully.

vaibhavk1992 · 2026-03-02T17:26:17Z

I have tried to address all the comments @vinishjail97 @the-other-tim-brown

vinishjail97 · 2026-03-02T17:59:18Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+  }
+
+  @Override
+  public void init(TargetTable targetTable, Configuration configuration) {


init() creates a new Engine from configuration, silently discarding the engine injected via the constructor. If someone builds a DeltaKernelConversionTarget with a custom engine (e.g. in tests or via direct construction), calling init() will override it without any warning. The two initialization paths (constructor vs init()) are competing and confusing — should either merge them or explicitly document when each is used.

+1 to this. @vaibhavk1992 can you address this?

@vaibhavk1992 any updates on whether this can be addressed?

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

vinishjail97 · 2026-03-02T17:59:18Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+                  }
+                } catch (Exception e) {
+                  // Log and continue to next commit
+                  log.warn(


log.warn("...", version, e.getMessage()) swallows the stack trace. Pass the exception as the last argument instead: log.warn("Failed to parse commit metadata for version {}", version, e). On-call engineers debugging a production issue won't know where the parse failure originated.

vinishjail97 · 2026-03-02T17:59:18Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+      try {
+        Table table = Table.forPath(engine, basePath);
+        this.latestSchema = table.getLatestSnapshot(engine).getSchema();
+      } catch (Exception e) {


catch (Exception e) silently sets this.latestSchema = null. If the error is a network issue, permissions error, or anything other than "table doesn't exist", this will silently proceed with a null schema and cause an NPE in addColumn() at line 366 (latestSchema.add(field)). Should catch only the specific "table not found" exception from Delta Kernel, not the broad Exception.

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelSync.java

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelDataFileUpdatesExtractor.java

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelReadWriteIntegration.java

…Kernel integration

vaibhavk1992 · 2026-03-11T16:49:42Z

@vinishjail97 I have implemented all the above suggestions.

the-other-tim-brown · 2026-03-15T18:10:42Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+      for (RowBackedAction action : actions) {
+
+        if (action instanceof io.delta.kernel.internal.actions.AddFile) {
+          io.delta.kernel.internal.actions.AddFile addFile =


Can these classes from the delta kernel be imported?

the-other-tim-brown · 2026-03-15T18:10:52Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+      Map<String, String> configMap = new HashMap<>();
+
+      configMap.put(TableSyncMetadata.XTABLE_METADATA, metadata.toJson());
+      configMap.put(


Is there a constant we can use here?

the-other-tim-brown · 2026-03-29T14:29:36Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+        DeltaKernelDataFileUpdatesExtractor.builder()
+            .engine(engine)
+            .basePath(targetTable.getBasePath())
+            // Column statistics are not needed for conversion operations


When we convert from one format to another, we actually do want the statistics. This allows query engines to leverage them for their planning operations for improved efficiency.

the-other-tim-brown · 2026-03-29T14:38:46Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+        if (action instanceof AddFile) {
+          AddFile addFile = (AddFile) action;
+          Row wrappedRow =
+              io.delta.kernel.internal.actions.SingleAction.createAddFileSingleAction(


Let's start using imports throughout the PR please. Do a sanity check of the files and make sure you are using them throughout. Highlighting every line leads to a lot of noise on the reviews.

the-other-tim-brown · 2026-03-29T15:02:51Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelUtils.java

+  public static boolean tableExists(Engine engine, String basePath) {
+    try {
+      Table table = Table.forPath(engine, basePath);
+      table.getLatestSnapshot(engine);


This looks like it will load the snapshot, is there a more lightweight way to do this?

the-other-tim-brown · 2026-03-29T15:06:23Z

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelSchemaExtractor.java

        DeltaKernelSchemaExtractor.getInstance().toInternalSchema(structRepresentation));
  }
+
+  // ========== Tests for fromInternalSchema() - New Tests ==========


Remove this comment for New Tests?

the-other-tim-brown · 2026-03-29T15:07:03Z

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelSchemaExtractor.java

+
+  // ========== Tests for fromInternalSchema() - New Tests ==========
+
+  @Test


Let's have tests with nested fields, lists and maps as well

the-other-tim-brown · 2026-03-29T15:11:22Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java

+
+  @Override
+  public Optional<String> getTargetCommitIdentifier(String sourceIdentifier) {
+    // Delta Kernel 4.0.0 does not support commit tags in commitInfo, which are required for


Let's make sure there is an issue in XTable tracking this as well.

the-other-tim-brown · 2026-03-29T15:24:23Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileUpdatesExtractor.java

+  private Seq<RowBackedAction> applyDiff(
+      Set<? extends InternalFile> filesAdded,
+      Collection<RowBackedAction> removeFileActions,
+      InternalSchema tableSchema,


tableSchema is unused, can it be removed from the method signature?

the-other-tim-brown · 2026-03-29T15:24:44Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileUpdatesExtractor.java

+        diff.getFilesAdded(),
+        diff.getFilesRemoved(),
+        tableSchema,
+        table.getPath(engine),


Should this be set to basePath?

the-other-tim-brown · 2026-03-29T15:28:44Z

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelDataFileUpdatesExtractor.java

+
+    // Verify we have AddFile actions
+    boolean hasAddFile = actionList.stream().anyMatch(action -> action instanceof AddFile);
+    assertTrue(hasAddFile, "Should contain AddFile actions");


Can we assert on the content of the AddFile to make sure it is aligned with our expectations?

the-other-tim-brown · 2026-03-29T15:30:01Z

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelDataFileUpdatesExtractor.java

+
+    // Note: file2 should not appear in actions as it's unchanged
+    // file1 should appear as RemoveFile as it's not in the new sync
+    System.out.println(


@vaibhavk1992 can you address this? We don't want printing in our tests

the-other-tim-brown · 2026-03-29T15:30:23Z

xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelDataFileUpdatesExtractor.java

+        actionList.stream().filter(action -> action instanceof RemoveFile).count();
+
+    // Verify: Should have AddFile for file3 (new file)
+    assertTrue(addFileCount >= 1, "Should have at least 1 AddFile action for new file (file3)");


These counts should be strict. Only 1 file is expected to be added and 1 removed

vaibhavk1992 and others added 30 commits May 14, 2025 10:43

single commit

dfea6a4

adding delta kernel

b75bc7c

adding the test file

16134b3

adding workable code for iteration over data

3929e95

adding Kernel 4.0 code

c6379b5

adding the working code with xtable that check getcurrenttable

6deb5f7

Merge branch 'main' into test-4

05d9984

adding the dependecies

c7ba4b9

adding getcurrentsnapshot code

0ff36a5

spotless fix

18ab9d6

spotless fix 2

e906091

spotless fix 2

e00241c

fixed partitioned test case

3fdfd31

setting junit parallel execution to true

e0102e3

testInsertsUpsertsAndDeletes test case addition,internal datatype add…

381722a

…itions,big fixes

added the fix for table basepath listing wrong paths

809bfe8

added the fix for table basepath listing wrong paths

40172f2

adding all tests

e0b7829

adding refactored code

9ac022a

spotless fix

73f33b6

fix change extraction

bee3e8a

adding the commitbacklog test cases changes

e75bb55

Merge branch 'apache:main' into test-4

21044af

adding a test case testConvertFromDeltaPartitionFormat

e212f52

adding a test case testConvertFromDeltaPartitionFormat

988cda1

adding the KernelPartitionExtractor test under kernel

1705ce4

commiting schema extractor and stats extrator

8f81109

adding unit test cases with the request changes on the PR

49ebf21

spotless fix

70fe0e3

spotless fix

fba7e0e

vaibhavk1992 added 3 commits February 13, 2026 19:38

corrected the test cases

de57c88

spotless fix

0345140

adding read write integration test case

b9f27af

vaibhavk1992 requested a review from the-other-tim-brown February 21, 2026 09:55

the-other-tim-brown reviewed Feb 23, 2026

View reviewed changes

vinishjail97 reviewed Feb 23, 2026

View reviewed changes

the-other-tim-brown reviewed Feb 23, 2026

View reviewed changes

vaibhavk1992 added 9 commits March 2, 2026 20:37

addressed comments over PR

2af1236

addressed comments over PR

2d0e16e

addressed comments over PR

ab0417c

adding data types

935d835

dummy commit to trigger actions

23f6321

Apply spotless formatting to remove wildcard imports

d622ae7

Fixed wildcard imports in Delta Kernel test files to comply with spotless rules enforced in upstream commit 5c25674.

adding read write integration test case

d405870

vaibhavk1992 requested review from the-other-tim-brown and vinishjail97 March 2, 2026 17:26

vinishjail97 reviewed Mar 2, 2026

View reviewed changes

Fix exception handling, Scala/Java mixing, and test quality in Delta …

37caddf

…Kernel integration

vaibhavk1992 requested a review from vinishjail97 March 11, 2026 16:49

the-other-tim-brown reviewed Mar 15, 2026

View reviewed changes

vaibhavk1992 added 2 commits March 16, 2026 11:36

added fixes

3915e22

adding fixes

5868207

vaibhavk1992 requested a review from the-other-tim-brown March 19, 2026 11:30

the-other-tim-brown reviewed Mar 29, 2026

View reviewed changes


		// ========== Tests for fromInternalSchema() - New Tests ==========

		@Test

Conversation

vaibhavk1992 commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

the-other-tim-brown commented Feb 23, 2026

Uh oh!

the-other-tim-brown left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vaibhavk1992 commented Mar 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vaibhavk1992 commented Mar 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

vaibhavk1992 commented Feb 6, 2026 •

edited

Loading