add delta lake to iceberg blueprint and IT by derrickaw · Pull Request #4103 · GoogleCloudPlatform/DataflowTemplates

derrickaw · 2026-08-03T21:06:20Z

No description provided.

codecov · 2026-08-03T21:10:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.01%. Comparing base (3515682) to head (3cfd16c).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #4103   +/-   ##
=========================================
  Coverage     38.01%   38.01%           
  Complexity      511      511           
=========================================
  Files           195      195           
  Lines         11761    11761           
  Branches       1181     1181           
=========================================
  Hits           4471     4471           
  Misses         6924     6924           
  Partials        366      366

Components	Coverage Δ
spanner-templates	`∅ <ø> (∅)`
spanner-import-export	`∅ <ø> (∅)`
spanner-live-forward-migration	`∅ <ø> (∅)`
spanner-live-reverse-replication	`∅ <ø> (∅)`
spanner-bulk-migration	`∅ <ø> (∅)`
gcs-spanner-dv	`∅ <ø> (∅)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gemini-code-assist · 2026-08-18T17:48:00Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a new Dataflow template that enables data migration from Delta Lake tables to Apache Iceberg tables. It includes the necessary YAML definitions, Java interface files, and updates to the template generation tooling to support this new pipeline. Additionally, an integration test has been included to verify the pipeline's correctness.

Highlights

New Dataflow Template: Added a new Delta Lake to Iceberg batch pipeline template to the repository.
Template Configuration: Introduced new YAML configuration files and a Java interface for the Delta Lake to Iceberg template.
Build Tooling Updates: Updated the Python generation script to dynamically handle imports and default parameters for Java templates.
Integration Testing: Added a comprehensive integration test for the new template to ensure end-to-end functionality.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the Delta Lake to Iceberg batch pipeline template, including its YAML definition, Java metadata interface, documentation, and integration tests. It also updates the Python template generator to conditionally import the Beam SDK Default options. The feedback suggests improving the YAML syntax for requirements to use standard list notation and dynamically calculating the parquet file size in the integration test to avoid hardcoding.

chamikaramj

Thanks!

chamikaramj · 2026-08-21T22:47:55Z

+import org.apache.beam.sdk.options.Validation;
+
+@Template(
+    name = "DeltaLake_To_Iceberg_Yaml",


Similar to what we have in Job builder currently. Let's introduce a Lakehouse specific sink (that wraps Iceberg sink) and use that from this and other blueprints that write to Lakehouse. Also, pls update the naming (step names, documentation etc.) accordingly.

chamikaramj · 2026-08-21T22:50:37Z

+
+  @TemplateParameter.Text(
+      order = 3,
+      name = "table",


This is the Lakehouse table ? Lat's make this clear in the naming pls.

chamikaramj · 2026-08-21T22:51:14Z

+
+  @TemplateParameter.Text(
+      order = 5,
+      name = "catalogProperties",


Same for this and other configs. Let's make it clear if we are configuring Delta Lake or Lakehouse here.

updated all configs associated with iceberg and changed to lakehouse

chamikaramj · 2026-08-21T22:53:08Z

+        "import com.google.cloud.teleport.metadata.TemplateCategory;",
+        "import com.google.cloud.teleport.metadata.TemplateParameter;",
+    ]
+    if has_defaults:


Can't we just add these to the list without the condition ?

there are a few templates/blueprints that don't have defaults so this is still needed for now

chamikaramj · 2026-08-21T22:54:00Z

@@ -0,0 +1,75 @@
+template:
+  name: "DeltaLake_To_Iceberg_Yaml"


Ditto regarding naming with "Lakehouse".

chamikaramj · 2026-08-21T22:55:33Z

+    gcsClient.createArtifact(deltaTableDir + "/part-00000.parquet", parquetBytes);
+
+    // Create and upload Delta Lake transaction log
+    String commitContent =


Please use the Delta client to write instead of manually writing commit logs. For example,

https://github.com/apache/beam/blob/7de4d3ab73b6951178f6eb625b55597b2ee9081b/sdks/java/io/delta/src/test/java/org/apache/beam/sdk/io/delta/DeltaIOIT.java#L143

…ke2LHIT

pull-request-size Bot added the size/L label Aug 3, 2026

derrickaw changed the title ~~add delta lake to iceberg IT~~ add delta lake to iceberg blueprint and IT Aug 3, 2026

derrickaw added the addition New feature or request label Aug 3, 2026

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from f0d7c7e to d642c4b Compare August 8, 2026 01:59

pull-request-size Bot added size/XL and removed size/L labels Aug 17, 2026

derrickaw added 7 commits August 18, 2026 16:37

add delta lake to iceberg IT

66fc208

fix spotless and rerun auto-generation

92aadfa

switch to interrupt call

9caddc2

change to a dynamic import to fix error

ece2e53

add forgotten yaml definition file

58cc7cc

add generated README for DeltaLakeToIceberg template

6f0eb7d

switch back to stop method for direct runner

353123c

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from f6ff2b9 to 353123c Compare August 18, 2026 16:37

derrickaw marked this pull request as ready for review August 18, 2026 17:47

gemini-code-assist Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread yaml/src/main/yaml/DeltaLakeToIceberg.yaml Outdated

Comment thread yaml/src/test/java/com/google/cloud/teleport/templates/yaml/DeltaLakeToLakehouseYamlIT.java Outdated

derrickaw added 2 commits August 18, 2026 18:20

address gemini comment

0b6fa4a

spotless

a9b0bcf

derrickaw force-pushed the 20260803_addDeltaLake2LHIT branch from e6ed587 to a9b0bcf Compare August 19, 2026 16:00

derrickaw requested a review from chamikaramj August 21, 2026 17:08

chamikaramj requested changes Aug 21, 2026

View reviewed changes

derrickaw added 7 commits August 25, 2026 14:25

initial comments fixes

9fdced5

change iceberg to lakehouse

ee83e6e

change to a wrapped writetoiceberg via writetolakehouse

99d9dd5

update yaml file with wrapper transform

0dc18f3

remove old writetolakehouse transform idea

e35c1d2

spotless

a0cf3b4

change to renamed transform

a323e94

derrickaw added 4 commits August 28, 2026 14:15

address Delta client comment

bf3e10c

spotless

f1580c4

Merge remote-tracking branch 'upstream/main' into 20260803_addDeltaLa…

498d496

…ke2LHIT

fix generated file

3cfd16c

Uh oh!

Conversation

derrickaw commented Aug 3, 2026

Uh oh!

codecov Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gemini-code-assist Bot commented Aug 18, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Aug 3, 2026 •

edited

Loading