Skip to content

[DO NOT MERGE] Validate beam 2.76.0 RC2 - #4135

Open
Amar3tto wants to merge 2 commits into
GoogleCloudPlatform:mainfrom
akvelon:validate-beam-276-rc2
Open

[DO NOT MERGE] Validate beam 2.76.0 RC2#4135
Amar3tto wants to merge 2 commits into
GoogleCloudPlatform:mainfrom
akvelon:validate-beam-276-rc2

Conversation

@Amar3tto

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request prepares the repository for the validation of Apache Beam 2.76.0 RC2. It involves bumping the core Beam version, updating the associated Maven repository, and cleaning up Python dependency lists to ensure compatibility and consistency with the new release candidate.

Highlights

  • Apache Beam Version Update: Updated the Apache Beam version to 2.76.0 (RC2) across the pom.xml and various Python requirement files to ensure the project uses the latest release candidate.
  • Dependency Management: Refined Python dependencies by removing unused packages like betterproto, envoy-data-plane, and grpcio-tools, while adding google-cloud-dataflow-client.
  • Build Configuration: Updated the 'validateCandidate' profile in pom.xml to target all JDK versions and configured the Apache Beam Maven repository for the new release candidate.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates Apache Beam and its linked dependencies to version 2.76.0 (specifically 2.76.0rc2 for Python) across Maven and Python requirements files. It also modifies the validateCandidate Maven profile activation. Feedback points out that the JDK activation range [1.) introduced in pom.xml is syntactically invalid in Maven and should be corrected to a valid range such as [1.8,).

Comment thread pom.xml
<id>validateCandidate</id>
<activation>
<activeByDefault>false</activeByDefault>
<jdk>[1.)</jdk>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The JDK activation range [1.) is syntactically invalid in Maven. A valid version range requires a comma to separate the lower and upper bounds (e.g., [1.8,) or [1,)). Please correct this to a valid range such as [1.8,) or [11,) depending on the minimum supported JDK version.

Suggested change
<jdk>[1.)</jdk>
<jdk>[1.8,)</jdk>

@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.21%. Comparing base (9d8862d) to head (5ace88e).
⚠️ Report is 23 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4135      +/-   ##
============================================
- Coverage     56.21%   56.21%   -0.01%     
  Complexity     7340     7340              
============================================
  Files          1126     1126              
  Lines         68748    68748              
  Branches       7784     7784              
============================================
- Hits          38650    38649       -1     
- Misses        27599    27602       +3     
+ Partials       2499     2497       -2     
Components Coverage Δ
spanner-templates 87.64% <ø> (+<0.01%) ⬆️
spanner-import-export 69.03% <ø> (ø)
spanner-live-forward-migration 89.59% <ø> (+0.01%) ⬆️
spanner-live-reverse-replication 83.72% <ø> (+0.01%) ⬆️
spanner-bulk-migration 92.30% <ø> (+<0.01%) ⬆️
gcs-spanner-dv 88.93% <ø> (+0.01%) ⬆️
see 3 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Amar3tto

Copy link
Copy Markdown
Collaborator Author

PostgreSQLDatastreamToSpannerInheritanceIT.testPostgreSqlInheritance:193->validateResult:264 expected that contains unordered record (and case insensitive) {ID=1, NAME=Parent Row 1}, but only had [{name=Child Row 1, id=2}, {name=Grandchild Row 1, id=3}]

[ERROR] com.google.cloud.teleport.v2.templates.DataStreamToBigQueryIT.testDataStreamMySqlToBigQuery -- Time elapsed: 2400 s <<< ERROR!

[ERROR] com.google.cloud.teleport.v2.templates.FirestoreToFirestoreIT.testFirestoreToFirestore_withReadTime -- Time elapsed: 593.5 s <<< ERROR!

[ERROR] com.google.cloud.teleport.v2.templates.ManagedIOToManagedIOIT.testBigQueryToBigQuery -- Time elapsed: 626.9 s <<< ERROR!

[ERROR] =com.google.cloud.teleport.v2.templates.StreamingDataGeneratorIT.testFakeMessagesToGcsWithSchemaTemplate -- Time elapsed: 0.872 s <<< ERROR!

[ERROR] Run 1: StreamingDataGeneratorIT>TemplateTestBase.setUpBase:277->TemplateTestBase.getSpecPath:353 » UncheckedExecution com.google.cloud.storage.StorageException: com.google.api.client.http.HttpResponseException: 503 Service Unavailable GET http://metadata.google.internal/computeMetadata/v1/universe/universe-domain

@Abacn

Abacn commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

testBigQueryToBigQuery:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1888)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1375)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserialize(AbstractDeserializer.java:274)
	at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1816)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:609)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:604)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:229)
	at jdk.proxy2/jdk.proxy2.$Proxy19.getGoogleCloudStorageReadOptions(Unknown Source)
	at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1$GcsUtilFactory.create(GcsUtilV1.java:157)
	at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.<init>(GcsUtil.java:101)

Caused by apache/beam@eb9e7a3

Due to that ManagedIO is still on Beam 2.75.0 and doesn't support deserialize GcsOption

@Abacn

Abacn commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

DataStreamToBigQuery tests

pipeline and checkers was successful, but stuck at clean up:

[Time-limited test] ERROR org.apache.beam.it.gcp.datastream.DatastreamResourceManager - Failed to delete connection profile testdatastreammysqltobigquery-jdbc-pr_20260817-200328_wm5twc.
dev.failsafe.FailsafeException: java.lang.InterruptedException
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:196)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at org.apache.beam.it.gcp.datastream.DatastreamResourceManager.cleanupAll(DatastreamResourceManager.java:670)
	at org.apache.beam.it.common.utils.ResourceManagerUtils.cleanResources(ResourceManagerUtils.java:165)

It appears the test is also highly flaky on main branch:

https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/datastream-pr.yml

Likely unrelated to the Beam version bump. Need to fix the test on main branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants