[flink] Verify paimon-flink's compatibility with Hadoop 3.x #7023
Conversation
| mvn clean install -DskipTests | ||
|
|
||
| # build paimon flink 1.x with hadoop 3.x | ||
| mvn clean package -DskipTests -Pflink1,hadoop3 |
There was a problem hiding this comment.
Why we need to build with -P hadoop3? What changed?
There was a problem hiding this comment.
Hadoop 2.x relies on commons-lang, which might be vulnerable to CVE-2025-48924. Some of our Paimon users have asked to provide a paimon-flink version with CVEs like have been fixed.
I'll update this to the description of this PR.
There was a problem hiding this comment.
Do you mean we will bundle the corresponding dependencies into our JAR file based on the Hadoop version?
There was a problem hiding this comment.
No, the official Paimon jars will not bundle hadoop 3.x dependencies.
This PR is more like providing a guarantee that if there is another project that relies on paimon-flink and Hadoop 3.x at the same time, this project is not supposed to have Hadoop version compatibility issue.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
I think we don't need to add tests. Having this profile is sufficient. We seem to have never encountered any incompatibility with Hadoop 3, so adding tests is not very meaningful.
The current tests are very unstable once there are multiple ones, and the mvn build encounters various conflicts.
3 participants
Purpose
In order to fix CVE vulnerabilites like CVE-2025-48924, this PR adds a profile and CI pipelines to verify paimon-flink's compatibility with Hadoop 3.x, so that if users want to use paimon-flink in a Hadoop 3.x project, they can compile the project with -Phadoop3 on their own and get the needed dependencies.
Given that Paimon still promises compatibility with Hadoop 2.x, Support for Hadoop 3.x is only added as an optional profile, instead of being set as the default dependency.
Currently only Flink 1.x has been verified against the newly introduced profile.
Tests
API and Format
This PR does not change API or format.
Documentation
It is documented in Flink quickstart how to build paimon-flink with hadoop 3.x.