Skip to content

HADOOP-19843. Regenerate tracked proto2 source code#8350

Open
pan3793 wants to merge 7 commits intoapache:trunkfrom
pan3793:proto2-generated
Open

HADOOP-19843. Regenerate tracked proto2 source code#8350
pan3793 wants to merge 7 commits intoapache:trunkfrom
pan3793:proto2-generated

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented Mar 18, 2026

Description of PR

Regenerate tracked proto2 source code for non-x86_64 platforms.

The issue was originally identified by @steveloughran, see background at https://lists.apache.org/thread/4sn3bb2qo9vz2kqgblhx3wdc35fkc3bd

Additionally, rename the folder arm-java to proto2-generated and change the profile activation condition from arm arch to non-x86_64 arch, because there are other platforms, e.g., rsicv64, that have a similar problem with arm, we should make it more generic.

How was this patch tested?

Pass CI for x86_64 platform verification.

And manually tested on M1 MacBook with dev container

$ ./start-build-env.sh
...
$ uname -m
aarch64
$ ./mvnw clean package -DskipTests -DskipShade
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:01 min
[INFO] Finished at: 2026-03-20T08:03:28Z
[INFO] ------------------------------------------------------------------------

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (HADOOP-19843)?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

No AI usage.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 19m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 9s Maven dependency ordering for branch
+1 💚 mvninstall 53m 57s trunk passed
+1 💚 compile 21m 15s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 20m 27s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 5m 54s trunk passed
+1 💚 mvnsite 20m 23s trunk passed
+1 💚 javadoc 10m 48s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 11m 1s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
-1 ❌ spotbugs 45m 32s /branch-spotbugs-root-warnings.html root in trunk has 92 extant spotbugs warnings.
+1 💚 shadedclient 35m 50s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 41s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 47s /patch-mvninstall-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvninstall 4m 12s /patch-mvninstall-root.txt root in the patch failed.
-1 ❌ compile 1m 17s /patch-compile-root-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt root in the patch failed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04.
-1 ❌ javac 1m 17s /patch-compile-root-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt root in the patch failed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04.
-1 ❌ compile 1m 23s /patch-compile-root-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt root in the patch failed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1.
-1 ❌ javac 1m 23s /patch-compile-root-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt root in the patch failed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1.
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 132 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 5m 47s the patch passed
-1 ❌ mvnsite 1m 18s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 javadoc 10m 50s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 11m 4s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
-1 ❌ spotbugs 0m 53s /patch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ spotbugs 2m 2s /patch-spotbugs-root.txt root in the patch failed.
-1 ❌ shadedclient 43m 30s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 7m 18s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
306m 2s
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/1/artifact/out/Dockerfile
GITHUB PR #8350
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname Linux f4b5db93f120 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / bd401e3
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/1/testReport/
Max. process+thread count 570 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/1/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 10m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 10s Maven dependency ordering for branch
+1 💚 mvninstall 54m 30s trunk passed
+1 💚 compile 20m 53s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 20m 36s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 5m 55s trunk passed
+1 💚 mvnsite 20m 14s trunk passed
+1 💚 javadoc 10m 45s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 11m 4s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
-1 ❌ spotbugs 45m 26s /branch-spotbugs-root-warnings.html root in trunk has 92 extant spotbugs warnings.
+1 💚 shadedclient 36m 32s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 46s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 47s /patch-mvninstall-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvninstall 4m 23s /patch-mvninstall-root.txt root in the patch failed.
-1 ❌ compile 1m 24s /patch-compile-root-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt root in the patch failed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04.
-1 ❌ javac 1m 24s /patch-compile-root-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt root in the patch failed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04.
-1 ❌ compile 1m 24s /patch-compile-root-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt root in the patch failed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1.
-1 ❌ javac 1m 24s /patch-compile-root-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt root in the patch failed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1.
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 132 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 5m 42s the patch passed
-1 ❌ mvnsite 1m 17s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 javadoc 10m 51s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 11m 9s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
-1 ❌ spotbugs 0m 56s /patch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ spotbugs 2m 5s /patch-spotbugs-root.txt root in the patch failed.
-1 ❌ shadedclient 43m 28s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 7m 20s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
297m 45s
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/2/artifact/out/Dockerfile
GITHUB PR #8350
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname Linux 216dfc2c60fb 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / bd401e3
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/2/testReport/
Max. process+thread count 578 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/2/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 22s Maven dependency ordering for branch
+1 💚 mvninstall 25m 59s trunk passed
+1 💚 compile 8m 21s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 8m 42s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 2m 56s trunk passed
+1 💚 mvnsite 9m 46s trunk passed
+1 💚 javadoc 5m 52s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 5m 47s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
-1 ❌ spotbugs 18m 57s /branch-spotbugs-root-warnings.html root in trunk has 92 extant spotbugs warnings.
+1 💚 shadedclient 15m 48s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 19m 55s the patch passed
+1 💚 compile 8m 32s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 8m 32s the patch passed
+1 💚 compile 8m 51s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 8m 51s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 132 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 3m 2s the patch passed
+1 💚 mvnsite 9m 29s the patch passed
+1 💚 javadoc 5m 51s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 5m 52s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 20m 38s the patch passed
+1 💚 shadedclient 33m 18s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 240m 17s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
454m 31s
Reason Tests
Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/3/artifact/out/Dockerfile
GITHUB PR #8350
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname Linux 5accf0ccf833 5.15.0-141-generic #151-Ubuntu SMP Sun May 18 21:35:19 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1aae752
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/3/testReport/
Max. process+thread count 4294 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/3/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how you've made it work for more platforms and got RAT to stop complaining.

@ajfabbri
Copy link
Contributor

I confirmed that generating ProtobufRpcEngineProtos.java on x86 matches the file in this PR.

Any ideas what the patch unit root failures are about?

+1 pending a good CI run.

@ajfabbri
Copy link
Contributor

Created #8358. My best guess for the CI failure is hitting nproc ulimit in Yetus CI container. We need a better way to re-trigger CI.

@ajfabbri ajfabbri self-requested a review March 19, 2026 20:14
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

That vm oom problem is known intermittent and obviously unrelated

There may be trailing spaces, but given this is autogenerated stuff, not our problem

*
* <pre>
** Name of the RPC method
** Name of the RPC method
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is where all the EOLs are coming from.

@pan3793 this is what came out of protoc? in which case we go with what was generated.

Copy link
Member Author

@pan3793 pan3793 Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveloughran, yes, I just copied the generated file and committed it to git directly, without touching the file content.

@pan3793
Copy link
Member Author

pan3793 commented Mar 20, 2026

last commit fixed the grammar in README.txt, no code change.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 7s Maven dependency ordering for branch
+1 💚 mvninstall 51m 12s trunk passed
+1 💚 compile 17m 25s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 17m 52s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 5m 51s trunk passed
+1 💚 mvnsite 19m 46s trunk passed
+1 💚 javadoc 9m 35s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 9m 34s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+0 🆗 spotbugs 0m 23s branch/hadoop-build-tools no spotbugs output file (spotbugsXml.xml)
-1 ❌ spotbugs 37m 51s /branch-spotbugs-root-warnings.html root in trunk has 92 extant spotbugs warnings.
+1 💚 shadedclient 35m 12s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 43s Maven dependency ordering for patch
+1 💚 mvninstall 44m 35s the patch passed
+1 💚 compile 17m 41s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 17m 41s the patch passed
+1 💚 compile 18m 11s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 18m 11s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 132 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 5m 50s the patch passed
+1 💚 mvnsite 19m 44s the patch passed
+1 💚 javadoc 9m 32s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 9m 33s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+0 🆗 spotbugs 0m 23s hadoop-build-tools has no data from spotbugs
+1 💚 shadedclient 69m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 822m 6s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 49s The patch does not generate ASF License warnings.
1238m 29s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/5/artifact/out/Dockerfile
GITHUB PR #8350
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname Linux 644c60c67ac1 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ec71ea7
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/5/testReport/
Max. process+thread count 3751 (vs. ulimit of 5500)
modules C: hadoop-build-tools hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8350/5/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants