Skip to content

HADOOP-18878: deprecate fs.s3a.multipart.purge#8353

Open
ajfabbri wants to merge 1 commit intoapache:trunkfrom
ajfabbri:af/hadoop-18878
Open

HADOOP-18878: deprecate fs.s3a.multipart.purge#8353
ajfabbri wants to merge 1 commit intoapache:trunkfrom
ajfabbri:af/hadoop-18878

Conversation

@ajfabbri
Copy link
Contributor

Description of PR

From HADOOP-18878:

the fs.s3a.multipart.purge option has been in for a long time, to help avoid running up costs from incomplete uploads, especially during testing.

but

adds overhead on startup (list + delete)
dangerous if one client has a very short lifespan; all active s3a committer jobs will be broken
obsoleted by s3 lifecycle rules
and "hadoop s3guard uploads" cli.
proposed: deprecate for the next release, delete after.

How was this patch tested?

s3a integration tests run against localstack S3.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [na ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 22s Maven dependency ordering for branch
+1 💚 mvninstall 42m 41s trunk passed
+1 💚 compile 15m 36s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 16m 14s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 5m 20s trunk passed
+1 💚 mvnsite 3m 6s trunk passed
+1 💚 javadoc 2m 23s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 2m 24s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 4m 41s trunk passed
+1 💚 shadedclient 28m 46s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 1m 50s the patch passed
+1 💚 compile 15m 7s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 15m 7s the patch passed
+1 💚 compile 16m 19s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 16m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 5m 19s the patch passed
+1 💚 mvnsite 3m 7s the patch passed
+1 💚 javadoc 2m 25s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 2m 20s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 5m 5s the patch passed
+1 💚 shadedclient 28m 55s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 22m 30s hadoop-common in the patch passed.
+1 💚 unit 3m 42s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
239m 35s
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8353/1/artifact/out/Dockerfile
GITHUB PR #8353
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname Linux bb24007cb6cc 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2fb0f80
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8353/1/testReport/
Max. process+thread count 3121 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8353/1/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@ajfabbri ajfabbri marked this pull request as ready for review March 19, 2026 19:34
@steveloughran
Copy link
Contributor

which s3 store did you test against?

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just need that declaration of which s3 store you ran the tests against, and what maven parameters you supplied

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants