Skip to content

fix: don't set global credential system properties for third-party S3 targets - #83

Merged
TheMeinerLP merged 2 commits into
mainfrom
fix/avoid-global-credential-system-properties
Jul 19, 2026
Merged

fix: don't set global credential system properties for third-party S3 targets#83
TheMeinerLP merged 2 commits into
mainfrom
fix/avoid-global-credential-system-properties

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Owner

Summary

For third-party/S3-compatible endpoints (Ceph RGW, R2, MinIO, ...), credentials already travel scoped to the filesystem instance via the s3x:// URI's userInfo (accessKeyId:secretAccessKey). S3FileSystemFactory.build() additionally wrote the same credentials to the process-wide aws.accessKeyId/aws.secretAccessKey system properties unconditionally, before branching on thirdParty.

That's redundant for the third-party path, and unsafe: system properties are global mutable JVM state. Any credential resolution elsewhere in the same JVM that falls back to the default AWS credentials chain — rather than using the URI-scoped credentials this method sets up — reads whatever value was written there most recently, independent of which storage's request is actually in flight.

What prompted this

Running BlueMapS3Storage against Ceph RGW in production, I started seeing an elevated Ceph RGW failed-request-rate alert. Investigation (RGW access logs, radosgw-admin bucket stats/user info, Kubernetes secret contents) ruled out:

  • Stale/rotated credentials — the access key in the mounted config matches the active RGW key exactly.
  • Bucket ownership — bucket stats confirms bluemap owns the bucket.
  • A recent config/deploy change — nothing changed in the days around when the failures started.

What stood out: 99.8% of the failing requests were GET .../tiles/<...> reads, rejected with no identifiable authenticated user at all (blank, not "wrong user") — and the original crash that kicked off the investigation was a 409 thrown from S3ReadAheadByteChannel (the async read-ahead path in aws-java-nio-spi-for-s3), not from the synchronous request path.

That pointed at the async path resolving credentials differently than the synchronous one — and S3FileSystemFactory writing credentials to global system properties, unconditionally, is the one place in this codebase where that kind of cross-path interference could happen.

Fix

Only set aws.accessKeyId/aws.secretAccessKey on the real-AWS-S3 branch, which has no URI-based credential mechanism and genuinely needs them. The third-party/S3X branch now relies solely on the URI-embedded credentials it already had.

Confidence level

Honest caveat: this is a hypothesis-driven fix, not a confirmed-and-reproduced one — the failure is intermittent (roughly 2-17 failed req/s in bursts, not constant), and I don't have a minimal repro isolating the exact internal code path in aws-java-nio-spi-for-s3 that falls back to the system-property-based default credentials chain. The change itself is low-risk (it only removes a redundant write on the third-party path; the real-AWS-S3 path is untouched), so I think it's worth merging and monitoring rather than blocking on a full repro.

Test plan

  • ./gradlew compileJava — builds clean
  • ./gradlew spotlessCheck — formatting passes
  • Monitor Ceph RGW failed-request-rate after deploying this to the production server that surfaced the issue — should drop back to baseline (<1 req/s) if this was the cause

… targets

For third-party/S3-compatible endpoints (e.g. Ceph RGW), credentials
already travel scoped to the filesystem instance via the s3x:// URI's
userInfo. The code additionally wrote them to the process-wide
aws.accessKeyId/aws.secretAccessKey system properties unconditionally,
which is redundant there and unsafe: any other credential resolution
in the same JVM that falls back to the default AWS credentials chain
(observed: the async read-ahead path, S3ReadAheadByteChannel) would
read whatever value this method last wrote, independent of which
storage's request is actually in flight.

Observed in production against Ceph RGW: intermittent, unpredictable
403s isolated to the async read-ahead path, with the rejected requests
carrying no identifiable user at all - consistent with a credential
resolution racing against an overwritten system property rather than
a real permissions problem (bucket ownership and the access key itself
were verified correct on the RGW side).

Left the system-property assignment in place for the real-AWS-S3
branch, which has no URI-based credential mechanism and still needs it.

This is a hypothesis-driven fix based on production log analysis
(RGW access logs, bucket/user state, and code review) - not confirmed
against a live repro, since the failure is intermittent. Worth
monitoring after merge.
The detailed why (production symptoms, ruled-out causes, the specific
async-path theory) belongs in the PR, not as a wall of comments in the
code itself.
@TheMeinerLP
TheMeinerLP force-pushed the fix/avoid-global-credential-system-properties branch from 0a49d4c to fa927c5 Compare July 19, 2026 10:33
@TheMeinerLP
TheMeinerLP merged commit a83df04 into main Jul 19, 2026
3 checks passed
TheMeinerLP added a commit to OneLiteFeatherNET/Kubernetes-FLUX that referenced this pull request Jul 19, 2026
Includes the credential-scoping fix (TheMeinerLP/BlueMapS3Storage#83)
for the intermittent Ceph RGW 403s traced earlier - third-party S3
targets no longer leak credentials through global JVM system
properties.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant