From d4e411758354ac14d4556acefd20acfe70f9c4be Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Mon, 13 Jul 2026 22:22:45 +0200 Subject: [PATCH 1/3] feat: add Cloudflare R2 support via account-id and optional list caching MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports the useful parts of Tresillo2017's R2 fork (compare/main...Tresillo2017:BlueMapS3Storage:main) cleanly: that fork's diff turned out to contain no actual Java changes despite its CHANGELOG_R2.md describing new R2Storage/R2Configuration classes, just three overlapping markdown files and an accidentally committed node_modules/. R2 is S3-compatible, so it already works through the existing themeinerlp:s3 storage type. Rather than duplicating that as a parallel R2-specific implementation, this adds: - account-id: optional Cloudflare account ID. When set (and endpoint-url is left empty), the endpoint is derived automatically as https://.r2.cloudflarestorage.com and path-style access is forced on, since R2 only supports path-style. - list-cache-ttl-seconds: optional TTL cache for mapIds() (map enumeration), since directory listings are a billed "Class A" operation on R2. Off by default (0); only affects map enumeration at init/reload, not tile reads/writes. (Fixes a pre-existing resource leak in mapIds() along the way — the underlying DirectoryStream was never closed.) Consolidates the fork's R2_GUIDE.md/R2_QUICK_START.md/CHANGELOG_R2.md into a single docs/cloudflare-r2.md, and documents both new options plus the existing checksum-validation option (added in #73 but never added to the README) in the main config table. --- README.md | 22 +++++ docs/cloudflare-r2.md | 84 +++++++++++++++++++ .../bluemap/s3/storage/S3Configuration.java | 4 + .../s3/storage/S3FileSystemFactory.java | 12 ++- .../bluemap/s3/storage/S3Storage.java | 39 +++++++-- .../s3/storage/S3StorageConfiguration.java | 28 +++++++ 6 files changed, 180 insertions(+), 9 deletions(-) create mode 100644 docs/cloudflare-r2.md diff --git a/README.md b/README.md index f54949f8..c967e09b 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ A storage addon for [BlueMap](https://github.com/BlueMap-Minecraft/BlueMap) that BlueMapS3Storage is an addon for BlueMap that provides the ability to store map data in S3-compatible storage services, such as: - Amazon S3 +- Cloudflare R2 (see [docs/cloudflare-r2.md](docs/cloudflare-r2.md) for R2-specific setup) - MinIO - DigitalOcean Spaces - Backblaze B2 @@ -107,6 +108,9 @@ force-path-style: true | `endpoint-url` | Optional: The endpoint URL for S3-compatible services (leave empty for AWS S3) | `http://localhost:9000` | | `root-path` | Optional: The root path in the S3 bucket where BlueMap data will be stored | `.` | | `force-path-style` | Optional: Force path style access for S3 (needed for MinIO) | `false` | +| `checksum-validation` | Controls when the AWS SDK attaches/validates request and response checksums (`when_required` or `when_supported`). Since SDK 2.30 the default is `when_supported`, which many S3-compatible stores (Ceph RGW included) reject with a bare 400 Bad Request. Set to `when_supported` only if you're on real AWS S3 or a provider you've confirmed handles it. | `when_required` | +| `account-id` | Optional: Cloudflare account ID. If set and `endpoint-url` is left empty, the endpoint is derived automatically as `https://.r2.cloudflarestorage.com` with path-style access enabled. See [docs/cloudflare-r2.md](docs/cloudflare-r2.md). | (empty) | +| `list-cache-ttl-seconds` | Optional: How long to cache the list of available maps (`mapIds()`). Directory listings are a billed operation on some providers (e.g. R2); this list rarely changes at runtime. `0` disables caching. | `0` | ## Usage Examples @@ -148,6 +152,24 @@ root-path: "." force-path-style: true ``` +### Cloudflare R2 + +```hocon +## ## +## BlueMap ## +## Storage-Config ## +## ## + +storage-type: "themeinerlp:s3" +compression: gzip +bucket-name: "bluemap-storage" +account-id: "your-cloudflare-account-id" +access-key-id: "your-r2-access-key" +secret-access-key: "your-r2-secret-key" +``` + +See [docs/cloudflare-r2.md](docs/cloudflare-r2.md) for full setup steps, the listing-cache option, and troubleshooting. + ### DigitalOcean Spaces ```hocon diff --git a/docs/cloudflare-r2.md b/docs/cloudflare-r2.md new file mode 100644 index 00000000..a18eadaa --- /dev/null +++ b/docs/cloudflare-r2.md @@ -0,0 +1,84 @@ +# Cloudflare R2 + +R2 is S3-compatible, so it works through the same `themeinerlp:s3` storage type as any other +S3-compatible provider. There's no separate `themeinerlp:r2` storage type; two small +R2-specific conveniences are built into the regular S3 config instead. + +## Setup + +1. Create a bucket in the Cloudflare dashboard under **R2**. +2. Under **Manage R2 API Tokens**, create a token with **Object Read & Write** permissions and + copy the Access Key ID / Secret Access Key immediately, you won't see them again. +3. Copy your **Account ID** from the R2 overview page. +4. Create a storage config file: + + | Platform | Path | + |---|---| + | Spigot/Paper | `./plugins/BlueMap/storages/r2.conf` | + | Sponge/Forge/Fabric | `./config/bluemap/storages/r2.conf` | + | CLI | `./config/storages/r2.conf` | + + ```hocon + storage-type: "themeinerlp:s3" + compression: gzip + bucket-name: "your-bucket-name" + account-id: "your-account-id" + access-key-id: "your-r2-access-key" + secret-access-key: "your-r2-secret-key" + ``` + + Setting `account-id` derives the endpoint (`https://.r2.cloudflarestorage.com`) + and enables path-style access automatically, since R2 only supports path-style. Leave + `endpoint-url` and `force-path-style` unset when using `account-id` this way. + + If you'd rather set the endpoint yourself (e.g. for a jurisdictional/EU-restricted bucket), + skip `account-id` and use `endpoint-url`/`force-path-style` directly instead, same as any + other S3-compatible provider: + + ```hocon + storage-type: "themeinerlp:s3" + compression: gzip + bucket-name: "your-bucket-name" + access-key-id: "your-r2-access-key" + secret-access-key: "your-r2-secret-key" + region: "auto" + endpoint-url: "https://your-account-id.eu.r2.cloudflarestorage.com" + force-path-style: true + ``` + +5. Reference the storage in your main BlueMap config: + + ```hocon + storages: { + r2: "storages/r2.conf" + } + + maps: [ + { id: "world", storage: "r2" } + ] + ``` + +6. Restart or reload BlueMap and confirm objects show up in the bucket. + +## Cost note: directory listing cache + +Directory listings (used for `mapIds()`, i.e. once per storage init/reload to enumerate +configured maps, not per-tile) are a billed "Class A" operation on R2. Since that list rarely +changes at runtime, you can cache it for a while: + +```hocon +list-cache-ttl-seconds: 300 +``` + +`0` (default) disables caching. This only affects map enumeration, not tile reads/writes. + +## Troubleshooting + +**`SignatureDoesNotMatch` / 403.** Almost always a copy-paste issue: regenerate the R2 API +token, make sure the secret key isn't wrapped in extra quotes or has trailing whitespace, and +double check the account ID is the exact 32-character hex string from the R2 overview page +(not the bucket name or a zone ID). + +**`PutObject => 400` / tile saves failing silently.** Not R2-specific, see the +[`checksum-validation`](../README.md#configuration-options) option. R2 (like most non-AWS +S3-compatible stores) needs `checksum-validation: when_required` (the default). diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java index de333803..efb7afed 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java @@ -35,4 +35,8 @@ public sealed interface S3Configuration permits S3StorageConfiguration { String getChecksumValidation(); + String getAccountId(); + + int getListCacheTtlSeconds(); + } diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java index ce6112ab..d940a48d 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java @@ -39,7 +39,11 @@ public static S3Fs build(S3Configuration cfg) { throw new IllegalArgumentException("bucketName is required"); } - final boolean thirdParty = cfg.getEndpointUrl() != null && !cfg.getEndpointUrl().isBlank(); + final boolean noExplicitEndpoint = cfg.getEndpointUrl() == null || cfg.getEndpointUrl().isBlank(); + final boolean usingAccountId = noExplicitEndpoint && cfg.getAccountId() != null && !cfg.getAccountId().isBlank(); + final String effectiveEndpoint = + usingAccountId ? "https://" + cfg.getAccountId() + ".r2.cloudflarestorage.com" : cfg.getEndpointUrl(); + final boolean thirdParty = effectiveEndpoint != null && !effectiveEndpoint.isBlank(); try { final URI uri; System.setProperty(AWS_REGION_KEY, cfg.getRegion() != null ? cfg.getRegion() : DEFAULT_AWS_REGION); @@ -54,11 +58,13 @@ public static S3Fs build(S3Configuration cfg) { System.setProperty("aws.requestChecksumCalculation", cfg.getChecksumValidation()); System.setProperty("aws.responseChecksumValidation", cfg.getChecksumValidation()); if (thirdParty) { - var url = URI.create(cfg.getEndpointUrl()); + var url = URI.create(effectiveEndpoint); if (!url.toString().startsWith("https")) { System.setProperty("s3.spi.endpoint-protocol", "http"); } - if (cfg.forcePathStyle()) { + // R2 only supports path-style access, so force it when we derived the endpoint + // from account-id, regardless of what force-path-style is set to. + if (cfg.forcePathStyle() || usingAccountId) { System.setProperty("s3.spi.force-path-style", "true"); } PROVIDER = new S3XFileSystemProvider(); diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Storage.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Storage.java index 2cb84b94..e55df24e 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Storage.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Storage.java @@ -17,6 +17,7 @@ */ package dev.themeinerlp.bluemap.s3.storage; +import com.github.benmanes.caffeine.cache.Cache; import com.github.benmanes.caffeine.cache.Caffeine; import com.github.benmanes.caffeine.cache.LoadingCache; import de.bluecolored.bluemap.core.storage.MapStorage; @@ -24,23 +25,37 @@ import de.bluecolored.bluemap.core.storage.compression.Compression; import de.bluecolored.bluemap.core.storage.file.FileMapStorage; import java.io.IOException; +import java.io.UncheckedIOException; import java.nio.file.FileSystem; import java.nio.file.Files; import java.nio.file.Path; +import java.time.Duration; +import java.util.List; import java.util.stream.Stream; public final class S3Storage implements Storage { + private static final String MAP_IDS_CACHE_KEY = "mapIds"; + private final S3Configuration configuration; private final Compression compression; private FileSystem s3FileSystem; private boolean closed = false; private final LoadingCache mapStorages; + // Directory listings are a billed "Class A" operation on R2 and similar providers, so + // mapIds() (which rarely changes at runtime) can optionally be cached for a while. Null + // when list-cache-ttl-seconds is 0, meaning caching stays off unless explicitly configured. + private final Cache> mapIdsCache; public S3Storage(S3Configuration configuration, Compression compression) { this.configuration = configuration; this.compression = compression; mapStorages = Caffeine.newBuilder().build(this::create); + int listCacheTtlSeconds = configuration.getListCacheTtlSeconds(); + mapIdsCache = + listCacheTtlSeconds > 0 + ? Caffeine.newBuilder().expireAfterWrite(Duration.ofSeconds(listCacheTtlSeconds)).build() + : null; } @Override @@ -83,12 +98,24 @@ public Stream mapIds() throws IOException { if (isClosed()) { throw new IOException("Storage is closed"); } - - // List all directories in the root path - return Files.list(getRooPath()) - .filter(Files::isDirectory) - .map(Path::getFileName) - .map(Path::toString); + + try { + List ids = + mapIdsCache != null + ? mapIdsCache.get(MAP_IDS_CACHE_KEY, key -> listMapIdsUncached()) + : listMapIdsUncached(); + return ids.stream(); + } catch (UncheckedIOException e) { + throw e.getCause(); + } + } + + private List listMapIdsUncached() { + try (Stream entries = Files.list(getRooPath())) { + return entries.filter(Files::isDirectory).map(Path::getFileName).map(Path::toString).toList(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } } @Override diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java index 843ffb4a..e9a59247 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java @@ -66,6 +66,24 @@ public final class S3StorageConfiguration extends StorageConfig implements S3Con """) private String checksumValidation = "when_required"; + @Comment(""" + Optional: Cloudflare account ID. If set and endpoint-url is left empty, the + endpoint is derived automatically as https://.r2.cloudflarestorage.com + and path-style access is enabled (required by R2). Leave empty for AWS S3 or any + other S3-compatible provider - use endpoint-url directly for those instead. + """) + @DebugDump(exclude = true) + private String accountId = ""; + + @Comment(""" + How long (in seconds) to cache the result of listing available maps (mapIds()). + 0 disables caching (default). Directory listings are a billed "Class A" operation + on R2 and similar providers, so a short cache here cuts down on repeated listing + calls; this list rarely changes at runtime (only when a map is added/removed), so + a stale cache for a few minutes is normally harmless. + """) + private int listCacheTtlSeconds = 0; + @Override public Storage createStorage() throws ConfigurationException { // Validate required configuration @@ -135,4 +153,14 @@ public boolean forcePathStyle() { public String getChecksumValidation() { return ChecksumValidationMode.parse(checksumValidation).propertyValue(); } + + @Override + public String getAccountId() { + return accountId; + } + + @Override + public int getListCacheTtlSeconds() { + return Math.max(0, listCacheTtlSeconds); + } } From 4cf4dc981589630b02eeb6b76154617e76367bd6 Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Mon, 13 Jul 2026 22:31:00 +0200 Subject: [PATCH 2/3] refactor: extract provider-specific config into ProviderProfile S3Configuration was starting to accumulate provider-specific optional fields (account-id for R2, alongside checksum-validation and the generic fields) that only ever matter for one particular backend. Adding the next provider's shorthand would mean growing the shared interface again. Introduces a small ProviderProfile abstraction instead: each profile knows how to recognize its own shorthand config and resolve it into the plain endpoint-url/force-path-style/region settings S3FileSystemFactory already understood. The factory itself no longer has any R2-specific logic - it just asks ProviderProfiles.resolve() for the effective connection settings. - ProviderProfile: the resolve contract + ResolvedEndpoint record - R2Profile: derives the R2 endpoint from account-id, forces path-style - GenericS3Profile: passes cfg's own endpoint-url/force-path-style/ region through unchanged - the fallback, tried last - ProviderProfiles: tries each profile in order, first match wins No config/behavior change for existing users. Adding a future provider's own shorthand (e.g. a hypothetical B2 profile) now means adding one small profile class, not touching the shared config interface or the factory. --- README.md | 1 + docs/cloudflare-r2.md | 4 ++ .../bluemap/s3/storage/GenericS3Profile.java | 43 +++++++++++ .../bluemap/s3/storage/ProviderProfile.java | 43 +++++++++++ .../bluemap/s3/storage/ProviderProfiles.java | 72 +++++++++++++++++++ .../bluemap/s3/storage/R2Profile.java | 49 +++++++++++++ .../bluemap/s3/storage/S3Configuration.java | 2 + .../s3/storage/S3FileSystemFactory.java | 16 ++--- .../s3/storage/S3StorageConfiguration.java | 13 ++++ 9 files changed, 233 insertions(+), 10 deletions(-) create mode 100644 src/main/java/dev/themeinerlp/bluemap/s3/storage/GenericS3Profile.java create mode 100644 src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfile.java create mode 100644 src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfiles.java create mode 100644 src/main/java/dev/themeinerlp/bluemap/s3/storage/R2Profile.java diff --git a/README.md b/README.md index c967e09b..be90aebf 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,7 @@ force-path-style: true | `root-path` | Optional: The root path in the S3 bucket where BlueMap data will be stored | `.` | | `force-path-style` | Optional: Force path style access for S3 (needed for MinIO) | `false` | | `checksum-validation` | Controls when the AWS SDK attaches/validates request and response checksums (`when_required` or `when_supported`). Since SDK 2.30 the default is `when_supported`, which many S3-compatible stores (Ceph RGW included) reject with a bare 400 Bad Request. Set to `when_supported` only if you're on real AWS S3 or a provider you've confirmed handles it. | `when_required` | +| `provider` | Optional: explicitly picks the provider profile (`r2` or `generic`) instead of auto-detecting from which fields are set. Only needed if you want to be explicit; see [docs/cloudflare-r2.md](docs/cloudflare-r2.md). | (auto-detect) | | `account-id` | Optional: Cloudflare account ID. If set and `endpoint-url` is left empty, the endpoint is derived automatically as `https://.r2.cloudflarestorage.com` with path-style access enabled. See [docs/cloudflare-r2.md](docs/cloudflare-r2.md). | (empty) | | `list-cache-ttl-seconds` | Optional: How long to cache the list of available maps (`mapIds()`). Directory listings are a billed operation on some providers (e.g. R2); this list rarely changes at runtime. `0` disables caching. | `0` | diff --git a/docs/cloudflare-r2.md b/docs/cloudflare-r2.md index a18eadaa..9f724f73 100644 --- a/docs/cloudflare-r2.md +++ b/docs/cloudflare-r2.md @@ -31,6 +31,10 @@ R2-specific conveniences are built into the regular S3 config instead. and enables path-style access automatically, since R2 only supports path-style. Leave `endpoint-url` and `force-path-style` unset when using `account-id` this way. + This is auto-detected: setting `account-id` (with no `endpoint-url`) is enough on its own. + Add `provider: "r2"` explicitly only if you want to be unambiguous about it, e.g. in a + config generated by tooling rather than typed by hand. + If you'd rather set the endpoint yourself (e.g. for a jurisdictional/EU-restricted bucket), skip `account-id` and use `endpoint-url`/`force-path-style` directly instead, same as any other S3-compatible provider: diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/GenericS3Profile.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/GenericS3Profile.java new file mode 100644 index 00000000..ee690a2e --- /dev/null +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/GenericS3Profile.java @@ -0,0 +1,43 @@ +/** + * A simple storage implementation for bluemap to save data into s3 storage solution. + * Copyright (C) 2025 TheMeinerLP and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package dev.themeinerlp.bluemap.s3.storage; + +/** + * Fallback profile: passes the configured endpoint-url/force-path-style/region through + * unchanged. Always applies, and {@link ProviderProfiles} tries it last so any provider-specific + * profile gets a chance to take over first. + */ +final class GenericS3Profile implements ProviderProfile { + + static final String ID = "generic"; + + @Override + public String id() { + return ID; + } + + @Override + public boolean appliesTo(S3Configuration cfg) { + return true; + } + + @Override + public ResolvedEndpoint resolve(S3Configuration cfg) { + return new ResolvedEndpoint(cfg.getEndpointUrl(), cfg.forcePathStyle(), cfg.getRegion()); + } +} diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfile.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfile.java new file mode 100644 index 00000000..8ef5ae92 --- /dev/null +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfile.java @@ -0,0 +1,43 @@ +/** + * A simple storage implementation for bluemap to save data into s3 storage solution. + * Copyright (C) 2025 TheMeinerLP and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package dev.themeinerlp.bluemap.s3.storage; + +/** + * Resolves a provider's own shorthand config (e.g. Cloudflare R2's account-id) into the plain + * connection settings {@link S3FileSystemFactory} actually needs. Keeps provider-specific + * conventions out of the generic S3 engine: add a new profile here when a provider gets its + * own convenience config, register it in {@link ProviderProfiles}, and the factory itself + * never needs to change. + */ +interface ProviderProfile { + + /** Stable name users can set explicitly via the provider config option, e.g. "r2". */ + String id(); + + /** + * Whether this profile's own shorthand config is present, so it should be picked + * automatically when the user didn't set provider explicitly. Only consulted as a + * fallback - see {@link ProviderProfiles#resolve(S3Configuration)}. + */ + boolean appliesTo(S3Configuration cfg); + + /** Only called once this profile has been selected, either explicitly or via {@link #appliesTo}. */ + ResolvedEndpoint resolve(S3Configuration cfg); + + record ResolvedEndpoint(String endpointUrl, boolean forcePathStyle, String region) {} +} diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfiles.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfiles.java new file mode 100644 index 00000000..e9ccae45 --- /dev/null +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/ProviderProfiles.java @@ -0,0 +1,72 @@ +/** + * A simple storage implementation for bluemap to save data into s3 storage solution. + * Copyright (C) 2025 TheMeinerLP and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package dev.themeinerlp.bluemap.s3.storage; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * Selects a {@link ProviderProfile} and resolves it against the given config. + * + *

If {@code provider} is set explicitly, that profile is used directly, an unmistakable, + * explicit strategy choice, and an unknown name fails fast rather than silently falling back. + * If left unset, profiles are tried in order and the first one whose {@link + * ProviderProfile#appliesTo(S3Configuration)} matches wins, so existing zero-config setups + * (e.g. just setting account-id) keep working without needing to also learn about + * {@code provider}. {@link GenericS3Profile} always applies and is tried last. + */ +final class ProviderProfiles { + + private static final List PROFILES = List.of(new R2Profile(), new GenericS3Profile()); + + private static final Map BY_ID = new LinkedHashMap<>(); + + static { + for (ProviderProfile profile : PROFILES) { + BY_ID.put(profile.id(), profile); + } + } + + private ProviderProfiles() {} + + static ProviderProfile.ResolvedEndpoint resolve(S3Configuration cfg) { + return select(cfg).resolve(cfg); + } + + private static ProviderProfile select(S3Configuration cfg) { + String explicit = cfg.getProvider(); + if (explicit != null && !explicit.isBlank()) { + ProviderProfile profile = BY_ID.get(explicit.trim().toLowerCase()); + if (profile == null) { + throw new IllegalArgumentException( + "Unknown provider '" + + explicit + + "', expected one of: " + + String.join(", ", BY_ID.keySet())); + } + return profile; + } + + return PROFILES.stream() + .filter(profile -> profile.appliesTo(cfg)) + .findFirst() + .orElseThrow(() -> new IllegalStateException( + "No provider profile matched (GenericS3Profile should always match)")); + } +} diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/R2Profile.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/R2Profile.java new file mode 100644 index 00000000..962a44e5 --- /dev/null +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/R2Profile.java @@ -0,0 +1,49 @@ +/** + * A simple storage implementation for bluemap to save data into s3 storage solution. + * Copyright (C) 2025 TheMeinerLP and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package dev.themeinerlp.bluemap.s3.storage; + +/** + * Cloudflare R2: derives the endpoint from account-id so users don't have to look up/type the + * full https://<account-id>.r2.cloudflarestorage.com URL themselves. Applies only when + * account-id is set and no explicit endpoint-url was given, so setting endpoint-url always + * still wins (e.g. for R2's jurisdictional/EU-restricted endpoints). + * + *

R2 only supports path-style access, so this always forces it on regardless of + * force-path-style. + */ +final class R2Profile implements ProviderProfile { + + static final String ID = "r2"; + + @Override + public String id() { + return ID; + } + + @Override + public boolean appliesTo(S3Configuration cfg) { + boolean noExplicitEndpoint = cfg.getEndpointUrl() == null || cfg.getEndpointUrl().isBlank(); + return noExplicitEndpoint && cfg.getAccountId() != null && !cfg.getAccountId().isBlank(); + } + + @Override + public ResolvedEndpoint resolve(S3Configuration cfg) { + String endpoint = "https://" + cfg.getAccountId() + ".r2.cloudflarestorage.com"; + return new ResolvedEndpoint(endpoint, true, cfg.getRegion()); + } +} diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java index efb7afed..1803c575 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3Configuration.java @@ -39,4 +39,6 @@ public sealed interface S3Configuration permits S3StorageConfiguration { int getListCacheTtlSeconds(); + String getProvider(); + } diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java index d940a48d..9a4827cc 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3FileSystemFactory.java @@ -39,14 +39,12 @@ public static S3Fs build(S3Configuration cfg) { throw new IllegalArgumentException("bucketName is required"); } - final boolean noExplicitEndpoint = cfg.getEndpointUrl() == null || cfg.getEndpointUrl().isBlank(); - final boolean usingAccountId = noExplicitEndpoint && cfg.getAccountId() != null && !cfg.getAccountId().isBlank(); - final String effectiveEndpoint = - usingAccountId ? "https://" + cfg.getAccountId() + ".r2.cloudflarestorage.com" : cfg.getEndpointUrl(); - final boolean thirdParty = effectiveEndpoint != null && !effectiveEndpoint.isBlank(); + final ProviderProfile.ResolvedEndpoint resolved = ProviderProfiles.resolve(cfg); + final boolean thirdParty = resolved.endpointUrl() != null && !resolved.endpointUrl().isBlank(); try { final URI uri; - System.setProperty(AWS_REGION_KEY, cfg.getRegion() != null ? cfg.getRegion() : DEFAULT_AWS_REGION); + String region = resolved.region() != null && !resolved.region().isBlank() ? resolved.region() : DEFAULT_AWS_REGION; + System.setProperty(AWS_REGION_KEY, region); System.setProperty("aws.accessKeyId", cfg.getAccessKeyId()); System.setProperty("aws.secretAccessKey", cfg.getSecretAccessKey()); // AWS SDK for Java 2.30.0+ defaults to attaching a flexible checksum (e.g. a CRC32 @@ -58,13 +56,11 @@ public static S3Fs build(S3Configuration cfg) { System.setProperty("aws.requestChecksumCalculation", cfg.getChecksumValidation()); System.setProperty("aws.responseChecksumValidation", cfg.getChecksumValidation()); if (thirdParty) { - var url = URI.create(effectiveEndpoint); + var url = URI.create(resolved.endpointUrl()); if (!url.toString().startsWith("https")) { System.setProperty("s3.spi.endpoint-protocol", "http"); } - // R2 only supports path-style access, so force it when we derived the endpoint - // from account-id, regardless of what force-path-style is set to. - if (cfg.forcePathStyle() || usingAccountId) { + if (resolved.forcePathStyle()) { System.setProperty("s3.spi.force-path-style", "true"); } PROVIDER = new S3XFileSystemProvider(); diff --git a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java index e9a59247..5f7b9b5e 100644 --- a/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java +++ b/src/main/java/dev/themeinerlp/bluemap/s3/storage/S3StorageConfiguration.java @@ -66,6 +66,14 @@ public final class S3StorageConfiguration extends StorageConfig implements S3Con """) private String checksumValidation = "when_required"; + @Comment(""" + Optional: explicitly picks which provider profile resolves this config, one of: + "r2", "generic". Leave empty to auto-detect: setting account-id below picks "r2" + automatically, otherwise "generic" (uses endpoint-url/force-path-style as-is). + Only needed if you want to be explicit rather than rely on auto-detection. + """) + private String provider = ""; + @Comment(""" Optional: Cloudflare account ID. If set and endpoint-url is left empty, the endpoint is derived automatically as https://.r2.cloudflarestorage.com @@ -163,4 +171,9 @@ public String getAccountId() { public int getListCacheTtlSeconds() { return Math.max(0, listCacheTtlSeconds); } + + @Override + public String getProvider() { + return provider; + } } From b1cb5b4bfdcb4b543f74e71ed4523aeb899129d8 Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Mon, 13 Jul 2026 22:41:19 +0200 Subject: [PATCH 3/3] docs: slim README down, move deep docs to the wiki README is now a short intro + quick start, linking out to the wiki (https://github.com/TheMeinerLP/BlueMapS3Storage/wiki) for the full config reference, provider examples, the Cloudflare R2 guide, and build instructions. Removes docs/cloudflare-r2.md, now superseded by the wiki's Cloudflare-R2 page (kept in sync, same content). --- README.md | 214 ++++-------------------------------------- docs/cloudflare-r2.md | 88 ----------------- 2 files changed, 20 insertions(+), 282 deletions(-) delete mode 100644 docs/cloudflare-r2.md diff --git a/README.md b/README.md index be90aebf..72b64dc0 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,11 @@ # BlueMapS3Storage -A storage addon for [BlueMap](https://github.com/BlueMap-Minecraft/BlueMap) that enables storing map data in S3-compatible storage services. +A storage addon for [BlueMap](https://github.com/BlueMap-Minecraft/BlueMap) that stores map data +in S3-compatible storage services (Amazon S3, Cloudflare R2, MinIO, DigitalOcean Spaces, +Backblaze B2, or anything else that speaks the S3 API) instead of the local filesystem. -## Overview - -BlueMapS3Storage is an addon for BlueMap that provides the ability to store map data in S3-compatible storage services, such as: - -- Amazon S3 -- Cloudflare R2 (see [docs/cloudflare-r2.md](docs/cloudflare-r2.md) for R2-specific setup) -- MinIO -- DigitalOcean Spaces -- Backblaze B2 -- Any other S3-compatible storage service - -This addon is particularly useful for: -- Servers with distributed architectures -- Environments where local storage is limited or not persistent -- Setups that require high availability and redundancy -- Multi-server networks that need to share the same map data - -## Features - -- Store BlueMap data in any S3-compatible storage service -- Configure custom endpoints for self-hosted S3 solutions -- Support for path-style access for compatibility with various S3 implementations (like MinIO) -- Seamless integration with BlueMap's storage system +Useful for distributed server architectures, environments without persistent local storage, and +multi-server networks that need to share the same map data. ## Requirements @@ -32,181 +13,26 @@ This addon is particularly useful for: - Java 21 or higher - Access to an S3-compatible storage service -## Installation -### Spigot/Paper - -1. Download the latest release of BlueMapS3Storage from the [releases page](https://github.com/TheMeinerLP/BlueMapS3Storage/releases) -2. Place the JAR file in the `./plugins/BlueMap/packs/` directory of your server -3. Restart your server or reload BlueMap - -### Sponge, Forge, Fabric -1. Download the latest release of BlueMapS3Storage from the [releases page](https://github.com/TheMeinerLP/BlueMapS3Storage/releases) -2. Place the JAR file in the `./config/bluemap/packs/` directory of your server -3. Restart your server or reload BlueMap - -### CLI -1. Download the latest release of BlueMapS3Storage from the [releases page](https://github.com/TheMeinerLP/BlueMapS3Storage/releases) -2. Place the JAR file in the `./config/packs/` directory of your server -3. Restart your server or reload BlueMap -## Configuration - -To use S3 storage with BlueMap, you need to create or modify the S3 storage configuration file. Create a file named `s3.conf` in the `plugins/BlueMap/storages` directory with the following content: - -```hocon -## ## -## BlueMap ## -## Storage-Config ## -## ## - -# The storage-type of this storage. -# Depending on this setting, different config-entries are allowed/expected in this config file. -# Don't change this value! (If you want a different storage-type, check out the other example-configs) -storage-type: "themeinerlp:s3" - -# The compression-type that bluemap will use to compress generated map-data. -# Available compression-types are: -# - gzip -# - zstd -# - deflate -# - none -# The default is: gzip -compression: gzip - -# The S3 storage -bucket-name: "bluemap-storage" - -# AWS region or "Minio" for MinIO -region: "Minio" - -# S3 credentials -access-key-id: "your-access-key" -secret-access-key: "your-secret-key" - -# Optional: Custom endpoint URL for S3-compatible services -# Leave empty for AWS S3 -endpoint-url: "http://localhost:9000" - -# Optional: The root path in the S3 bucket where BlueMap data will be stored -# Default is "." (root of the bucket) -root-path: "." - -# Optional: Force path style access for S3 (needed for MinIO) -# Default is false (use virtual-hosted style) -force-path-style: true -``` - -### Configuration Options - -| Option | Description | Default | -|--------|-------------|---------| -| `storage-type` | The storage type identifier (don't change this value) | `themeinerlp:s3` | -| `compression` | The compression type to use for storing data (options: "gzip", "zstd", "deflate", "none") | `gzip` | -| `bucket-name` | The name of the S3 bucket to use | `bluemap-storage` | -| `region` | The AWS region where the bucket is located | `Minio` | -| `access-key-id` | The AWS access key ID for authentication | `bluemap` | -| `secret-access-key` | The AWS secret access key for authentication | `bluemap-secret` | -| `endpoint-url` | Optional: The endpoint URL for S3-compatible services (leave empty for AWS S3) | `http://localhost:9000` | -| `root-path` | Optional: The root path in the S3 bucket where BlueMap data will be stored | `.` | -| `force-path-style` | Optional: Force path style access for S3 (needed for MinIO) | `false` | -| `checksum-validation` | Controls when the AWS SDK attaches/validates request and response checksums (`when_required` or `when_supported`). Since SDK 2.30 the default is `when_supported`, which many S3-compatible stores (Ceph RGW included) reject with a bare 400 Bad Request. Set to `when_supported` only if you're on real AWS S3 or a provider you've confirmed handles it. | `when_required` | -| `provider` | Optional: explicitly picks the provider profile (`r2` or `generic`) instead of auto-detecting from which fields are set. Only needed if you want to be explicit; see [docs/cloudflare-r2.md](docs/cloudflare-r2.md). | (auto-detect) | -| `account-id` | Optional: Cloudflare account ID. If set and `endpoint-url` is left empty, the endpoint is derived automatically as `https://.r2.cloudflarestorage.com` with path-style access enabled. See [docs/cloudflare-r2.md](docs/cloudflare-r2.md). | (empty) | -| `list-cache-ttl-seconds` | Optional: How long to cache the list of available maps (`mapIds()`). Directory listings are a billed operation on some providers (e.g. R2); this list rarely changes at runtime. `0` disables caching. | `0` | - -## Usage Examples - -### AWS S3 - -```hocon -## ## -## BlueMap ## -## Storage-Config ## -## ## - -storage-type: "themeinerlp:s3" -compression: gzip -bucket-name: "my-bluemap-bucket" -region: "us-east-1" -access-key-id: "AKIAIOSFODNN7EXAMPLE" -secret-access-key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" -endpoint-url: "" -root-path: "." -force-path-style: false -``` - -### MinIO - -```hocon -## ## -## BlueMap ## -## Storage-Config ## -## ## - -storage-type: "themeinerlp:s3" -compression: gzip -bucket-name: "bluemap" -region: "us-east-1" -access-key-id: "minioadmin" -secret-access-key: "minioadmin" -endpoint-url: "http://minio-server:9000" -root-path: "." -force-path-style: true -``` - -### Cloudflare R2 - -```hocon -## ## -## BlueMap ## -## Storage-Config ## -## ## - -storage-type: "themeinerlp:s3" -compression: gzip -bucket-name: "bluemap-storage" -account-id: "your-cloudflare-account-id" -access-key-id: "your-r2-access-key" -secret-access-key: "your-r2-secret-key" -``` - -See [docs/cloudflare-r2.md](docs/cloudflare-r2.md) for full setup steps, the listing-cache option, and troubleshooting. - -### DigitalOcean Spaces - -```hocon -## ## -## BlueMap ## -## Storage-Config ## -## ## - -storage-type: "themeinerlp:s3" -compression: gzip -bucket-name: "bluemap-maps" -region: "nyc3" -access-key-id: "your-spaces-key" -secret-access-key: "your-spaces-secret" -endpoint-url: "https://nyc3.digitaloceanspaces.com" -root-path: "." -force-path-style: false -``` - -## Building from Source +## Quick start -1. Clone the repository: - ``` - git clone https://github.com/TheMeinerLP/BlueMapS3Storage.git - ``` +1. Download the latest release from the [releases page](https://github.com/TheMeinerLP/BlueMapS3Storage/releases) + and drop the jar in `./plugins/BlueMap/packs/` (Spigot/Paper), `./config/bluemap/packs/` + (Sponge/Forge/Fabric), or `./config/packs/` (CLI). +2. Create a storage config file with at least `storage-type`, `bucket-name`, `access-key-id`, + `secret-access-key`, and `endpoint-url` (leave empty for AWS S3). +3. Reference the storage from your BlueMap config and restart/reload. -2. Build the project using Gradle: - ``` - ./gradlew shadowJar - ``` +See the **[Wiki](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki)** for everything else: -3. The built JAR file will be located in the `build/libs` directory. +- [Installation](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki/Installation) — exact paths per platform +- [Configuration](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki/Configuration) — full option reference +- [Provider Examples](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki/Provider-Examples) — AWS S3, MinIO, DigitalOcean Spaces +- [Cloudflare R2](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki/Cloudflare-R2) — R2-specific setup, cost notes, troubleshooting +- [Building From Source](https://github.com/TheMeinerLP/BlueMapS3Storage/wiki/Building-From-Source) ## License -This project is licensed under the [GNU Affero General Public License v3.0 (AGPL-3.0)](LICENSE). +[GNU Affero General Public License v3.0 (AGPL-3.0)](LICENSE) ## Credits @@ -216,4 +42,4 @@ This project is licensed under the [GNU Affero General Public License v3.0 (AGPL ## Support -If you encounter any issues or have questions, please [open an issue](https://github.com/TheMeinerLP/BlueMapS3Storage/issues) on the GitHub repository. +[Open an issue](https://github.com/TheMeinerLP/BlueMapS3Storage/issues) on GitHub. diff --git a/docs/cloudflare-r2.md b/docs/cloudflare-r2.md deleted file mode 100644 index 9f724f73..00000000 --- a/docs/cloudflare-r2.md +++ /dev/null @@ -1,88 +0,0 @@ -# Cloudflare R2 - -R2 is S3-compatible, so it works through the same `themeinerlp:s3` storage type as any other -S3-compatible provider. There's no separate `themeinerlp:r2` storage type; two small -R2-specific conveniences are built into the regular S3 config instead. - -## Setup - -1. Create a bucket in the Cloudflare dashboard under **R2**. -2. Under **Manage R2 API Tokens**, create a token with **Object Read & Write** permissions and - copy the Access Key ID / Secret Access Key immediately, you won't see them again. -3. Copy your **Account ID** from the R2 overview page. -4. Create a storage config file: - - | Platform | Path | - |---|---| - | Spigot/Paper | `./plugins/BlueMap/storages/r2.conf` | - | Sponge/Forge/Fabric | `./config/bluemap/storages/r2.conf` | - | CLI | `./config/storages/r2.conf` | - - ```hocon - storage-type: "themeinerlp:s3" - compression: gzip - bucket-name: "your-bucket-name" - account-id: "your-account-id" - access-key-id: "your-r2-access-key" - secret-access-key: "your-r2-secret-key" - ``` - - Setting `account-id` derives the endpoint (`https://.r2.cloudflarestorage.com`) - and enables path-style access automatically, since R2 only supports path-style. Leave - `endpoint-url` and `force-path-style` unset when using `account-id` this way. - - This is auto-detected: setting `account-id` (with no `endpoint-url`) is enough on its own. - Add `provider: "r2"` explicitly only if you want to be unambiguous about it, e.g. in a - config generated by tooling rather than typed by hand. - - If you'd rather set the endpoint yourself (e.g. for a jurisdictional/EU-restricted bucket), - skip `account-id` and use `endpoint-url`/`force-path-style` directly instead, same as any - other S3-compatible provider: - - ```hocon - storage-type: "themeinerlp:s3" - compression: gzip - bucket-name: "your-bucket-name" - access-key-id: "your-r2-access-key" - secret-access-key: "your-r2-secret-key" - region: "auto" - endpoint-url: "https://your-account-id.eu.r2.cloudflarestorage.com" - force-path-style: true - ``` - -5. Reference the storage in your main BlueMap config: - - ```hocon - storages: { - r2: "storages/r2.conf" - } - - maps: [ - { id: "world", storage: "r2" } - ] - ``` - -6. Restart or reload BlueMap and confirm objects show up in the bucket. - -## Cost note: directory listing cache - -Directory listings (used for `mapIds()`, i.e. once per storage init/reload to enumerate -configured maps, not per-tile) are a billed "Class A" operation on R2. Since that list rarely -changes at runtime, you can cache it for a while: - -```hocon -list-cache-ttl-seconds: 300 -``` - -`0` (default) disables caching. This only affects map enumeration, not tile reads/writes. - -## Troubleshooting - -**`SignatureDoesNotMatch` / 403.** Almost always a copy-paste issue: regenerate the R2 API -token, make sure the secret key isn't wrapped in extra quotes or has trailing whitespace, and -double check the account ID is the exact 32-character hex string from the R2 overview page -(not the bucket name or a zone ID). - -**`PutObject => 400` / tile saves failing silently.** Not R2-specific, see the -[`checksum-validation`](../README.md#configuration-options) option. R2 (like most non-AWS -S3-compatible stores) needs `checksum-validation: when_required` (the default).