Skip to content

feat: add support for setting S3 storage class on initial object write #1241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shibd merged 9 commits into from
Mar 4, 2026
Merged

feat: add support for setting S3 storage class on initial object write #1241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1692c64
feat: allow setting S3 storage class on PutObject requests
mattsre Jan 27, 2026
0f726b0
pass through storage class as string, update S3 docs
mattsre Feb 6, 2026
99addd6
add integration tests for setting s3 storage class
mattsre Feb 11, 2026
9b16b6c
Use testcontainers 1.21.4 and testcontainers-bom
lhotari Feb 25, 2026
232dae1
Use recent junit-jupiter version
lhotari Feb 25, 2026
5da831a
Remove separate maven config for integration tests
lhotari Feb 25, 2026
82f38af
Upgrade some maven plugins to ensure proper support for junit-jupiter
lhotari Feb 25, 2026
853ca98
Fix build output in GitHub Actions workflow
lhotari Feb 25, 2026
92723b5
Fix running junit4 tests
lhotari Feb 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pr-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
run: mvn license:check checkstyle:check spotbugs:check

- name: Build with Maven
run: mvn install test -DfailIfNoTests=false
run: mvn -B -ntp install test -DfailIfNoTests=false

- name: package surefire artifacts
if: failure()
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
.project
.settings/
.recommenders/
.factorypath

# Intellij
.idea/
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ The Cloud Storage sink connector supports the following properties.
| `role` | String | False | null | The Cloud Storage role. |
| `roleSessionName` | String | False | null | The Cloud Storage role session name. |
| `endpoint` | String | True | null | The Cloud Storage endpoint. |
| `s3StorageClass` | String | False | "STANDARD" | The S3 storage class to use when writing objects. Only applies when `provider` is `s3v2`. The value is passed directly to the S3 API and must be a valid [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) string (e.g. `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `GLACIER_IR`, `DEEP_ARCHIVE`). |
| `bucket` | String | True | null | The Cloud Storage bucket. |
| `formatType` | String | False | "json" | The data format type. Available options are JSON, Avro, Bytes, or Parquet. By default, it is set to JSON. |
| `partitionerType` | String | False | "partition" | The partitioning type. It can be configured by topic partitions or by time. By default, the partition type is configured by topic partitions. |
Expand Down
4 changes: 3 additions & 1 deletion docs/aws-s3-sink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ pulsarctl sinks create \
"bucket": "Your bucket name",
"region": "Your AWS S3 region",
"formatType": "json",
"partitionerType": "PARTITION"
"partitionerType": "PARTITION",
"s3StorageClass": "STANDARD"
}'
```

Expand Down Expand Up @@ -134,6 +135,7 @@ Before using the AWS S3 sink connector, you need to configure it. This table out
| `partitionerType` | String | False | false | null | The partitioning type. It can be configured by topic `PARTITION` or `TIME`. By default, the partition type is configured by topic partitions. |
| `region` | String | False | false | null | The AWS S3 region. Either the endpoint or region must be set. |
| `endpoint` | String | False | false | null | The AWS S3 endpoint. Either the endpoint or region must be set. |
| `s3StorageClass` | String | False | false | "STANDARD" | The S3 storage class to use when writing objects. Only applies when `provider` is `s3v2`. The value is passed directly to the S3 API and must be a valid [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) string (e.g. `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `GLACIER_IR`, `DEEP_ARCHIVE`, `REDUCED_REDUNDANCY`). |
| `role` | String | False | false | null | The AWS role. |
| `roleSessionName` | String | False | false | null | The AWS role session name. |
| `timePartitionPattern` | String | False | false | "yyyy-MM-dd" | The format pattern of the time-based partitioning. For details, refer to the Java date and time format. |
Expand Down
43 changes: 36 additions & 7 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,14 @@
<awaitility.version>4.2.0</awaitility.version>
<mockito.version>3.7.7</mockito.version>
<powermock.version>2.0.2</powermock.version>
<testcontainers.version>1.15.2</testcontainers.version>
<testcontainers.version>1.21.4</testcontainers.version>
<junit-jupiter.version>5.14.3</junit-jupiter.version>

<!-- build plugin dependencies -->
<license.plugin.version>3.0</license.plugin.version>
<maven-checkstyle-plugin.version>3.1.1</maven-checkstyle-plugin.version>
<maven-compiler-plugin.version>3.10.1</maven-compiler-plugin.version>
<maven-surefire-plugin.version>3.0.0-M3</maven-surefire-plugin.version>
<maven-compiler-plugin.version>3.15.0</maven-compiler-plugin.version>
<maven-surefire-plugin.version>3.5.3</maven-surefire-plugin.version>
<nifi.nar.plugin.version>1.2.0</nifi.nar.plugin.version>
<puppycrawl.checkstyle.version>8.29</puppycrawl.checkstyle.version>
<spotbugs-maven-plugin.version>4.2.2</spotbugs-maven-plugin.version>
Expand Down Expand Up @@ -240,13 +241,20 @@
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<artifactId>testcontainers-bom</artifactId>
<version>${testcontainers.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>pulsar</artifactId>
<version>${testcontainers.version}</version>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${junit-jupiter.version}</version>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<version>${junit-jupiter.version}</version>
</dependency>
Comment thread
lhotari marked this conversation as resolved.
<dependency>
<groupId>software.amazon.awssdk</groupId>
Expand Down Expand Up @@ -605,6 +613,26 @@
<version>${pulsar.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>localstack</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<scope>test</scope>
</dependency>

</dependencies>

Expand All @@ -630,6 +658,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>
Expand Down
5 changes: 5 additions & 0 deletions src/main/java/org/apache/pulsar/io/jcloud/BlobStoreAbstractConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ public class BlobStoreAbstractConfig implements Serializable {
// #### common configuration ####
private boolean usePathStyleUrl = true;
private String awsCannedAcl = "";
private String s3StorageClass = "STANDARD";
private boolean skipFailedMessages = false;

// #### partitioner configuration ####
Expand Down Expand Up @@ -126,6 +127,10 @@ public void validate() {
checkArgument(isNotBlank(region) || isNotBlank(endpoint),
"Either the aws-end-point or aws-region must be set.");
}
if (provider.equalsIgnoreCase(PROVIDER_AWSS3V2)) {
checkArgument(isNotBlank(s3StorageClass),
"s3StorageClass property must not be empty for s3v2 provider.");
}
if (isNotBlank(endpoint)) {
checkArgument(hasURIScheme(endpoint), "endpoint property needs to specify URI scheme.");
}
Expand Down
7 changes: 7 additions & 0 deletions src/main/java/org/apache/pulsar/io/jcloud/writer/S3BlobWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ public class S3BlobWriter implements BlobWriter {
private final S3Client s3;
private final String bucket;
private ObjectCannedACL acl;
private String storageClass;

public S3BlobWriter(CloudStorageSinkConfig sinkConfig) {

Expand All @@ -67,6 +68,9 @@ public S3BlobWriter(CloudStorageSinkConfig sinkConfig) {
if (StringUtils.isNotEmpty(sinkConfig.getAwsCannedAcl())) {
acl = ObjectCannedACL.fromValue(sinkConfig.getAwsCannedAcl());
}
if (StringUtils.isNotEmpty(sinkConfig.getS3StorageClass())) {
storageClass = sinkConfig.getS3StorageClass();
}
}

@Override
Expand All @@ -75,6 +79,9 @@ public void uploadBlob(String key, ByteBuffer payload) throws IOException {
if (acl != null) {
req.acl(acl);
}
if (storageClass != null) {
req.storageClass(storageClass);
}

s3.putObject(req.build(), RequestBody.fromByteBuffer(payload));
}
Expand Down
Loading