extstore s3 driver by cconstable · Pull Request #2129 · temporalio/sdk-typescript

cconstable · 2026-06-17T16:00:03Z

What was changed?

Adds the built-in S3 driver for External Storage
@temporalio/external-storage-s3: S3StorageDriver + S3StorageDriverClient interface, no
AWS dependency.
@temporalio/external-storage-s3-aws-sdk: AwsSdkS3StorageDriverClient, backed by
@aws-sdk/client-s3 (peer dep).
@experimental not yet wired into the converter pipeline.

Why?

The two-package split keeps the AWS SDK out of core, so users can supply their own S3
client.

Checklist

New tests added

cconstable · 2026-06-18T16:44:01Z

+{
+  "name": "@temporalio/external-storage-s3-aws-sdk",
+  "version": "1.18.1",
+  "private": true,


prevents npm publish from exposing these until extstore is released. we'll want to flip this later.

cconstable · 2026-06-18T16:57:23Z

+async function runAllWithAbortOnError<T>(
+  external: AbortSignal | undefined,
+  makeTasks: (signal: AbortSignal) => Promise<T>[]
+): Promise<T[]> {
+  const controller = new AbortController();
+  const signal = external ? AbortSignal.any([external, controller.signal]) : controller.signal;
+  const tasks = makeTasks(signal);
+  try {
+    return await Promise.all(tasks);
+  } catch (err) {
+    controller.abort();
+    await Promise.allSettled(tasks);
+    throw err;
+  }
+}


n.b.

S3 driver's runAllWithAbortOnError is concurrent and cancels in-flight sibling requests on the first failure (AWS SDK allows for cancelling mid-request).

GCS driver's allSettledOrThrow is concurrent but doesn't attempt mid-flight cancellation. @google-cloud/storage can't cancel an in-flight request. Abort in GCS is only check up-front.

jmaeagle99

LGTM with a few minor comments. Please seek an TS maintainer for review as well.

jmaeagle99 · 2026-06-18T18:46:06Z

+
+  describe(): Record<string, string> {
+    const region = this.client.config?.region;
+    return typeof region === 'string' && region ? { clientRegion: region } : {};


region might also be a Provider<string | undefined> as documented at https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-client-s3/Interface/ClientInputEndpointParameters/

Do we want to invoke the provider, check if the promise is completed (assuming the promise is cached), and read the value? If this is even possible.

My understanding is that region will be set to the string when the promise resolves which is essentially what you are suggesting. We can't check promise status.

jmaeagle99 · 2026-06-18T18:57:24Z

+ * @experimental
+ */
+export class AwsSdkS3StorageDriverClient implements S3StorageDriverClient {
+  constructor(private readonly client: AwsS3Client) {}


I think the way we are using this client, it does not support multipart upload. We might either need to depend on another package or write it ourselves using this client. There is varying recommendations of at what threshold multipart should be used, but it looks like AWS says to definitely use it at 100 MB or larger, which is above our max payload size. So maybe something to defer for now, what for feedback and/or measure with experimentation.

Good callout. I've documented this. Agree on deferring for now.

THardy98 · 2026-06-23T17:39:37Z

+
+> ⚠️ **This package is experimental and may be subject to change.** ⚠️
+
+`@temporalio/external-storage-s3` stores and retrieves Temporal payloads in Amazon S3 via the [External Storage](../../README.md) feature.


Why does External Storage link to the README? did you mean docs: https://docs.temporal.io/external-storage

Yea I suppose that is better! I originally had this linking to the in-repo docs.

THardy98 · 2026-06-23T18:01:23Z

nice README!

THardy98 · 2026-06-23T20:03:38Z

+    await Promise.allSettled(tasks);
+    throw err;
+  }
+}


Remind me - what do we do when an operation fails, is it a task failure and then we retry?

Just thinking of the scenario when store fails partway through. Subsequent retries should attempt to store only the payloads where objectExists == false ?

What about if retries are exhausted and we've partly completed the store operation? I guess there isn't much we can do

Yes. We fail the task and it must be retried at the task-level. If we are trying store many payloads and one fails, the whole thing fails.

Right now, none of the drivers will retry. I think we should have some sensible retry policy to address transient failures. Will chat with the team about this.

…ould still occur and waste time/bandwidth.

jmaeagle99 · 2026-06-25T23:32:05Z

+  async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
+    const contextSegments = buildContextSegments(context.target);
+    return runAllWithAbortOnError(context.abortSignal, (signal) => {
+      const uploads = new Map<string, Promise<void>>();


That's an interesting optimization! Should we do the same for downloads too?

cconstable requested a review from a team as a code owner June 17, 2026 16:00

cconstable changed the base branch from main to extstore-foundation June 17, 2026 16:04

cconstable force-pushed the extstore-s3-driver branch 2 times, most recently from 6f23506 to 2f945b0 Compare June 17, 2026 16:37

cconstable force-pushed the extstore-foundation branch from 1b6c2b9 to 447552b Compare June 17, 2026 16:38

Base automatically changed from extstore-foundation to main June 17, 2026 20:09

cconstable force-pushed the extstore-s3-driver branch from fa18b39 to a9e4554 Compare June 18, 2026 15:38

cconstable assigned THardy98 and jmaeagle99 Jun 18, 2026

cconstable commented Jun 18, 2026

View reviewed changes

jmaeagle99 reviewed Jun 18, 2026

View reviewed changes

THardy98 reviewed Jun 23, 2026

View reviewed changes

cconstable added 10 commits June 25, 2026 11:44

feat(extstore): add generic s3 driver interface.

31ab61f

feat(extstore): add aws-sdk s3 driver package.

346b9cf

fix formatting issues

17bd404

improve s3 driver utility function names.

ebccb71

make extstore driver packages private

dc7b236

tighten extstore s3 driver tests.

a6c963f

remove explicit unwrapping in extstore tests.

5a15795

remove unnecessary extstore s3client alias.

165d454

Address small nits and update tests.

cfd9c1f

Add upload deduping to avoid race. Uploads are idempotent but races c…

e5ae665

…ould still occur and waste time/bandwidth.

cconstable force-pushed the extstore-s3-driver branch from eed8180 to e5ae665 Compare June 25, 2026 15:45

jmaeagle99 approved these changes Jun 25, 2026

View reviewed changes


		> ⚠️ This package is experimental and may be subject to change. ⚠️

		`@temporalio/external-storage-s3` stores and retrieves Temporal payloads in Amazon S3 via the [External Storage](../../README.md) feature.

Uh oh!

Conversation

cconstable commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed?

Why?

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmaeagle99 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cconstable commented Jun 17, 2026 •

edited

Loading