diff --git a/src/destinations/overview.mdx b/src/destinations/overview.mdx index a522abb3..ff42f2ae 100644 --- a/src/destinations/overview.mdx +++ b/src/destinations/overview.mdx @@ -2,7 +2,7 @@ title: Overview --- -Ampersand currently supports webhook and Kinesis destinations. Destinations allow you to route data synced from SaaS instances via [Read Actions](/read-actions) or [Subscribe Actions](/subscribe-actions), and to receive real-time [Notifications](/notifications) about important lifecycle events in your projects. +Ampersand currently supports webhook, Kinesis, and S3 destinations. Destinations allow you to route data synced from SaaS instances via [Read Actions](/read-actions) or [Subscribe Actions](/subscribe-actions), and to receive real-time [Notifications](/notifications) about important lifecycle events in your projects. ## Add a destination to the Ampersand Dashboard @@ -35,7 +35,8 @@ Destinations can also be used to receive [notifications](/notifications) about i ## Supported destinations * [Webhook destinations](/destinations/webhooks) -* [Kinesis destinations](/destinations/kinesis) +* [Amazon Kinesis destinations](/destinations/kinesis) +* [Amazon S3 destinations](/destinations/s3) ## Other Destinations @@ -43,7 +44,6 @@ We have many other destination types on the roadmap, including: * Postgres * Ampersand-hosted Postgres -* Amazon S3 * Amazon SQS * Google Cloud Storage * Google PubSub diff --git a/src/destinations/s3.mdx b/src/destinations/s3.mdx new file mode 100644 index 00000000..6cc1b704 --- /dev/null +++ b/src/destinations/s3.mdx @@ -0,0 +1,303 @@ +--- +title: Amazon S3 destinations +--- + +For more information on destinations, see the [Destinations](/destinations) page. + +When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand can write the payload as objects to your Amazon S3 bucket. + +## Prerequisites + +Before setting up an S3 destination, ensure that you have: +- An AWS account with access to S3 +- An S3 bucket (or permissions to create one) +- AWS credentials with the following permission: + - `s3:PutObject` + +## Create an S3 destination + +### Step 1: Set up your S3 bucket + +If you don't already have an S3 bucket, create one in the AWS Console or using the AWS CLI: + +```bash +aws s3api create-bucket \ + --bucket ampersand-integration-bucket \ + --region us-west-2 \ + --create-bucket-configuration LocationConstraint=us-west-2 +``` + +### Step 2: Create AWS credentials + +Create an IAM user or role with permissions to write to your S3 bucket. You need: + +- **AWS Access Key ID** +- **AWS Secret Access Key** +- **AWS Session Token** (optional, for temporary credentials) + +**Example IAM policy:** + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "s3:PutObject", + "Resource": "arn:aws:s3:::ampersand-integration-bucket/*" + } + ] +} +``` + +### Step 3: Add the destination to Ampersand + +Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new S3 destination. + +You'll need to provide: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| **Destination name** | string | Yes | Alias to reference in your `amp.yaml` file | +| **Bucket** | string | Yes | Name of your S3 bucket | +| **Region** | string | Yes | AWS region where your bucket is located (e.g., `us-west-2`) | +| **AWS Access Key ID** | string | Yes | AWS access key with S3 permissions | +| **AWS Secret Access Key** | string | Yes | AWS secret access key | +| **AWS Session Token** | string | No | Session token for temporary credentials | +| **Object key template** | string | No | [JMESPath](https://jmespath.org) template for object key naming | +| **Storage class** | string | No | S3 storage class for written objects (defaults to `STANDARD`) | + + +Ampersand encrypts and stores your AWS credentials securely. + + +## Refer to the destination in your integration + +After creating your S3 destination, reference it in your `amp.yaml` file: + +```yaml +specVersion: 1.0.0 +integrations: + - name: salesforceToS3 + displayName: Salesforce to S3 + provider: salesforce + read: + objects: + - objectName: account + destination: ampersandS3Bucket + - objectName: contact + destination: ampersandS3Bucket +``` + +## Message format + +Ampersand writes each message as a JSON object to your S3 bucket. All data is wrapped in a top-level `data` object. + +In addition to the object body, each S3 object carries event metadata as [S3 object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html) (`x-amz-meta-*` headers), including `projectId`, `installationId`, `connectionId`, `destinationId`, `operationId`, `objectName`, `event-id`, `timestamp`, and `topic`. + +### Read action messages + +```json +{ + "data": { + "action": "read", + "projectId": "9482e676-4874-43a4-beea-fa8081d13a07", + "provider": "hubspot", + "groupRef": "group-id-1", + "groupName": "group-name-1", + "consumerRef": "user-id", + "consumerName": "user-name", + "installationId": "085353b1-07f1-4209-b471-7e028c0a68c8", + "installationUpdateTime": "2025-10-14T02:58:55.595861Z", + "objectName": "contacts", + "operationId": "0199e0a8-149d-7ee9-9f50-0514403517f9", + "operationTime": "2025-10-14T02:58:59.323405308Z", + "result": [ + { + "fields": { + "id": "438", + "firstname": "Maria", + "lastname": "Johnson", + "email": "maria@example.com", + "city": "Brisbane" + }, + "raw": { + "archived": false, + "createdAt": "2023-10-26T17:55:48.301Z", + "id": "438", + "properties": { + "firstname": "Maria", + "lastname": "Johnson", + "email": "maria@example.com", + "city": "Brisbane", + "createdate": "2023-10-26T17:55:48.301Z" + } + } + } + ] + } +} +``` + +**Key fields:** + +- `action`: The action type (`read` or `subscribe`) +- `projectId`: Your Ampersand project identifier +- `provider`: The SaaS provider name (e.g., `hubspot`, `salesforce`) +- `groupRef` / `groupName`: Customer group identifier and name +- `consumerRef` / `consumerName`: End user identifier and name +- `installationId`: The installation instance ID +- `installationUpdateTime`: When the installation was last updated +- `objectName`: The object being synced +- `operationId`: Unique identifier for this sync operation +- `operationTime`: When the operation completed +- `result`: Array of records synced + - `fields`: Normalized field data + - `raw`: Original API response from the provider + +### Subscribe action messages + +Subscribe action messages follow the same structure as read actions, but include additional event-related fields within each result entry: + +```json +{ + "data": { + "action": "subscribe", + "projectId": "9482e676-4874-43a4-beea-fa8081d13a07", + "provider": "salesforce", + "groupRef": "webhook-demo-group-id", + "groupName": "webhook-demo-group-name", + "consumerRef": "user-id", + "consumerName": "user-name", + "installationId": "0c9230e1-8fbe-4b28-bf10-2beee8fbf4ce", + "installationUpdateTime": "2025-04-10T23:00:52.618406Z", + "objectName": "company", + "operationTime": "2025-04-10T23:22:25.000Z", + "result": [ + { + "fields": { + "id": "001Dp00000ZDgmxIAD", + "annualrevenue": null, + "website": null + }, + "subscribeEventType": "update", + "providerEventType": "UPDATE", + "raw": { + "Id": "001Dp00000ZDgmxIAD", + "AnnualRevenue": null, + "Description": "Notes about the account", + "Name": "Acme Corp", + "Website": null, + "attributes": { + "type": "Account", + "url": "/services/data/v59.0/sobjects/Account/001Dp00000ZDgmxIAD" + } + }, + "rawEvent": { + "ChangeEventHeader": { + "changeOrigin": "com/salesforce/api/soap/63.0;client=SfdcInternalAPI/", + "changeType": "UPDATE", + "changedFields": ["Description", "LastModifiedDate"], + "commitNumber": 12041091891110, + "commitTimestamp": 1744327345000, + "commitUser": "005Dp000003Cd1SIAS", + "entityName": "Account", + "recordId": "001Dp00000ZDgmxIAD", + "sequenceNumber": 1, + "transactionKey": "00051705-2e99-d1e5-7e7a-48e3af682d07" + }, + "Description": "Notes about the account", + "LastModifiedDate": "2025-04-10T23:22:25.000Z" + } + } + ] + } +} +``` + +**Additional fields in subscribe actions:** + +- `subscribeEventType`: Normalized event type (`create`, `update`, `delete`, `associationUpdate`) +- `providerEventType`: Raw event type from the provider API +- `rawEvent`: Original webhook event from the provider (when available) + +## Object key configuration + +Each message is written as a separate object in your bucket. You can customize the object key (the object's path within the bucket) using a JMESPath template. JMESPath is a query language for JSON. See [jmespath.org](https://jmespath.org) for the full specification. + +You can specify an object key template when creating an S3 destination in the [Ampersand Dashboard](https://dashboard.withampersand.com/projects/_/destinations). If you don't specify one, the default key is the message timestamp followed by the message ID: + +``` +2025-10-14T02:58:59.323405308Z_0199e0a8-149d-7ee9-9f50-0514403517f9.json +``` + +### Template context + +Your template can reference three namespaces: + +| Namespace | Fields | +|-----------|--------| +| `metadata` | `projectId`, `installationId`, `connectionId`, `destinationId`, `operationId`, `objectName`, `event-id`, `timestamp`, `topic` | +| `time` | `year`, `month`, `day`, `hour`, `minute`, `second`, `date`, `datetime`, `unix`, `rfc3339`, `rfc3339_nano` | +| `data` | The message payload (e.g., `data.installationId` — see [Message format](#message-format) above) | + + +JMESPath is case-sensitive: `metadata.operationId` works, `metadata.operationid` does not. Fields containing a hyphen must be quoted, e.g. `metadata."event-id"`. Templates are validated when you create or update the destination. + + +### Custom object key examples + +**Group objects by synced object name:** +``` +join('/', [metadata.objectName, metadata.operationId]) +``` + +**Partition by date:** +``` +join('/', [metadata.objectName, time.date, metadata."event-id"]) +``` + +**Add a file extension:** +``` +join('', [time.rfc3339_nano, '_', metadata."event-id", '.json']) +``` + +**Use a field from the payload:** +``` +join('/', [data.installationId, metadata.operationId]) +``` + + +Make sure your template produces a unique key for every message — include `metadata."event-id"` or `metadata.operationId`. If two messages evaluate to the same key, the later object overwrites the earlier one. + + +## Storage class + +By default, objects are written with the `STANDARD` storage class. You can choose a different class when creating the destination, such as `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `GLACIER_IR`, or `DEEP_ARCHIVE`. See [the AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for a comparison. + +## Troubleshooting + +### Objects not appearing in S3 + +**Check destination configuration:** + +1. Verify that the bucket name and region are correct in the [Ampersand Dashboard](https://dashboard.withampersand.com/projects/_/destinations). +2. Test your AWS credentials: + ```bash + echo '{}' > test.json + aws s3api put-object \ + --bucket your-bucket-name \ + --key ampersand-test.json \ + --body test.json + ``` +3. Check that IAM permissions for the credential include `s3:PutObject` on the bucket. + +## Limitations + +- **Message size**: Unlike Kinesis, S3 destinations have no practical message size limit — large payloads are written directly to your bucket. +- **One object per message**: Each message becomes a separate S3 object. High-volume syncs produce many small objects, which affects S3 request costs. +- **Ordering**: S3 objects are independent; there are no ordering guarantees. Use the `timestamp` object metadata or a time-based key template to order messages. + + +If you have questions about scaling, contact `support@withampersand.com`. + diff --git a/src/docs.json b/src/docs.json index f14b6995..0c3b2cf1 100644 --- a/src/docs.json +++ b/src/docs.json @@ -58,7 +58,8 @@ "pages": [ "destinations/overview", "destinations/webhooks", - "destinations/kinesis" + "destinations/kinesis", + "destinations/s3" ] }, { diff --git a/src/generate-docs.ts b/src/generate-docs.ts index 399a2123..2b83c791 100644 --- a/src/generate-docs.ts +++ b/src/generate-docs.ts @@ -290,6 +290,7 @@ const baseConfig = { "destinations/overview", "destinations/webhooks", "destinations/kinesis", + "destinations/s3", ] }, { diff --git a/src/notifications/notification-payloads.mdx b/src/notifications/notification-payloads.mdx index fba5bf6f..111359d6 100644 --- a/src/notifications/notification-payloads.mdx +++ b/src/notifications/notification-payloads.mdx @@ -5,7 +5,7 @@ title: Notification payloads The payload structure depends on your destination type: - **Webhook destinations**: Payloads include `notificationType` and `data` fields. -- **Kinesis destinations**: Payloads contain the `data` object. The `notificationType` is included in the event metadata. +- **Kinesis and Amazon S3 destinations**: Payloads contain the `data` object. The `notificationType` is included in the event metadata (for S3, this is the object's metadata). You can find the OpenAPI spec for notification payloads [on GitHub](https://github.com/amp-labs/openapi/blob/main/notifications/notifications.yaml), or refer to the examples below: @@ -99,7 +99,7 @@ If the configuration object is too large to include inline, we will generate a s } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -151,7 +151,7 @@ Note: `config` contains the complete installation configuration object including } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -202,7 +202,7 @@ Note: `config` contains the complete installation configuration object including } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -257,7 +257,7 @@ Note: `config` contains the complete installation configuration object including } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -303,7 +303,7 @@ Note: `fromTimestamp` and `untilTimestamp` indicate the time range of the trigge } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -350,7 +350,7 @@ Note: `error` describes what went wrong during the triggered read. } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -431,7 +431,7 @@ Note: For partial failures, `success` is `false` with both `successfulRecordIds` } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -465,7 +465,7 @@ Note: For partial failures, `success` is `false` with both `successfulRecordIds` } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", @@ -495,7 +495,7 @@ Note: For partial failures, `success` is `false` with both `successfulRecordIds` } ``` -**Kinesis destination payload:** +**Kinesis and S3 destination payload:** ```json { "projectId": "8f4a2b1c-3d5e-4f6a-9b0c-1d2e3f4a5b6c", diff --git a/src/notifications/overview.mdx b/src/notifications/overview.mdx index 0426be7a..2191171e 100644 --- a/src/notifications/overview.mdx +++ b/src/notifications/overview.mdx @@ -55,7 +55,7 @@ There are two ways to set up notifications: - Go to [Destinations](https://dashboard.withampersand.com/projects/_/settings/destinations/) - Click **New Destination** and configure: - **Name**: A descriptive name (e.g., "production-alerts") - - **Type**: Choose webhook, Kinesis, or log destination + - **Type**: Choose webhook, Kinesis, Amazon S3, or log destination - **Configuration**: Provide the endpoint URL or stream details - Click **Create Destination**