awslabs · shiladitya-bits · May 27, 2026
diff --git a/tools/ddb_migration/README.md b/tools/ddb_migration/README.md
@@ -0,0 +1,131 @@
+# DynamoDB Zero-Downtime Migration
+
+Zero-downtime migration between DynamoDB tables using DynamoDB Streams, Export to S3, and conditional writes for conflict resolution.
+
+## When to use this
+
+- Migrating to Global Tables with Multi-Region Strong Consistency (MRSC)
+- Moving a table to a different AWS account
+- Restructuring key schema or adding GSIs that require a new table
+- Consolidating multiple tables into a single-table design
+- Any scenario where you need a new table but can't stop writes
+
+## How it works
+
+Three overlapping phases:
+
+1. **Stream Capture & Bulk Copy** -- Enable streams, export to S3, backfill into target (parallel with stream replay)
+2. **Stream Catch-up** -- Lambda replays live changes with conditional writes
+3. **Convergence & Switchover** -- Verify consistency, pause briefly, flip routing
+
+Conflict resolution uses a `_migration_ts` attribute with conditional writes. Backfill writes `_migration_ts = 0` (lowest priority). Stream replay writes `_migration_ts = ApproximateCreationDateTime` (always wins over backfill). Later timestamps always win over earlier ones.
+
+## Package structure
+
+```
+ddb_migration/
+  lambda/
+    stream_replay.py    # Lambda handler for DynamoDB Streams -> target table
+  glue/
+    backfill_job.py     # AWS Glue job for large tables (100+ GiB)
+  scripts/
+    backfill.py         # Standalone backfill script (tables < 100 GiB)
+    convergence_check.py # Pre-cutover verification
+  iam/
+    policies.json       # IAM policy templates for each component
+  transform.py          # Shared item transformation (customize for schema changes)
+  deploy.sh             # One-command infrastructure setup
+  requirements.txt
+```
+
+## Quick start
+
+```bash
+# 1. Configure
+export SOURCE_TABLE=OrdersV1
+export TARGET_TABLE=OrdersV2
+export PARTITION_KEY=pk
+export SORT_KEY=sk
+export REGION=us-east-1
+
+# 2. Deploy infrastructure (creates target table, Lambda, alarms)
+./deploy.sh
+
+# 3. Export source table
+aws dynamodb export-table-to-point-in-time \
+  --table-arn arn:aws:dynamodb:$REGION:$(aws sts get-caller-identity --query Account --output text):table/$SOURCE_TABLE \
+  --s3-bucket $EXPORT_BUCKET --s3-prefix exports/ \
+  --export-format DYNAMODB_JSON --region $REGION
+
+# 4. Run backfill (while stream replay Lambda is already processing)
+export EXPORT_BUCKET=ddb-migration-$(aws sts get-caller-identity --query Account --output text)-$REGION
+export EXPORT_PREFIX=exports/
+python scripts/backfill.py
+
+# 5. Verify convergence
+python scripts/convergence_check.py
+
+# 6. Cutover: flip your application routing to TARGET_TABLE
+```
+
+## For large tables (100+ GiB)
+
+Use the Glue job instead of the standalone script:
+
+```bash
+aws glue create-job \
+  --name migration-backfill \
+  --role AWSGlueServiceRole-Migration \
+  --command '{"Name":"pythonshell","ScriptLocation":"s3://bucket/glue/backfill_job.py","PythonVersion":"3.9"}' \
+  --default-arguments '{
+    "--TARGET_TABLE":"OrdersV2",
+    "--PARTITION_KEY":"pk",
+    "--EXPORT_BUCKET":"my-bucket",
+    "--EXPORT_PREFIX":"exports/",
+    "--TARGET_REGION":"us-east-1",
+    "--LAMBDA_FUNCTION":"migration-stream-replay",
+    "--additional-python-modules":"boto3"
+  }' \
+  --max-capacity 1.0
+
+aws glue start-job-run --job-name migration-backfill
+```
+
+Or use [Bulk Executor](../bulk_executor) for the simple path (maintenance window, no conditional writes needed):
+```bash
+./bulk load-export --table OrdersV2 --s3-path "s3://bucket/exports/AWSDynamoDB/export-id"
+```
+
+## Schema changes
+
+If your target table has a different schema, edit `transform.py`:
+
+```python
+def transform(item, source_event=None):
+    # Example: rename attribute
+    item['order_id'] = item.pop('orderId', item.get('order_id'))
+    # Example: add computed field
+    item['gsi1pk'] = f"TENANT#{item['tenant_id']}"
+    return item
+```
+
+Both the Lambda and backfill import this module, ensuring identical transformations.
+
+## Cross-account migration
+
+1. Add a resource-based policy on the target table (see `iam/policies.json` -> `CrossAccountTarget`)
+2. Set `TARGET_REGION` and configure the Lambda to assume a role in the target account
+3. The S3 export bucket needs a bucket policy allowing the target account to read
+
+## Prerequisites
+
+- Python 3.9+
+- boto3
+- Source table with PITR enabled
+- AWS CLI v2
+
+## Related
+
+- [Zero-Downtime Migration to MRSC](https://quip-amazon.com/4fhmAPBfxor6) -- MRSC-specific walkthrough
+- [Bulk Executor](../bulk_executor) -- for simple-path migrations (maintenance window)
+- [AWS Glue DynamoDB export blog](https://aws.amazon.com/blogs/database/filter-transform-and-load-your-dynamodb-table-exports-using-aws-glue/)
diff --git a/tools/ddb_migration/deploy.sh b/tools/ddb_migration/deploy.sh
@@ -0,0 +1,147 @@
+#!/bin/bash
+# Deploy infrastructure for DynamoDB zero-downtime migration.
+#
+# Creates: target table, stream replay Lambda, event source mapping,
+#          CloudWatch alarms, and optionally a Glue job for large tables.
+#
+# Usage:
+#   export SOURCE_TABLE=OrdersV1
+#   export TARGET_TABLE=OrdersV2
+#   export PARTITION_KEY=pk
+#   export REGION=us-east-1
+#   ./deploy.sh
+#
+# For cross-account: also set TARGET_ACCOUNT and TARGET_ROLE
+
+set -euo pipefail
+
+REGION=${REGION:-us-east-1}
+ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
+SOURCE_TABLE=${SOURCE_TABLE:?SOURCE_TABLE required}
+TARGET_TABLE=${TARGET_TABLE:?TARGET_TABLE required}
+PARTITION_KEY=${PARTITION_KEY:-pk}
+SORT_KEY=${SORT_KEY:-}
+LAMBDA_NAME="migration-stream-replay"
+EXPORT_BUCKET=${EXPORT_BUCKET:-ddb-migration-${ACCOUNT_ID}-${REGION}}
+
+echo "=== DynamoDB Zero-Downtime Migration Setup ==="
+echo "Source: $SOURCE_TABLE | Target: $TARGET_TABLE | Region: $REGION"
+echo ""
+
+# --- 1. Create target table ---
+echo "1. Creating target table..."
+KEY_SCHEMA="AttributeName=$PARTITION_KEY,KeyType=HASH"
+ATTR_DEFS="AttributeName=$PARTITION_KEY,AttributeType=S"
+if [ -n "${SORT_KEY:-}" ]; then
+  KEY_SCHEMA="$KEY_SCHEMA AttributeName=$SORT_KEY,KeyType=RANGE"
+  ATTR_DEFS="$ATTR_DEFS AttributeName=$SORT_KEY,AttributeType=S"
+fi
+
+aws dynamodb create-table \
+  --table-name "$TARGET_TABLE" \
+  --attribute-definitions $ATTR_DEFS \
+  --key-schema $KEY_SCHEMA \
+  --billing-mode PAY_PER_REQUEST \
+  --region "$REGION" > /dev/null 2>&1 || echo "   (already exists)"
+aws dynamodb wait table-exists --table-name "$TARGET_TABLE" --region "$REGION"
+echo "   Done."
+
+# --- 2. Enable streams on source ---
+echo "2. Enabling streams on $SOURCE_TABLE..."
+aws dynamodb update-table \
+  --table-name "$SOURCE_TABLE" \
+  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \
+  --region "$REGION" > /dev/null 2>&1 || echo "   (already enabled)"
+echo "   Done."
+
+# --- 3. Create S3 bucket for export ---
+echo "3. Creating export bucket ($EXPORT_BUCKET)..."
+if [ "$REGION" = "us-east-1" ]; then
+  aws s3api create-bucket --bucket "$EXPORT_BUCKET" --region "$REGION" 2>/dev/null || true
+else
+  aws s3api create-bucket --bucket "$EXPORT_BUCKET" --region "$REGION" \
+    --create-bucket-configuration LocationConstraint="$REGION" 2>/dev/null || true
+fi
+echo "   Done."
+
+# --- 4. Deploy Lambda ---
+echo "4. Deploying stream replay Lambda..."
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR/lambda"
+zip -j function.zip stream_replay.py ../transform.py > /dev/null
+
+aws lambda create-function \
+  --function-name "$LAMBDA_NAME" \
+  --runtime python3.12 \
+  --handler stream_replay.handler \
+  --role "arn:aws:iam::${ACCOUNT_ID}:role/migration-stream-replay-role" \
+  --environment "Variables={TARGET_TABLE=$TARGET_TABLE,PARTITION_KEY=$PARTITION_KEY,TARGET_REGION=$REGION}" \
+  --timeout 300 \
+  --memory-size 512 \
+  --zip-file fileb://function.zip \
+  --region "$REGION" > /dev/null 2>&1 || \
+  aws lambda update-function-code \
+    --function-name "$LAMBDA_NAME" \
+    --zip-file fileb://function.zip \
+    --region "$REGION" > /dev/null
+
+rm -f function.zip
+cd "$SCRIPT_DIR"
+echo "   Done."
+
+# --- 5. Create event source mapping ---
+echo "5. Creating event source mapping..."
+STREAM_ARN=$(aws dynamodb describe-table \
+  --table-name "$SOURCE_TABLE" \
+  --query 'Table.LatestStreamArn' --output text --region "$REGION")
+
+aws lambda create-event-source-mapping \
+  --function-name "$LAMBDA_NAME" \
+  --event-source-arn "$STREAM_ARN" \
+  --starting-position TRIM_HORIZON \
+  --batch-size 100 \
+  --maximum-batching-window-in-seconds 5 \
+  --bisect-batch-on-function-error \
+  --maximum-retry-attempts 3 \
+  --function-response-types ReportBatchItemFailures \
+  --region "$REGION" > /dev/null 2>&1 || echo "   (already exists)"
+echo "   Done."
+
+# --- 6. CloudWatch alarms ---
+echo "6. Creating iterator age alarms..."
+aws cloudwatch put-metric-alarm \
+  --alarm-name "Migration-IteratorAge-Warning" \
+  --metric-name IteratorAge \
+  --namespace AWS/Lambda \
+  --dimensions "Name=FunctionName,Value=$LAMBDA_NAME" \
+  --statistic Maximum --period 60 --evaluation-periods 5 \
+  --threshold 43200000 \
+  --comparison-operator GreaterThanThreshold \
+  --region "$REGION" 2>/dev/null
+
+aws cloudwatch put-metric-alarm \
+  --alarm-name "Migration-IteratorAge-Critical" \
+  --metric-name IteratorAge \
+  --namespace AWS/Lambda \
+  --dimensions "Name=FunctionName,Value=$LAMBDA_NAME" \
+  --statistic Maximum --period 60 --evaluation-periods 5 \
+  --threshold 72000000 \
+  --comparison-operator GreaterThanThreshold \
+  --region "$REGION" 2>/dev/null
+echo "   Done."
+
+echo ""
+echo "=== Setup complete ==="
+echo ""
+echo "Next steps:"
+echo "  1. Export:    aws dynamodb export-table-to-point-in-time \\"
+echo "                 --table-arn arn:aws:dynamodb:$REGION:$ACCOUNT_ID:table/$SOURCE_TABLE \\"
+echo "                 --s3-bucket $EXPORT_BUCKET --s3-prefix exports/ \\"
+echo "                 --export-format DYNAMODB_JSON --region $REGION"
+echo ""
+echo "  2. Backfill:  export EXPORT_BUCKET=$EXPORT_BUCKET EXPORT_PREFIX=exports/"
+echo "               python scripts/backfill.py"
+echo ""
+echo "  3. Monitor:   python scripts/convergence_check.py"
+echo ""
+echo "  4. Cutover:   Flip your application routing to $TARGET_TABLE"