tools/ddb_migration: add zero-downtime migration toolkit + one-click demo

tools/ddb_migration/.gitignore

@@ -0,0 +1,15 @@
+.DS_Store
+__pycache__/
+*.py[cod]
+.venv
+venv
+.pytest_cache
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+
+# Demo artifacts
+demo/config.env
+demo/.demo_state
+*.zip

tools/ddb_migration/AGENTS.md

@@ -0,0 +1,45 @@
+# Agents Guide — `tools/ddb_migration`
+
+Conventions for future contributors (human or AI) extending this toolkit.
+
+## Module boundaries
+
+- **`transform.py`** — single source of truth for any per-item transform. Both `lambda/stream_replay.py` and `scripts/backfill.py` import it. Keep it free of AWS SDK calls.
+- **`lambda/stream_replay.py`** — runs in Lambda. No filesystem writes, no `print()` for structured data (use the `_log()` helper). Every record write must be conditional on `_migration_ts`.
+- **`scripts/backfill.py`** — runs locally or in EC2/CodeBuild. Same conditional-write contract as the Lambda. Always sets `_migration_ts = 0`.
+- **`scripts/convergence_check.py`** — gate (exits 0 / 1). Do not add interactive prompts.
+- **`scripts/cleanup.py`** — idempotent post-cutover. Safe to re-run.
+- **`scripts/verify_cutover.py`** — read-only. Never writes to source or target.
+
+## Conditional-write contract (load-bearing)
+
+Every write to the target table goes through:
+
+```python
+table.put_item(
+    Item=item,
+    ConditionExpression='attribute_not_exists(#pk) OR #ts < :ts',
+    ExpressionAttributeNames={'#pk': partition_key, '#ts': '_migration_ts'},
+    ExpressionAttributeValues={':ts': migration_ts},
+)
+```
+
+`migration_ts` rules:
+- backfill writes: `0`
+- stream replay `INSERT`/`MODIFY`: `event['dynamodb']['ApproximateCreationDateTime']`
+- stream replay `REMOVE` (tombstone): `event['dynamodb']['ApproximateCreationDateTime']`
+
+Do not bypass this. Do not introduce a third writer with a different timestamp source.
+
+## Adding a new script
+
+1. Add `scripts/your_script.py` with a `main()` that returns an `int` exit code.
+2. Add `tests/test_your_script.py` using moto fixtures from `conftest.py`.
+3. Document in `README.md` under "Scripts."
+4. If it provisions AWS resources, hook it into `deploy.sh` AND `teardown.sh`.
+
+## Testing
+
+- Unit tests use `moto`. No real AWS calls.
+- DynamoDB Streams events are constructed as fixtures (moto's stream support is partial); we mock the Lambda handler's boto3 client when needed.
+- `make test` must stay green before any commit.

tools/ddb_migration/Makefile

@@ -0,0 +1,44 @@
+.DEFAULT_GOAL := help
+.PHONY: help install test coverage lint clean
+
+VENV := .venv
+PIP := $(VENV)/bin/pip
+PYTEST := $(VENV)/bin/pytest
+
+PYTHON := $(shell command -v python3.11 || command -v python3.10 || command -v python3)
+
+help:
+	@echo "Targets:"
+	@echo "  make install   Create venv and install dev dependencies"
+	@echo "  make test      Run unit tests with coverage summary"
+	@echo "  make coverage  Run tests with detailed missing-line coverage"
+	@echo "  make lint      Syntax check shell + python sources"
+	@echo "  make clean     Remove venv and caches"
+
+install:
+	$(PYTHON) -m venv $(VENV)
+	$(PIP) install --upgrade pip
+	$(PIP) install -r requirements-dev.txt
+
+test:
+	@$(PYTEST) -q --tb=short --cov=scripts --cov=lambda --cov-branch --cov-report= 2>&1 | tail -10
+	@$(VENV)/bin/coverage json -o .coverage.json --quiet 2>/dev/null && \
+	  $(VENV)/bin/python -c "import json; t=json.load(open('.coverage.json'))['totals']; \
+	    print(); \
+	    print(f\"  LINE COVERAGE:   {t['percent_statements_covered']:5.1f}%  ({t['covered_lines']}/{t['num_statements']} statements covered)\"); \
+	    print(f\"  BRANCH COVERAGE: {t['percent_branches_covered']:5.1f}%  ({t['covered_branches']}/{t['num_branches']} branches covered)\"); \
+	    print()" && rm -f .coverage.json
+
+coverage:
+	$(PYTEST) --cov=scripts --cov=lambda --cov-branch --cov-report=term-missing
+
+lint:
+	@for f in deploy.sh teardown.sh demo/run_demo.sh; do bash -n $$f && echo "  OK $$f" || exit 1; done
+	@find . -name '*.py' -not -path './.venv/*' -exec $(VENV)/bin/python -m py_compile {} +
+	@echo "  OK python syntax"
+
+clean:
+	rm -rf $(VENV) .pytest_cache
+	find . -type d -name __pycache__ -exec rm -rf {} +
+	find . -type f -name .coverage -delete
+	find . -type f -name .coverage.json -delete

tools/ddb_migration/NOTICE

@@ -0,0 +1,2 @@
+DynamoDB Zero-Downtime Migration Toolkit
+Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.

tools/ddb_migration/README.md

@@ -0,0 +1,210 @@
+# DynamoDB Zero-Downtime Migration Toolkit
+
+General-purpose, production-grade tooling for migrating an Amazon DynamoDB
+table to a new table without taking writes offline. Built on three native
+DynamoDB features — Export to S3, Streams, and conditional writes — and a
+single-attribute conflict-resolution scheme.
+
+Use cases:
+
+* Adopting Multi-Region Strong Consistency (MRSC) on Global Tables (requires
+  starting from an empty replica).
+* Cross-account migrations (workload consolidation, account splits, compliance).
+* Schema changes — partition→composite key, attribute renames, new GSI keys.
+* Billing-mode transitions on large tables without surprise throttling.
+* Single-table consolidation across small tables.
+
+## Architecture
+
+Three overlapping phases:
+
+1. **Capture & bulk copy** — enable Streams on the source, export it to S3,
+   start the stream-replay Lambda, run `backfill.py` to load the export into
+   the target. Backfill writes go in with `_migration_ts=0`.
+2. **Catch-up** — the Lambda replays every live source-table change to the
+   target, with `_migration_ts = ApproximateCreationDateTime` (always > 0).
+   The conditional expression `attribute_not_exists(#pk) OR #ts < :ts` ensures
+   newer timestamps always win, so backfill writes never overwrite live writes
+   and stale events never overwrite newer ones.
+3. **Convergence & cutover** — `convergence_check.py` blocks until iterator age
+   is near zero, the DLQ is empty, and Scan COUNT on both tables agrees. Then
+   you flip routing.
+
+REMOVE events are written as `_tombstone=True` items (not deletes) so the
+in-flight backfill cannot resurrect them. Post-cutover, `cleanup.py` enables
+DynamoDB TTL to expire them automatically.
+
+## Layout
+
+```
+tools/ddb_migration/
+├── deploy.sh / teardown.sh         One-command provision / clean-up
+├── transform.py                    Shared per-item transform (customize here)
+├── lambda/stream_replay.py         Streams → target, conditional writes
+├── scripts/
+│   ├── backfill.py                 S3 export → target, parallel + throttled
+│   ├── convergence_check.py        Pre-cutover gate (exit 0 / 1)
+│   ├── cleanup.py                  Post-cutover removal of migration metadata
+│   └── verify_cutover.py           Sample-based source ↔ target verifier
+├── iam/policies.json               Reference IAM policy templates
+├── demo/                           One-click demo (real AWS resources)
+└── tests/                          pytest unit tests with moto
+```
+
+## Prerequisites
+
+* Python 3.10+ and `pip install -r requirements.txt`.
+* AWS CLI v2, configured for the source-table account.
+* PITR enabled on the source table (for the export). `deploy.sh` does not
+  enable this for you on production tables — verify before running.
+* Permissions to create IAM roles, Lambda functions, S3 buckets, SQS queues,
+  SNS topics, and CloudWatch alarms.
+
+## Quick start (same account)
+
+```sh
+cd tools/ddb_migration
+make install
+source .venv/bin/activate
+
+export SOURCE_TABLE=my-prod-table
+export TARGET_TABLE=my-prod-table-v2
+export PARTITION_KEY=customer_id
+export SORT_KEY=order_id          # optional
+export REGION=us-east-1
+
+# 1. Provision Lambda, IAM role, DLQ, SNS topic, alarms, target table.
+./deploy.sh
+
+# 2. Trigger an S3 export (the deploy.sh output prints the exact command).
+
+# 3. Run the backfill once the export completes.
+EXPORT_BUCKET=ddb-migration-<account>-us-east-1 \
+  python scripts/backfill.py
+
+# 4. Wait for stream replay to drain. Run the gate.
+DLQ_URL=<from deploy.sh output> python scripts/convergence_check.py
+
+# 5. Sample-verify before flipping app routing.
+python scripts/verify_cutover.py --sample-size 1000
+
+# 6. Flip your application's table reference. Resume traffic.
+
+# 7. After 7-14 days of validation:
+python scripts/cleanup.py
+./teardown.sh CONFIRM=yes        # removes Lambda/role/DLQ/SNS/alarms/bucket
+```
+
+## Configuration
+
+`deploy.sh` reads from environment variables or from `./config.env` if present.
+A non-exhaustive list:
+
+| Var | Default | Notes |
+|-----|---------|-------|
+| `SOURCE_TABLE` | required | Existing table |
+| `TARGET_TABLE` | required | Created by deploy.sh unless cross-account |
+| `PARTITION_KEY` | `pk` | Source-table partition-key attribute name |
+| `PARTITION_KEY_TYPE` | `S` | `S`, `N`, or `B` |
+| `SORT_KEY` | (unset) | Leave empty for hash-only schema |
+| `SORT_KEY_TYPE` | `S` | |
+| `REGION` | `us-east-1` | |
+| `LAMBDA_FUNCTION_NAME` | `ddb-migration-stream-replay` | |
+| `LAMBDA_ROLE_NAME` | `ddb-migration-stream-replay-role` | |
+| `DLQ_NAME` | `ddb-migration-dlq` | |
+| `SNS_TOPIC_NAME` | `ddb-migration-alerts` | Subscribe an endpoint after deploy |
+| `ITERATOR_AGE_WARN_MS` | `43200000` (12 h) | Warning alarm threshold |
+| `ITERATOR_AGE_CRIT_MS` | `72000000` (20 h) | Critical alarm threshold |
+| `TARGET_ACCOUNT` | (unset) | Set for cross-account; switches mode |
+| `TARGET_ROLE_ARN` | (unset) | Required when `TARGET_ACCOUNT` differs |
+| `TRANSFORM_MODULE` | (unset) | Custom Python module path; falls back to bundled `transform.py` |
+
+`backfill.py` and `convergence_check.py` accept the same env vars plus their
+own CLI flags (`--dry-run`, `--ignore-count-drift`, `--max-iterator-age-ms`,
+etc.). Run any of them with `--help` for the full list.
+
+## Customizing the transform
+
+Edit `transform.py` (or set `TRANSFORM_MODULE` to a different module). The
+function runs in the Lambda, in `backfill.py`, and in `verify_cutover.py` —
+all three must share the same logic, which is why it is one file.
+
+```python
+def transform(item, source_event=None):
+    # Rename a column.
+    if "user_id" in item:
+        item["customer_id"] = item.pop("user_id")
+    # Compute a new GSI key.
+    item["status_idx"] = f"{item['status']}#{item['created_at']}"
+    return item   # or return None to skip the item entirely
+```
+
+## Cross-account migrations
+
+When the target table lives in a different account:
+
+1. In the **target** account, create a role `ddb-migration-target-writer` that
+   trusts the source-account stream-replay role and grants
+   `dynamodb:PutItem` + `dynamodb:UpdateItem` on the target table. The
+   resource-based policy template is in `iam/policies.json` →
+   `CrossAccountTargetTablePolicy`.
+2. In the **source** account, set `TARGET_ACCOUNT` and `TARGET_ROLE_ARN`
+   before running `deploy.sh`. The Lambda's inline policy will include
+   `sts:AssumeRole` for that role.
+3. To run `backfill.py` from the target account against the source-account
+   export bucket, attach `iam/policies.json:CrossAccountExportBucketPolicy` to
+   the bucket.
+
+Test IAM end-to-end before you start the migration. Permission failures
+mid-stream waste the 24-hour stream-retention window.
+
+## Convergence gates
+
+`convergence_check.py` runs three checks in sequence:
+
+1. **Iterator age** is below `--max-iterator-age-ms` (default 1000 ms).
+2. **DLQ** is empty (visible + in-flight).
+3. **Scan COUNT** on source vs. target is within `--count-drift-pct`
+   (default 0.5%). The target scan excludes tombstones, so deleted-and-replayed
+   items don't inflate the count. Use `--ignore-count-drift` to skip this check
+   on very large tables where the Scan would be expensive.
+
+The script exits non-zero on any failure, so you can use it in CI:
+
+```sh
+python scripts/convergence_check.py || { echo "not ready"; exit 1; }
+```
+
+## Demo
+
+`demo/run_demo.sh` provisions a tiny source/target pair, seeds 10K items,
+drives live writes during the migration, and verifies the cutover. End-to-end
+runtime ~12 min, cost <$1. See `demo/README.md`.
+
+## Limitations
+
+* Streams retain records for 24 h. If the Lambda falls behind by that long,
+  data is lost — alarms fire at 12 h (warning) and 20 h (critical).
+* `Scan COUNT` for very large tables is expensive and time-consuming. Allow
+  several minutes for tables over 100 GiB.
+* Glue-based backfill for tables larger than ~100 GiB is not bundled;
+  `backfill.py` parallelizes within one host. For very large tables, fan it
+  out across multiple hosts or write a small Glue wrapper.
+* Tombstones live for `--tombstone-ttl-days` (default 7) after `cleanup.py`
+  runs. They are eventually deleted by DynamoDB TTL.
+* Rollback is *not* a flag flip back. Once you cut over, returning to the
+  source requires deploying a reverse-replay Lambda before the cutover so the
+  source stays current.
+
+## Tests
+
+`make test` runs the suite with moto-backed mocks; no AWS calls. `make
+coverage` for a per-line report. The integration path (`demo/run_demo.sh`) is
+gated behind `DDB_MIGRATION_DEMO_CONFIRM=yes` and requires real AWS creds.
+
+## References
+
+* [AWS docs — Export to S3](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.html)
+* [AWS docs — DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html)
+* [AWS docs — Conditional writes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html)
+* [Bulk Executor for DynamoDB](https://github.com/awslabs/amazon-dynamodb-tools/tree/main/tools/bulk_executor) — sibling tool for non-zero-downtime bulk operations.

tools/ddb_migration/demo/README.md

@@ -0,0 +1,59 @@
+# `tools/ddb_migration/demo` — One-click migration demo
+
+Provisions a small source + target pair, seeds 10,000 items, drives live writes
+during the migration, runs the convergence gate, and verifies the cutover.
+
+End-to-end: ~12–15 minutes. Cost: <$1 in a US region.
+
+## Prerequisites
+
+* An AWS account with admin-equivalent permissions (the demo creates IAM roles,
+  Lambda, DynamoDB tables, S3 bucket, SNS topic, SQS queue, CloudWatch alarms).
+* `aws` CLI v2 configured (`AWS_PROFILE` set).
+* Python 3.10+ with the parent tool's dependencies installed:
+  ```sh
+  cd tools/ddb_migration
+  make install
+  source .venv/bin/activate
+  ```
+
+## Run it
+
+```sh
+cp demo/config.example.env demo/config.env
+# edit demo/config.env: set AWS_PROFILE and REGION
+DDB_MIGRATION_DEMO_CONFIRM=yes ./demo/run_demo.sh
+```
+
+## What you'll see
+
+```
+========== 1/9  Creating source table ddb-migration-demo-source ==========
+========== 2/9  Provisioning migration infrastructure (deploy.sh) ==========
+[deploy] Ensuring target table ddb-migration-demo-target exists in us-east-1
+...
+========== 9/9  Verifying sample of items ==========
+  matched=500  missing=0  diverged=0  total=500
+VERIFY OK
+
+DEMO PASSED
+```
+
+## Clean up
+
+```sh
+CONFIRM=yes ./teardown.sh
+aws dynamodb delete-table --table-name ddb-migration-demo-source --region us-east-1
+aws dynamodb delete-table --table-name ddb-migration-demo-target --region us-east-1
+```
+
+## What it actually demonstrates
+
+1. Source table receives live writes throughout the migration window.
+2. The S3 export captures the source at a point in time.
+3. `backfill.py` loads the export into the target with `_migration_ts=0`.
+4. The Lambda is replaying live writes (with newer `_migration_ts`) in parallel.
+5. Conflict resolution: backfill writes never overwrite live updates.
+6. Tombstones prevent the backfill from resurrecting deleted items.
+7. The convergence gate blocks cutover until iterator age, DLQ, and counts agree.
+8. The verifier samples 500 items and proves source ↔ target parity.

tools/ddb_migration/demo/config.example.env

@@ -0,0 +1,15 @@
+# Copy to ./config.env and fill in the blanks before running run_demo.sh.
+# Both deploy.sh and run_demo.sh source this file automatically.
+
+# Required.
+export AWS_PROFILE=your-dev-profile
+export REGION=us-east-1
+
+# Demo-specific overrides — leave defaults unless you want different names.
+export SOURCE_TABLE=ddb-migration-demo-source
+export TARGET_TABLE=ddb-migration-demo-target
+export PARTITION_KEY=pk
+export SORT_KEY=sk
+export DEMO_ITEM_COUNT=10000
+export DEMO_LIVE_WRITE_RATE=5
+export DEMO_LIVE_WRITE_DURATION_SECS=120