tools/ddb_migration: add zero-downtime migration toolkit + one-click demo - #164
Draft
chauhansarthak wants to merge 1 commit into
Draft
tools/ddb_migration: add zero-downtime migration toolkit + one-click demo#164chauhansarthak wants to merge 1 commit into
chauhansarthak wants to merge 1 commit into
Conversation
…demo
General-purpose tooling for migrating an Amazon DynamoDB table to a new table
without taking writes offline. Built on Streams + S3 Export + conditional
writes with a single-attribute (_migration_ts) conflict-resolution scheme.
Components:
- lambda/stream_replay.py Streams replay with newer-wins conditional puts
and tombstones for REMOVE events. TRANSFORM_MODULE
env var honored for pluggable transforms.
- scripts/backfill.py Parallel S3-export -> target loader with iterator-
age circuit breaker (configurable cadence) and
--dry-run.
- scripts/convergence_check.py Pre-cutover gate: iterator age, DLQ depth,
and Scan COUNT drift. Exits non-zero on failure.
- scripts/cleanup.py Post-cutover: TTL-based tombstone expiry and
_migration_ts attribute removal.
- scripts/verify_cutover.py Sample-based source <-> target parity check.
- transform.py Shared customizable per-item transform.
- deploy.sh / teardown.sh Idempotent provisioning of Lambda, IAM role,
DLQ, SNS topic, and CloudWatch alarms (with
actions). Cross-account branch via TARGET_ACCOUNT
+ TARGET_ROLE_ARN.
- iam/policies.json Reference IAM templates including cross-account.
- demo/ One-click demo: seed -> live-write -> migrate ->
verify, gated behind DDB_MIGRATION_DEMO_CONFIRM.
- tests/ 35 moto-backed unit tests; 69% line coverage.
Conflict-resolution invariant: every target write carries _migration_ts.
Backfill writes use 0 (lowest priority); stream replay uses the source event
ApproximateCreationDateTime. The condition
attribute_not_exists(#pk) OR #ts < :ts
ensures newer timestamps always win, regardless of arrival order.
REMOVE events become _tombstone=True items so the in-flight backfill cannot
resurrect deletions. Post-cutover cleanup.py sets a TTL attribute so DynamoDB
auto-expires the tombstones after the validation period.
Contributor
|
Hello! Thanks for your contribution! Please see this comment. I will review the PR and discuss with the team where is the best place to put this contribution. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tools/ddb_migration/— a general-purpose toolkit for migrating an Amazon DynamoDB table to a new table without taking writes offline. Built on three native DynamoDB features (Export to S3 + Streams + conditional writes) with a single-attribute (_migration_ts) conflict-resolution scheme.Use cases: MRSC adoption on Global Tables, cross-account migrations, schema changes, billing-mode transitions on large tables, single-table consolidation.
Architecture
Three overlapping phases:
backfill.py(writes with_migration_ts=0)._migration_ts = ApproximateCreationDateTime. Conditionalattribute_not_exists(#pk) OR #ts < :tsensures newer wins.convergence_check.pyblocks until iterator age, DLQ, and Scan COUNT all agree.REMOVE events become
_tombstone=Trueitems so the in-flight backfill cannot resurrect deletes;cleanup.pyenables DynamoDB TTL post-cutover to auto-expire them.Files
lambda/stream_replay.pyTRANSFORM_MODULE, structured JSON loggingscripts/backfill.py--dry-runscripts/convergence_check.pyscripts/cleanup.py_migration_tsremovalscripts/verify_cutover.pytransform.pydeploy.sh/teardown.sh--alarm-actions)iam/policies.jsondemo/DDB_MIGRATION_DEMO_CONFIRM=yes)tests/Test plan
make install && make test— 37 tests pass with motobash -nsyntax check ondeploy.sh,teardown.sh,demo/run_demo.shpython -m py_compileon every.pyfile_migration_tsfor every item — conflict resolution working as designed)verify_cutover.py --sample-size 500→ 500 matched, 0 missing, 0 divergedteardown.sh CONFIRM=yescleanly removed all provisioned resourcesNotes
backfill.pyparallelizes within one host. README documents the limitation.bulk_executor,dax_calculator): tool-level NOTICE, Makefile, pytest.ini, requirements.txt + requirements-dev.txt, no per-file license headers.By submitting this PR I confirm that my contribution is licensed under Apache 2.0.