chore: add scripts to scrub and import partial db dumps - #2110
Open
netomi wants to merge 1 commit into
Open
Conversation
Member
|
wow. Will try out these locally |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds two developer scripts to help (1) scrub sensitive fields from partial Postgres COPY-TEXT dumps under db/dump, and (2) import those dumps into a local Postgres instance, including optional Flyway-version targeting/auto-detection.
Changes:
- Added
scripts/scrub-db-dump.shto spin up a scratch Postgres, import selected dump tables, redact sensitive columns, and write the redacted data back to the dump directory. - Added
scripts/import-db-dump.shto load the dump into a target Postgres in a single transaction, including schema-version auto-detection and sequence fixups.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| scripts/scrub-db-dump.sh | New scrubber script: detects dump-era schema, imports key tables into scratch DB, redacts sensitive fields, and replaces dump CSVs. |
| scripts/import-db-dump.sh | New importer script: detects dump-era column layout via scratch DB, migrates/validates target state, deletes + reloads dump tables transactionally, and resets sequences. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+220
to
+221
| UPDATE personal_access_token | ||
| SET value = encode(sha256(('redacted-pat-' || id)::bytea), 'hex'); |
Comment on lines
+226
to
+231
| UPDATE user_data | ||
| SET email = 'user' || id || '@example.invalid', | ||
| full_name = 'Redacted User ' || id, | ||
| avatar_url = NULL, | ||
| eclipse_token = NULL, | ||
| eclipse_person_id = NULL; |
| # rather than from custom bookkeeping. | ||
| mapfile -t VERSIONS < <( | ||
| find "${MIGRATIONS_DIR}" -maxdepth 1 -name 'V*__*' -printf '%f\n' \ | ||
| | sed -E 's/^V([0-9_]+)__.*/\1/; s/_/./' \ |
|
|
||
| mapfile -t VERSIONS < <( | ||
| find "${MIGRATIONS_DIR}" -maxdepth 1 -name 'V*__*' -printf '%f\n' \ | ||
| | sed -E 's/^V([0-9_]+)__.*/\1/; s/_/./' \ |
Comment on lines
+327
to
+345
| for t in "${TABLES[@]}"; do | ||
| IFS=',' read -r -a dump_cols <<< "${COLUMN_LISTS[${t}]}" | ||
| gap_cols=$(target_psql -t -c " | ||
| select column_name from information_schema.columns | ||
| where table_schema='public' and table_name='${t}' | ||
| and is_nullable='NO' and column_default is null | ||
| and column_name not in ($(printf "'%s'," "${dump_cols[@]}" | sed 's/,$//')); | ||
| " | tr -d ' ' | grep -v '^$' || true) | ||
| while IFS= read -r col; do | ||
| [ -z "${col}" ] && continue | ||
| key="${t}.${col}" | ||
| GAP_COLUMNS["${key}"]=1 | ||
| if [ -z "${KNOWN_BACKFILLS[${key}]:-}" ]; then | ||
| UNKNOWN_GAPS+=("${key}") | ||
| else | ||
| echo " ${key}: no value in the dump, not nullable, no default - will backfill (known)" | ||
| fi | ||
| done <<< "${gap_cols}" | ||
| done |
Contributor
Author
|
I did use the scripts in the following way:
After that run the local server with a later migration version, which will then automatically migrate to that version. That also allows to verify that the migration e.g. on a production DB will work as expected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds 2 scripts to easily import partial db dumps:
scripts/scrub-db-dump.shscripts/import-db-dump.shthe first script scrubs all sensitive and PII information from a dump located in
db/dump.the second script imports the dump in the some location to a locally running postgresql instance
both scripts support a
-tparameter to indicate the target db migration version.