Replace manual config confirmations with automated cross-reference readiness check

README.md

@@ -7,6 +7,7 @@ This repository contains a tool that takes a JSON file as input, containing a li
 ## Table of Contents
 
 - [Getting Started](#getting-started)
+- [Exporting Users](#exporting-users)
 - [Migrating OAuth Connections](#migrating-oauth-connections)
 - [Handle Existing User IDs and Foreign Key Constraints](#handle-existing-user-ids-and-foreign-key-constraints)
 - [Configuration](#configuration)
@@ -15,6 +16,7 @@ This repository contains a tool that takes a JSON file as input, containing a li
 
 ### Documentation
 
+- [Exporting Users](docs/exporting-users.md)
 - [Schema Fields Reference](docs/schema-fields.md)
 - [Creating Custom Transformers](docs/creating-transformers.md)
 - [AI Migration Prompt](prompts/migration-prompt.md)
@@ -169,6 +171,14 @@ bun migrate -y \
 - `--file` (or `-f`)
 - `CLERK_SECRET_KEY` (via `--clerk-secret-key`, environment variable, or `.env` file)
 
+## Exporting Users
+
+Some platforms require exporting users directly from their database before migrating. See the [Exporting Users](docs/exporting-users.md) guide for setup, CLI options, and troubleshooting.
+
+```bash
+bun export:supabase
+```
+
 ## Migrating OAuth Connections
 
 OAuth connections can not be directly migrated. The creation of the connection requires the user to consent, which can't happen on a migration like this. Instead you can rely on Clerk's [Account Linking](https://clerk.com/docs/guides/configure/auth-strategies/social-connections/account-linking) to handle this.

docs/exporting-users.md

@@ -0,0 +1,130 @@
+# Exporting Users
+
+Some platforms require exporting users directly from their database before migrating to Clerk. This guide covers the available export commands and how to use them.
+
+## Supabase
+
+The Supabase export connects directly to your Supabase Postgres database and exports users from the `auth.users` table. This is the recommended approach because it includes `encrypted_password` (bcrypt hashes), which are not available through the Supabase Admin API.
+
+### Prerequisites
+
+- A Supabase project with **Auth enabled** (Authentication section in the Supabase Dashboard)
+- A Postgres connection string from your Supabase project
+
+### Getting Your Connection String
+
+1. Open your Supabase project dashboard
+2. Click the **Connect** button
+3. Copy one of the connection strings below
+
+**Pooler connection** (recommended):
+
+```
+postgres://postgres.[REF]:[PASSWORD]@aws-0-[REGION].pooler.supabase.com:6543/postgres
+```
+
+**Direct connection** (requires the IPv4 add-on):
+
+```
+postgresql://postgres:[PASSWORD]@db.[REF].supabase.co:5432/postgres
+```
+
+Replace `[PASSWORD]` with your database password. If your password contains special characters (`@`, `#`, `%`, etc.), they must be URL-encoded (e.g. `@` becomes `%40`).
+
+### Environment Variables
+
+| Variable          | Description                |
+| ----------------- | -------------------------- |
+| `SUPABASE_DB_URL` | Postgres connection string |
+
+Add this to your `.env` file to skip the interactive prompt:
+
+```bash
+SUPABASE_DB_URL=postgres://postgres.ref:password@aws-0-us-east-1.pooler.supabase.com:6543/postgres
+```
+
+### Usage
+
+```bash
+bun export:supabase
+```
+
+If no connection string is found in environment variables, the CLI will prompt you to enter one.
+
+### CLI Options
+
+| Option            | Description                                               |
+| ----------------- | --------------------------------------------------------- |
+| `--db-url <url>`  | Postgres connection string (takes priority over env vars) |
+| `--output <path>` | Output file path (default: `supabase-export.json`)        |
+
+```bash
+# Specify connection string and output file
+bun export:supabase --db-url "postgres://..." --output users.json
+```
+
+### Connection String Priority
+
+The export resolves the connection string in this order:
+
+1. `--db-url` CLI flag
+2. `SUPABASE_DB_URL` environment variable
+3. Interactive prompt
+
+### Output
+
+The export produces a JSON file (default: `supabase-export.json`) containing an array of user objects with the following fields:
+
+| Field                | Description                                                        |
+| -------------------- | ------------------------------------------------------------------ |
+| `id`                 | Supabase user ID                                                   |
+| `email`              | User email address                                                 |
+| `email_confirmed_at` | Email verification timestamp                                       |
+| `encrypted_password` | Bcrypt password hash                                               |
+| `phone`              | Phone number                                                       |
+| `phone_confirmed_at` | Phone verification timestamp                                       |
+| `first_name`         | Extracted from `display_name`, `first_name`, or `name` in metadata |
+| `last_name`          | Extracted from `last_name` in metadata                             |
+| `raw_user_meta_data` | Full user metadata object                                          |
+| `raw_app_meta_data`  | Full app metadata object                                           |
+| `created_at`         | Account creation timestamp                                         |
+
+After exporting, a field coverage summary shows how many users have each field populated.
+
+### Next Step
+
+Once the export is complete, run the migration:
+
+```bash
+bun migrate --transformer supabase --file supabase-export.json
+```
+
+### Troubleshooting
+
+#### ENOTFOUND — hostname could not be resolved
+
+The project ref in the connection string is incorrect. Verify it matches your Supabase project by checking the URL in your Supabase Dashboard.
+
+#### ETIMEDOUT or ENETUNREACH — connection timed out
+
+Direct connections (`db.[REF].supabase.co`) require the IPv4 add-on. Use a pooler connection instead, or enable IPv4 in Supabase Dashboard under Settings > Add-Ons.
+
+#### Authentication failed
+
+The database password is incorrect. Reset it in Supabase Dashboard under Settings > Database > Database Password. If the new password contains special characters, URL-encode them in the connection string.
+
+#### Could not read from auth.users
+
+The `auth.users` table is created automatically when Supabase Auth is enabled. Ensure Auth is enabled in Supabase Dashboard under Authentication, and that you are connecting with the `postgres` role (not an application-level role).
+
+#### Connection string cannot be parsed as a URL
+
+The connection string contains characters that break URL parsing. Ensure special characters in the password are URL-encoded:
+
+| Character | Encoded |
+| --------- | ------- |
+| `@`       | `%40`   |
+| `#`       | `%23`   |
+| `%`       | `%25`   |
+| `?`       | `%3F`   |
+| space     | `%20`   |

package.json

@@ -10,6 +10,7 @@
     "clean-logs": "bun ./src/clean-logs/index.ts",
     "convert-logs": "bun ./src/convert-logs/index.ts",
     "delete": "bun ./src/delete/index.ts",
+    "export:supabase": "bun ./src/export/index.ts",
     "format": "prettier . --write",
     "format:test": "prettier . --check",
     "lint": "eslint .",
@@ -27,18 +28,22 @@
   "dependencies": {
     "@clack/prompts": "^1.0.0-alpha.9",
     "@clerk/backend": "^2.29.3",
+    "@clerk/nextjs": "^6.37.3",
     "@clerk/types": "^4.101.11",
     "bun": "^1.3.6",
     "csv-parser": "^3.2.0",
     "dotenv": "16.6.1",
+    "jose": "^6.1.3",
     "mime-types": "^3.0.2",
     "p-limit": "^7.2.0",
+    "pg": "^8.18.0",
     "picocolors": "^1.1.1",
     "zod": "^4.3.5"
   },
   "devDependencies": {
     "@types/bun": "^1.3.6",
     "@types/mime-types": "^3.0.1",
+    "@types/pg": "^8.16.0",
     "@typescript-eslint/eslint-plugin": "^8.53.1",
     "@typescript-eslint/parser": "^8.53.1",
     "eslint": "^9.39.2",

prompts/migration-prompt.md

@@ -19,11 +19,18 @@ Follow these steps EXACTLY in order. Do NOT skip any steps.
 
 ### Step 1: Verify Environment
 
-Before doing ANYTHING else:
+Before proceeding, check if dependencies are installed. If not:
+1. Use `bun install` to install dependencies.
+
+After confirming dependencies are installed and before doing ANYTHING else:
+1. Check if dependencies are installed, and if not use `bun install` to install dependencies.
 1. Check if `.env` file exists with `CLERK_SECRET_KEY`
-2. If missing, IMMEDIATELY ask for the key (Clerk Dashboard → API Keys → Secret keys, or https://dashboard.clerk.com/~/api-keys)
-3. Create/update the `.env` file with the provided key
-4. Do NOT proceed until the key is configured
+1. If missing, IMMEDIATELY ask for the key (Clerk Dashboard → API Keys → Secret keys, or https://dashboard.clerk.com/~/api-keys)
+1. Create/update the `.env` file with the provided key
+1. Do NOT proceed until the key is configured
+
+
+**DO NOT** move to step 1 until the dependenices are installed and the `CLERK_SECRET_KEY` is present in the `.env` file
 
 ### Step 2: Analyze the Data File
 
@@ -40,9 +47,10 @@ Read a sample of the file to understand its structure. Look for signature fields
 ### Step 3A: If a Transformer Matches
 
 1. Tell me which transformer will be used
-2. Summarize the field mappings that will be applied
-3. Ask if I want to proceed with the migration
-4. If confirmed, run:
+1. Summarize the field mappings that will be applied
+1. Use `displayCrossReference()` and related code to display a mnigration readiness table to the user.
+1. Ask if I want to proceed with the migration
+1. If confirmed, run:
    ```bash
    bun migrate -y --transformer [transformer-key] --file [file-path]
    ```
@@ -52,19 +60,11 @@ Read a sample of the file to understand its structure. Look for signature fields
 If the data doesn't match any existing transformer, you MUST:
 
 1. **Inform the user**: Explain that no existing transformer matches their data format
-2. **List the fields found**: Show all fields discovered in their data file
-3. **Create a custom transformer**: Generate a transformer file at `src/transformers/[platform-name].ts` using the 'transformer' skill or the `prompts/transformer-prompt.md`
-4. **MANDATORY - Register the transformer**:
-   - Add an import to `src/transformers/index.ts`
-   - Add the transformer to the `transformers` array
-
-   **THIS STEP IS NOT OPTIONAL.** If you skip registration:
-   - The transformer will NOT appear in the CLI's platform selection
-   - The `bun delete` command will NOT find migrated users
-   - Users will see "Found 0 migrated users to delete" after migration
-
-5. **Run tests**: Execute `bun run test` to verify the transformer is properly registered
-6. **Run the migration**: After tests pass, run the migration command
+1. **List the fields found**: Show all fields discovered in their data file
+1. **Create a custom transformer**: Use the `transformer` skill. If the `transformer` skill is not available use `.claude/skills/transformer/SKILL.md` or `prompts/transformer-prompt.md`. Do not try and create a transformer without using the skill or prompt.
+1. **Run tests**: Execute `bun run test` to verify the transformer is properly registered
+1. Use `displayCrossReference()` and related code to display a mnigration readiness table to the user.  Always display this after any field mapping summary.
+1. **Run the migration**: After tests pass, run the migration command
 
 ### Step 4: Post-Migration Verification
 

prompts/transformer-prompt.md

@@ -13,10 +13,13 @@ I need to create a custom transformer for the Clerk user migration tool. Please
 
 ## Environment Setup
 
+Before proceeding, check if dependencies are installed. If not:
+1. Use `bun install` to install dependencies.
+
 Before generating the transformer, check if a `.env` file exists with `CLERK_SECRET_KEY`. If not:
 1. Ask the user to provide their CLERK_SECRET_KEY (found in Clerk Dashboard → API Keys → Secret keys)
-2. Create the `.env` file with the provided key
-3. Continue with the transformer generation without stopping
+1. Create the `.env` file with the provided key
+1. Continue with the transformer generation without stopping
 
 Do not ask "would you like me to create one?" - just ask for the key directly and create the file.
 
@@ -31,15 +34,15 @@ Do not ask "would you like me to create one?" - just ask for the key directly an
    - If not, ask me for the key (found in Clerk Dashboard → API Keys → Secret keys, or https://dashboard.clerk.com/~/api-keys)
    - Create/update the `.env` file with the key
 
-2. Analyze the JSON/CSV structure to identify:
+1. Analyze the JSON/CSV structure to identify:
    - User ID field (maps to `userId`)
    - Email field(s) and verification status
    - Phone field(s) and verification status
    - Name fields (first name, last name, or combined name)
    - Password field and hash algorithm
    - Any metadata fields
 
-3. Generate a complete transformer file following this structure:
+1. Generate a complete transformer file following this structure:
 
 ```typescript
 // src/transformers/[platform-name].ts
@@ -61,7 +64,7 @@ const [platformName]Transformer = {
 export default [platformName]Transformer;
 ```
 
-4. **CRITICAL - Register the transformer**: After creating the transformer file, you MUST register it in `src/transformers/index.ts`. This is NOT optional. The migration and delete commands will fail silently if the transformer is not registered.
+1. **CRITICAL - Register the transformer**: After creating the transformer file, you MUST register it in `src/transformers/index.ts`. This is NOT optional. The migration and delete commands will fail silently if the transformer is not registered.
 
    Add both an import and include it in the exports array:
 
@@ -80,6 +83,17 @@ export default [platformName]Transformer;
    - The transformer will NOT appear in the CLI's platform selection
    - The `bun delete` command will NOT be able to find migrated users
    - Users will see "Found 0 migrated users to delete" even after successful migration
+
+   **THIS STEP IS NOT OPTIONAL.** If you skip registration:
+   - The transformer will NOT appear in the CLI's platform selection
+   - The `bun delete` command will NOT find migrated users
+   - Users will see "Found 0 migrated users to delete" after migration
+
+1. **Run tests**: Execute `bun run test` to verify the transformer is properly registered
+1. Use `displayCrossReference()` and related code to display a mnigration readiness table to the user. Always display this after any field mapping summary.
+1. **Run the migration**: After tests pass, run the migration command
+
+
 ````
 
 ## Questions to Answer
@@ -194,14 +208,14 @@ After generating your transformer, verify these steps were completed:
 
 ### 1. Transformer File Created
 
-- [ ] File exists at `src/transformers/[platform-name].ts`
-- [ ] Has a default export with `key`, `value`, `label`, `description`, and `transformer` fields
-- [ ] The `transformer` object maps source fields to Clerk fields (including a field that maps to `userId`)
+- \[ ] File exists at `src/transformers/[platform-name].ts`
+- \[ ] Has a default export with `key`, `value`, `label`, `description`, and `transformer` fields
+- \[ ] The `transformer` object maps source fields to Clerk fields (including a field that maps to `userId`)
 
 ### 2. Transformer Registered (CRITICAL)
 
-- [ ] Import added to `src/transformers/index.ts`
-- [ ] Transformer added to the `transformers` array export
+- \[ ] Import added to `src/transformers/index.ts`
+- \[ ] Transformer added to the `transformers` array export
 
 **If you skip registration, the delete command will fail to find migrated users!**