Skip to content

ENSDb Compare tool for database exports and comparisons #892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
djstrong wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

ENSDb Compare tool for database exports and comparisons #892

djstrong wants to merge 1 commit into from

Conversation

@djstrong
Copy link
Contributor

@djstrong djstrong commented Jul 30, 2025 •
edited
Loading

  • Introduced a new CLI tool for comparing ENS databases using Drizzle ORM and PostgreSQL.
  • Implemented features for exporting tables in various formats (CSV, JSON, binary) with compression options.
  • Added functionality for benchmarking export performance and generating comparison reports.

Questions:

  1. Should we compare only some tables and some columns like in graphql diff tool? Or compare everything?
  2. Should the tool work on databases with different number of blocks indexed? If yes then open question is how to export or compare data to a given block number.

I have run comparison on the same database exported with some time delay:

Comparison Summary:
Database A: export2drizzle/
Database B: export3drizzle/
Tables Compared: 29
Total Differences: 49
Comparison Time: 22266ms

Table Results:
╔══════════════════════════════╤═════════╤═════════╤═══════════╤═══════════╤════════╤════════╗
║ Table                        │ Total A │ Total B │ Identical │ Different │ Only A │ Only B ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ abi_changed                  │ 125     │ 125     │ 125       │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ accounts                     │ 11180   │ 11182   │ 11180     │ 0         │ 0      │ 2      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ addr_changed                 │ 22881   │ 22884   │ 22881     │ 0         │ 0      │ 3      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ authorisation_changed        │ 0       │ 0       │ 0         │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ contenthash_changed          │ 2985    │ 2985    │ 2985      │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ domains                      │ 81256   │ 81260   │ 81252     │ 4         │ 0      │ 4      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ expiry_extended              │ 87      │ 87      │ 87        │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ ext_resolver_address_records │ 21563   │ 21566   │ 21563     │ 0         │ 0      │ 3      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ ext_resolver_text_records    │ 178147  │ 178147  │ 178147    │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ fuses_set                    │ 1006    │ 1006    │ 1006      │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ interface_changed            │ 233     │ 233     │ 233       │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ multicoin_addr_changed       │ 22928   │ 22931   │ 22928     │ 0         │ 0      │ 3      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_changed                 │ 20163   │ 20164   │ 20163     │ 0         │ 0      │ 1      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_registered              │ 68334   │ 68336   │ 68334     │ 0         │ 0      │ 2      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_renewed                 │ 1172    │ 1172    │ 1172      │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_transferred             │ 2501    │ 2502    │ 2501      │ 0         │ 0      │ 1      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_unwrapped               │ 824     │ 824     │ 824       │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ name_wrapped                 │ 67373   │ 67374   │ 67373     │ 0         │ 0      │ 1      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ new_owners                   │ 90200   │ 90206   │ 90200     │ 0         │ 0      │ 6      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ new_resolvers                │ 72518   │ 72524   │ 72518     │ 0         │ 0      │ 6      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ new_ttls                     │ 14      │ 14      │ 14        │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ pubkey_changed               │ 1064    │ 1065    │ 1064      │ 0         │ 0      │ 1      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ registrations                │ 68181   │ 68183   │ 68181     │ 0         │ 0      │ 2      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ resolvers                    │ 85943   │ 85949   │ 85943     │ 0         │ 0      │ 6      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ text_changed                 │ 186524  │ 186524  │ 186524    │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ transfers                    │ 2136    │ 2138    │ 2136      │ 0         │ 0      │ 2      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ version_changed              │ 62      │ 62      │ 62        │ 0         │ 0      │ 0      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ wrapped_domains              │ 66549   │ 66550   │ 66549     │ 0         │ 0      │ 1      ║
╟──────────────────────────────┼─────────┼─────────┼───────────┼───────────┼────────┼────────╢
║ wrapped_transfers            │ 70768   │ 70769   │ 70768     │ 0         │ 0      │ 1      ║
╚══════════════════════════════╧═════════╧═════════╧═══════════╧═══════════╧════════╧════════╝

Database B is the later one so it has more data and almost all is identical except 4 domains where subdomain_count has increased.
comparison_report.json
So if we want to compare databases with different number of blocks indexed then at least we need to add some special comparison method for subdomain_count but there will be more issues.

Export fails on larger databases.

@djstrong djstrong self-assigned this Jul 30, 2025
@vercel
Copy link

vercel bot commented Jul 30, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Project Deployment Preview Comments Updated (UTC)
admin.ensnode.io ❌ Failed Aug 13, 2025 2:05pm
ensnode.io ❌ Failed Aug 13, 2025 2:05pm
ensrainbow.io ❌ Failed Aug 13, 2025 2:05pm

@changeset-bot
Copy link

changeset-bot bot commented Jul 30, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: 99daddd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@djstrong djstrong added the ensdb ENSDb related label Jul 30, 2025
@djstrong djstrong added this to ENSNode Jul 30, 2025
@djstrong djstrong moved this to In Progress in ENSNode Jul 30, 2025
@djstrong
feat: add ENSDb Compare tool for database exports and comparisons
99daddd 
- Introduced a new CLI tool for comparing ENS databases using Drizzle ORM and PostgreSQL.
- Implemented features for exporting tables in various formats (CSV, JSON, binary) with compression options.
- Added functionality for benchmarking export performance and generating comparison reports.
- Updated package.json to support new dependencies and tools.
- Enhanced pnpm workspace configuration to include the new tool.

This commit lays the groundwork for efficient database management and comparison in ENS projects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@djstrong djstrong

Labels

ensdb ENSDb related

Projects

ENSNode
Status: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@djstrong