Migrate l1-registry from git submodule to npm package#89
Merged
Conversation
- Update package.json to use @l1beat/l1-registry^1.0.0 from GitHub Packages - Configure .npmrc for GitHub Packages authentication (token via NPM_TOKEN env var) - Update src/services/registryService.js to use npm package path resolution - Remove .gitmodules and l1-registry submodule - Verify registry loads successfully with npm package Benefits: - Eliminates submodule update issues - Enables automatic dependency management - Works on all deployment platforms - Supports semantic versioning and rollbacks Tests: Registry API tests passing, no breakage in core functionality Setup: - Set NPM_TOKEN environment variable with GitHub PAT before running npm install
- Run tests on all branches with MongoDB service - Node.js 18 with npm dependency caching - Configure npm authentication for GitHub Packages - Multi-stage pipeline: test, build, deploy-preview, deploy-production - Uses GITHUB_TOKEN for secure package access - Deploy to preview on develop, production on main
- Update MongoDB image to 7.0 with proper health check - Add environment variables for API keys and endpoints - Add MongoDB readiness check before running tests - Configure proper test database URI - Add GLACIER_API_KEY as repository secret (required for tests)
… timeouts
- Parallelize chain updates: process 15 chains concurrently using p-limit
- Reduce hourly cron from ~4 hours to ~10-15 min (90-95% faster)
- Apply to both initial load and hourly updates
- Fix teleporter weekly timeout errors
- Increase timeout from 30s to 60s for weekly updates
- Add retry logic with exponential backoff (up to 3 retries)
- Improve teleporter logging to differentiate daily vs weekly updates
- Add [TELEPORTER DAILY] and [TELEPORTER WEEKLY] prefixes
- Add per-page timing and progress tracking
- Add estimated time remaining for weekly updates
- Replace console.log with logger in chainController
Multiple simultaneous requests were creating duplicate update processes due to
non-atomic check-and-create operations. This caused wasted API calls, inconsistent
states, and confusing logs.
Changes:
- Use atomic findOneAndUpdate for lock acquisition (prevents race conditions)
- Fix stale timeouts based on real performance (60min daily, 8hrs weekly)
* Daily: 30-40 min typical, was timing out at 10 min
* Weekly: 4-6 hours typical, was timing out at 30 min
- Fix pageCount ReferenceError in error handlers with safe access checks
- Add clear logging for lock acquisition vs rejection
Impact: Eliminates duplicate updates, prevents premature cleanup, fixes crashes
Or even shorter:
fix: prevent teleporter race conditions and improve error handling
- Use atomic findOneAndUpdate to prevent duplicate updates
- Fix stale timeouts: 60min daily (was 10min), 8hrs weekly (was 30min)
- Fix pageCount ReferenceError with safe variable access in catch blocks
- Add lock acquisition/rejection logging
Eliminates duplicate updates running simultaneously and prevents crashes
during error handling.
- Reduce teleporter message page size from 100 to 50 to prevent timeout errors - Improve validator fetching logs in chainService with request details - Remove SnowPeer fallback for validator fetching
- Refactor concurrent chain updates to collect results instead of incrementing shared variables - Ensure accurate success/failure reporting in logs - Remove misleading progress counter from concurrent logs
… timeouts - Increase daily and weekly timeouts to 120s (from 60s/90s) - Reduce page size from 50 to 25 to lighten request load - Address persistent Glacier API timeouts during data fetching
…cier errors - Separate state document creation from lock acquisition to prevent duplicates - Use updateOne with (upsert) to ensure document exists - Use findOneAndUpdate WITHOUT upsert for atomic lock acquisition - Restore pageSize to 100 (reduced to 25 during high load) - Add retry logic for network errors (socket hang up, ECONNRESET, ETIMEDOUT) - Add retry logic for HTTP errors (502 Bad Gateway, 503, 504) - Retry up to 3 times with exponential backoff for transient failures
- Check both error.code and error.cause?.code for error detection - Fixes ECONNRESET errors that have code in cause object - Add errorCode to retry warning logs for better debugging
… cleanup - Implement "zombie-aware" locking in teleporterService to abort updates if lock ownership is lost - Update fixStaleUpdates in app.js to respect update durations (60m for daily, 8h for weekly) before killing processes - Add unique index to updateType in TeleporterUpdateState schema to prevent duplicate state records - Clean up duplicate state records before starting new updates - Prevent weekly updates from being killed prematurely on server restart
- Add ensureTeleporterUniqueIndex function to app.js startup - Automatically clean up duplicate update state records on boot - Enforce unique index creation on TeleporterUpdateState collection - Prevent "duplicate key error" or silent index failures on existing DBs
- Add retry logic to ensureTeleporterUniqueIndex to handle race conditions during duplicate cleanup - Update initializeDataUpdates to return explicit success/failure status objects for accurate tracking - Improve logging for initialization results (success/failure counts)
…in cleanup - Fix fetchValidators calls to use evmChainId as fallback parameter instead of subnetId for both parameters, enabling the alternative validator endpoint lookup when Glacier API fails - Change cron job from Promise.all to Promise.allSettled for consistent resilience with initialization phase, ensuring all chains are updated even when individual updates fail - Add removeLegacyChains startup function to clean up X-Chain and P-Chain documents that shouldn't be exposed in the L1 API
…rrors Prevent server from running with corrupted state by throwing when ensureTeleporterUniqueIndex fails, and wrap startup in try-catch with process.exit(1) for critical errors.
- Add heartbeat mechanism in fetchICMMessages() that updates lastUpdatedAt every 10 pages (~2-3 min) to signal the update is still alive - Increase stale timeout in teleporterController from 5 minutes to 4 hours for weekly updates (which take 2-4+ hours to complete) - Increase response status display threshold from 5 to 15 minutes - Add in-progress check in getWeeklyMessageCounts() before triggering background updates to prevent competing update triggers The 5-minute stale timeout was causing API requests to mark valid in-progress weekly updates as failed, triggering new updates that would steal the lock and cause the original update to abort. This cycle repeated indefinitely, preventing weekly updates from ever completing.
- Add 429 (Too Many Requests) to retryable errors in fetchICMMessages() with longer backoff times (30s, 45s, 60s) to handle Glacier API rate limits - Change fixStaleUpdates() to reset ALL in_progress teleporter updates on server startup unconditionally, since processes are gone after restart - Remove time-based stale check (60min/8h thresholds) that left zombie in_progress states blocking new updates This fixes weekly updates that never complete due to: 1. 429 rate limit errors causing immediate failure instead of retry 2. Zombie in_progress states surviving server restarts
- Remove localhost URLs from production CORS configuration - Remove deprecated useNewUrlParser and useUnifiedTopology MongoDB options - Remove duplicate slug indexes from blogPost and author schemas (already created by unique: true constraint) Fixes Mongoose duplicate index warnings and MongoDB driver deprecation warnings.
- Fix chainService evmChainId casting error by only checking numeric IDs - Add chainId field to Chain model and services as alias for blockchainId - Update chainModel tests to use blockchainId as primary field - Fix TPS route handlers to properly handle null/undefined data - Update test assertions to match actual backend behavior
- Created populate-native-tokens.js script
- Fetches chain data from Glacier API
- Migrates assets[] to nativeToken{} schema
- Populates missing descriptions and RPC URLs
- Added npm script: populate-tokens
- Updated Mongoose model: replaced assets[] with nativeToken{}
- Updated registryService: parse nativeToken from registry data
- Updated database sync: store nativeToken instead of assets
Schema change:
- Old: assets: [{symbol, name, decimals}]
- New: nativeToken: {symbol, name, decimals}
- Update Node.js version from 18 to 20 to meet package requirements - Add packages:read permission to access GitHub Packages - Resolves 401 authentication errors and unsupported engine warnings
- Replace GITHUB_TOKEN with NPM_TOKEN for package access - Fixes 401 authentication errors in workflow
- Package is now public, no auth token required - Simplified npm configuration for GitHub Packages
- Published l1beat-l1-registry to npm registry - Remove GitHub Packages authentication requirement - No tokens needed - works everywhere automatically
- Updated registryService to use l1beat-l1-registry package - Added assets field to Chain model schema - Fixed null checks in tpsRoutes and cumulativeTxCountRoutes - Updated tests to match backend behavior (validatorCount vs validators) - Fixed MongoDB connection handling in chainService tests - All 195 tests now passing
- Updated chainService tests to handle empty database in CI - Updated blog sync tests to accept 500 status (RSS may fail in CI) - Tests now pass in both local and CI environments
- Add isL1 field to database sync operation - Add sybilResistanceType field to database sync operation - Fields were parsed from registry but not persisted to DB - Fixes API returning null for these fields despite registry data
Migrate l1-registry from git submodule to npm package
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the git submodule dependency with the published
l1beat-l1-registrynpm package, simplifying dependency management and enabling automatic updates.Changes
🏗️ Infrastructure
.gitmodules,l1-registry/)l1beat-l1-registry@^1.1.0npm package.npmrcfor GitHub Package Registry.github/workflows/ci.yml)✨ New Fields
isL1– Distinguishes L1 chains from subnets (74 L1s, 42 subnets)sybilResistanceType– Consensus type (Proof of Stake / Proof of Authority)nativeToken.logoUri– Native token logo URLs📝 Updated Files
src/services/registryService.js– Import from npm package, sync new fieldssrc/models/chain.js– Added new schema fieldsTesting