Database Converter

A sophisticated .NET 9.0 Blazor Server application designed for one-way database synchronization between different database systems.

Project Overview

The Database Converter provides a web-based interface for configuring, scheduling, and monitoring database conversion jobs with advanced features like real-time monitoring, batch processing, and incremental synchronization.

Features

• Multi-Database Support: SQL Server, MySQL, PostgreSQL
• Flexible Synchronization Options:
  • Full table conversion
  • Incremental synchronization with change detection
  • Table and column filtering
  • Schema creation and data-only modes
• Advanced Scheduling: Cron-based job scheduling via Hangfire
• Real-time Monitoring: Progress tracking with detailed statistics
• Performance Optimizations: Batch processing, bulk operations, memory management
• Comprehensive Error Handling: Table-level isolation, row-level thresholds, automatic retries
• Docker Support: Ready for containerized deployment

Architecture

The application follows a clean architecture with clear separation of concerns:

1. Data Layer

• Core Entities:
  • SyncJob: Stores job configuration including source/destination connections, table/column mappings, scheduling, and execution tracking
  • SyncJobLog: Records execution history with detailed logs and error messages
  • AppDbContext: EF Core context with SQLite storage and JSON serialization for complex types

2. Service Layer

• SyncJobService: Manages job CRUD operations, Hangfire integration, and job status management
• DatabaseScanService: Handles database schema discovery for multiple database types

3. Operations Layer

• SyncJobExecutor: Core synchronization engine that handles the actual data conversion between databases
  • Supports SQL Server → MySQL/PostgreSQL, MySQL → SQL Server, and PostgreSQL → SQL Server
  • Implements different sync modes (full, incremental) with various change detection methods
  • Handles batching, error handling, and progress tracking

4. UI Layer

• SyncJobs.razor: Main dashboard showing job status, execution controls, and logs
• SyncJobDialog.razor: Configuration interface for creating/editing jobs with database scanning
• LogDetailsDialog.razor: Detailed log viewer

Key Technical Features

Database Synchronization

• Pure SQL-based conversion with comprehensive data type mapping
• Support for schema creation and data-only modes
• Table and column level filtering
• Batch processing for large datasets

Incremental Synchronization

• Multiple change detection methods:
  • Hash-based comparison for balanced performance/accuracy
  • Timestamp-based for fastest performance with large datasets
  • Full sync with optimized bulk operations for reliability
• Primary key-based merge with deletion detection

Performance Optimizations

• Configurable batch sizes (100-50,000 records)
• Memory management with periodic garbage collection
• Bulk insert operations for improved performance
• Connection pooling and resource management

Error Handling & Resilience

• Table-level error isolation (continues processing other tables)
• Row-level error handling with configurable thresholds
• Automatic retry logic for transient failures
• Detailed error logging with stack traces

Job Scheduling

• Cron-based scheduling via Hangfire
• Job status tracking and cancellation
• Automatic registration of existing jobs on startup

Real-time Monitoring

• Progress tracking with table/row counts
• Estimated completion time
• Live error reporting
• Background job monitoring without UI blocking

Deployment

The application is designed for Docker deployment:

• Uses a configurable storage directory for persistent data
• Separate SQLite databases for application data and Hangfire
• Environment-specific configuration support

For detailed Docker deployment instructions, see DOCKER-README.md.

Getting Started

Prerequisites

• .NET 9.0 SDK
• Docker (optional, for containerized deployment)

Local Development

# Clone the repository
git clone https://github.com/yourusername/database-converter.git
cd database-converter

# Build and run the application
dotnet build
dotnet run

Docker Deployment

# Build Docker image
docker build -t database-converter .

# Run with Docker Compose
docker-compose up --build -d

Technology Stack

• .NET 9.0 - Core framework
• Blazor Server - Web UI framework
• MudBlazor - UI component library
• Entity Framework Core - ORM with SQLite for application data
• Hangfire - Background job scheduling and execution
• Docker - Containerization support

Areas for Future Enhancement

1. Testing: Adding automated tests for core functionality
2. Security: Implementing encryption for stored database credentials
3. Database Support: Expanding to additional database types
4. Documentation: Adding more inline code documentation

License

MIT License